-----============= acceptance-small: sanity-flr ============----- Tue Aug 11 16:32:19 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg108-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:32:38 (1786480358) === oleg108-client.virtnet: executing check_config_client /mnt/lustre oleg108-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg108-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b62904bf000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b62904bf000.idle_timeout=debug disable quota as required oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:33:22 (1786480402) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8230] == sanity-flr test 0a: lfs mirror create with -N option == 16:33:25 (1786480405) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:33:43 (1786480423) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:33:46 (1786480426) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:33:49 (1786480429) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:34:11 (1786480451) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:34:14 (1786480454) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 16:34:17 (1786480457) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b62904bf000.stats=clear osc.lustre-OST0001-osc-ffff9b62904bf000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:34:25 (1786480465) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:34:34 (1786480474) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:34:42 (1786480482) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:34:45 (1786480485) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:34:53 (1786480493) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 16:35:01 (1786480501) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:35:08 (1786480508) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:35:14 (1786480514) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:35:21 (1786480521) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:35:31 (1786480531) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:35:38 (1786480538) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.569785 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643056 s, 16.3 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:35:47 (1786480547) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.450652 s, 27.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:35:55 (1786480555) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:36:05 (1786480565) Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg108-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (130s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:38:15 (1786480695) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.676767 s, 17.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:38:56 (1786480736) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (67s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:40:03 (1786480803) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.74062 s, 19.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 11 16:40 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.762s user 0m0.000s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b62904bf000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.949s user 0m0.001s sys 0m0.185s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.993s user 0m0.002s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b62904bf000.ost_server_uuid in FULL state after 0 sec PASS 33b (80s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:41:23 (1786480883) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:41:25 (1786480885) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:41:28 (1786480888) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 16:41:30 (1786480890) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 16:41:36 (1786480896) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.440894 s, 28.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:41:47 (1786480907) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241883 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175103 s, 59.9 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:41:53 (1786480913) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:41:59 (1786480919) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112622 s, 909 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000951121 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00308756 s, 332 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00189878 s, 539 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 16:42:07 (1786480927) create test files with size 15737027 .. 0b6f23a093e7d1f6cb70c571b01be429 - dc0b767719c9f67089800852c38a5515 - a69c29bd077852a880b05c26b147bbf1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre PASS 37 (17s) == sanity-flr test 38: resync ============================ 16:42:24 (1786480944) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:42:37 (1786480957) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:42:43 (1786480963) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11712 s, 9.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421675 s, 24.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 16:42:52 (1786480972) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117514 s, 17.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.308715 s, 13.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220339 s, 9.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 16:43:06 (1786480986) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 16:43:09 (1786480989) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:43:12 (1786480992) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261939 s, 40.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 16:43:19 (1786480999) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.322146 s, 9.8 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:43:29 (1786481009) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:43:39 (1786481019) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.228205 s, 45.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:43:48 (1786481028) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b6284b0f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.270065 s, 38.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b62908cc000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:44:01 (1786481041) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:44:34 (1786481074) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:45:07 (1786481107) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:45:10 (1786481110) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.168693 s, 18.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.142303 s, 22.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf8b304d (nosync,stale) Mirror 2: 0xf8b304d (nosync,stale) ** verify mirror 2 doesn't change original checksum: c1a7e1634b9293a772db1934b9548f27 - mirror 1 checksum: 1ce19eb2dbb801cceda1190cdb05b4ca - mirror 2 checksum: c1a7e1634b9293a772db1934b9548f27 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-11T16:45:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-11T16:45:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c1a7e1634b9293a772db1934b9548f27 - mirror 1 checksum: 1ce19eb2dbb801cceda1190cdb05b4ca - mirror 2 checksum: 1ce19eb2dbb801cceda1190cdb05b4ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-11T16:45:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-11T16:45:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:45:19 (1786481119) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304973 s, 34.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 314368.. 315391: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (5s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:45:24 (1786481124) Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 16:45:34 (1786481134) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server Failover mds1 to oleg108-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 16:45:55 (1786481155) targets are mounted 16:45:55 (1786481155) facet_failover done oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025633 s, 40.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (44s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:46:08 (1786481168) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0212229 s, 965 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0132432 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (7s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:46:15 (1786481175) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0189568 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00899118 s, 2.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:46:23 (1786481183) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0176488 s, 928 kB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:46:29 (1786481189) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.028676 s, 714 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:46:35 (1786481195) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00438516 s, 934 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0625269 s, 65.5 kB/s PASS 60a (12s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:46:47 (1786481207) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:47:14 (1786481234) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:47:44 (1786481264) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:48:07 (1786481287) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:48:31 (1786481311) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 16:48:38 (1786481318) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57554 mirror split pid 57555 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57554 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57555 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 16:49:46 (1786481386) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 1387 skipping zero size read 000003 1786481388.153959 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x12dc7 000004 1786481388.510743 trunc from 00000000 to 0x012dc6 (0x12dc7 bytes) truncating to largest ever: 0x2a235 000005 1786481388.554839 trunc from 0x012dc7 to 0x02a234 (0x1746e bytes) 000006 1786481388.583028 read 0x016d20 thru 0x020100 (0x93e1 bytes) 000012 1786481388.598932 write 0x03ac40 thru 0x03ffff (0x53c0 bytes) 000014 1786481388.601844 fallocate 0x0123b6 thru 0x020a9b (0xe6e6 bytes) 000015 1786481388.608820 write 0x03833b thru 0x03ffff (0x7cc5 bytes) fallocating to largest ever: 0x2e60c 000016 1786481388.615558 fallocate 0x025ccd thru 0x02e60b (0x893f bytes) 000017 1786481388.622729 trunc from 0x02aaee to 0x033d47 (0x925a bytes) 000018 1786481388.628624 trunc from 0x032177 to 0x0351a2 (0x302c bytes) 000019 1786481388.635075 trunc from 0x040000 to 0x024120 (0x1bedf bytes) 000020 1786481388.654689 fallocate 0x009b74 thru 0x00cc47 (0x30d4 bytes) 000021 1786481388.665372 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24121): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24121): Mirror 2: 0xc94cbb83 (stale) CRC-32 checksum value for chunk [0, 0x24121): Mirror 1: 0xc94cbb83 Mirror 3: 0xc94cbb83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24121: skipped 000022 1786481389.455062 mapwrite 0x0291c6 thru 0x034ba5 (0xb9e0 bytes) 000023 1786481389.617205 trunc from 0x001728 to 0x0098a5 (0x817e bytes) 000024 1786481389.625429 mapwrite 0x026127 thru 0x0291d5 (0x30af bytes) 000025 1786481389.668599 fallocate 0x00aee3 thru 0x00c26a (0x1388 bytes) 000026 1786481389.675046 mapwrite 0x003bab thru 0x00f2b5 (0xb70b bytes) 000027 1786481389.721532 write 0x036e16 thru 0x03ffff (0x91ea bytes) 000028 1786481389.723863 trunc from 0x022619 to 0x02b151 (0x8b39 bytes) 000029 1786481389.730379 trunc from 0x01fd01 to 0x0298f2 (0x9bf2 bytes) 000030 1786481389.736543 punch from 0x00b9cc to 0x01493e (0x8f73 bytes) 000031 1786481389.742857 read 0x00d011 thru 0x016c5b (0x9c4b bytes) 000032 1786481389.752974 read 0x03eb30 thru 0x03ffff (0x14d0 bytes) 000033 1786481389.753986 read 0x0233b5 thru 0x02740b (0x4057 bytes) 000034 1786481389.766060 punch from 0x0364ff to 0x036e69 (0x96b bytes) 000035 1786481389.787470 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000036 1786481390.230358 trunc from 0x040000 to 0x021005 (0x1effa bytes) 000038 1786481390.262883 mapwrite 0x01058b thru 0x01348c (0x2f02 bytes) 000039 1786481390.311560 mapwrite 0x00bb30 thru 0x01450c (0x89dd bytes) 000040 1786481390.350884 punch from 0x01dae7 to 0x021005 (0x351f bytes) 000041 1786481390.361027 mapwrite 0x00b363 thru 0x01273b (0x73d9 bytes) 000042 1786481390.394754 write 0x023d17 thru 0x0251d5 (0x14bf bytes) 000043 1786481390.397031 fallocate 0x0128bd thru 0x01acb3 (0x83f7 bytes) 000044 1786481390.402396 trunc from 0x0251d6 to 0x021398 (0x3e3d bytes) truncating to largest ever: 0x335ea 000046 1786481390.434711 trunc from 0x021399 to 0x0335e9 (0x12251 bytes) 000047 1786481390.456856 trunc from 0x0335ea to 0x0261c8 (0xd421 bytes) 000048 1786481390.484853 write 0x0041a2 thru 0x0048bc (0x71b bytes) 000049 1786481390.493690 punch from 0x01073e to 0x01e8df (0xe1a2 bytes) 000050 1786481390.499982 fallocate 0x00dadd thru 0x0180a0 (0xa5c4 bytes) 000051 1786481390.510313 write 0x00417d thru 0x014172 (0xfff6 bytes) 000052 1786481390.516927 read 0x002f90 thru 0x01097c (0xd9ed bytes) fallocating to largest ever: 0x40000 000053 1786481390.524588 fallocate 0x03f6e3 thru 0x03ffff (0x91d bytes) 000055 1786481390.538589 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd8fbd0d3 Mirror 4: 0xd8fbd0d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000056 1786481391.327491 write 0x031565 thru 0x034aa5 (0x3541 bytes) 000057 1786481391.384769 mapwrite 0x0118ed thru 0x011aed (0x201 bytes) 000058 1786481391.406966 mapwrite 0x00c11a thru 0x01094d (0x4834 bytes) 000059 1786481391.443758 fallocate 0x038a0a thru 0x03ad94 (0x238b bytes) 000061 1786481391.449791 write 0x02464a thru 0x02881d (0x41d4 bytes) 000062 1786481391.459981 mapwrite 0x02e44a thru 0x02fcd1 (0x1888 bytes) 000063 1786481391.484287 fallocate 0x035fb3 thru 0x03cfb2 (0x7000 bytes) 000065 1786481391.490446 mapwrite 0x02f04c thru 0x03a819 (0xb7ce bytes) 000066 1786481391.519180 fallocate 0x01eb44 thru 0x023884 (0x4d41 bytes) 000067 1786481391.525354 read 0x026f0c thru 0x03407c (0xd171 bytes) 000068 1786481391.526767 trunc from 0x040000 to 0x000528 (0x3fad7 bytes) 000069 1786481391.569443 trunc from 0x000529 to 0x0316c9 (0x311a1 bytes) 000071 1786481391.583542 fallocate 0x03bf3c thru 0x03caff (0xbc4 bytes) 000073 1786481391.592451 punch from 0x011761 to 0x016b1e (0x53be bytes) 000074 1786481391.600871 punch from 0x0236cb to 0x025d53 (0x2689 bytes) 000075 1786481391.615975 MIRROR_RESYNC 3 mirrors 000076 1786481392.218098 trunc from 0x0118a9 to 0x01f427 (0xdb7f bytes) 000077 1786481392.255757 trunc from 0x0316ca to 0x029801 (0x7ec8 bytes) 000078 1786481392.278660 read 0x0280f2 thru 0x029801 (0x1710 bytes) 000079 1786481392.288514 write 0x03415c thru 0x03b05c (0x6f01 bytes) 000080 1786481392.291389 read 0x014df7 thru 0x01d2e6 (0x84f0 bytes) 000081 1786481392.304617 trunc from 0x026a7b to 0x031c89 (0xb20f bytes) 000082 1786481392.309733 punch from 0x020798 to 0x02be93 (0xb6fc bytes) 000085 1786481392.318030 write 0x0335e6 thru 0x03ffff (0xca1a bytes) 000087 1786481392.321987 mapwrite 0x0222a9 thru 0x02feea (0xdc42 bytes) 000088 1786481392.438016 fallocate 0x03e437 thru 0x03ffff (0x1bc9 bytes) 000089 1786481392.442394 trunc from 0x00e4aa to 0x0139cf (0x5526 bytes) 000090 1786481392.448213 write 0x0225dd thru 0x026ad6 (0x44fa bytes) 000091 1786481392.449586 read 0x01402a thru 0x0200c1 (0xc098 bytes) 000092 1786481392.463220 write 0x038a46 thru 0x03ce9d (0x4458 bytes) 000093 1786481392.464983 fallocate 0x0259c4 thru 0x02f256 (0x9893 bytes) 000094 1786481392.469424 punch from 0x01108c to 0x01a8ff (0x9874 bytes) 000095 1786481392.474800 mapwrite 0x019fda thru 0x022c35 (0x8c5c bytes) 000096 1786481392.511052 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa9c7ac96 Mirror 5: 0xa9c7ac96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000097 1786481393.215539 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa9c7ac96 Mirror 5: 0xa9c7ac96 Mirror 6: 0xa9c7ac96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000098 1786481394.154346 punch from 0x026918 to 0x02f69b (0x8d84 bytes) 000099 1786481394.199492 write 0x00e260 thru 0x019c46 (0xb9e7 bytes) 000100 1786481394.225609 mapwrite 0x038c8b thru 0x03e9b5 (0x5d2b bytes) 000101 1786481394.305548 mapwrite 0x00be45 thru 0x010f9c (0x5158 bytes) 000102 1786481394.376547 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xa9c7ac96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xa9c7ac96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000103 1786481395.113208 mapread 0x02bae6 thru 0x038230 (0xc74b bytes) 000104 1786481395.201247 mapwrite 0x01d82d thru 0x025d70 (0x8544 bytes) 000105 1786481395.244927 MIRROR_RESYNC 4 mirrors 000106 1786481396.492105 fallocate 0x03c6cc thru 0x03ffff (0x3934 bytes) 000107 1786481396.533965 trunc from 0x040000 to 0x01b2bd (0x24d42 bytes) 000108 1786481396.555616 mapread 0x005305 thru 0x00e4d5 (0x91d1 bytes) 000109 1786481396.575851 mapread 0x01439e thru 0x01b2bd (0x6f20 bytes) 000110 1786481396.590930 fallocate 0x0099d6 thru 0x014b1e (0xb149 bytes) 000111 1786481396.597372 fallocate 0x0352c7 thru 0x03f965 (0xa69f bytes) 000112 1786481396.609635 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b2be): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b2be): Mirror 3: 0x3e87d0fb (stale) Mirror 4: 0x3e87d0fb (stale) Mirror 6: 0x3e87d0fb (stale) CRC-32 checksum value for chunk [0, 0x1b2be): Mirror 1: 0x3e87d0fb Mirror 7: 0x3e87d0fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b2be: skipped 000113 1786481397.658434 trunc from 0x018dcc to 0x01b2bd (0x24f2 bytes) 000114 1786481397.719761 trunc from 0x00f6d6 to 0x018938 (0x9263 bytes) 000115 1786481397.738367 MIRROR_RESYNC 5 mirrors 000116 1786481399.211493 write 0x00e014 thru 0x015c32 (0x7c1f bytes) 000117 1786481399.243654 trunc from 0x007dca to 0x013649 (0xb880 bytes) 000119 1786481399.269575 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b2be): Mirror 3: 0x93a337e2 (stale) Mirror 4: 0x93a337e2 (stale) Mirror 7: 0x93a337e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b2be): Mirror 3: 0x93a337e2 (stale) Mirror 4: 0x93a337e2 (stale) Mirror 7: 0x93a337e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b2be: skipped 000120 1786481399.686681 trunc from 0x014387 to 0x0189ac (0x4626 bytes) 000121 1786481399.694476 read 0x012111 thru 0x0198c3 (0x77b3 bytes) 000124 1786481399.713263 MIRROR_RESYNC 4 mirrors 000125 1786481400.941267 read 0x0035b7 thru 0x00c97a (0x93c4 bytes) 000126 1786481400.950651 fallocate 0x02dfc9 thru 0x034ccd (0x6d05 bytes) 000127 1786481401.002801 write 0x03ecb1 thru 0x03ffff (0x134f bytes) 000128 1786481401.003987 mapread 0x037877 thru 0x03f76b (0x7ef5 bytes) 000131 1786481401.034838 fallocate 0x026afd thru 0x028269 (0x176d bytes) 000132 1786481401.043367 trunc from 0x025c61 to 0x0303b6 (0xa756 bytes) 000133 1786481401.053758 mapread 0x019e83 thru 0x01b431 (0x15af bytes) 000134 1786481401.068845 mapread 0x033de1 thru 0x03ffff (0xc21f bytes) 000135 1786481401.074757 mapread 0x03647d thru 0x03ffff (0x9b83 bytes) 000136 1786481401.076170 mapwrite 0x01aab6 thru 0x022e39 (0x8384 bytes) 000137 1786481401.138866 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xce06479a (stale) Mirror 7: 0xce06479a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xce06479a (stale) Mirror 7: 0xce06479a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000138 1786481401.676293 write 0x00b95b thru 0x00bda8 (0x44e bytes) 000139 1786481401.693784 read 0x03e3af thru 0x03ffff (0x1c51 bytes) 000140 1786481401.717322 MIRROR_RESYNC 3 mirrors 000141 1786481403.091941 write 0x00515a thru 0x0140e6 (0xef8d bytes) 000142 1786481403.193553 mapwrite 0x01da37 thru 0x028c58 (0xb222 bytes) 000143 1786481403.304153 fallocate 0x02c695 thru 0x038d72 (0xc6de bytes) 000144 1786481403.318459 write 0x0221e1 thru 0x0267c4 (0x45e4 bytes) 000145 1786481403.322364 mapread 0x016160 thru 0x025d85 (0xfc26 bytes) 000147 1786481403.333910 mapwrite 0x00a472 thru 0x00dfeb (0x3b7a bytes) 000148 1786481403.353577 trunc from 0x040000 to 0x03875f (0x78a0 bytes) 000149 1786481403.375472 trunc from 0x003e2d to 0x0129ff (0xebd3 bytes) 000150 1786481403.383175 read 0x01b934 thru 0x02803d (0xc70a bytes) 000151 1786481403.387133 fallocate 0x0134c3 thru 0x017283 (0x3dc1 bytes) 000152 1786481403.394220 mapwrite 0x014d7b thru 0x0164f4 (0x177a bytes) 000153 1786481403.409604 mapwrite 0x03ee8d thru 0x03ffff (0x1173 bytes) 000154 1786481403.462611 punch from 0x003013 to 0x012bde (0xfbcc bytes) 000155 1786481403.467810 mapwrite 0x03e06e thru 0x03ffff (0x1f92 bytes) 000156 1786481403.483578 read 0x01e83c thru 0x025dca (0x758f bytes) 000157 1786481403.484438 trunc from 0x040000 to 0x034894 (0xb76b bytes) 000158 1786481403.504287 trunc from 0x01322c to 0x014cd8 (0x1aad bytes) 000160 1786481403.510587 punch from 0x022e8f to 0x02c642 (0x97b4 bytes) 000161 1786481403.516275 punch from 0x02b0e5 to 0x02f9ea (0x4906 bytes) 000162 1786481403.520744 read 0x00bab6 thru 0x016884 (0xadcf bytes) 000163 1786481403.536835 trunc from 0x034895 to 0x032fc4 (0x18d0 bytes) 000164 1786481403.561411 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32fc5): Mirror 3: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fc5): Mirror 3: 0x5d5cf179 (stale) Mirror 7: 0x5d5cf179 (stale) CRC-32 checksum value for chunk [0, 0x32fc5): Mirror 1: 0xf0041f1b Mirror 8: 0xf0041f1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fc5: skipped 000165 1786481405.157220 mapwrite 0x0378c4 thru 0x03fcfe (0x843b bytes) 000166 1786481405.309768 trunc from 0x03fcff to 0x026ba4 (0x1915a bytes) 000167 1786481405.338153 mapwrite 0x02c605 thru 0x02cfb8 (0x9b4 bytes) 000168 1786481405.439375 trunc from 0x02cfb9 to 0x017c5b (0x1535d bytes) 000169 1786481405.485363 punch from 0x010bd5 to 0x017c5b (0x7087 bytes) 000170 1786481405.501034 mapread 0x00d56a thru 0x017c5b (0xa6f2 bytes) 000171 1786481405.543270 mapwrite 0x021860 thru 0x0258f3 (0x4094 bytes) 000172 1786481405.612772 read 0x00858f thru 0x0178e1 (0xf353 bytes) 000173 1786481405.614841 punch from 0x01c649 to 0x0258f3 (0x92ab bytes) 000174 1786481405.632830 trunc from 0x0258f4 to 0x03b801 (0x15f0e bytes) 000175 1786481405.665865 mapwrite 0x0129b5 thru 0x01de4f (0xb49b bytes) 000176 1786481405.744315 read 0x0076df thru 0x00dba4 (0x64c6 bytes) 000178 1786481405.745782 trunc from 0x03b802 to 0x000a99 (0x3ad68 bytes) 000179 1786481405.781910 punch from 0x0006f9 to 0x000a99 (0x3a1 bytes) 000180 1786481405.795890 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa9a): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa9a): Mirror 3: 0xdc8d352b (stale) Mirror 7: 0xdc8d352b (stale) Mirror 8: 0xdc8d352b (stale) CRC-32 checksum value for chunk [0, 0xa9a): Mirror 1: 0xdc8d352b Mirror 9: 0xdc8d352b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa9a: skipped 000181 1786481406.857543 fallocate 0x011ae6 thru 0x018b02 (0x701d bytes) 000182 1786481406.902563 fallocate 0x03edb8 thru 0x03ffff (0x1248 bytes) 000183 1786481406.910327 fallocate 0x0088f7 thru 0x00cea2 (0x45ac bytes) 000184 1786481406.926694 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18b03): Mirror 3: 0xfa31ecd2 (stale) Mirror 7: 0xfa31ecd2 (stale) Mirror 9: 0xe93735f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18b03): Mirror 3: 0xfa31ecd2 (stale) Mirror 7: 0xfa31ecd2 (stale) Mirror 9: 0xe93735f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18b03: skipped 000186 1786481407.296295 fallocate 0x02fb14 thru 0x03c2a5 (0xc792 bytes) 000188 1786481407.302935 read 0x0026ba thru 0x002968 (0x2af bytes) 000189 1786481407.315775 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 3: 0x18437a8 (stale) Mirror 7: 0x18437a8 (stale) Mirror 9: 0xb556eb9b (stale) CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 1: 0xb556eb9b Mirror 10: 0xb556eb9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2a6: skipped 000190 1786481408.453065 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 3: 0x18437a8 (stale) Mirror 7: 0x18437a8 (stale) Mirror 9: 0xb556eb9b (stale) CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 1: 0xb556eb9b Mirror 10: 0xb556eb9b Mirror 11: 0xb556eb9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2a6: skipped 000191 1786481409.923255 trunc from 0x02f7a5 to 0x037da5 (0x8601 bytes) 000192 1786481409.998958 trunc from 0x03c2a6 to 0x016755 (0x25b50 bytes) 000193 1786481410.028445 punch from 0x0166ab to 0x016755 (0xab bytes) 000194 1786481410.036201 read 0x010b62 thru 0x016755 (0x5bf4 bytes) 000195 1786481410.080305 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x16756): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16756): Mirror 3: 0x8f4c2f4c (stale) Mirror 7: 0x8f4c2f4c (stale) Mirror 9: 0x42e9ec3f (stale) Mirror 10: 0x42e9ec3f (stale) Mirror 11: 0x42e9ec3f (stale) CRC-32 checksum value for chunk [0, 0x16756): Mirror 1: 0x42e9ec3f Mirror 12: 0x42e9ec3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16756: skipped 000196 1786481411.308331 fallocate 0x01f766 thru 0x01f8cc (0x167 bytes) 000197 1786481411.400873 trunc from 0x016756 to 0x002d3c (0x13a19 bytes) 000198 1786481411.483212 trunc from 0x0012b7 to 0x002d3c (0x1a86 bytes) 000199 1786481411.490910 write 0x00d10f thru 0x0122a5 (0x5197 bytes) 000200 1786481411.494018 trunc from 0x0122a6 to 0x00ac9f (0x7606 bytes) 000201 1786481411.515096 mapwrite 0x00db44 thru 0x01d332 (0xf7ef bytes) 000202 1786481411.617358 trunc from 0x01d333 to 0x02a32f (0xcffd bytes) 000203 1786481411.640543 read 0x00a5b3 thru 0x0155f9 (0xb047 bytes) 000204 1786481411.641617 trunc from 0x01f0ff to 0x02a32f (0xb231 bytes) 000205 1786481411.649356 fallocate 0x011081 thru 0x01a1be (0x913e bytes) 000206 1786481411.658452 mapread 0x00ee97 thru 0x01852c (0x9696 bytes) 000207 1786481411.661757 write 0x00f354 thru 0x0187c6 (0x9473 bytes) 000208 1786481411.670671 read 0x009e21 thru 0x00c170 (0x2350 bytes) 000209 1786481411.677589 MIRROR_RESYNC 7 mirrors 000210 1786481413.257172 trunc from 0x01c4fb to 0x02a32f (0xde35 bytes) 000211 1786481413.312718 punch from 0x00dcd2 to 0x01327e (0x55ad bytes) 000212 1786481413.319022 read 0x01ce99 thru 0x02a32f (0xd497 bytes) 000215 1786481413.340632 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a330): Mirror 3: 0x57f78ceb (stale) Mirror 7: 0x57f78ceb (stale) Mirror 9: 0x57f78ceb (stale) Mirror 10: 0x57f78ceb (stale) Mirror 11: 0x57f78ceb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a330): Mirror 3: 0x57f78ceb (stale) Mirror 7: 0x57f78ceb (stale) Mirror 9: 0x57f78ceb (stale) Mirror 10: 0x57f78ceb (stale) Mirror 11: 0x57f78ceb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a330: skipped 000216 1786481413.903025 punch from 0x00ddcf to 0x0120f1 (0x4323 bytes) 000217 1786481413.912918 trunc from 0x00ed30 to 0x017719 (0x89ea bytes) 000219 1786481413.927493 read 0x0156f1 thru 0x01a0b0 (0x49c0 bytes) 000220 1786481413.941396 fallocate 0x0367b2 thru 0x03ffff (0x984e bytes) 000221 1786481413.953462 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2a330): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a330): Mirror 3: 0x57f78ceb (stale) Mirror 7: 0x57f78ceb (stale) Mirror 9: 0x57f78ceb (stale) Mirror 10: 0x57f78ceb (stale) Mirror 11: 0x57f78ceb (stale) CRC-32 checksum value for chunk [0, 0x2a330): Mirror 1: 0x9bf9f393 Mirror 12: 0x9bf9f393 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a330: skipped 000223 1786481415.329177 mapread 0x01f64d thru 0x0246b5 (0x5069 bytes) 000224 1786481415.366911 write 0x0012ea thru 0x00c1b3 (0xaeca bytes) 000225 1786481415.424584 trunc from 0x0129b7 to 0x020e18 (0xe462 bytes) 000226 1786481415.430818 mapread 0x01123b thru 0x01324a (0x2010 bytes) 000228 1786481415.442450 mapwrite 0x03c947 thru 0x03ffff (0x36b9 bytes) 000229 1786481415.515716 read 0x00dfa4 thru 0x01192a (0x3987 bytes) 000230 1786481415.516725 mapread 0x036322 thru 0x03ffff (0x9cde bytes) 000231 1786481415.521958 mapwrite 0x00ff26 thru 0x0117e4 (0x18bf bytes) 000232 1786481415.537801 mapwrite 0x00ba37 thru 0x0137a9 (0x7d73 bytes) 000233 1786481415.615223 trunc from 0x040000 to 0x022ffb (0x1d004 bytes) 000234 1786481415.655483 mapwrite 0x00b3a0 thru 0x014c57 (0x98b8 bytes) 000235 1786481415.728987 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x22ffc): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22ffc): Mirror 3: 0xd7208f8b (stale) Mirror 7: 0xd7208f8b (stale) Mirror 9: 0xd7208f8b (stale) Mirror 10: 0xd7208f8b (stale) Mirror 11: 0xd7208f8b (stale) Mirror 12: 0x5e70cc6c (stale) CRC-32 checksum value for chunk [0, 0x22ffc): Mirror 1: 0x1fa2a6ce Mirror 13: 0x1fa2a6ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22ffc: skipped 000236 1786481417.202390 write 0x006ee7 thru 0x007273 (0x38d bytes) 000237 1786481417.303444 write 0x0294ed thru 0x02af86 (0x1a9a bytes) 000238 1786481417.305559 trunc from 0x02af87 to 0x023793 (0x77f3 bytes) 000239 1786481417.336848 read 0x021a22 thru 0x023793 (0x1d72 bytes) 000240 1786481417.346046 punch from 0x003316 to 0x007393 (0x407e bytes) 000242 1786481417.362908 mapread 0x002b43 thru 0x00439f (0x185d bytes) 000243 1786481417.390152 punch from 0x0221ae to 0x023793 (0x15e6 bytes) 000244 1786481417.395919 write 0x00fabb thru 0x01aff3 (0xb539 bytes) 000245 1786481417.410521 trunc from 0x023794 to 0x03a50b (0x16d78 bytes) 000246 1786481417.426496 trunc from 0x0019c7 to 0x0110ec (0xf726 bytes) 000247 1786481417.445030 fallocate 0x03bc6b thru 0x03ffff (0x4395 bytes) 000248 1786481417.450509 write 0x01feef thru 0x0259be (0x5ad0 bytes) 000249 1786481417.463736 read 0x02e736 thru 0x03a50b (0xbdd6 bytes) 000250 1786481417.473874 punch from 0x031a84 to 0x03a50b (0x8a88 bytes) 000251 1786481417.479306 punch from 0x003262 to 0x00355e (0x2fd bytes) 000252 1786481417.489504 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a50c): Mirror 7: 0x8d219729 (stale) Mirror 9: 0x8d219729 (stale) Mirror 10: 0x8d219729 (stale) Mirror 11: 0x8d219729 (stale) Mirror 12: 0x6b2053ad (stale) Mirror 13: 0xc5f27e1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a50c): Mirror 7: 0x8d219729 (stale) Mirror 9: 0x8d219729 (stale) Mirror 10: 0x8d219729 (stale) Mirror 11: 0x8d219729 (stale) Mirror 12: 0x6b2053ad (stale) Mirror 13: 0xc5f27e1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a50c: skipped 000255 1786481418.149867 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a50c): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a50c): Mirror 7: 0x8d219729 (stale) Mirror 9: 0x8d219729 (stale) Mirror 10: 0x8d219729 (stale) Mirror 11: 0x8d219729 (stale) Mirror 12: 0x6b2053ad (stale) Mirror 13: 0xc5f27e1b (stale) CRC-32 checksum value for chunk [0, 0x3a50c): Mirror 1: 0xcfc52be1 Mirror 14: 0xcfc52be1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a50c: skipped 000256 1786481419.737395 punch from 0x000540 to 0x004feb (0x4aac bytes) 000257 1786481419.791939 fallocate 0x010781 thru 0x01df47 (0xd7c7 bytes) 000258 1786481419.798412 trunc from 0x010aa7 to 0x01a557 (0x9ab1 bytes) 000259 1786481419.815972 write 0x035707 thru 0x03d6b7 (0x7fb1 bytes) 000260 1786481419.837042 write 0x014148 thru 0x01f9fd (0xb8b6 bytes) 000261 1786481419.849854 write 0x03c59b thru 0x03ffff (0x3a65 bytes) 000262 1786481419.851340 mapwrite 0x00555f thru 0x0070ff (0x1ba1 bytes) 000263 1786481419.922684 mapwrite 0x037f63 thru 0x03ffff (0x809d bytes) 000264 1786481419.951370 write 0x000ab5 thru 0x00bb94 (0xb0e0 bytes) 000265 1786481419.958279 fallocate 0x011666 thru 0x01e95f (0xd2fa bytes) 000266 1786481419.971037 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x8310ddf1 (stale) Mirror 9: 0x8310ddf1 (stale) Mirror 10: 0x8310ddf1 (stale) Mirror 11: 0x8310ddf1 (stale) Mirror 12: 0xe36459b1 (stale) Mirror 13: 0x48c22c7f (stale) Mirror 14: 0x757e22d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x90f7f52b Mirror 15: 0x90f7f52b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000268 1786481421.347943 read 0x0188b9 thru 0x01ce0b (0x4553 bytes) 000270 1786481421.404932 read 0x03ffb2 thru 0x03ffff (0x4e bytes) 000271 1786481421.429076 write 0x00a9e0 thru 0x0195b5 (0xebd6 bytes) 000272 1786481421.504490 mapread 0x00c57c thru 0x00caa5 (0x52a bytes) 000273 1786481421.506926 mapread 0x03a4f1 thru 0x03ffff (0x5b0f bytes) 000274 1786481421.537023 mapread 0x021bd6 thru 0x024f75 (0x33a0 bytes) 000275 1786481421.567196 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x8310ddf1 (stale) Mirror 9: 0x8310ddf1 (stale) Mirror 10: 0x8310ddf1 (stale) Mirror 11: 0x8310ddf1 (stale) Mirror 12: 0xe36459b1 (stale) Mirror 13: 0x48c22c7f (stale) Mirror 14: 0x757e22d8 (stale) Mirror 15: 0x90f7f52b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x50722e71 Mirror 16: 0x50722e71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000276 1786481423.002004 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x8310ddf1 (stale) Mirror 9: 0x8310ddf1 (stale) Mirror 10: 0x8310ddf1 (stale) Mirror 11: 0x8310ddf1 (stale) Mirror 12: 0xe36459b1 (stale) Mirror 13: 0x48c22c7f (stale) Mirror 14: 0x757e22d8 (stale) Mirror 15: 0x90f7f52b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x50722e71 Mirror 16: 0x50722e71 Mirror 17: 0x50722e71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1786481424.199226 mapwrite 0x02131e thru 0x029616 (0x82f9 bytes) 000278 1786481424.387385 trunc from 0x00c9c6 to 0x0119ed (0x5028 bytes) 000279 1786481424.405290 mapwrite 0x0179fd thru 0x0214ab (0x9aaf bytes) 000280 1786481424.510026 punch from 0x01818b to 0x019f7c (0x1df2 bytes) 000281 1786481424.531611 fallocate 0x021bfe thru 0x0315d1 (0xf9d4 bytes) 000282 1786481424.547882 mapwrite 0x01a5ef thru 0x0218e8 (0x72fa bytes) 000283 1786481424.589464 trunc from 0x040000 to 0x01aa0d (0x255f2 bytes) 000284 1786481424.662046 read 0x00d889 thru 0x015aad (0x8225 bytes) 000285 1786481424.678660 mapread 0x0060a7 thru 0x012440 (0xc39a bytes) 000286 1786481424.700195 fallocate 0x00773c thru 0x00bba3 (0x4468 bytes) 000287 1786481424.709313 mapread 0x01852c thru 0x01aa0d (0x24e2 bytes) 000288 1786481424.749643 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1aa0e): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aa0e): Mirror 7: 0x59fecb13 (stale) Mirror 9: 0x59fecb13 (stale) Mirror 10: 0x59fecb13 (stale) Mirror 11: 0x59fecb13 (stale) Mirror 12: 0xedd2264f (stale) Mirror 13: 0x7280d1cf (stale) Mirror 14: 0x60a53564 (stale) Mirror 15: 0xde7a2493 (stale) Mirror 16: 0xdb3cfb5d (stale) Mirror 17: 0xdb3cfb5d (stale) CRC-32 checksum value for chunk [0, 0x1aa0e): Mirror 1: 0x9e70c9bd Mirror 18: 0x9e70c9bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aa0e: skipped 000289 1786481426.388969 write 0x00ed7d thru 0x00f888 (0xb0c bytes) 000290 1786481426.479665 MIRROR_RESYNC 12 mirrors 000291 1786481429.497279 trunc from 0x01aa0e to 0x01eb26 (0x4119 bytes) 000292 1786481429.637429 mapread 0x00c545 thru 0x010ba1 (0x465d bytes) 000293 1786481429.709228 trunc from 0x01eb27 to 0x004477 (0x1a6af bytes) 000294 1786481429.755500 trunc from 0x004478 to 0x02b929 (0x274b2 bytes) 000295 1786481429.788434 MIRROR_RESYNC 12 mirrors 000296 1786481431.401763 fallocate 0x011edc thru 0x01cb67 (0xac8c bytes) 000297 1786481431.445831 read 0x020153 thru 0x0295bc (0x946a bytes) 000298 1786481431.461652 MIRROR_RESYNC 12 mirrors 000299 1786481433.603978 MIRROR_RESYNC 12 mirrors 000300 1786481434.548271 read 0x01ef23 thru 0x0299ee (0xaacc bytes) 000301 1786481434.559274 trunc from 0x02a9e3 to 0x02b929 (0xf47 bytes) 000302 1786481434.619978 read 0x02a682 thru 0x02b929 (0x12a8 bytes) 000303 1786481434.636849 trunc from 0x008011 to 0x00c48d (0x447d bytes) 000304 1786481434.647555 fallocate 0x00a76f thru 0x018c22 (0xe4b4 bytes) 000305 1786481434.654052 mapread 0x01cddb thru 0x020bb6 (0x3ddc bytes) 000306 1786481434.659812 trunc from 0x005837 to 0x007f0c (0x26d6 bytes) 000307 1786481434.667843 fallocate 0x00c85c thru 0x0192b9 (0xca5e bytes) 000308 1786481434.677167 write 0x02db6c thru 0x0371ec (0x9681 bytes) 000309 1786481434.678963 mapwrite 0x01e081 thru 0x02aa45 (0xc9c5 bytes) 000310 1786481434.723708 punch from 0x001ab7 to 0x004f3e (0x3488 bytes) 000311 1786481434.730207 read 0x01a4cc thru 0x0211d3 (0x6d08 bytes) 000312 1786481434.731225 write 0x03bce6 thru 0x03db58 (0x1e73 bytes) 000313 1786481434.733433 punch from 0x012d14 to 0x018f99 (0x6286 bytes) 000314 1786481434.738596 mapread 0x0210b0 thru 0x02ce82 (0xbdd3 bytes) 000315 1786481434.753078 fallocate 0x03091c thru 0x036c80 (0x6365 bytes) 000316 1786481434.758980 fallocate 0x000940 thru 0x00d1c5 (0xc886 bytes) 000317 1786481434.764758 trunc from 0x03db59 to 0x02bcfe (0x11e5a bytes) 000318 1786481434.789588 read 0x01bc43 thru 0x01cda9 (0x1167 bytes) 000319 1786481434.795254 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bcff): Mirror 7: 0xd1509633 (stale) Mirror 9: 0xd1509633 (stale) Mirror 10: 0xd1509633 (stale) Mirror 11: 0xd1509633 (stale) Mirror 13: 0xd1509633 (stale) Mirror 14: 0xd1509633 (stale) Mirror 15: 0xd1509633 (stale) Mirror 16: 0xd1509633 (stale) Mirror 17: 0xd1509633 (stale) Mirror 18: 0xd1509633 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcff): Mirror 7: 0xd1509633 (stale) Mirror 9: 0xd1509633 (stale) Mirror 10: 0xd1509633 (stale) Mirror 11: 0xd1509633 (stale) Mirror 13: 0xd1509633 (stale) Mirror 14: 0xd1509633 (stale) Mirror 15: 0xd1509633 (stale) Mirror 16: 0xd1509633 (stale) Mirror 17: 0xd1509633 (stale) Mirror 18: 0xd1509633 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcff: skipped 000320 1786481435.324767 trunc from 0x018ffc to 0x02351b (0xa520 bytes) 000321 1786481435.332670 mapwrite 0x0045aa thru 0x00b8bb (0x7312 bytes) 000322 1786481435.374064 punch from 0x019f61 to 0x0286ff (0xe79f bytes) 000323 1786481435.379645 read 0x00ef67 thru 0x00f10d (0x1a7 bytes) 000324 1786481435.380915 mapread 0x000a98 thru 0x0101ee (0xf757 bytes) 000325 1786481435.391752 MIRROR_RESYNC 11 mirrors 000326 1786481438.284689 mapwrite 0x01d84d thru 0x02b817 (0xdfcb bytes) 000327 1786481438.451831 trunc from 0x02bcff to 0x02b09b (0xc63 bytes) 000328 1786481438.482972 write 0x0196ab thru 0x0260a1 (0xc9f7 bytes) 000329 1786481438.490329 fallocate 0x015129 thru 0x01fe56 (0xad2e bytes) 000330 1786481438.499964 mapwrite 0x03e6f4 thru 0x03ffff (0x190c bytes) 000331 1786481438.572410 fallocate 0x00a4e2 thru 0x00eaa4 (0x45c3 bytes) 000332 1786481438.591190 punch from 0x03b662 to 0x03ffff (0x499e bytes) 000334 1786481438.599458 write 0x037899 thru 0x03ffff (0x8767 bytes) 000335 1786481438.602335 punch from 0x0378be to 0x03acf3 (0x3436 bytes) 000336 1786481438.617461 trunc from 0x040000 to 0x01c124 (0x23edb bytes) 000337 1786481438.639829 write 0x02993e thru 0x03709d (0xd760 bytes) 000338 1786481438.644097 mapread 0x00e21a thru 0x01630b (0x80f2 bytes) 000340 1786481438.662029 mapread 0x034f9b thru 0x03709d (0x2103 bytes) 000341 1786481438.664839 mapread 0x015638 thru 0x023103 (0xdacc bytes) 000342 1786481438.679646 trunc from 0x03709e to 0x015af0 (0x215ad bytes) 000343 1786481438.724799 write 0x025a51 thru 0x02b4e5 (0x5a95 bytes) 000344 1786481438.736419 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2b4e6): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b4e6): Mirror 7: 0x9066b399 (stale) Mirror 9: 0x9066b399 (stale) Mirror 10: 0x9066b399 (stale) Mirror 11: 0x9066b399 (stale) Mirror 13: 0x9066b399 (stale) Mirror 14: 0x9066b399 (stale) Mirror 15: 0x9066b399 (stale) Mirror 16: 0x9066b399 (stale) Mirror 17: 0x9066b399 (stale) Mirror 18: 0x9066b399 (stale) CRC-32 checksum value for chunk [0, 0x2b4e6): Mirror 1: 0x1705ed55 Mirror 19: 0x1705ed55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b4e6: skipped 000345 1786481440.161093 trunc from 0x02b4e6 to 0x03669d (0xb1b8 bytes) 000347 1786481440.245604 trunc from 0x03669e to 0x026cca (0xf9d3 bytes) 000348 1786481440.271955 punch from 0x00f3cc to 0x01decc (0xeb01 bytes) 000349 1786481440.279105 mapwrite 0x001199 thru 0x003290 (0x20f8 bytes) 000350 1786481440.323329 trunc from 0x026ccb to 0x024d97 (0x1f33 bytes) 000351 1786481440.344710 mapread 0x012728 thru 0x019edc (0x77b5 bytes) 000352 1786481440.357173 punch from 0x00a1b0 to 0x00b228 (0x1079 bytes) 000353 1786481440.368676 fallocate 0x0067b0 thru 0x009392 (0x2be3 bytes) 000354 1786481440.378664 punch from 0x023b9e to 0x024d97 (0x11fa bytes) 000355 1786481440.386088 mapread 0x021436 thru 0x022111 (0xcdc bytes) 000357 1786481440.404936 fallocate 0x03ed65 thru 0x03ffff (0x129b bytes) 000358 1786481440.412071 trunc from 0x0080ff to 0x009f48 (0x1e4a bytes) 000359 1786481440.423547 mapwrite 0x01f68b thru 0x01fbe4 (0x55a bytes) 000360 1786481440.441851 mapread 0x020c07 thru 0x021c76 (0x1070 bytes) 000361 1786481440.443853 read 0x00d605 thru 0x019684 (0xc080 bytes) 000362 1786481440.446717 punch from 0x003a09 to 0x00b11c (0x7714 bytes) 000363 1786481440.455675 mapwrite 0x013667 thru 0x014a0b (0x13a5 bytes) 000364 1786481440.481559 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x24d98): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d98): Mirror 7: 0x772e9bfb (stale) Mirror 9: 0x772e9bfb (stale) Mirror 10: 0x772e9bfb (stale) Mirror 11: 0x772e9bfb (stale) Mirror 13: 0x772e9bfb (stale) Mirror 14: 0x772e9bfb (stale) Mirror 15: 0x772e9bfb (stale) Mirror 16: 0x772e9bfb (stale) Mirror 17: 0x772e9bfb (stale) Mirror 18: 0x772e9bfb (stale) Mirror 19: 0x772e9bfb (stale) CRC-32 checksum value for chunk [0, 0x24d98): Mirror 1: 0x540c0c7b Mirror 20: 0x540c0c7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d98: skipped 000365 1786481441.742531 mapwrite 0x026ddb thru 0x028615 (0x183b bytes) 000366 1786481441.908911 mapwrite 0x0282c2 thru 0x02d36d (0x50ac bytes) 000367 1786481441.977964 mapread 0x01d891 thru 0x02bdb9 (0xe529 bytes) 000368 1786481442.008756 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2d36e): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d36e): Mirror 7: 0x340243d1 (stale) Mirror 9: 0x340243d1 (stale) Mirror 10: 0x340243d1 (stale) Mirror 11: 0x340243d1 (stale) Mirror 13: 0x340243d1 (stale) Mirror 14: 0x340243d1 (stale) Mirror 15: 0x340243d1 (stale) Mirror 16: 0x340243d1 (stale) Mirror 17: 0x340243d1 (stale) Mirror 18: 0x340243d1 (stale) Mirror 19: 0xe4dd7af5 (stale) Mirror 20: 0xb25fb021 (stale) CRC-32 checksum value for chunk [0, 0x2d36e): Mirror 1: 0x19bd7d6b Mirror 21: 0x19bd7d6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d36e: skipped 000369 1786481443.792424 mapread 0x013cc0 thru 0x01a438 (0x6779 bytes) 000370 1786481443.853488 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d36e): Mirror 9: 0x340243d1 (stale) Mirror 10: 0x340243d1 (stale) Mirror 11: 0x340243d1 (stale) Mirror 13: 0x340243d1 (stale) Mirror 14: 0x340243d1 (stale) Mirror 15: 0x340243d1 (stale) Mirror 16: 0x340243d1 (stale) Mirror 17: 0x340243d1 (stale) Mirror 18: 0x340243d1 (stale) Mirror 19: 0xe4dd7af5 (stale) Mirror 20: 0xb25fb021 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d36e): Mirror 9: 0x340243d1 (stale) Mirror 10: 0x340243d1 (stale) Mirror 11: 0x340243d1 (stale) Mirror 13: 0x340243d1 (stale) Mirror 14: 0x340243d1 (stale) Mirror 15: 0x340243d1 (stale) Mirror 16: 0x340243d1 (stale) Mirror 17: 0x340243d1 (stale) Mirror 18: 0x340243d1 (stale) Mirror 19: 0xe4dd7af5 (stale) Mirror 20: 0xb25fb021 (stale) CRC-32 checksum value for chunk [0, 0x2d36e): Mirror 1: 0x19bd7d6b Mirror 21: 0x19bd7d6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d36e: skipped 000371 1786481445.302960 punch from 0x0249b5 to 0x026694 (0x1ce0 bytes) 000372 1786481445.389913 write 0x017cc8 thru 0x01cce6 (0x501f bytes) 000373 1786481445.413468 trunc from 0x02d36e to 0x0138d9 (0x19a94 bytes) 000374 1786481445.445018 read 0x006439 thru 0x00e8ef (0x84b7 bytes) 000375 1786481445.479593 fallocate 0x01fffe thru 0x026ffd (0x7000 bytes) 000376 1786481445.495771 MIRROR_RESYNC 13 mirrors 000377 1786481448.792758 read 0x0004c6 thru 0x00e8ce (0xe409 bytes) 000378 1786481448.806619 mapread 0x0080bb thru 0x00f7ba (0x7700 bytes) 000379 1786481448.813286 punch from 0x01127c to 0x011bfe (0x983 bytes) 000380 1786481448.861386 mapread 0x005d9f thru 0x007b9f (0x1e01 bytes) 000381 1786481448.877675 read 0x01cf63 thru 0x026ffd (0xa09b bytes) 000382 1786481448.890270 punch from 0x00ee00 to 0x019239 (0xa43a bytes) 000383 1786481448.896933 read 0x00e6f6 thru 0x01d7b6 (0xf0c1 bytes) 000384 1786481448.911653 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x26ffe): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ffe): Mirror 9: 0x862bd496 (stale) Mirror 10: 0x862bd496 (stale) Mirror 11: 0x862bd496 (stale) Mirror 13: 0x862bd496 (stale) Mirror 14: 0x862bd496 (stale) Mirror 15: 0x862bd496 (stale) Mirror 16: 0x862bd496 (stale) Mirror 17: 0x862bd496 (stale) Mirror 18: 0x862bd496 (stale) Mirror 19: 0x862bd496 (stale) Mirror 20: 0x862bd496 (stale) Mirror 21: 0x862bd496 (stale) CRC-32 checksum value for chunk [0, 0x26ffe): Mirror 1: 0x61f2f388 Mirror 22: 0x61f2f388 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ffe: skipped 000386 1786481450.043689 trunc from 0x026ffe to 0x03f4de (0x184e1 bytes) 000387 1786481450.098918 trunc from 0x0166d9 to 0x01716f (0xa97 bytes) 000388 1786481450.103778 mapwrite 0x00f8e0 thru 0x018fe4 (0x9705 bytes) 000389 1786481450.145840 write 0x01796f thru 0x01f82d (0x7ebf bytes) 000391 1786481450.152677 read 0x0340f1 thru 0x03e953 (0xa863 bytes) 000392 1786481450.161442 fallocate 0x03b6cb thru 0x03ffff (0x4935 bytes) 000393 1786481450.166683 mapread 0x01ddfa thru 0x024d14 (0x6f1b bytes) 000394 1786481450.176655 fallocate 0x028b9d thru 0x0294ae (0x912 bytes) 000395 1786481450.182194 mapread 0x0010cf thru 0x009c17 (0x8b49 bytes) 000396 1786481450.191222 punch from 0x0327ba to 0x035624 (0x2e6b bytes) 000397 1786481450.196836 trunc from 0x03a897 to 0x03ffff (0x5769 bytes) 000398 1786481450.202423 punch from 0x023530 to 0x02b40c (0x7edd bytes) 000399 1786481450.207731 mapread 0x003677 thru 0x00875b (0x50e5 bytes) 000400 1786481450.208976 mapwrite 0x02b71c thru 0x038b08 (0xd3ed bytes) 000401 1786481450.254621 write 0x033a5d thru 0x0374a0 (0x3a44 bytes) 000402 1786481450.270653 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb430cb6a (stale) Mirror 10: 0xb430cb6a (stale) Mirror 11: 0xb430cb6a (stale) Mirror 13: 0xb430cb6a (stale) Mirror 14: 0xb430cb6a (stale) Mirror 15: 0xb430cb6a (stale) Mirror 16: 0xb430cb6a (stale) Mirror 18: 0xb430cb6a (stale) Mirror 19: 0xb430cb6a (stale) Mirror 20: 0xb430cb6a (stale) Mirror 21: 0xb430cb6a (stale) Mirror 22: 0x4bc884fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb430cb6a (stale) Mirror 10: 0xb430cb6a (stale) Mirror 11: 0xb430cb6a (stale) Mirror 13: 0xb430cb6a (stale) Mirror 14: 0xb430cb6a (stale) Mirror 15: 0xb430cb6a (stale) Mirror 16: 0xb430cb6a (stale) Mirror 18: 0xb430cb6a (stale) Mirror 19: 0xb430cb6a (stale) Mirror 20: 0xb430cb6a (stale) Mirror 21: 0xb430cb6a (stale) Mirror 22: 0x4bc884fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000403 1786481451.206871 trunc from 0x040000 to 0x01a144 (0x25ebb bytes) 000404 1786481451.242813 punch from 0x0050da to 0x010890 (0xb7b7 bytes) 000405 1786481451.248828 trunc from 0x0094e6 to 0x016a43 (0xd55e bytes) 000406 1786481451.255711 punch from 0x0186f2 to 0x018fd8 (0x8e7 bytes) 000408 1786481451.265995 punch from 0x001bfd to 0x00da8e (0xbe92 bytes) 000409 1786481451.272299 write 0x03c67a thru 0x03de6d (0x17f4 bytes) 000410 1786481451.273104 trunc from 0x03de6e to 0x02e636 (0xf837 bytes) 000411 1786481451.299896 punch from 0x00a5f8 to 0x017a6c (0xd475 bytes) 000412 1786481451.305859 fallocate 0x038d86 thru 0x039d02 (0xf7d bytes) 000413 1786481451.310053 mapread 0x025cc4 thru 0x02e68e (0x89cb bytes) 000414 1786481451.331336 mapread 0x007143 thru 0x011e4b (0xad09 bytes) 000415 1786481451.353741 MIRROR_RESYNC 13 mirrors 000416 1786481453.504058 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) 000417 1786481454.063066 write 0x01854b thru 0x0257c1 (0xd277 bytes) 000418 1786481454.139096 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39d03): Mirror 9: 0xa096fbec (stale) Mirror 10: 0xa096fbec (stale) Mirror 11: 0xa096fbec (stale) Mirror 13: 0xa096fbec (stale) Mirror 15: 0xa096fbec (stale) Mirror 16: 0xa096fbec (stale) Mirror 19: 0xa096fbec (stale) Mirror 20: 0xa096fbec (stale) Mirror 21: 0xa096fbec (stale) Mirror 22: 0xa096fbec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d03): Mirror 9: 0xa096fbec (stale) Mirror 10: 0xa096fbec (stale) Mirror 11: 0xa096fbec (stale) Mirror 13: 0xa096fbec (stale) Mirror 15: 0xa096fbec (stale) Mirror 16: 0xa096fbec (stale) Mirror 19: 0xa096fbec (stale) Mirror 20: 0xa096fbec (stale) Mirror 21: 0xa096fbec (stale) Mirror 22: 0xa096fbec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d03: skipped 000419 1786481454.705403 punch from 0x01fae1 to 0x027469 (0x7989 bytes) 000421 1786481454.711794 trunc from 0x039d03 to 0x02fcc8 (0xa03a bytes) 000422 1786481454.730827 punch from 0x006c0b to 0x00ed2c (0x8122 bytes) 000423 1786481454.736419 trunc from 0x02fcc9 to 0x02af15 (0x4db3 bytes) 000424 1786481454.756315 punch from 0x0117d3 to 0x01815a (0x6988 bytes) 000425 1786481454.761905 trunc from 0x010231 to 0x0143da (0x41aa bytes) 000427 1786481454.768641 read 0x000645 thru 0x00b24d (0xac09 bytes) 000428 1786481454.782571 trunc from 0x02af16 to 0x009df2 (0x21123 bytes) 000429 1786481454.804422 punch from 0x0082f6 to 0x009df2 (0x1afd bytes) 000430 1786481454.809919 read 0x00765c thru 0x0079bf (0x364 bytes) 000431 1786481454.815291 MIRROR_RESYNC 11 mirrors 000432 1786481456.072386 mapread 0x0079aa thru 0x009df2 (0x2449 bytes) 000433 1786481456.094589 write 0x0005e7 thru 0x004bcd (0x45e7 bytes) 000434 1786481456.138696 write 0x01c139 thru 0x02a420 (0xe2e8 bytes) 000435 1786481456.140674 trunc from 0x02a421 to 0x00c55d (0x1dec3 bytes) 000436 1786481456.165796 read 0x00ac8f thru 0x00c55d (0x18cf bytes) 000437 1786481456.171438 read 0x0099c0 thru 0x00c55d (0x2b9e bytes) 000439 1786481456.181679 write 0x01d8b9 thru 0x01e6e7 (0xe2f bytes) 000440 1786481456.183783 write 0x02d918 thru 0x03c568 (0xec51 bytes) 000441 1786481456.186765 punch from 0x019ed5 to 0x01ada6 (0xed2 bytes) 000443 1786481456.191775 trunc from 0x011d8e to 0x017149 (0x53bc bytes) 000444 1786481456.196662 read 0x0096f3 thru 0x00d34e (0x3c5c bytes) 000445 1786481456.223259 MIRROR_RESYNC 11 mirrors 000446 1786481458.144273 punch from 0x02c176 to 0x0382a9 (0xc134 bytes) 000448 1786481458.183584 mapwrite 0x02b50c thru 0x0315e1 (0x60d6 bytes) 000449 1786481458.224854 write 0x01c36f thru 0x01eb8b (0x281d bytes) 000450 1786481458.236662 read 0x01d6fb thru 0x021255 (0x3b5b bytes) 000451 1786481458.242743 write 0x00e584 thru 0x014aa7 (0x6524 bytes) 000452 1786481458.254646 write 0x02e3a6 thru 0x03b986 (0xd5e1 bytes) 000453 1786481458.263772 fallocate 0x006a0a thru 0x00f7f2 (0x8de9 bytes) 000454 1786481458.268772 mapwrite 0x014efd thru 0x01e588 (0x968c bytes) 000455 1786481458.298711 write 0x013126 thru 0x021710 (0xe5eb bytes) 000456 1786481458.300499 mapread 0x033988 thru 0x03c568 (0x8be1 bytes) 000457 1786481458.308007 trunc from 0x01f43e to 0x021bec (0x27af bytes) 000458 1786481458.324331 read 0x03c54e thru 0x03c568 (0x1b bytes) 000459 1786481458.325157 write 0x02fb6c thru 0x034d4d (0x51e2 bytes) 000460 1786481458.327663 mapread 0x034f67 thru 0x03beb4 (0x6f4e bytes) 000461 1786481458.328929 read 0x022ba8 thru 0x02ff22 (0xd37b bytes) 000462 1786481458.343103 trunc from 0x02053c to 0x0301f9 (0xfcbe bytes) 000463 1786481458.362939 trunc from 0x03c569 to 0x002354 (0x3a214 bytes) 000464 1786481458.382636 trunc from 0x002355 to 0x0392b6 (0x36f62 bytes) 000465 1786481458.404407 trunc from 0x033cc2 to 0x0392b6 (0x55f5 bytes) 000466 1786481458.410100 trunc from 0x0392b7 to 0x005e66 (0x33450 bytes) 000468 1786481458.430938 MIRROR_RESYNC 11 mirrors 000469 1786481459.928804 fallocate 0x02b473 thru 0x03154e (0x60dc bytes) 000470 1786481459.971944 mapread 0x004037 thru 0x005e66 (0x1e30 bytes) 000471 1786481459.995740 MIRROR_RESYNC 11 mirrors 000472 1786481461.373099 punch from 0x001b97 to 0x005e66 (0x42d0 bytes) 000473 1786481461.410344 mapread 0x0058fe thru 0x005e66 (0x569 bytes) 000474 1786481461.418858 trunc from 0x005e67 to 0x00a0af (0x4249 bytes) 000475 1786481461.440254 punch from 0x003538 to 0x004155 (0xc1e bytes) 000476 1786481461.448924 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa0b0): Mirror 9: 0xd8ea0e42 (stale) Mirror 10: 0xd8ea0e42 (stale) Mirror 11: 0xd8ea0e42 (stale) Mirror 13: 0xd8ea0e42 (stale) Mirror 15: 0xd8ea0e42 (stale) Mirror 16: 0xd8ea0e42 (stale) Mirror 19: 0xd8ea0e42 (stale) Mirror 20: 0xd8ea0e42 (stale) Mirror 22: 0xd8ea0e42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa0b0): Mirror 9: 0xd8ea0e42 (stale) Mirror 10: 0xd8ea0e42 (stale) Mirror 11: 0xd8ea0e42 (stale) Mirror 13: 0xd8ea0e42 (stale) Mirror 15: 0xd8ea0e42 (stale) Mirror 16: 0xd8ea0e42 (stale) Mirror 19: 0xd8ea0e42 (stale) Mirror 20: 0xd8ea0e42 (stale) Mirror 22: 0xd8ea0e42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa0b0: skipped 000477 1786481461.886941 mapwrite 0x028839 thru 0x037d6a (0xf532 bytes) 000478 1786481461.964929 punch from 0x0178e8 to 0x01d5a1 (0x5cba bytes) 000479 1786481461.971807 punch from 0x00d926 to 0x01276f (0x4e4a bytes) 000480 1786481461.982260 MIRROR_RESYNC 10 mirrors 000481 1786481463.687955 punch from 0x01ef62 to 0x01fb6e (0xc0d bytes) 000482 1786481463.740010 mapwrite 0x013712 thru 0x01846e (0x4d5d bytes) 000483 1786481463.808570 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37d6b): Mirror 9: 0x252c618b (stale) Mirror 10: 0x252c618b (stale) Mirror 11: 0x252c618b (stale) Mirror 13: 0x252c618b (stale) Mirror 16: 0x252c618b (stale) Mirror 19: 0x252c618b (stale) Mirror 20: 0x252c618b (stale) Mirror 22: 0x252c618b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d6b): Mirror 9: 0x252c618b (stale) Mirror 10: 0x252c618b (stale) Mirror 11: 0x252c618b (stale) Mirror 13: 0x252c618b (stale) Mirror 16: 0x252c618b (stale) Mirror 19: 0x252c618b (stale) Mirror 20: 0x252c618b (stale) Mirror 22: 0x252c618b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d6b: skipped 000484 1786481464.710438 trunc from 0x02e7a2 to 0x037a07 (0x9266 bytes) 000486 1786481464.730430 write 0x01f904 thru 0x026297 (0x6994 bytes) 000487 1786481464.752393 read 0x011086 thru 0x014615 (0x3590 bytes) 000488 1786481464.760030 write 0x0288d0 thru 0x02aefa (0x262b bytes) 000489 1786481464.773058 mapread 0x008e9f thru 0x01432b (0xb48d bytes) 000490 1786481464.796335 write 0x001f90 thru 0x0032bd (0x132e bytes) 000491 1786481464.803207 trunc from 0x037d6b to 0x016ce4 (0x21086 bytes) 000492 1786481464.829826 MIRROR_RESYNC 9 mirrors 000493 1786481466.302701 write 0x03aa37 thru 0x03ffff (0x55c9 bytes) 000495 1786481466.352557 trunc from 0x040000 to 0x0271b2 (0x18e4d bytes) 000496 1786481466.385694 punch from 0x0177b8 to 0x01da9b (0x62e4 bytes) 000497 1786481466.395933 mapread 0x00bd5a thru 0x00c65d (0x904 bytes) 000498 1786481466.410625 trunc from 0x024df5 to 0x0271b2 (0x23be bytes) 000500 1786481466.418413 punch from 0x00c8b6 to 0x014f14 (0x865f bytes) 000501 1786481466.426422 read 0x01f517 thru 0x01f627 (0x111 bytes) 000502 1786481466.434055 write 0x019597 thru 0x027cc0 (0xe72a bytes) 000503 1786481466.456910 write 0x01be76 thru 0x020114 (0x429f bytes) 000504 1786481466.459428 read 0x025479 thru 0x027cc0 (0x2848 bytes) 000505 1786481466.460633 trunc from 0x0128ab to 0x0137a2 (0xef8 bytes) 000506 1786481466.465918 trunc from 0x0084ea to 0x0160ac (0xdbc3 bytes) 000507 1786481466.482641 trunc from 0x027cc1 to 0x031e98 (0xa1d8 bytes) 000508 1786481466.506296 trunc from 0x014521 to 0x01c5f0 (0x80d0 bytes) 000510 1786481466.546982 read 0x013969 thru 0x0185af (0x4c47 bytes) 000511 1786481466.575808 punch from 0x0286de to 0x031013 (0x8936 bytes) 000512 1786481466.590305 read 0x030309 thru 0x031e98 (0x1b90 bytes) 000513 1786481466.611963 trunc from 0x00fc5f to 0x018fa0 (0x9342 bytes) 000514 1786481466.625936 mapwrite 0x014c37 thru 0x01cc01 (0x7fcb bytes) 000515 1786481466.732420 trunc from 0x031e99 to 0x0293be (0x8ada bytes) 000516 1786481466.753834 trunc from 0x01003c to 0x0102b2 (0x277 bytes) 000517 1786481466.761378 trunc from 0x0293bf to 0x02f198 (0x5dda bytes) 000518 1786481466.782798 read 0x000ba8 thru 0x0060dd (0x5536 bytes) 000519 1786481466.788312 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2f199): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f199): Mirror 9: 0xf4b42381 (stale) Mirror 10: 0xf4b42381 (stale) Mirror 11: 0xf4b42381 (stale) Mirror 13: 0xf4b42381 (stale) Mirror 16: 0xf4b42381 (stale) Mirror 19: 0xf4b42381 (stale) Mirror 20: 0xf4b42381 (stale) Mirror 22: 0xf4b42381 (stale) CRC-32 checksum value for chunk [0, 0x2f199): Mirror 1: 0x18eb400a Mirror 23: 0x18eb400a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f199: skipped 000520 1786481467.875563 mapread 0x01cd3c thru 0x0282b9 (0xb57e bytes) 000521 1786481467.939562 trunc from 0x02f199 to 0x00d198 (0x22000 bytes) 000522 1786481468.011518 fallocate 0x00d744 thru 0x0126a5 (0x4f62 bytes) 000523 1786481468.018457 write 0x0288c7 thru 0x02e0d9 (0x5813 bytes) 000524 1786481468.022229 mapwrite 0x026701 thru 0x033336 (0xcc36 bytes) 000525 1786481468.100270 read 0x012541 thru 0x016cc1 (0x4781 bytes) 000526 1786481468.110579 trunc from 0x033337 to 0x03610b (0x2dd5 bytes) 000527 1786481468.131350 mapread 0x034fb4 thru 0x03610b (0x1158 bytes) 000530 1786481468.139999 trunc from 0x01501c to 0x01ccaf (0x7c94 bytes) 000531 1786481468.144438 trunc from 0x026715 to 0x029601 (0x2eed bytes) 000532 1786481468.154048 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3610c): Mirror 9: 0xe5ab9f12 (stale) Mirror 10: 0xe5ab9f12 (stale) Mirror 11: 0xe5ab9f12 (stale) Mirror 16: 0xe5ab9f12 (stale) Mirror 19: 0xe5ab9f12 (stale) Mirror 20: 0xe5ab9f12 (stale) Mirror 22: 0xe5ab9f12 (stale) Mirror 23: 0xe1fe9aa5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3610c): Mirror 9: 0xe5ab9f12 (stale) Mirror 10: 0xe5ab9f12 (stale) Mirror 11: 0xe5ab9f12 (stale) Mirror 16: 0xe5ab9f12 (stale) Mirror 19: 0xe5ab9f12 (stale) Mirror 20: 0xe5ab9f12 (stale) Mirror 22: 0xe5ab9f12 (stale) Mirror 23: 0xe1fe9aa5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3610c: skipped 000533 1786481468.822298 write 0x0178cb thru 0x019fbd (0x26f3 bytes) 000534 1786481468.840733 punch from 0x01a2cc to 0x021f6e (0x7ca3 bytes) 000535 1786481468.846854 read 0x000469 thru 0x003112 (0x2caa bytes) 000538 1786481468.857502 read 0x01d7d8 thru 0x01effd (0x1826 bytes) 000539 1786481468.875843 trunc from 0x03610c to 0x031bc4 (0x4547 bytes) 000540 1786481468.891856 punch from 0x026860 to 0x02fb1e (0x92bf bytes) 000541 1786481468.899325 fallocate 0x0044c7 thru 0x00d4bc (0x8ff6 bytes) 000542 1786481468.905414 write 0x011759 thru 0x013bfa (0x24a2 bytes) 000543 1786481468.907188 trunc from 0x031bc5 to 0x039e05 (0x8241 bytes) 000544 1786481468.928940 write 0x0034da thru 0x00a8e9 (0x7410 bytes) 000545 1786481468.930474 fallocate 0x0090b7 thru 0x011ba6 (0x8af0 bytes) 000546 1786481468.937130 write 0x010c5d thru 0x015a3e (0x4de2 bytes) 000547 1786481468.938619 trunc from 0x01eed5 to 0x02173a (0x2866 bytes) 000548 1786481468.949230 mapread 0x01caec thru 0x02a7c1 (0xdcd6 bytes) 000549 1786481469.000216 read 0x0269a1 thru 0x03662f (0xfc8f bytes) 000551 1786481469.010414 trunc from 0x039e06 to 0x02762d (0x127d8 bytes) 000552 1786481469.029466 write 0x02d404 thru 0x0322cc (0x4ec9 bytes) 000553 1786481469.032455 trunc from 0x00ac48 to 0x00d872 (0x2c2b bytes) 000554 1786481469.054035 punch from 0x0105c4 to 0x017c7b (0x76b8 bytes) 000555 1786481469.063918 write 0x010f82 thru 0x01f920 (0xe99f bytes) 000556 1786481469.073886 mapwrite 0x03bf29 thru 0x03ffff (0x40d7 bytes) 000557 1786481469.121588 read 0x029a48 thru 0x02dba4 (0x415d bytes) 000558 1786481469.130294 read 0x03c642 thru 0x03ffff (0x39be bytes) 000559 1786481469.130867 mapwrite 0x00a821 thru 0x00ae47 (0x627 bytes) 000560 1786481469.139318 punch from 0x031f47 to 0x0370e9 (0x51a3 bytes) 000561 1786481469.143368 fallocate 0x00b9a4 thru 0x0167a4 (0xae01 bytes) 000562 1786481469.153062 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf545587c (stale) Mirror 10: 0xf545587c (stale) Mirror 11: 0xf545587c (stale) Mirror 16: 0xf545587c (stale) Mirror 19: 0xf545587c (stale) Mirror 20: 0xf545587c (stale) Mirror 22: 0xf545587c (stale) Mirror 23: 0x67e4f76c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x34bf36d0 Mirror 24: 0x34bf36d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000563 1786481470.406503 punch from 0x0220a6 to 0x02693e (0x4899 bytes) 000564 1786481470.457551 write 0x016236 thru 0x0238af (0xd67a bytes) 000566 1786481470.489534 read 0x024d09 thru 0x02ff2f (0xb227 bytes) 000567 1786481470.507989 mapwrite 0x003dd2 thru 0x013434 (0xf663 bytes) 000568 1786481470.560988 mapwrite 0x00ab6e thru 0x012bcb (0x805e bytes) 000569 1786481470.579190 trunc from 0x01ac53 to 0x02359c (0x894a bytes) 000570 1786481470.584530 mapwrite 0x009a5d thru 0x017512 (0xdab6 bytes) 000571 1786481470.610263 mapwrite 0x01e333 thru 0x01e424 (0xf2 bytes) 000572 1786481470.639925 write 0x007ec8 thru 0x013444 (0xb57d bytes) 000573 1786481470.641407 trunc from 0x01451c to 0x01ddb7 (0x989c bytes) 000574 1786481470.658366 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf545587c (stale) Mirror 10: 0xf545587c (stale) Mirror 11: 0xf545587c (stale) Mirror 16: 0xf545587c (stale) Mirror 19: 0xf545587c (stale) Mirror 20: 0xf545587c (stale) Mirror 22: 0xf545587c (stale) Mirror 23: 0x67e4f76c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf545587c (stale) Mirror 10: 0xf545587c (stale) Mirror 11: 0xf545587c (stale) Mirror 16: 0xf545587c (stale) Mirror 19: 0xf545587c (stale) Mirror 20: 0xf545587c (stale) Mirror 22: 0xf545587c (stale) Mirror 23: 0x67e4f76c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000575 1786481471.216215 trunc from 0x040000 to 0x023702 (0x1c8fd bytes) 000576 1786481471.270360 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x23703): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23703): Mirror 9: 0x3b83e263 (stale) Mirror 10: 0x3b83e263 (stale) Mirror 11: 0x3b83e263 (stale) Mirror 16: 0x3b83e263 (stale) Mirror 19: 0x3b83e263 (stale) Mirror 20: 0x3b83e263 (stale) Mirror 22: 0x3b83e263 (stale) Mirror 23: 0x146a5a2c (stale) CRC-32 checksum value for chunk [0, 0x23703): Mirror 1: 0x9a3bbb74 Mirror 24: 0x9a3bbb74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23703: skipped 000577 1786481472.229490 fallocate 0x01242e thru 0x01eafc (0xc6cf bytes) 000578 1786481472.274781 mapread 0x01762e thru 0x0212ab (0x9c7e bytes) 000579 1786481472.300765 fallocate 0x02a73e thru 0x02d54d (0x2e10 bytes) 000580 1786481472.305883 mapread 0x02177c thru 0x023702 (0x1f87 bytes) 000581 1786481472.311191 mapread 0x01070f thru 0x01a1f5 (0x9ae7 bytes) 000582 1786481472.320066 write 0x012492 thru 0x014397 (0x1f06 bytes) 000583 1786481472.325637 mapread 0x00e300 thru 0x00f455 (0x1156 bytes) 000584 1786481472.326950 fallocate 0x010999 thru 0x01f209 (0xe871 bytes) 000585 1786481472.332963 trunc from 0x001064 to 0x00bf76 (0xaf13 bytes) 000586 1786481472.338404 punch from 0x020552 to 0x02201d (0x1acc bytes) 000587 1786481472.343625 fallocate 0x032c3a thru 0x03d603 (0xa9ca bytes) 000588 1786481472.349456 trunc from 0x023703 to 0x03158c (0xde8a bytes) 000589 1786481472.371804 mapread 0x00e281 thru 0x01c044 (0xddc4 bytes) 000590 1786481472.373994 write 0x00be2c thru 0x01479e (0x8973 bytes) 000591 1786481472.379756 fallocate 0x00e837 thru 0x01ca6d (0xe237 bytes) 000592 1786481472.385156 write 0x03f91a thru 0x03ffff (0x6e6 bytes) 000593 1786481472.387184 write 0x03b019 thru 0x03ffff (0x4fe7 bytes) 000594 1786481472.396352 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf545587c (stale) Mirror 10: 0xf545587c (stale) Mirror 11: 0xf545587c (stale) Mirror 16: 0xf545587c (stale) Mirror 19: 0xf545587c (stale) Mirror 20: 0xf545587c (stale) Mirror 22: 0xf545587c (stale) Mirror 23: 0x67e4f76c (stale) Mirror 24: 0x9b802398 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x33c7215c Mirror 25: 0x33c7215c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000595 1786481473.427019 trunc from 0x040000 to 0x020969 (0x1f696 bytes) 000596 1786481473.536182 punch from 0x015f8b to 0x020969 (0xa9df bytes) 000597 1786481473.544954 mapwrite 0x031dfa thru 0x03a7e4 (0x89eb bytes) 000598 1786481473.669879 trunc from 0x03a7e5 to 0x0236eb (0x170f9 bytes) 000599 1786481473.725092 mapwrite 0x01eedb thru 0x023cfa (0x4e20 bytes) 000600 1786481473.804708 write 0x0183c6 thru 0x0223c7 (0xa002 bytes) 000601 1786481473.805806 punch from 0x012037 to 0x016976 (0x4940 bytes) 000602 1786481473.811465 trunc from 0x0178fe to 0x01d0ae (0x57b1 bytes) 000603 1786481473.823104 mapwrite 0x02ca4e thru 0x0352d5 (0x8888 bytes) 000604 1786481473.879033 fallocate 0x0207a6 thru 0x02fe25 (0xf680 bytes) 000605 1786481473.892570 trunc from 0x0352d6 to 0x0023a9 (0x32f2c bytes) 000606 1786481473.922920 MIRROR_RESYNC 11 mirrors 000607 1786481475.508882 mapread 0x00221e thru 0x0023a9 (0x18c bytes) 000608 1786481475.538642 MIRROR_EXTEND to 12 mirrors 000609 1786481476.220182 mapwrite 0x03dc62 thru 0x03ffff (0x239e bytes) 000610 1786481476.360844 write 0x03b247 thru 0x03ffff (0x4db9 bytes) 000611 1786481476.362435 mapwrite 0x00b6b2 thru 0x011358 (0x5ca7 bytes) 000612 1786481476.391973 trunc from 0x0223c6 to 0x02eb98 (0xc7d3 bytes) 000613 1786481476.397066 trunc from 0x040000 to 0x014cc5 (0x2b33a bytes) 000614 1786481476.417990 mapread 0x01022a thru 0x014cc5 (0x4a9c bytes) 000615 1786481476.424760 trunc from 0x014cc6 to 0x0200bf (0xb3fa bytes) 000616 1786481476.447456 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x200c0): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200c0): Mirror 9: 0xae53e8b6 (stale) Mirror 10: 0xae53e8b6 (stale) Mirror 11: 0xae53e8b6 (stale) Mirror 16: 0xae53e8b6 (stale) Mirror 19: 0xae53e8b6 (stale) Mirror 20: 0xae53e8b6 (stale) Mirror 22: 0xae53e8b6 (stale) Mirror 23: 0xae53e8b6 (stale) Mirror 24: 0xae53e8b6 (stale) Mirror 25: 0xae53e8b6 (stale) Mirror 26: 0xae53e8b6 (stale) CRC-32 checksum value for chunk [0, 0x200c0): Mirror 1: 0x8db3f11b Mirror 27: 0x8db3f11b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200c0: skipped 000617 1786481477.956389 mapread 0x002ec8 thru 0x012b35 (0xfc6e bytes) 000618 1786481478.022085 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x200c0): Mirror 10: 0xae53e8b6 (stale) Mirror 11: 0xae53e8b6 (stale) Mirror 16: 0xae53e8b6 (stale) Mirror 19: 0xae53e8b6 (stale) Mirror 20: 0xae53e8b6 (stale) Mirror 22: 0xae53e8b6 (stale) Mirror 23: 0xae53e8b6 (stale) Mirror 24: 0xae53e8b6 (stale) Mirror 25: 0xae53e8b6 (stale) Mirror 26: 0xae53e8b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200c0): Mirror 10: 0xae53e8b6 (stale) Mirror 11: 0xae53e8b6 (stale) Mirror 16: 0xae53e8b6 (stale) Mirror 19: 0xae53e8b6 (stale) Mirror 20: 0xae53e8b6 (stale) Mirror 22: 0xae53e8b6 (stale) Mirror 23: 0xae53e8b6 (stale) Mirror 24: 0xae53e8b6 (stale) Mirror 25: 0xae53e8b6 (stale) Mirror 26: 0xae53e8b6 (stale) CRC-32 checksum value for chunk [0, 0x200c0): Mirror 1: 0x8db3f11b Mirror 27: 0x8db3f11b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200c0: skipped 000620 1786481479.106255 punch from 0x01e91b to 0x0200bf (0x17a5 bytes) 000621 1786481479.171072 mapread 0x00e319 thru 0x011463 (0x314b bytes) 000622 1786481479.193834 punch from 0x010339 to 0x01985f (0x9527 bytes) 000623 1786481479.200439 mapread 0x019a18 thru 0x0200bf (0x66a8 bytes) 000624 1786481479.212406 punch from 0x0057c1 to 0x00e3f8 (0x8c38 bytes) 000625 1786481479.217753 punch from 0x00c867 to 0x01b0e1 (0xe87b bytes) 000626 1786481479.227216 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x200c0): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200c0): Mirror 10: 0xae53e8b6 (stale) Mirror 11: 0xae53e8b6 (stale) Mirror 16: 0xae53e8b6 (stale) Mirror 19: 0xae53e8b6 (stale) Mirror 20: 0xae53e8b6 (stale) Mirror 22: 0xae53e8b6 (stale) Mirror 23: 0xae53e8b6 (stale) Mirror 24: 0xae53e8b6 (stale) Mirror 25: 0xae53e8b6 (stale) Mirror 26: 0xae53e8b6 (stale) Mirror 27: 0x8db3f11b (stale) CRC-32 checksum value for chunk [0, 0x200c0): Mirror 1: 0xae53e8b6 Mirror 28: 0xae53e8b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200c0: skipped 000627 1786481480.251959 trunc from 0x0200c0 to 0x00b097 (0x15028 bytes) 000628 1786481480.321813 write 0x00da91 thru 0x00fc18 (0x2188 bytes) 000629 1786481480.325154 mapwrite 0x009ad6 thru 0x00c175 (0x26a0 bytes) 000630 1786481480.379514 trunc from 0x00fc19 to 0x00f4f2 (0x726 bytes) 000631 1786481480.412394 MIRROR_RESYNC 13 mirrors 000632 1786481482.604545 mapwrite 0x037ea9 thru 0x03ffff (0x8157 bytes) 000633 1786481482.697311 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x3280ea95 (stale) Mirror 11: 0x3280ea95 (stale) Mirror 16: 0x3280ea95 (stale) Mirror 19: 0x3280ea95 (stale) Mirror 20: 0x3280ea95 (stale) Mirror 22: 0x3280ea95 (stale) Mirror 23: 0x3280ea95 (stale) Mirror 24: 0x3280ea95 (stale) Mirror 25: 0x3280ea95 (stale) Mirror 26: 0x3280ea95 (stale) Mirror 27: 0x3280ea95 (stale) Mirror 28: 0x3280ea95 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa1cf50a0 Mirror 29: 0xa1cf50a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1786481484.288587 punch from 0x006f0b to 0x01557d (0xe673 bytes) 000636 1786481484.423043 MIRROR_RESYNC 14 mirrors 000637 1786481486.958676 trunc from 0x040000 to 0x0181f1 (0x27e0e bytes) 000639 1786481487.025933 fallocate 0x0000d5 thru 0x00d867 (0xd793 bytes) 000640 1786481487.036175 write 0x03b489 thru 0x03ffff (0x4b77 bytes) 000641 1786481487.037735 mapread 0x001669 thru 0x010542 (0xeeda bytes) 000642 1786481487.073161 trunc from 0x02502b to 0x02a264 (0x523a bytes) 000643 1786481487.088181 mapread 0x0122cd thru 0x012656 (0x38a bytes) 000644 1786481487.098671 trunc from 0x024e04 to 0x032d7a (0xdf77 bytes) 000645 1786481487.107980 fallocate 0x030444 thru 0x03be2e (0xb9eb bytes) 000647 1786481487.119792 punch from 0x0374e1 to 0x03d05e (0x5b7e bytes) 000648 1786481487.139743 mapread 0x0260da thru 0x028622 (0x2549 bytes) 000649 1786481487.160473 punch from 0x028b3b to 0x02c836 (0x3cfc bytes) 000650 1786481487.178629 punch from 0x03083d to 0x03d5f5 (0xcdb9 bytes) 000651 1786481487.187203 punch from 0x00afa2 to 0x00b739 (0x798 bytes) 000653 1786481487.199207 mapwrite 0x03b48e thru 0x03ffff (0x4b72 bytes) 000654 1786481487.251636 fallocate 0x007344 thru 0x010a9a (0x9757 bytes) 000655 1786481487.259880 trunc from 0x014361 to 0x021fd7 (0xdc77 bytes) 000656 1786481487.267267 fallocate 0x018612 thru 0x02198f (0x937e bytes) 000657 1786481487.274805 mapread 0x0373c8 thru 0x038c0b (0x1844 bytes) 000658 1786481487.279600 punch from 0x024096 to 0x02e819 (0xa784 bytes) 000659 1786481487.290861 write 0x01205d thru 0x0211dd (0xf181 bytes) 000660 1786481487.312304 mapread 0x035087 thru 0x0367b3 (0x172d bytes) 000661 1786481487.314898 mapwrite 0x00e7b4 thru 0x01d839 (0xf086 bytes) 000662 1786481487.372153 punch from 0x013795 to 0x016f20 (0x378c bytes) 000663 1786481487.381794 trunc from 0x037c9c to 0x03ffff (0x8364 bytes) 000664 1786481487.404796 MIRROR_RESYNC 14 mirrors 000665 1786481491.091955 trunc from 0x01eabd to 0x028a27 (0x9f6b bytes) 000667 1786481491.141431 mapwrite 0x029676 thru 0x03283e (0x91c9 bytes) 000668 1786481491.189238 fallocate 0x02e41f thru 0x03d3d7 (0xefb9 bytes) 000669 1786481491.195761 trunc from 0x03aa20 to 0x03c3b5 (0x1996 bytes) 000670 1786481491.200332 fallocate 0x0352d9 thru 0x03ffff (0xad27 bytes) 000671 1786481491.205600 read 0x01c69a thru 0x024573 (0x7eda bytes) 000672 1786481491.215209 trunc from 0x016a64 to 0x022275 (0xb812 bytes) 000673 1786481491.219940 punch from 0x03a874 to 0x03ffff (0x578c bytes) 000674 1786481491.225772 trunc from 0x040000 to 0x03983f (0x67c0 bytes) 000675 1786481491.249470 write 0x012d55 thru 0x01645a (0x3706 bytes) 000676 1786481491.261526 fallocate 0x02e8c2 thru 0x030cf8 (0x2437 bytes) 000677 1786481491.269233 trunc from 0x039840 to 0x03718d (0x26b2 bytes) 000678 1786481491.288258 write 0x02331d thru 0x02c0da (0x8dbe bytes) 000679 1786481491.294267 read 0x015f70 thru 0x01caf3 (0x6b84 bytes) 000680 1786481491.307840 mapread 0x02e2c2 thru 0x03718d (0x8ecc bytes) 000681 1786481491.309324 mapread 0x00271c thru 0x008715 (0x5ffa bytes) 000683 1786481491.317451 mapwrite 0x0213b7 thru 0x02bdb3 (0xa9fd bytes) 000684 1786481491.349663 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3718e): Mirror 10: 0x64f83a17 (stale) Mirror 11: 0x64f83a17 (stale) Mirror 16: 0x64f83a17 (stale) Mirror 19: 0x64f83a17 (stale) Mirror 20: 0x64f83a17 (stale) Mirror 22: 0x64f83a17 (stale) Mirror 23: 0x64f83a17 (stale) Mirror 24: 0x64f83a17 (stale) Mirror 25: 0x64f83a17 (stale) Mirror 27: 0x64f83a17 (stale) Mirror 28: 0x64f83a17 (stale) Mirror 29: 0x64f83a17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3718e): Mirror 10: 0x64f83a17 (stale) Mirror 11: 0x64f83a17 (stale) Mirror 16: 0x64f83a17 (stale) Mirror 19: 0x64f83a17 (stale) Mirror 20: 0x64f83a17 (stale) Mirror 22: 0x64f83a17 (stale) Mirror 23: 0x64f83a17 (stale) Mirror 24: 0x64f83a17 (stale) Mirror 25: 0x64f83a17 (stale) Mirror 27: 0x64f83a17 (stale) Mirror 28: 0x64f83a17 (stale) Mirror 29: 0x64f83a17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3718e: skipped 000685 1786481491.980549 read 0x03130a thru 0x03718d (0x5e84 bytes) 000686 1786481491.991020 trunc from 0x03718e to 0x027ea7 (0xf2e6 bytes) 000687 1786481492.016318 trunc from 0x025639 to 0x027ea7 (0x286f bytes) 000689 1786481492.027522 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x27ea8): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ea8): Mirror 10: 0x411329cb (stale) Mirror 11: 0x411329cb (stale) Mirror 16: 0x411329cb (stale) Mirror 19: 0x411329cb (stale) Mirror 20: 0x411329cb (stale) Mirror 22: 0x411329cb (stale) Mirror 23: 0x411329cb (stale) Mirror 24: 0x411329cb (stale) Mirror 25: 0x411329cb (stale) Mirror 27: 0x411329cb (stale) Mirror 28: 0x411329cb (stale) Mirror 29: 0x411329cb (stale) CRC-32 checksum value for chunk [0, 0x27ea8): Mirror 1: 0x9cd8dbef Mirror 30: 0x9cd8dbef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ea8: skipped 000690 1786481493.562885 write 0x036b63 thru 0x03c74b (0x5be9 bytes) 000691 1786481493.614636 write 0x020ff1 thru 0x02c6b0 (0xb6c0 bytes) 000692 1786481493.625698 punch from 0x039409 to 0x03c74b (0x3343 bytes) 000693 1786481493.649822 fallocate 0x0081e3 thru 0x013f08 (0xbd26 bytes) 000694 1786481493.654996 mapread 0x01e979 thru 0x02debd (0xf545 bytes) 000695 1786481493.675329 fallocate 0x02b328 thru 0x030e29 (0x5b02 bytes) 000696 1786481493.687652 MIRROR_RESYNC 14 mirrors 000697 1786481498.344752 write 0x0260a0 thru 0x02abb2 (0x4b13 bytes) 000698 1786481498.379621 fallocate 0x008281 thru 0x0100a1 (0x7e21 bytes) 000699 1786481498.388593 trunc from 0x0049a6 to 0x00d9c4 (0x901f bytes) 000700 1786481498.397144 write 0x02c7bf thru 0x039150 (0xc992 bytes) 000701 1786481498.408044 read 0x0382e0 thru 0x03c74b (0x446c bytes) 000702 1786481498.415675 mapread 0x02f66a thru 0x03c74b (0xd0e2 bytes) 000703 1786481498.417638 punch from 0x017dee to 0x026329 (0xe53c bytes) 000704 1786481498.440666 trunc from 0x02e206 to 0x0339e8 (0x57e3 bytes) 000705 1786481498.451846 MIRROR_RESYNC 14 mirrors 000706 1786481502.209484 fallocate 0x00134d thru 0x00bca4 (0xa958 bytes) 000707 1786481502.261774 read 0x015cb0 thru 0x023652 (0xd9a3 bytes) 000708 1786481502.270983 mapread 0x01d309 thru 0x0266e7 (0x93df bytes) 000709 1786481502.305777 read 0x02fa4f thru 0x038b9d (0x914f bytes) 000710 1786481502.317958 trunc from 0x03c74c to 0x02e884 (0xdec7 bytes) 000711 1786481502.345424 punch from 0x01a9aa to 0x028eb7 (0xe50e bytes) 000712 1786481502.352636 read 0x0183b5 thru 0x020910 (0x855c bytes) 000713 1786481502.392464 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e885): Mirror 10: 0xbf36a6c0 (stale) Mirror 11: 0xbf36a6c0 (stale) Mirror 16: 0xbf36a6c0 (stale) Mirror 19: 0xbf36a6c0 (stale) Mirror 20: 0xbf36a6c0 (stale) Mirror 22: 0xbf36a6c0 (stale) Mirror 23: 0xbf36a6c0 (stale) Mirror 24: 0xbf36a6c0 (stale) Mirror 27: 0xbf36a6c0 (stale) Mirror 28: 0xbf36a6c0 (stale) Mirror 29: 0xbf36a6c0 (stale) Mirror 30: 0xbf36a6c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e885): Mirror 10: 0xbf36a6c0 (stale) Mirror 11: 0xbf36a6c0 (stale) Mirror 16: 0xbf36a6c0 (stale) Mirror 19: 0xbf36a6c0 (stale) Mirror 20: 0xbf36a6c0 (stale) Mirror 22: 0xbf36a6c0 (stale) Mirror 23: 0xbf36a6c0 (stale) Mirror 24: 0xbf36a6c0 (stale) Mirror 27: 0xbf36a6c0 (stale) Mirror 28: 0xbf36a6c0 (stale) Mirror 29: 0xbf36a6c0 (stale) Mirror 30: 0xbf36a6c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e885: skipped 000714 1786481503.305377 write 0x025951 thru 0x032c0f (0xd2bf bytes) 000715 1786481503.317804 fallocate 0x009bb7 thru 0x019437 (0xf881 bytes) 000716 1786481503.324348 write 0x006836 thru 0x009f13 (0x36de bytes) 000717 1786481503.335520 mapread 0x0282b3 thru 0x032c0f (0xa95d bytes) 000718 1786481503.338291 read 0x00b8f0 thru 0x01af44 (0xf655 bytes) 000719 1786481503.349953 read 0x000b21 thru 0x00508b (0x456b bytes) 000720 1786481503.359846 mapread 0x00298d thru 0x00db68 (0xb1dc bytes) 000721 1786481503.365699 mapread 0x00445c thru 0x01424f (0xfdf4 bytes) 000722 1786481503.368677 trunc from 0x019231 to 0x01d19d (0x3f6d bytes) 000723 1786481503.373696 punch from 0x0168a1 to 0x025364 (0xeac4 bytes) 000724 1786481503.389206 trunc from 0x032c10 to 0x0344e6 (0x18d7 bytes) 000725 1786481503.402969 mapread 0x026678 thru 0x02cfdc (0x6965 bytes) 000726 1786481503.404292 punch from 0x02556d to 0x02a545 (0x4fd9 bytes) 000727 1786481503.410360 read 0x015f99 thru 0x01a03f (0x40a7 bytes) 000728 1786481503.420475 trunc from 0x0256de to 0x02c870 (0x7193 bytes) 000729 1786481503.425904 fallocate 0x020a25 thru 0x02947a (0x8a56 bytes) 000730 1786481503.431105 trunc from 0x0344e7 to 0x02c3d9 (0x810d bytes) 000731 1786481503.445529 mapwrite 0x01cdc8 thru 0x01d729 (0x962 bytes) 000732 1786481503.458734 fallocate 0x003c38 thru 0x005396 (0x175f bytes) 000733 1786481503.463692 write 0x01f848 thru 0x02d5c3 (0xdd7c bytes) 000734 1786481503.466752 read 0x02052b thru 0x02d5c3 (0xd099 bytes) 000735 1786481503.467984 mapread 0x02d540 thru 0x02d5c3 (0x84 bytes) 000736 1786481503.469528 fallocate 0x01d6fe thru 0x0248fe (0x7201 bytes) 000737 1786481503.474745 read 0x019cb1 thru 0x021a73 (0x7dc3 bytes) 000738 1786481503.476530 write 0x025f59 thru 0x035759 (0xf801 bytes) 000739 1786481503.479440 read 0x02696a thru 0x033ad6 (0xd16d bytes) 000740 1786481503.480464 trunc from 0x03575a to 0x00339e (0x323bb bytes) 000741 1786481503.507348 mapread 0x003371 thru 0x00339e (0x2e bytes) 000742 1786481503.508829 mapread 0x002830 thru 0x00339e (0xb6f bytes) 000743 1786481503.511013 mapwrite 0x0274b8 thru 0x02ce3d (0x5986 bytes) 000744 1786481503.560899 read 0x026842 thru 0x02c63d (0x5dfc bytes) 000745 1786481503.561934 fallocate 0x01b037 thru 0x028546 (0xd510 bytes) 000746 1786481503.568854 read 0x01c72f thru 0x01f6db (0x2fad bytes) 000747 1786481503.569996 punch from 0x016b1a to 0x02263d (0xbb24 bytes) 000748 1786481503.575452 mapread 0x01db7b thru 0x021717 (0x3b9d bytes) 000749 1786481503.586755 trunc from 0x02ce3e to 0x007b31 (0x2530c bytes) 000750 1786481503.608688 punch from 0x004758 to 0x007b31 (0x33da bytes) 000751 1786481503.613864 fallocate 0x00a20b thru 0x017ec9 (0xdcbf bytes) 000752 1786481503.619482 mapwrite 0x028c1c thru 0x029fc3 (0x13a8 bytes) 000753 1786481503.656186 punch from 0x00345a to 0x006ce9 (0x3890 bytes) 000754 1786481503.663435 write 0x02487d thru 0x02d8b7 (0x903b bytes) 000756 1786481503.670360 punch from 0x00a90c to 0x015e93 (0xb588 bytes) 000757 1786481503.675744 mapread 0x01a86c thru 0x02a0cb (0xf860 bytes) 000758 1786481503.730495 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d8b8): Mirror 10: 0xbe6be608 (stale) Mirror 11: 0xbe6be608 (stale) Mirror 16: 0xbe6be608 (stale) Mirror 19: 0xbe6be608 (stale) Mirror 20: 0xbe6be608 (stale) Mirror 23: 0xbe6be608 (stale) Mirror 24: 0xbe6be608 (stale) Mirror 27: 0xbe6be608 (stale) Mirror 28: 0xbe6be608 (stale) Mirror 29: 0xbe6be608 (stale) Mirror 30: 0xbe6be608 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d8b8): Mirror 10: 0xbe6be608 (stale) Mirror 11: 0xbe6be608 (stale) Mirror 16: 0xbe6be608 (stale) Mirror 19: 0xbe6be608 (stale) Mirror 20: 0xbe6be608 (stale) Mirror 23: 0xbe6be608 (stale) Mirror 24: 0xbe6be608 (stale) Mirror 27: 0xbe6be608 (stale) Mirror 28: 0xbe6be608 (stale) Mirror 29: 0xbe6be608 (stale) Mirror 30: 0xbe6be608 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d8b8: skipped 000759 1786481504.430187 trunc from 0x02d8b8 to 0x020b74 (0xcd43 bytes) 000760 1786481504.481993 fallocate 0x00381a thru 0x006b1a (0x3301 bytes) 000761 1786481504.496962 mapwrite 0x0365e4 thru 0x03ffff (0x9a1c bytes) 000762 1786481504.585802 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x2c2246dc (stale) Mirror 11: 0x2c2246dc (stale) Mirror 16: 0x2c2246dc (stale) Mirror 19: 0x2c2246dc (stale) Mirror 20: 0x2c2246dc (stale) Mirror 24: 0x2c2246dc (stale) Mirror 27: 0x2c2246dc (stale) Mirror 28: 0x2c2246dc (stale) Mirror 29: 0x2c2246dc (stale) Mirror 30: 0x2c2246dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x2c2246dc (stale) Mirror 11: 0x2c2246dc (stale) Mirror 16: 0x2c2246dc (stale) Mirror 19: 0x2c2246dc (stale) Mirror 20: 0x2c2246dc (stale) Mirror 24: 0x2c2246dc (stale) Mirror 27: 0x2c2246dc (stale) Mirror 28: 0x2c2246dc (stale) Mirror 29: 0x2c2246dc (stale) Mirror 30: 0x2c2246dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1786481505.363574 fallocate 0x008594 thru 0x01001a (0x7a87 bytes) 000764 1786481505.370378 write 0x0265a9 thru 0x02df84 (0x79dc bytes) 000765 1786481505.387432 mapread 0x003d64 thru 0x0103d7 (0xc674 bytes) 000766 1786481505.408853 trunc from 0x040000 to 0x02d34f (0x12cb0 bytes) 000767 1786481505.424900 write 0x013987 thru 0x01cfa7 (0x9621 bytes) 000768 1786481505.439628 read 0x00845c thru 0x00a23c (0x1de1 bytes) 000769 1786481505.440650 trunc from 0x02d350 to 0x022bd8 (0xa777 bytes) 000770 1786481505.462043 mapread 0x0150b8 thru 0x0213f3 (0xc33c bytes) 000771 1786481505.476913 write 0x012fad thru 0x01b534 (0x8588 bytes) 000772 1786481505.483882 trunc from 0x022bd9 to 0x00194e (0x2128a bytes) 000773 1786481505.506736 mapwrite 0x02f921 thru 0x03c27f (0xc95f bytes) 000774 1786481505.569982 fallocate 0x004326 thru 0x01236d (0xe048 bytes) 000775 1786481505.577091 write 0x009312 thru 0x00b178 (0x1e67 bytes) 000776 1786481505.593882 trunc from 0x02ffd3 to 0x03c27f (0xc2ad bytes) 000777 1786481505.604895 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c280): Mirror 10: 0xbc24f496 (stale) Mirror 11: 0xbc24f496 (stale) Mirror 19: 0xbc24f496 (stale) Mirror 20: 0xbc24f496 (stale) Mirror 24: 0xbc24f496 (stale) Mirror 27: 0xbc24f496 (stale) Mirror 28: 0xbc24f496 (stale) Mirror 29: 0xbc24f496 (stale) Mirror 30: 0xbc24f496 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c280): Mirror 10: 0xbc24f496 (stale) Mirror 11: 0xbc24f496 (stale) Mirror 19: 0xbc24f496 (stale) Mirror 20: 0xbc24f496 (stale) Mirror 24: 0xbc24f496 (stale) Mirror 27: 0xbc24f496 (stale) Mirror 28: 0xbc24f496 (stale) Mirror 29: 0xbc24f496 (stale) Mirror 30: 0xbc24f496 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c280: skipped 000778 1786481506.309691 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c280): Mirror 10: 0xbc24f496 (stale) Mirror 11: 0xbc24f496 (stale) Mirror 19: 0xbc24f496 (stale) Mirror 20: 0xbc24f496 (stale) Mirror 24: 0xbc24f496 (stale) Mirror 27: 0xbc24f496 (stale) Mirror 28: 0xbc24f496 (stale) Mirror 30: 0xbc24f496 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c280): Mirror 10: 0xbc24f496 (stale) Mirror 11: 0xbc24f496 (stale) Mirror 19: 0xbc24f496 (stale) Mirror 20: 0xbc24f496 (stale) Mirror 24: 0xbc24f496 (stale) Mirror 27: 0xbc24f496 (stale) Mirror 28: 0xbc24f496 (stale) Mirror 30: 0xbc24f496 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c280: skipped 000780 1786481506.836432 fallocate 0x00f1bb thru 0x01683f (0x7685 bytes) 000781 1786481506.844500 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3c280): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c280): Mirror 10: 0xbc24f496 (stale) Mirror 11: 0xbc24f496 (stale) Mirror 19: 0xbc24f496 (stale) Mirror 20: 0xbc24f496 (stale) Mirror 24: 0xbc24f496 (stale) Mirror 27: 0xbc24f496 (stale) Mirror 28: 0xbc24f496 (stale) Mirror 30: 0xbc24f496 (stale) CRC-32 checksum value for chunk [0, 0x3c280): Mirror 1: 0xbf18c2a1 Mirror 31: 0xbf18c2a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c280: skipped 000782 1786481507.837782 fallocate 0x009651 thru 0x015d41 (0xc6f1 bytes) 000783 1786481507.889964 trunc from 0x03c280 to 0x005249 (0x37036 bytes) 000784 1786481507.917300 mapwrite 0x017b4f thru 0x019efc (0x23ae bytes) 000785 1786481507.963962 fallocate 0x03011c thru 0x0386bd (0x85a2 bytes) 000786 1786481507.975800 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x19efd): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19efd): Mirror 10: 0x56abfb16 (stale) Mirror 11: 0x56abfb16 (stale) Mirror 19: 0x56abfb16 (stale) Mirror 20: 0x56abfb16 (stale) Mirror 24: 0x56abfb16 (stale) Mirror 27: 0x56abfb16 (stale) Mirror 28: 0x56abfb16 (stale) Mirror 30: 0x56abfb16 (stale) Mirror 31: 0x33237e85 (stale) CRC-32 checksum value for chunk [0, 0x19efd): Mirror 1: 0xda2a2f8b Mirror 32: 0xda2a2f8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19efd: skipped 000787 1786481509.036593 write 0x02c620 thru 0x036744 (0xa125 bytes) 000788 1786481509.095791 punch from 0x0046f0 to 0x00dc34 (0x9545 bytes) 000789 1786481509.103759 punch from 0x023e60 to 0x0323e5 (0xe586 bytes) 000790 1786481509.123614 fallocate 0x02a46c thru 0x031477 (0x700c bytes) 000791 1786481509.129646 fallocate 0x027bde thru 0x031474 (0x9897 bytes) 000792 1786481509.136863 mapread 0x0100ea thru 0x019970 (0x9887 bytes) 000793 1786481509.158064 fallocate 0x001e6b thru 0x008973 (0x6b09 bytes) 000794 1786481509.164423 mapwrite 0x031852 thru 0x03ffff (0xe7ae bytes) 000795 1786481509.240567 trunc from 0x00f630 to 0x0152d6 (0x5ca7 bytes) 000796 1786481509.250771 trunc from 0x040000 to 0x03a391 (0x5c6e bytes) 000797 1786481509.268266 trunc from 0x03a392 to 0x015a8c (0x24905 bytes) 000798 1786481509.299538 read 0x003edc thru 0x009fda (0x60ff bytes) 000799 1786481509.312785 mapread 0x012751 thru 0x015a8c (0x333c bytes) 000800 1786481509.329527 trunc from 0x01109c to 0x015926 (0x488b bytes) 000801 1786481509.336788 read 0x00c290 thru 0x0110a4 (0x4e15 bytes) 000802 1786481509.353178 punch from 0x007a5c to 0x008587 (0xb2c bytes) 000803 1786481509.367687 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x15a8d): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15a8d): Mirror 10: 0xdb8c0d7d (stale) Mirror 11: 0xdb8c0d7d (stale) Mirror 19: 0xdb8c0d7d (stale) Mirror 20: 0xdb8c0d7d (stale) Mirror 24: 0xdb8c0d7d (stale) Mirror 27: 0xdb8c0d7d (stale) Mirror 28: 0xdb8c0d7d (stale) Mirror 30: 0xdb8c0d7d (stale) Mirror 31: 0xad64f5f5 (stale) Mirror 32: 0x6e6ce4cf (stale) CRC-32 checksum value for chunk [0, 0x15a8d): Mirror 1: 0x6e6ce4cf Mirror 33: 0x6e6ce4cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15a8d: skipped 000804 1786481510.489690 mapwrite 0x02616a thru 0x0329d4 (0xc86b bytes) 000805 1786481510.642834 MIRROR_RESYNC 12 mirrors 000806 1786481512.916241 read 0x021977 thru 0x030a50 (0xf0da bytes) 000807 1786481512.927412 punch from 0x020bf5 to 0x026f59 (0x6365 bytes) 000808 1786481512.979897 write 0x01cb6c thru 0x020eb5 (0x434a bytes) 000809 1786481512.997609 mapread 0x0253f5 thru 0x02e476 (0x9082 bytes) 000810 1786481513.044511 punch from 0x01026f to 0x0159ea (0x577c bytes) 000811 1786481513.050757 read 0x01528f thru 0x021021 (0xbd93 bytes) 000812 1786481513.072370 trunc from 0x0329d5 to 0x01ff0a (0x12aca bytes) 000813 1786481513.096194 read 0x01c79b thru 0x01ff0a (0x3770 bytes) 000814 1786481513.096978 mapwrite 0x003e93 thru 0x006eb7 (0x3025 bytes) 000815 1786481513.118740 write 0x0108b8 thru 0x0156d6 (0x4e1f bytes) 000816 1786481513.122262 write 0x00753f thru 0x01643f (0xef01 bytes) 000817 1786481513.125327 punch from 0x003713 to 0x006e77 (0x3765 bytes) 000819 1786481513.134526 punch from 0x01ee0a to 0x01ff0a (0x1101 bytes) 000820 1786481513.144725 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ff0b): Mirror 10: 0x54ef5f4d (stale) Mirror 11: 0x54ef5f4d (stale) Mirror 19: 0x54ef5f4d (stale) Mirror 20: 0x54ef5f4d (stale) Mirror 27: 0x54ef5f4d (stale) Mirror 28: 0x54ef5f4d (stale) Mirror 30: 0x54ef5f4d (stale) Mirror 31: 0x54ef5f4d (stale) Mirror 32: 0x54ef5f4d (stale) Mirror 33: 0x54ef5f4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff0b): Mirror 10: 0x54ef5f4d (stale) Mirror 11: 0x54ef5f4d (stale) Mirror 19: 0x54ef5f4d (stale) Mirror 20: 0x54ef5f4d (stale) Mirror 27: 0x54ef5f4d (stale) Mirror 28: 0x54ef5f4d (stale) Mirror 30: 0x54ef5f4d (stale) Mirror 31: 0x54ef5f4d (stale) Mirror 32: 0x54ef5f4d (stale) Mirror 33: 0x54ef5f4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff0b: skipped 000821 1786481513.819561 write 0x01ff55 thru 0x02cbb6 (0xcc62 bytes) 000822 1786481513.832702 read 0x02626e thru 0x02cbb6 (0x6949 bytes) 000823 1786481513.834733 read 0x00f428 thru 0x011f5e (0x2b37 bytes) 000824 1786481513.846663 mapwrite 0x03f06c thru 0x03ffff (0xf94 bytes) 000825 1786481513.913983 read 0x00dd99 thru 0x01d265 (0xf4cd bytes) 000826 1786481513.916348 mapread 0x01a8aa thru 0x0251d4 (0xa92b bytes) 000827 1786481513.920829 mapwrite 0x00f0f7 thru 0x011ced (0x2bf7 bytes) 000828 1786481513.934163 punch from 0x015f16 to 0x024e25 (0xef10 bytes) 000829 1786481513.939379 trunc from 0x040000 to 0x018992 (0x2766d bytes) 000830 1786481513.959335 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18993): Mirror 10: 0x3f381647 (stale) Mirror 11: 0x3f381647 (stale) Mirror 19: 0x3f381647 (stale) Mirror 20: 0x3f381647 (stale) Mirror 27: 0x3f381647 (stale) Mirror 28: 0x3f381647 (stale) Mirror 30: 0x3f381647 (stale) Mirror 31: 0x3f381647 (stale) Mirror 32: 0x3f381647 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18993): Mirror 10: 0x3f381647 (stale) Mirror 11: 0x3f381647 (stale) Mirror 19: 0x3f381647 (stale) Mirror 20: 0x3f381647 (stale) Mirror 27: 0x3f381647 (stale) Mirror 28: 0x3f381647 (stale) Mirror 30: 0x3f381647 (stale) Mirror 31: 0x3f381647 (stale) Mirror 32: 0x3f381647 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18993: skipped 000831 1786481514.565884 fallocate 0x0125ea thru 0x01910a (0x6b21 bytes) 000832 1786481514.572712 mapread 0x0131b6 thru 0x01910a (0x5f55 bytes) 000833 1786481514.596777 write 0x018e95 thru 0x022f9d (0xa109 bytes) 000834 1786481514.604566 mapwrite 0x012bca thru 0x01de89 (0xb2c0 bytes) 000835 1786481514.623668 trunc from 0x01f172 to 0x01f999 (0x828 bytes) 000836 1786481514.632758 MIRROR_RESYNC 10 mirrors 000837 1786481516.551343 mapread 0x00f701 thru 0x014c8b (0x558b bytes) 000838 1786481516.575637 trunc from 0x022f9e to 0x0107ce (0x127cf bytes) 000839 1786481516.622561 mapread 0x004c78 thru 0x0107ce (0xbb57 bytes) 000840 1786481516.644353 trunc from 0x0107cf to 0x02b28a (0x1aabc bytes) 000841 1786481516.666621 fallocate 0x0264a9 thru 0x02ffd5 (0x9b2d bytes) 000842 1786481516.673404 mapwrite 0x0017fe thru 0x00b44d (0x9c50 bytes) 000843 1786481516.697009 trunc from 0x02ffd6 to 0x008ac3 (0x27512 bytes) 000844 1786481516.712595 fallocate 0x033c66 thru 0x03f3ab (0xb746 bytes) 000845 1786481516.719533 trunc from 0x008ff4 to 0x01400e (0xb01b bytes) 000846 1786481516.732498 MIRROR_RESYNC 10 mirrors 000847 1786481517.989889 trunc from 0x018829 to 0x020b3e (0x8316 bytes) 000848 1786481518.037179 trunc from 0x03f3ac to 0x01eba7 (0x20804 bytes) 000849 1786481518.063626 trunc from 0x00754b to 0x0116b8 (0xa16e bytes) 000850 1786481518.075451 mapwrite 0x03b2b8 thru 0x03ffff (0x4d48 bytes) 000851 1786481518.138432 read 0x02ee52 thru 0x038824 (0x99d3 bytes) 000853 1786481518.150673 punch from 0x029b75 to 0x02bb5e (0x1fea bytes) 000854 1786481518.160840 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xebc9a126 (stale) Mirror 19: 0xebc9a126 (stale) Mirror 20: 0xebc9a126 (stale) Mirror 27: 0xebc9a126 (stale) Mirror 28: 0xebc9a126 (stale) Mirror 30: 0xebc9a126 (stale) Mirror 31: 0xebc9a126 (stale) Mirror 32: 0xebc9a126 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xebc9a126 (stale) Mirror 19: 0xebc9a126 (stale) Mirror 20: 0xebc9a126 (stale) Mirror 27: 0xebc9a126 (stale) Mirror 28: 0xebc9a126 (stale) Mirror 30: 0xebc9a126 (stale) Mirror 31: 0xebc9a126 (stale) Mirror 32: 0xebc9a126 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1786481519.039414 punch from 0x025f6a to 0x0302db (0xa372 bytes) 000856 1786481519.048282 trunc from 0x040000 to 0x013a50 (0x2c5af bytes) 000857 1786481519.080438 mapwrite 0x00e5c6 thru 0x0175dc (0x9017 bytes) 000858 1786481519.171101 trunc from 0x00ecd2 to 0x014de8 (0x6117 bytes) 000859 1786481519.178692 trunc from 0x0175dd to 0x024d81 (0xd7a5 bytes) 000860 1786481519.207530 punch from 0x00261c to 0x01037a (0xdd5f bytes) 000862 1786481519.217467 fallocate 0x006d0f thru 0x00e86c (0x7b5e bytes) 000863 1786481519.230779 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24d82): Mirror 10: 0x3f68ccba (stale) Mirror 20: 0x3f68ccba (stale) Mirror 27: 0x3f68ccba (stale) Mirror 28: 0x3f68ccba (stale) Mirror 30: 0x3f68ccba (stale) Mirror 31: 0x3f68ccba (stale) Mirror 32: 0x3f68ccba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d82): Mirror 10: 0x3f68ccba (stale) Mirror 20: 0x3f68ccba (stale) Mirror 27: 0x3f68ccba (stale) Mirror 28: 0x3f68ccba (stale) Mirror 30: 0x3f68ccba (stale) Mirror 31: 0x3f68ccba (stale) Mirror 32: 0x3f68ccba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d82: skipped 000864 1786481519.948361 write 0x027d7c thru 0x033dac (0xc031 bytes) 000865 1786481519.959055 mapread 0x005f91 thru 0x007239 (0x12a9 bytes) 000866 1786481519.982071 mapread 0x030090 thru 0x0336f6 (0x3667 bytes) 000867 1786481519.983283 fallocate 0x02e199 thru 0x03b86c (0xd6d4 bytes) 000868 1786481519.991494 read 0x00b275 thru 0x016271 (0xaffd bytes) 000869 1786481520.006494 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b86d): Mirror 20: 0x5a28cc24 (stale) Mirror 27: 0x5a28cc24 (stale) Mirror 28: 0x5a28cc24 (stale) Mirror 30: 0x5a28cc24 (stale) Mirror 31: 0x5a28cc24 (stale) Mirror 32: 0x5a28cc24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b86d): Mirror 20: 0x5a28cc24 (stale) Mirror 27: 0x5a28cc24 (stale) Mirror 28: 0x5a28cc24 (stale) Mirror 30: 0x5a28cc24 (stale) Mirror 31: 0x5a28cc24 (stale) Mirror 32: 0x5a28cc24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b86d: skipped 000870 1786481520.626226 read 0x028894 thru 0x037bd2 (0xf33f bytes) 000871 1786481520.643719 trunc from 0x0377dd to 0x037c46 (0x46a bytes) 000872 1786481520.650579 read 0x0224c2 thru 0x030a0f (0xe54e bytes) 000873 1786481520.674386 mapread 0x006fc3 thru 0x00dda5 (0x6de3 bytes) 000874 1786481520.677252 trunc from 0x019a21 to 0x021fe9 (0x85c9 bytes) 000876 1786481520.686429 trunc from 0x03b86d to 0x029cc6 (0x11ba6 bytes) 000877 1786481520.708433 write 0x018519 thru 0x022854 (0xa33c bytes) 000878 1786481520.712451 mapwrite 0x00e685 thru 0x018b93 (0xa50f bytes) 000879 1786481520.747463 mapread 0x011fc5 thru 0x01f19a (0xd1d6 bytes) 000880 1786481520.749720 trunc from 0x029cc7 to 0x022484 (0x7842 bytes) 000881 1786481520.770430 fallocate 0x03d2eb thru 0x03d851 (0x567 bytes) 000882 1786481520.782371 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d852): Mirror 27: 0xf55a9889 (stale) Mirror 28: 0xf55a9889 (stale) Mirror 30: 0xf55a9889 (stale) Mirror 31: 0xf55a9889 (stale) Mirror 32: 0xf55a9889 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d852): Mirror 27: 0xf55a9889 (stale) Mirror 28: 0xf55a9889 (stale) Mirror 30: 0xf55a9889 (stale) Mirror 31: 0xf55a9889 (stale) Mirror 32: 0xf55a9889 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d852: skipped 000885 1786481521.549298 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d852): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d852): Mirror 27: 0xf55a9889 (stale) Mirror 28: 0xf55a9889 (stale) Mirror 30: 0xf55a9889 (stale) Mirror 31: 0xf55a9889 (stale) Mirror 32: 0xf55a9889 (stale) CRC-32 checksum value for chunk [0, 0x3d852): Mirror 1: 0x1b719ff8 Mirror 33: 0x1b719ff8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d852: skipped 000886 1786481523.039176 write 0x02bc50 thru 0x0349ef (0x8da0 bytes) 000887 1786481523.158973 trunc from 0x03d852 to 0x00ded4 (0x2f97d bytes) 000888 1786481523.173291 read 0x009ed2 thru 0x00ded4 (0x4003 bytes) 000889 1786481523.179419 mapwrite 0x007ce0 thru 0x01039d (0x86be bytes) 000890 1786481523.224671 fallocate 0x004daf thru 0x0058f9 (0xb4b bytes) 000891 1786481523.229025 trunc from 0x01039e to 0x02d2b0 (0x1cf13 bytes) 000892 1786481523.238221 mapwrite 0x00f931 thru 0x01c204 (0xc8d4 bytes) 000893 1786481523.276662 mapread 0x009fab thru 0x019763 (0xf7b9 bytes) 000894 1786481523.282559 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d2b1): Mirror 27: 0xd026bbd7 (stale) Mirror 28: 0xd026bbd7 (stale) Mirror 30: 0xd026bbd7 (stale) Mirror 32: 0xd026bbd7 (stale) Mirror 33: 0x7387a41f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d2b1): Mirror 27: 0xd026bbd7 (stale) Mirror 28: 0xd026bbd7 (stale) Mirror 30: 0xd026bbd7 (stale) Mirror 32: 0xd026bbd7 (stale) Mirror 33: 0x7387a41f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d2b1: skipped 000895 1786481523.889082 read 0x010cf4 thru 0x01d21e (0xc52b bytes) 000897 1786481523.908375 trunc from 0x0126ba to 0x0192eb (0x6c32 bytes) 000898 1786481523.925672 punch from 0x0225d5 to 0x0277be (0x51ea bytes) 000899 1786481523.952304 read 0x00e0b1 thru 0x013167 (0x50b7 bytes) 000900 1786481523.990181 trunc from 0x02d2b1 to 0x0111ab (0x1c105 bytes) 000901 1786481524.023058 MIRROR_RESYNC 6 mirrors 000902 1786481525.563672 punch from 0x006e17 to 0x0111ab (0xa395 bytes) 000903 1786481525.609853 trunc from 0x00da8d to 0x0111ab (0x371f bytes) 000904 1786481525.617696 write 0x03438f thru 0x03ffff (0xbc71 bytes) 000907 1786481525.626414 trunc from 0x040000 to 0x03de9c (0x2163 bytes) 000908 1786481525.652247 punch from 0x00a8e2 to 0x019660 (0xed7f bytes) 000909 1786481525.658090 write 0x03f1cf thru 0x03ffff (0xe31 bytes) 000910 1786481525.659663 mapread 0x00e866 thru 0x0112da (0x2a75 bytes) 000911 1786481525.691750 trunc from 0x040000 to 0x03a709 (0x58f6 bytes) 000912 1786481525.719374 trunc from 0x006578 to 0x007237 (0xcc0 bytes) 000913 1786481525.731982 MIRROR_RESYNC 6 mirrors 000914 1786481527.183922 MIRROR_EXTEND to 7 mirrors 000915 1786481528.451607 write 0x01cfe0 thru 0x021b23 (0x4b44 bytes) 000916 1786481528.574493 read 0x02bf1d thru 0x036764 (0xa848 bytes) 000917 1786481528.604780 mapread 0x03991a thru 0x03a709 (0xdf0 bytes) 000918 1786481528.635922 mapread 0x029776 thru 0x02da46 (0x42d1 bytes) 000920 1786481528.685625 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a70a): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a70a): Mirror 27: 0x31b596d (stale) Mirror 28: 0x31b596d (stale) Mirror 30: 0x31b596d (stale) Mirror 32: 0x31b596d (stale) Mirror 33: 0x31b596d (stale) Mirror 34: 0x31b596d (stale) CRC-32 checksum value for chunk [0, 0x3a70a): Mirror 1: 0xb77d5827 Mirror 35: 0xb77d5827 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a70a: skipped 000922 1786481529.962173 mapwrite 0x03e46f thru 0x03ffff (0x1b91 bytes) 000923 1786481530.019452 mapwrite 0x029c0b thru 0x03124e (0x7644 bytes) 000924 1786481530.046587 mapread 0x001ba5 thru 0x0040ce (0x252a bytes) 000925 1786481530.068027 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: 0xf3869442 (stale) Mirror 28: 0xf3869442 (stale) Mirror 30: 0xf3869442 (stale) Mirror 32: 0xf3869442 (stale) Mirror 33: 0xf3869442 (stale) Mirror 34: 0xf3869442 (stale) Mirror 35: 0xf3fac31d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbc7ed691 Mirror 36: 0xbc7ed691 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000926 1786481531.165298 punch from 0x01b07a to 0x01c307 (0x128e bytes) 000927 1786481531.226243 fallocate 0x01e0b6 thru 0x02bd24 (0xdc6f bytes) 000928 1786481531.258401 trunc from 0x008944 to 0x01119d (0x885a bytes) 000929 1786481531.267045 trunc from 0x00e2ee to 0x01c728 (0xe43b bytes) 000930 1786481531.291450 trunc from 0x040000 to 0x01ab3d (0x254c2 bytes) 000931 1786481531.405663 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1ab3e): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab3e): Mirror 27: 0xf85c1cdd (stale) Mirror 28: 0xf85c1cdd (stale) Mirror 30: 0xf85c1cdd (stale) Mirror 32: 0xf85c1cdd (stale) Mirror 33: 0xf85c1cdd (stale) Mirror 34: 0xf85c1cdd (stale) Mirror 35: 0xf85c1cdd (stale) Mirror 36: 0xf85c1cdd (stale) CRC-32 checksum value for chunk [0, 0x1ab3e): Mirror 1: 0xf85c1cdd Mirror 37: 0xf85c1cdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab3e: skipped 000932 1786481533.274632 punch from 0x003d9f to 0x0138e6 (0xfb48 bytes) 000933 1786481533.341420 fallocate 0x02c0fb thru 0x034878 (0x877e bytes) 000934 1786481533.355344 mapwrite 0x03e6a0 thru 0x03ffff (0x1960 bytes) 000935 1786481533.496415 punch from 0x00819a to 0x0127d4 (0xa63b bytes) 000936 1786481533.509972 trunc from 0x040000 to 0x00a13d (0x35ec2 bytes) 000938 1786481533.550079 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa13e): Mirror 27: 0x649226bf (stale) Mirror 28: 0x649226bf (stale) Mirror 30: 0x649226bf (stale) Mirror 33: 0x649226bf (stale) Mirror 34: 0x649226bf (stale) Mirror 35: 0x649226bf (stale) Mirror 36: 0x649226bf (stale) Mirror 37: 0x649226bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa13e): Mirror 27: 0x649226bf (stale) Mirror 28: 0x649226bf (stale) Mirror 30: 0x649226bf (stale) Mirror 33: 0x649226bf (stale) Mirror 34: 0x649226bf (stale) Mirror 35: 0x649226bf (stale) Mirror 36: 0x649226bf (stale) Mirror 37: 0x649226bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa13e: skipped 000939 1786481534.430448 read 0x00890c thru 0x00a13d (0x1832 bytes) 000940 1786481534.439079 mapwrite 0x005cad thru 0x0098aa (0x3bfe bytes) 000941 1786481534.492550 read 0x0014eb thru 0x005ad4 (0x45ea bytes) 000942 1786481534.493923 read 0x003a22 thru 0x005256 (0x1835 bytes) 000943 1786481534.495494 fallocate 0x02df41 thru 0x02e98e (0xa4e bytes) 000944 1786481534.503982 trunc from 0x02e98f to 0x0379d7 (0x9049 bytes) 000946 1786481534.552199 mapread 0x033eb4 thru 0x0379d7 (0x3b24 bytes) 000947 1786481534.576629 mapread 0x0246d6 thru 0x03115b (0xca86 bytes) 000950 1786481534.580673 write 0x016a6a thru 0x021f19 (0xb4b0 bytes) 000951 1786481534.583368 mapwrite 0x01f767 thru 0x022bf6 (0x3490 bytes) 000952 1786481534.609419 punch from 0x029b41 to 0x02a1d3 (0x693 bytes) 000953 1786481534.617156 mapwrite 0x00fe7b thru 0x0177d5 (0x795b bytes) 000954 1786481534.640532 read 0x013c5c thru 0x01e6a3 (0xaa48 bytes) 000955 1786481534.641225 mapread 0x028c86 thru 0x02e60a (0x5985 bytes) 000956 1786481534.659341 MIRROR_RESYNC 9 mirrors 000957 1786481537.410836 MIRROR_EXTEND to 10 mirrors 000958 1786481538.422816 MIRROR_RESYNC 10 mirrors 000959 1786481539.081239 mapread 0x020c33 thru 0x021253 (0x621 bytes) 000960 1786481539.093926 punch from 0x0067e6 to 0x016223 (0xfa3e bytes) 000961 1786481539.151077 mapwrite 0x039345 thru 0x03ffff (0x6cbb bytes) 000962 1786481539.218849 mapwrite 0x003b70 thru 0x0138b6 (0xfd47 bytes) 000963 1786481539.311814 punch from 0x00f768 to 0x012f7f (0x3818 bytes) 000964 1786481539.317407 write 0x011196 thru 0x017b97 (0x6a02 bytes) 000965 1786481539.329300 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: 0xcc2af07c (stale) Mirror 28: 0xcc2af07c (stale) Mirror 30: 0xcc2af07c (stale) Mirror 34: 0xcc2af07c (stale) Mirror 35: 0xcc2af07c (stale) Mirror 36: 0xcc2af07c (stale) Mirror 37: 0xcc2af07c (stale) Mirror 38: 0xcc2af07c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: 0xcc2af07c (stale) Mirror 28: 0xcc2af07c (stale) Mirror 30: 0xcc2af07c (stale) Mirror 34: 0xcc2af07c (stale) Mirror 35: 0xcc2af07c (stale) Mirror 36: 0xcc2af07c (stale) Mirror 37: 0xcc2af07c (stale) Mirror 38: 0xcc2af07c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000966 1786481539.692939 MIRROR_RESYNC 9 mirrors 000967 1786481541.703829 write 0x03db78 thru 0x03ffff (0x2488 bytes) 000968 1786481541.749979 mapread 0x02be15 thru 0x0368dc (0xaac8 bytes) 000969 1786481541.794884 read 0x016050 thru 0x01c2e5 (0x6296 bytes) 000970 1786481541.807487 trunc from 0x03e40e to 0x03ffff (0x1bf2 bytes) 000971 1786481541.824824 trunc from 0x040000 to 0x01e612 (0x219ed bytes) 000974 1786481541.869179 trunc from 0x01e613 to 0x010f80 (0xd692 bytes) 000975 1786481541.892306 trunc from 0x010f81 to 0x02aa13 (0x19a93 bytes) 000976 1786481541.918307 write 0x00b768 thru 0x012fa8 (0x7841 bytes) 000977 1786481541.933247 mapwrite 0x03ad87 thru 0x03ffff (0x5279 bytes) 000978 1786481542.002419 trunc from 0x040000 to 0x00a3dc (0x35c23 bytes) 000979 1786481542.027567 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa3dd): Mirror 28: 0x1149682c (stale) Mirror 30: 0x1149682c (stale) Mirror 34: 0x1149682c (stale) Mirror 35: 0x1149682c (stale) Mirror 36: 0x1149682c (stale) Mirror 37: 0x1149682c (stale) Mirror 38: 0x1149682c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa3dd): Mirror 28: 0x1149682c (stale) Mirror 30: 0x1149682c (stale) Mirror 34: 0x1149682c (stale) Mirror 35: 0x1149682c (stale) Mirror 36: 0x1149682c (stale) Mirror 37: 0x1149682c (stale) Mirror 38: 0x1149682c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa3dd: skipped 000982 1786481542.533714 read 0x0004c3 thru 0x00a3dc (0x9f1a bytes) 000983 1786481542.551161 write 0x027fe9 thru 0x02f297 (0x72af bytes) 000984 1786481542.556724 read 0x0261dd thru 0x02ea4d (0x8871 bytes) 000985 1786481542.581264 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2f298): Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f298): Mirror 28: 0xe283199 (stale) Mirror 30: 0xe283199 (stale) Mirror 34: 0xe283199 (stale) Mirror 35: 0xe283199 (stale) Mirror 36: 0xe283199 (stale) Mirror 37: 0xe283199 (stale) Mirror 38: 0xe283199 (stale) CRC-32 checksum value for chunk [0, 0x2f298): Mirror 1: 0x3ac726ac Mirror 39: 0x3ac726ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f298: skipped 000986 1786481543.603915 mapwrite 0x0211df thru 0x02a176 (0x8f98 bytes) 000987 1786481543.764295 write 0x03e1e9 thru 0x03ffff (0x1e17 bytes) 000988 1786481543.773199 mapread 0x02988d thru 0x0306f9 (0x6e6d bytes) 000989 1786481543.786356 read 0x03f45e thru 0x03ffff (0xba2 bytes) 000990 1786481543.788038 write 0x01a51d thru 0x022d35 (0x8819 bytes) 000991 1786481543.799227 punch from 0x02c3e2 to 0x0302be (0x3edd bytes) 000993 1786481543.810252 trunc from 0x040000 to 0x03cacc (0x3533 bytes) 000994 1786481543.837013 trunc from 0x03cacd to 0x028c4a (0x13e82 bytes) 000995 1786481543.856657 punch from 0x008fc0 to 0x00c15f (0x31a0 bytes) 000996 1786481543.870459 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c4b): Mirror 28: 0x392170da (stale) Mirror 30: 0x392170da (stale) Mirror 34: 0x392170da (stale) Mirror 35: 0x392170da (stale) Mirror 36: 0x392170da (stale) Mirror 37: 0x392170da (stale) Mirror 39: 0xb532672a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c4b): Mirror 28: 0x392170da (stale) Mirror 30: 0x392170da (stale) Mirror 34: 0x392170da (stale) Mirror 35: 0x392170da (stale) Mirror 36: 0x392170da (stale) Mirror 37: 0x392170da (stale) Mirror 39: 0xb532672a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c4b: skipped 000997 1786481544.590314 fallocate 0x002816 thru 0x00a132 (0x791d bytes) 000998 1786481544.607304 trunc from 0x009c5d to 0x00e89e (0x4c42 bytes) 000999 1786481544.642104 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x28c4b): Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 39: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c4b): Mirror 28: 0x392170da (stale) Mirror 30: 0x392170da (stale) Mirror 34: 0x392170da (stale) Mirror 35: 0x392170da (stale) Mirror 36: 0x392170da (stale) Mirror 37: 0x392170da (stale) Mirror 39: 0xb532672a (stale) CRC-32 checksum value for chunk [0, 0x28c4b): Mirror 1: 0xbfed2f6d Mirror 40: 0xbfed2f6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c4b: skipped 001000 1786481545.853585 trunc from 0x00205a to 0x00900e (0x6fb5 bytes) All operations completed A-OK! PASS 70a (165s) == sanity-flr test 71: check mirror extend parent fid ==== 16:52:31 (1786481551) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x21:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 16:52:40 (1786481560) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x340:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239809 s, 8.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x340:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x341:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701312 s, 15.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x340:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x341:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x340:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x341:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x342:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 16:52:49 (1786481569) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5358720 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 712544 .. Extending file size to 5859712 .. Extending file size to 8558080 .. Extending file size to 7574976 .. Extending file size to 8535808 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8784512 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8565600 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61451 61452 61456 61458 61460 done 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) 192.168.201.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:54:12 (1786481652) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5093568 .. Extending file size to 6861952 .. Extending file size to 7427072 .. Extending file size to 7523520 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8902272 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7515424 .. Extending file size to 8995168 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8996480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 62887 62889 62892 62893 62894 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) 192.168.201.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:55:29 (1786481729) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2816 1268488 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17656 3589364 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25492 3581528 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43148 7170892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2816 1268488 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17656 3589364 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25492 3581528 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43148 7170892 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15296 s, 13.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237226 s, 8.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe5f4972c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:55:52 (1786481752) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122207 s, 17.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:55:59 (1786481759) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x235:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x354:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x354:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x355:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x357:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:56:07 (1786481767) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0943996 s, 22.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x29:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:56:15 (1786481775) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0900454 s, 23.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x358:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x358:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x359:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:56:24 (1786481784) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0958707 s, 21.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x358:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x358:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35b:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:56:57 (1786481817) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11665 s, 9.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2b:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:57:45 (1786481865) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0932507 s, 22.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:58:34 (1786481914) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg108-server: Pool lustre.test_204f created Adding targets to pool oleg108-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg108-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100834 s, 20.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg108-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg108-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg108-server: Pool lustre.test_204f destroyed PASS 204f (62s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:59:36 (1786481976) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:59:42 (1786481982) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:59:48 (1786481988) oleg108-server: Pool lustre.test_206 created Adding targets to pool oleg108-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg108-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg108-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg108-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg108-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:00:11 (1786482011) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404204 s, 25.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308605 s, 34.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3f94fe29 Mirror 3: 0x3f94fe29 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:00:19 (1786482019) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:00:22 (1786482022) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:00:25 (1786482025) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0603379 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa10000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2000000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2000000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2000000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12a0000 Read this (4 bytes): BBBB 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) PASS 209a (29s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:00:54 (1786482054) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0270506 s, 303 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:00:59 (1786482059) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649236 s, 16.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x256 lmm_fid: [0x200000405:0x256:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x256 lmm_fid: [0x200000405:0x256:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x367:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611734 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 17:01:00.000000000 -0400 Modify: 2026-08-11 17:01:00.000000000 -0400 Change: 2026-08-11 17:01:01.000000000 -0400 Birth: 2026-08-11 17:01:01.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:01:06 (1786482066) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306577 s, 34.2 MB/s oleg108-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg108-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg108-server: O/280000401/d6/366: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:01:26 (1786482086) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.238901 s, 171 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0158503 s, 258 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 11 17:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x25b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 11 17:01 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:01:31 (1786482091) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:01:36 (1786482096) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1761 sec =========== 17:01:41 (1786482101) === sanity-flr: start cleanup 17:01:42 (1786482102) === === sanity-flr: finish cleanup 17:01:50 (1786482110) ===