-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 06:38:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg617-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:39:07 (1787827147) === oleg617-client.virtnet: executing check_config_client /mnt/lustre oleg617-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg617-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9bb1c8bc7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9bb1c8bc7800.idle_timeout=debug disable quota as required oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:39:48 (1787827188) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8225] == sanity-flr test 0a: lfs mirror create with -N option == 06:39:50 (1787827190) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:40:08 (1787827208) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:40:11 (1787827211) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:40:14 (1787827214) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:40:35 (1787827235) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:40:38 (1787827238) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:40:41 (1787827241) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9bb1c8bc7800.stats=clear osc.lustre-OST0001-osc-ffff9bb1c8bc7800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:40:48 (1787827248) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:41:00 (1787827260) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:41:07 (1787827267) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:41:11 (1787827271) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:41:20 (1787827280) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 06:41:29 (1787827289) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:41:37 (1787827297) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:41:45 (1787827305) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:41:53 (1787827313) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:42:03 (1787827323) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:42:10 (1787827330) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.538994 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145804 s, 7.2 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:42:19 (1787827339) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.739292 s, 28.4 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:42:28 (1787827348) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:42:38 (1787827358) Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg617-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg617-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (124s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:44:42 (1787827482) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.729706 s, 15.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:45:21 (1787827521) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg617-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:46:38 (1787827598) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.27057 s, 16.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 06:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.171s user 0m0.000s sys 0m0.198s Stopping /mnt/lustre-ost1 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb1c8bc7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.480s user 0m0.002s sys 0m0.163s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg617-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg617-server oleg617-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.233s user 0m0.000s sys 0m0.196s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg617-server: oleg617-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg617-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg617-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb1c8bc7800.ost_server_uuid in FULL state after 0 sec PASS 33b (88s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:48:06 (1787827686) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:48:09 (1787827689) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:48:12 (1787827692) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:48:15 (1787827695) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 06:48:24 (1787827704) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.610077 s, 20.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:48:38 (1787827718) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428494 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459399 s, 22.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:48:45 (1787827725) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:48:53 (1787827733) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000865228 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00148839 s, 688 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000957921 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00168533 s, 608 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 06:49:03 (1787827743) create test files with size 15752227 .. b2ee613e75f668682456ea4db8832d03 - f271501cd59d4db65a94a6be8cbe4e64 - 9fbc371289f6279acef37573689156d3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg617-client.virtnet /mnt/lustre (opts:) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 06:49:27 (1787827767) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:49:41 (1787827781) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:49:48 (1787827788) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113223 s, 9.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309898 s, 33.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 06:50:00 (1787827800) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13023 s, 16.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.458999 s, 9.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194435 s, 10.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 06:50:19 (1787827819) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:50:22 (1787827822) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:50:25 (1787827825) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348466 s, 30.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 06:50:32 (1787827832) llite.lustre-ffff9bb1d028e800.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 06:50:39 (1787827839) llite.lustre-ffff9bb1d028e800.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225307 s, 46.5 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496344 s, 21.1 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 06:50:48 (1787827848) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223949 s, 14.0 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:51:00 (1787827860) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:51:10 (1787827870) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.261774 s, 40.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg617-client.virtnet /mnt/lustre (opts:) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:51:22 (1787827882) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9bb1c6688000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.339311 s, 30.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg617-client.virtnet /mnt/lustre (opts:) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9bb1d82ee000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:51:37 (1787827897) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:52:15 (1787827935) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:52:50 (1787827970) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:52:54 (1787827974) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.254471 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.16543 s, 19.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8ba28cbd (nosync,stale) Mirror 2: 0x8ba28cbd (nosync,stale) ** verify mirror 2 doesn't change original checksum: d4df8e7b64d9d0263bfe215e3767c1b8 - mirror 1 checksum: a934dd46f6cc8a25b0a46a3cd0fe67b4 - mirror 2 checksum: d4df8e7b64d9d0263bfe215e3767c1b8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T06:52:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T06:52:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d4df8e7b64d9d0263bfe215e3767c1b8 - mirror 1 checksum: a934dd46f6cc8a25b0a46a3cd0fe67b4 - mirror 2 checksum: a934dd46f6cc8a25b0a46a3cd0fe67b4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T06:52:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T06:52:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:53:05 (1787827985) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363484 s, 28.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:53:12 (1787827992) Failing mds1 on oleg617-server Stopping /mnt/lustre-mds1 (opts:) on oleg617-server 06:53:24 (1787828004) shut down facet: mds1 facet_host: oleg617-server facet_failover_host: oleg617-server Failover mds1 to oleg617-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0000 06:53:57 (1787828037) targets are mounted 06:53:57 (1787828037) facet_failover done oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661879 s, 15.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:54:14 (1787828054) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0517823 s, 396 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.019031 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:54:25 (1787828065) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0687744 s, 298 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0148911 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:54:35 (1787828075) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00510197 s, 3.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:54:42 (1787828082) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0468216 s, 437 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:54:50 (1787828090) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00966641 s, 424 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0406178 s, 101 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:54:58 (1787828098) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:55:20 (1787828120) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:55:49 (1787828149) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:56:13 (1787828173) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:56:37 (1787828197) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 06:56:44 (1787828204) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58839 mirror split pid 58840 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5103: 58839 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5103: 58840 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 06:57:50 (1787828270) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 8271 skipping zero size read truncating to largest ever: 0x172f7 000002 1787828272.160572 trunc from 00000000 to 0x0172f6 (0x172f7 bytes) 000003 1787828272.181951 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x1d1d9 000004 1787828272.697871 fallocate 0x014e4a thru 0x01d1d8 (0x838f bytes) 000005 1787828272.746260 trunc from 0x007085 to 0x00a6df (0x365b bytes) fallocating to largest ever: 0x40000 000006 1787828272.753788 fallocate 0x0324bd thru 0x03ffff (0xdb43 bytes) 000007 1787828272.758587 read 0x01a0b0 thru 0x027981 (0xd8d2 bytes) 000008 1787828272.767754 trunc from 0x016042 to 0x02404f (0xe00e bytes) 000009 1787828272.772665 trunc from 0x040000 to 0x0357de (0xa821 bytes) 000010 1787828272.798431 write 0x03717b thru 0x03ffff (0x8e85 bytes) 000011 1787828272.801444 mapread 0x00fa81 thru 0x0187ab (0x8d2b bytes) 000012 1787828272.834782 trunc from 0x000720 to 0x00d5ed (0xcece bytes) 000013 1787828272.845225 trunc from 0x040000 to 0x010a0c (0x2f5f3 bytes) 000014 1787828272.870311 mapwrite 0x002c1b thru 0x00bee7 (0x92cd bytes) 000015 1787828272.913435 punch from 0x008de6 to 0x010a0c (0x7c27 bytes) 000016 1787828272.921856 punch from 0x00a811 to 0x010a0c (0x61fc bytes) 000017 1787828272.931799 punch from 0x010515 to 0x010a0c (0x4f8 bytes) 000018 1787828272.946424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) 000019 1787828273.143391 trunc from 0x010a0d to 0x004f16 (0xbaf6 bytes) 000021 1787828273.171400 trunc from 0x004f17 to 0x038983 (0x33a6d bytes) 000022 1787828273.194449 punch from 0x0119cb to 0x014621 (0x2c57 bytes) 000025 1787828273.209396 trunc from 0x02a27b to 0x02c996 (0x271c bytes) 000027 1787828273.216306 write 0x00c7ff thru 0x016b61 (0xa363 bytes) 000029 1787828273.245780 read 0x00b4b4 thru 0x0116cc (0x6219 bytes) 000030 1787828273.254554 mapread 0x01f88b thru 0x01fdcf (0x545 bytes) 000031 1787828273.262198 mapread 0x015354 thru 0x024db1 (0xfa5e bytes) 000032 1787828273.285630 mapread 0x0348f3 thru 0x038983 (0x4091 bytes) 000033 1787828273.295400 fallocate 0x00a43f thru 0x012495 (0x8057 bytes) 000034 1787828273.302463 trunc from 0x001879 to 0x0092f2 (0x7a7a bytes) 000035 1787828273.307711 punch from 0x019310 to 0x01e5a5 (0x5296 bytes) 000037 1787828273.314065 trunc from 0x028c6a to 0x02b80e (0x2ba5 bytes) 000038 1787828273.320715 trunc from 0x036b17 to 0x038983 (0x1e6d bytes) 000039 1787828273.329286 read 0x024362 thru 0x02a64d (0x62ec bytes) 000042 1787828273.343765 write 0x019d24 thru 0x02388c (0x9b69 bytes) 000043 1787828273.345299 fallocate 0x039216 thru 0x03f30a (0x60f5 bytes) 000044 1787828273.351436 read 0x006eda thru 0x01266f (0xb796 bytes) 000045 1787828273.352115 trunc from 0x00228a to 0x01220c (0xff83 bytes) 000046 1787828273.382094 fallocate 0x0131e2 thru 0x01d4a1 (0xa2c0 bytes) 000048 1787828273.391257 write 0x02883c thru 0x028b11 (0x2d6 bytes) 000049 1787828273.395443 trunc from 0x036f69 to 0x0375db (0x673 bytes) 000050 1787828273.401894 trunc from 0x029c5e to 0x030f03 (0x72a6 bytes) 000051 1787828273.407953 mapwrite 0x03b02e thru 0x03ffff (0x4fd2 bytes) 000052 1787828273.493810 mapread 0x010c52 thru 0x015c32 (0x4fe1 bytes) 000053 1787828273.494892 write 0x01386b thru 0x01aeda (0x7670 bytes) 000054 1787828273.498511 trunc from 0x040000 to 0x027d4f (0x182b0 bytes) 000055 1787828273.526888 write 0x015ec3 thru 0x01965e (0x379c bytes) 000056 1787828273.532214 mapwrite 0x035b73 thru 0x0396e1 (0x3b6f bytes) 000058 1787828273.634802 read 0x0088dc thru 0x0182c6 (0xf9eb bytes) 000059 1787828273.636347 punch from 0x023cfa to 0x0305d9 (0xc8e0 bytes) 000061 1787828273.644021 trunc from 0x0396e2 to 0x00f9ed (0x29cf4 bytes) 000062 1787828273.673396 punch from 0x00e6aa to 0x00f9ed (0x1344 bytes) 000063 1787828273.678938 read 0x00543f thru 0x00f9ed (0xa5af bytes) 000064 1787828273.686682 mapread 0x00852b thru 0x009229 (0xcff bytes) 000066 1787828273.688025 trunc from 0x00da86 to 0x00f9ed (0x1f68 bytes) 000067 1787828273.701662 fallocate 0x02cb13 thru 0x0392de (0xc7cc bytes) 000068 1787828273.711662 read 0x004015 thru 0x00919b (0x5187 bytes) 000069 1787828273.716308 fallocate 0x00d3cd thru 0x01274a (0x537e bytes) 000070 1787828273.731685 trunc from 0x0082ce to 0x0103c0 (0x80f3 bytes) 000071 1787828273.746925 fallocate 0x018cc2 thru 0x020b40 (0x7e7f bytes) 000072 1787828273.763941 read 0x002f77 thru 0x01274a (0xf7d4 bytes) 000073 1787828273.778863 mapwrite 0x003283 thru 0x005f96 (0x2d14 bytes) 000074 1787828273.803409 trunc from 0x01274b to 0x03cb57 (0x2a40d bytes) 000075 1787828273.824232 read 0x03441f thru 0x037713 (0x32f5 bytes) 000078 1787828273.848494 read 0x035df6 thru 0x03cb57 (0x6d62 bytes) 000079 1787828273.849771 trunc from 0x037b89 to 0x03cb57 (0x4fcf bytes) 000080 1787828273.859902 mapwrite 0x018d0f thru 0x01b16a (0x245c bytes) 000081 1787828273.889339 trunc from 0x03cb58 to 0x0142ef (0x28868 bytes) 000082 1787828273.909535 punch from 0x00ccfe to 0x00f778 (0x2a7b bytes) 000083 1787828273.917582 mapwrite 0x00130d thru 0x003a84 (0x2778 bytes) 000084 1787828273.942479 mapread 0x005909 thru 0x0142ef (0xe9e7 bytes) 000085 1787828273.956462 read 0x003cd5 thru 0x013a18 (0xfd44 bytes) 000086 1787828273.957319 punch from 0x00653a to 0x01403e (0xdb05 bytes) 000087 1787828273.965751 write 0x020bf8 thru 0x027925 (0x6d2e bytes) 000088 1787828273.969682 mapwrite 0x02892c thru 0x02bbce (0x32a3 bytes) 000089 1787828274.044413 read 0x01d4f0 thru 0x02428b (0x6d9c bytes) 000090 1787828274.045170 punch from 0x02494b to 0x02bbce (0x7284 bytes) 000091 1787828274.051287 write 0x03e625 thru 0x03ffff (0x19db bytes) 000092 1787828274.053264 mapwrite 0x038bd5 thru 0x03ffff (0x742b bytes) 000093 1787828274.094858 punch from 0x03b947 to 0x03ffff (0x46b9 bytes) 000094 1787828274.100350 mapread 0x0241ea thru 0x02b2da (0x70f1 bytes) 000095 1787828274.102537 trunc from 0x040000 to 0x00b56d (0x34a92 bytes) 000096 1787828274.117311 read 0x003cdd thru 0x009b29 (0x5e4d bytes) 000097 1787828274.117918 fallocate 0x0331c5 thru 0x03ffff (0xce3b bytes) 000098 1787828274.123055 fallocate 0x01b757 thru 0x023c0c (0x84b6 bytes) 000099 1787828274.127955 fallocate 0x017033 thru 0x02425d (0xd22b bytes) 000100 1787828274.138359 mapwrite 0x00b567 thru 0x0134d0 (0x7f6a bytes) 000101 1787828274.183091 mapread 0x01479f thru 0x018deb (0x464d bytes) 000104 1787828274.191248 MIRROR_EXTEND to 2 mirrors 000105 1787828274.855265 mapwrite 0x02bb83 thru 0x0319e2 (0x5e60 bytes) 000106 1787828274.997845 punch from 0x0261a6 to 0x026ecb (0xd26 bytes) 000108 1787828275.007995 write 0x00d99d thru 0x0145c7 (0x6c2b bytes) 000109 1787828275.024890 read 0x0243a3 thru 0x02e9e9 (0xa647 bytes) 000110 1787828275.031835 trunc from 0x012df5 to 0x01c5ce (0x97da bytes) 000111 1787828275.049395 mapread 0x013432 thru 0x01f7d2 (0xc3a1 bytes) 000112 1787828275.077260 fallocate 0x02667f thru 0x02c7df (0x6161 bytes) 000113 1787828275.085333 fallocate 0x0163bc thru 0x021bd8 (0xb81d bytes) 000114 1787828275.093781 write 0x039c1f thru 0x03ffff (0x63e1 bytes) 000115 1787828275.096292 mapwrite 0x02a04c thru 0x02f245 (0x51fa bytes) 000116 1787828275.119432 punch from 0x038979 to 0x03b822 (0x2eaa bytes) 000117 1787828275.127666 trunc from 0x040000 to 0x00ccff (0x33300 bytes) 000118 1787828275.146608 mapwrite 0x0392ce thru 0x03b0f2 (0x1e25 bytes) 000119 1787828275.186585 trunc from 0x03b0f3 to 0x004074 (0x3707e bytes) 000122 1787828275.208750 write 0x01a0bf thru 0x02966a (0xf5ac bytes) 000123 1787828275.211434 mapwrite 0x028ad2 thru 0x0352ab (0xc7da bytes) 000124 1787828275.263825 mapwrite 0x03b20e thru 0x03f549 (0x433c bytes) 000125 1787828275.301044 mapwrite 0x0266f2 thru 0x033371 (0xcc80 bytes) 000126 1787828275.327657 mapread 0x0238ff thru 0x030598 (0xcc9a bytes) 000127 1787828275.328593 read 0x0354e3 thru 0x0374a0 (0x1fbe bytes) 000128 1787828275.329128 trunc from 0x039cfe to 0x03f549 (0x584c bytes) 000129 1787828275.345702 MIRROR_RESYNC 2 mirrors 000130 1787828275.931580 read 0x03480e thru 0x03f549 (0xad3c bytes) 000131 1787828275.954864 read 0x034af7 thru 0x03f549 (0xaa53 bytes) 000132 1787828275.963084 MIRROR_EXTEND to 3 mirrors 000134 1787828276.704488 read 0x01fd17 thru 0x027b2b (0x7e15 bytes) 000135 1787828276.717700 read 0x03a05e thru 0x03f549 (0x54ec bytes) 000136 1787828276.730743 trunc from 0x01b379 to 0x02282d (0x74b5 bytes) 000137 1787828276.771016 trunc from 0x03f54a to 0x03a687 (0x4ec2 bytes) 000138 1787828276.811164 mapread 0x010cf3 thru 0x012f6c (0x227a bytes) 000139 1787828276.834867 fallocate 0x00853d thru 0x01225e (0x9d22 bytes) 000140 1787828276.848560 mapwrite 0x00fb49 thru 0x01b262 (0xb71a bytes) 000141 1787828276.932732 write 0x008543 thru 0x012228 (0x9ce6 bytes) 000142 1787828276.940678 mapwrite 0x0302f6 thru 0x034721 (0x442c bytes) 000143 1787828276.995935 trunc from 0x024e53 to 0x02a152 (0x5300 bytes) 000144 1787828277.017704 MIRROR_RESYNC 3 mirrors 000145 1787828278.112501 MIRROR_EXTEND to 4 mirrors 000146 1787828279.088477 MIRROR_RESYNC 4 mirrors 000147 1787828279.564891 trunc from 0x03a688 to 0x03dd3a (0x36b3 bytes) 000148 1787828279.605619 fallocate 0x0278c9 thru 0x02cb87 (0x52bf bytes) 000149 1787828279.611288 mapread 0x0033fc thru 0x0096ea (0x62ef bytes) 000150 1787828279.632492 fallocate 0x02963a thru 0x031157 (0x7b1e bytes) 000151 1787828279.640704 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd3b): Mirror 2: 0x723af651 (stale) Mirror 4: 0x723af651 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd3b): Mirror 2: 0x723af651 (stale) Mirror 4: 0x723af651 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd3b: skipped 000152 1787828279.998118 trunc from 0x03dd3b to 0x0255a3 (0x18797 bytes) 000153 1787828280.020394 mapread 0x01be99 thru 0x020e3c (0x4fa4 bytes) 000154 1787828280.040147 trunc from 0x0255a4 to 0x011cc5 (0x138de bytes) 000155 1787828280.063308 write 0x02b3e0 thru 0x02cbef (0x1810 bytes) 000156 1787828280.064876 write 0x027272 thru 0x02ce96 (0x5c25 bytes) 000157 1787828280.071072 mapread 0x002293 thru 0x00ac15 (0x8983 bytes) 000159 1787828280.089884 write 0x017599 thru 0x01db07 (0x656f bytes) 000160 1787828280.102901 mapwrite 0x01f6de thru 0x02d122 (0xda45 bytes) 000161 1787828280.209348 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d123): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d123): Mirror 2: 0x93bb85a9 (stale) Mirror 4: 0x93bb85a9 (stale) CRC-32 checksum value for chunk [0, 0x2d123): Mirror 1: 0x8d2ae88a Mirror 5: 0x8d2ae88a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d123: skipped 000162 1787828281.084845 fallocate 0x03f25e thru 0x03ffff (0xda2 bytes) 000165 1787828281.139289 mapread 0x038262 thru 0x03ade2 (0x2b81 bytes) 000166 1787828281.184940 read 0x00e320 thru 0x01d0ee (0xedcf bytes) 000167 1787828281.201870 read 0x0056b5 thru 0x013a4a (0xe396 bytes) 000168 1787828281.221555 mapread 0x03b68b thru 0x03ffff (0x4975 bytes) 000169 1787828281.226486 punch from 0x035a1d to 0x03e0b0 (0x8694 bytes) 000170 1787828281.238420 fallocate 0x014d95 thru 0x01badb (0x6d47 bytes) 000171 1787828281.248627 read 0x02e606 thru 0x03bbb3 (0xd5ae bytes) 000172 1787828281.285346 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4e794eb0 (stale) Mirror 4: 0x4e794eb0 (stale) Mirror 5: 0x15e47582 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x15e47582 Mirror 6: 0x15e47582 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000173 1787828282.351850 write 0x01b925 thru 0x01be38 (0x514 bytes) 000174 1787828282.433802 mapwrite 0x03b202 thru 0x03ffff (0x4dfe bytes) 000175 1787828282.476881 write 0x008007 thru 0x00c26c (0x4266 bytes) 000176 1787828282.485424 trunc from 0x040000 to 0x02c079 (0x13f86 bytes) 000177 1787828282.507639 fallocate 0x011f37 thru 0x01a209 (0x82d3 bytes) 000178 1787828282.512864 read 0x0239c7 thru 0x02997f (0x5fb9 bytes) 000179 1787828282.524278 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c07a): Mirror 4: 0x90ce6e9c (stale) Mirror 5: 0x9b967dff (stale) Mirror 6: 0x9b967dff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c07a): Mirror 4: 0x90ce6e9c (stale) Mirror 5: 0x9b967dff (stale) Mirror 6: 0x9b967dff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c07a: skipped 000182 1787828282.932064 fallocate 0x0148a1 thru 0x0154f2 (0xc52 bytes) 000184 1787828282.941718 read 0x027296 thru 0x02c079 (0x4de4 bytes) 000185 1787828282.958601 read 0x00f7dc thru 0x01da78 (0xe29d bytes) 000186 1787828283.015261 fallocate 0x028bc0 thru 0x034e96 (0xc2d7 bytes) 000188 1787828283.020476 write 0x02478a thru 0x02b0cd (0x6944 bytes) 000189 1787828283.026605 mapread 0x024d47 thru 0x032f08 (0xe1c2 bytes) 000190 1787828283.037291 trunc from 0x034e97 to 0x02f0d9 (0x5dbd bytes) 000191 1787828283.055167 punch from 0x024c26 to 0x024fe5 (0x3c0 bytes) 000192 1787828283.074485 fallocate 0x0221da thru 0x02a157 (0x7f7e bytes) 000193 1787828283.082834 fallocate 0x0254a6 thru 0x02e919 (0x9474 bytes) 000194 1787828283.091655 fallocate 0x008c21 thru 0x009202 (0x5e2 bytes) 000196 1787828283.100885 write 0x01ce7b thru 0x026772 (0x98f8 bytes) 000197 1787828283.103446 trunc from 0x02f0da to 0x003bcc (0x2b50d bytes) 000199 1787828283.121660 write 0x00a723 thru 0x013b08 (0x93e6 bytes) 000201 1787828283.126772 punch from 0x0000c4 to 0x000852 (0x78f bytes) 000202 1787828283.134335 write 0x03cc23 thru 0x03ffff (0x33dd bytes) 000203 1787828283.136447 trunc from 0x040000 to 0x02699e (0x19661 bytes) 000204 1787828283.160323 mapwrite 0x02077d thru 0x023248 (0x2acc bytes) 000205 1787828283.201473 mapread 0x020277 thru 0x0258b3 (0x563d bytes) 000206 1787828283.213532 mapwrite 0x013387 thru 0x01dee6 (0xab60 bytes) 000207 1787828283.303746 punch from 0x007a5f to 0x0162c2 (0xe864 bytes) 000208 1787828283.312748 mapwrite 0x03ebd7 thru 0x03ffff (0x1429 bytes) 000209 1787828283.370474 trunc from 0x002d49 to 0x00b65d (0x8915 bytes) 000210 1787828283.383859 mapread 0x039583 thru 0x03ffff (0x6a7d bytes) 000211 1787828283.385457 mapwrite 0x0203f2 thru 0x02833b (0x7f4a bytes) 000212 1787828283.433758 mapread 0x01290f thru 0x019204 (0x68f6 bytes) 000213 1787828283.499574 fallocate 0x03944e thru 0x03ffff (0x6bb2 bytes) 000214 1787828283.506490 write 0x020c1c thru 0x027f36 (0x731b bytes) 000215 1787828283.521330 MIRROR_RESYNC 4 mirrors 000218 1787828285.043208 trunc from 0x040000 to 0x021915 (0x1e6ea bytes) 000219 1787828285.120850 mapwrite 0x01b9b3 thru 0x01e73b (0x2d89 bytes) 000220 1787828285.207341 MIRROR_RESYNC 4 mirrors 000222 1787828286.265486 MIRROR_EXTEND to 5 mirrors 000223 1787828287.082165 mapwrite 0x018844 thru 0x0244a6 (0xbc63 bytes) 000224 1787828287.235294 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x244a7): Mirror 4: 0xa6ccc874 (stale) Mirror 5: 0xa6ccc874 (stale) Mirror 7: 0xa6ccc874 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x244a7): Mirror 4: 0xa6ccc874 (stale) Mirror 5: 0xa6ccc874 (stale) Mirror 7: 0xa6ccc874 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x244a7: skipped 000226 1787828287.557384 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x244a7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x244a7): Mirror 4: 0xa6ccc874 (stale) Mirror 5: 0xa6ccc874 (stale) Mirror 7: 0xa6ccc874 (stale) CRC-32 checksum value for chunk [0, 0x244a7): Mirror 1: 0xd65e165 Mirror 8: 0xd65e165 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x244a7: skipped 000227 1787828288.454867 trunc from 0x010b55 to 0x013849 (0x2cf5 bytes) 000228 1787828288.517991 write 0x03430a thru 0x034477 (0x16e bytes) 000229 1787828288.530877 mapread 0x009516 thru 0x014c19 (0xb704 bytes) 000230 1787828288.590682 mapread 0x002345 thru 0x00760c (0x52c8 bytes) 000231 1787828288.611306 punch from 0x0269ec to 0x033ae5 (0xd0fa bytes) 000232 1787828288.638559 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34478): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34478): Mirror 4: 0xdbe296e2 (stale) Mirror 5: 0xdbe296e2 (stale) Mirror 7: 0xdbe296e2 (stale) Mirror 8: 0x633629ff (stale) CRC-32 checksum value for chunk [0, 0x34478): Mirror 1: 0xce617117 Mirror 9: 0xce617117 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34478: skipped 000233 1787828289.843462 trunc from 0x034478 to 0x02c625 (0x7e52 bytes) 000234 1787828289.921024 punch from 0x01680e to 0x0243b0 (0xdba3 bytes) 000235 1787828289.947591 mapwrite 0x0120da thru 0x016123 (0x404a bytes) 000236 1787828290.041268 trunc from 0x02c626 to 0x0367a6 (0xa181 bytes) 000237 1787828290.095419 trunc from 0x0367a7 to 0x0272c4 (0xf4e2 bytes) 000238 1787828290.137464 trunc from 0x0272c5 to 0x0281bb (0xef7 bytes) 000239 1787828290.152002 trunc from 0x0281bc to 0x024ea0 (0x331b bytes) 000241 1787828290.176369 fallocate 0x02a0f3 thru 0x03361b (0x9529 bytes) 000242 1787828290.181313 fallocate 0x010f2c thru 0x020ead (0xff82 bytes) 000243 1787828290.186050 trunc from 0x011468 to 0x01cad4 (0xb66d bytes) 000244 1787828290.199225 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24ea1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ea1): Mirror 4: 0x6018dfb2 (stale) Mirror 5: 0x6018dfb2 (stale) Mirror 7: 0x6018dfb2 (stale) Mirror 8: 0x44945700 (stale) Mirror 9: 0x44945700 (stale) CRC-32 checksum value for chunk [0, 0x24ea1): Mirror 1: 0x724f1a37 Mirror 10: 0x724f1a37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ea1: skipped 000245 1787828291.681724 punch from 0x004ddd to 0x0066c8 (0x18ec bytes) 000247 1787828291.733955 trunc from 0x024ea1 to 0x034a5c (0xfbbc bytes) 000248 1787828291.789281 read 0x011b35 thru 0x01445b (0x2927 bytes) 000249 1787828291.800881 fallocate 0x01c5d0 thru 0x028ec6 (0xc8f7 bytes) 000250 1787828291.815368 mapread 0x0253b7 thru 0x034a5c (0xf6a6 bytes) 000251 1787828291.847861 trunc from 0x01844d to 0x01b073 (0x2c27 bytes) 000253 1787828291.853472 mapread 0x00823a thru 0x01132e (0x90f5 bytes) 000254 1787828291.862789 trunc from 0x034a5d to 0x0172fc (0x1d760 bytes) 000255 1787828291.877735 punch from 0x0052b1 to 0x013028 (0xdd78 bytes) 000257 1787828291.883889 mapwrite 0x02588c thru 0x02d2b0 (0x7a25 bytes) 000258 1787828291.967630 mapwrite 0x01b4b7 thru 0x02afe4 (0xfb2e bytes) 000259 1787828292.053453 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2d2b1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d2b1): Mirror 4: 0xc6aa34ca (stale) Mirror 5: 0xc6aa34ca (stale) Mirror 7: 0xc6aa34ca (stale) Mirror 8: 0xee53a6db (stale) Mirror 9: 0xee53a6db (stale) Mirror 10: 0xdc490afa (stale) CRC-32 checksum value for chunk [0, 0x2d2b1): Mirror 1: 0xf5c4745d Mirror 11: 0xf5c4745d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d2b1: skipped 000260 1787828293.184459 trunc from 0x01db3f to 0x026f3a (0x93fc bytes) 000261 1787828293.270247 write 0x038c1b thru 0x03ce5e (0x4244 bytes) 000262 1787828293.278292 trunc from 0x008d67 to 0x012f09 (0xa1a3 bytes) 000263 1787828293.287346 write 0x02a629 thru 0x030ca4 (0x667c bytes) 000264 1787828293.320731 mapwrite 0x011181 thru 0x013147 (0x1fc7 bytes) 000265 1787828293.407181 trunc from 0x010b27 to 0x01cf06 (0xc3e0 bytes) 000266 1787828293.416157 write 0x02e065 thru 0x038295 (0xa231 bytes) 000267 1787828293.427110 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ce5f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce5f): Mirror 4: 0xa58f3074 (stale) Mirror 5: 0xa58f3074 (stale) Mirror 7: 0xa58f3074 (stale) Mirror 8: 0x9bacb1d0 (stale) Mirror 9: 0xc8fba413 (stale) Mirror 10: 0x63bfed67 (stale) Mirror 11: 0xbd57f2cc (stale) CRC-32 checksum value for chunk [0, 0x3ce5f): Mirror 1: 0x8d0acf25 Mirror 12: 0x8d0acf25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce5f: skipped 000268 1787828294.696676 write 0x02f1e9 thru 0x03c9fe (0xd816 bytes) 000269 1787828294.815877 mapwrite 0x0295ef thru 0x038a4d (0xf45f bytes) 000270 1787828294.869204 mapwrite 0x01451e thru 0x020fef (0xcad2 bytes) 000271 1787828294.906101 read 0x02c8c5 thru 0x038004 (0xb740 bytes) 000273 1787828294.907576 punch from 0x03bb8c to 0x03ce5e (0x12d3 bytes) 000275 1787828294.913155 mapread 0x033973 thru 0x03ce5e (0x94ec bytes) 000276 1787828294.918739 fallocate 0x0200a8 thru 0x025643 (0x559c bytes) 000277 1787828294.924414 trunc from 0x022fe7 to 0x0323ed (0xf407 bytes) 000278 1787828294.929838 mapread 0x03208c thru 0x036750 (0x46c5 bytes) 000279 1787828294.938752 write 0x03dab5 thru 0x03ffff (0x254b bytes) 000280 1787828294.940137 write 0x033ab5 thru 0x03fa36 (0xbf82 bytes) 000281 1787828294.943127 trunc from 0x040000 to 0x03541f (0xabe0 bytes) 000282 1787828294.967637 read 0x01ef6b thru 0x01fb8b (0xc21 bytes) 000283 1787828294.968482 fallocate 0x01d282 thru 0x029b9a (0xc919 bytes) 000284 1787828294.983141 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x35420): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35420): Mirror 4: 0x2d7087ae (stale) Mirror 5: 0x2d7087ae (stale) Mirror 7: 0x2d7087ae (stale) Mirror 8: 0x20aaa5f9 (stale) Mirror 9: 0x5fbaf607 (stale) Mirror 10: 0x22649914 (stale) Mirror 11: 0xc6770ff0 (stale) Mirror 12: 0x922083c3 (stale) CRC-32 checksum value for chunk [0, 0x35420): Mirror 1: 0xf722e9c6 Mirror 13: 0xf722e9c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35420: skipped 000285 1787828296.387600 punch from 0x03254b to 0x03541f (0x2ed5 bytes) 000286 1787828296.431323 mapwrite 0x006c50 thru 0x007e01 (0x11b2 bytes) 000287 1787828296.466704 fallocate 0x02dd50 thru 0x036183 (0x8434 bytes) 000288 1787828296.472696 trunc from 0x00ad75 to 0x00b43f (0x6cb bytes) 000289 1787828296.477542 write 0x01af3d thru 0x025cff (0xadc3 bytes) 000290 1787828296.491797 read 0x014419 thru 0x01d7e4 (0x93cc bytes) 000291 1787828296.500428 punch from 0x010e2d to 0x0205b9 (0xf78d bytes) 000292 1787828296.515407 trunc from 0x01b6b2 to 0x022e4b (0x779a bytes) 000293 1787828296.525532 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36184): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36184): Mirror 4: 0x6f87dadb (stale) Mirror 5: 0x6f87dadb (stale) Mirror 7: 0x6f87dadb (stale) Mirror 8: 0xfcb10e98 (stale) Mirror 9: 0x6c879084 (stale) Mirror 10: 0x634afac7 (stale) Mirror 11: 0x155c8af1 (stale) Mirror 12: 0xd22d6a62 (stale) Mirror 13: 0x6c13c778 (stale) CRC-32 checksum value for chunk [0, 0x36184): Mirror 1: 0x63ba7b2e Mirror 14: 0x63ba7b2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36184: skipped 000294 1787828297.843803 MIRROR_RESYNC 11 mirrors 000296 1787828301.740755 fallocate 0x0212a3 thru 0x02b703 (0xa461 bytes) 000297 1787828301.810844 punch from 0x007a89 to 0x01240d (0xa985 bytes) 000299 1787828301.817701 punch from 0x007a13 to 0x014eed (0xd4db bytes) 000300 1787828301.823923 mapread 0x00cfdf thru 0x013ed2 (0x6ef4 bytes) 000301 1787828301.848544 fallocate 0x03c818 thru 0x03ffff (0x37e8 bytes) 000302 1787828301.855354 fallocate 0x011de3 thru 0x019df1 (0x800f bytes) 000303 1787828301.865006 trunc from 0x0085e8 to 0x014aa0 (0xc4b9 bytes) 000304 1787828301.871643 fallocate 0x03ce1c thru 0x03ffff (0x31e4 bytes) 000305 1787828301.878518 punch from 0x01cfc9 to 0x026974 (0x99ac bytes) 000306 1787828301.884898 write 0x028822 thru 0x02f345 (0x6b24 bytes) 000307 1787828301.903725 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36184): Mirror 4: 0x63ba7b2e (stale) Mirror 5: 0x63ba7b2e (stale) Mirror 8: 0x63ba7b2e (stale) Mirror 9: 0x63ba7b2e (stale) Mirror 10: 0x63ba7b2e (stale) Mirror 11: 0x63ba7b2e (stale) Mirror 12: 0x63ba7b2e (stale) Mirror 13: 0x63ba7b2e (stale) Mirror 14: 0x63ba7b2e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36184): Mirror 4: 0x63ba7b2e (stale) Mirror 5: 0x63ba7b2e (stale) Mirror 8: 0x63ba7b2e (stale) Mirror 9: 0x63ba7b2e (stale) Mirror 10: 0x63ba7b2e (stale) Mirror 11: 0x63ba7b2e (stale) Mirror 12: 0x63ba7b2e (stale) Mirror 13: 0x63ba7b2e (stale) Mirror 14: 0x63ba7b2e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36184: skipped 000308 1787828302.488919 mapwrite 0x003e50 thru 0x00afae (0x715f bytes) 000309 1787828302.525316 read 0x0114d5 thru 0x01f773 (0xe29f bytes) 000310 1787828302.541332 trunc from 0x036184 to 0x007e30 (0x2e353 bytes) 000311 1787828302.558615 punch from 0x007bee to 0x007e30 (0x243 bytes) 000312 1787828302.564362 fallocate 0x001c9b thru 0x007d49 (0x60af bytes) 000313 1787828302.568694 mapwrite 0x02af4c thru 0x038655 (0xd70a bytes) 000314 1787828302.634563 read 0x0037fb thru 0x00a7d4 (0x6fda bytes) 000315 1787828302.647278 punch from 0x004095 to 0x00c2b8 (0x8224 bytes) 000316 1787828302.653843 trunc from 0x00b58a to 0x0131fb (0x7c72 bytes) 000317 1787828302.674586 MIRROR_RESYNC 10 mirrors 000318 1787828304.896792 trunc from 0x038656 to 0x03e2ac (0x5c57 bytes) 000319 1787828304.954759 trunc from 0x005dfc to 0x011d8e (0xbf93 bytes) 000320 1787828304.963873 mapread 0x02f2f3 thru 0x034c23 (0x5931 bytes) 000322 1787828305.011521 mapwrite 0x02f895 thru 0x030950 (0x10bc bytes) 000323 1787828305.039264 MIRROR_RESYNC 10 mirrors 000324 1787828307.832704 read 0x0342f3 thru 0x03e2ac (0x9fba bytes) 000325 1787828307.843653 trunc from 0x02e059 to 0x030d75 (0x2d1d bytes) 000326 1787828307.897139 trunc from 0x00e27d to 0x013d94 (0x5b18 bytes) 000327 1787828307.903622 mapread 0x02f272 thru 0x038d19 (0x9aa8 bytes) 000328 1787828307.924252 mapread 0x02dc64 thru 0x0324fc (0x4899 bytes) 000329 1787828307.925531 trunc from 0x03e2ad to 0x000123 (0x3e189 bytes) 000330 1787828307.954510 fallocate 0x000afa thru 0x00a9ca (0x9ed1 bytes) 000331 1787828307.960703 punch from 0x0000df to 0x000123 (0x45 bytes) 000333 1787828307.967018 fallocate 0x00d168 thru 0x0157a2 (0x863b bytes) 000334 1787828307.972072 punch from 0x014cb4 to 0x0157a2 (0xaef bytes) 000335 1787828307.977425 write 0x01587c thru 0x01a1d1 (0x4956 bytes) 000336 1787828307.983603 mapread 0x00b6ed thru 0x019360 (0xdc74 bytes) 000337 1787828307.999469 mapread 0x0090c9 thru 0x00d46f (0x43a7 bytes) 000339 1787828308.006133 read 0x006812 thru 0x00dc38 (0x7427 bytes) 000340 1787828308.007404 punch from 0x015270 to 0x01a1d1 (0x4f62 bytes) 000341 1787828308.022275 mapread 0x00448b thru 0x0137ba (0xf330 bytes) 000343 1787828308.024072 read 0x00fabc thru 0x01a1d1 (0xa716 bytes) 000344 1787828308.048146 fallocate 0x01c850 thru 0x01d8ce (0x107f bytes) 000346 1787828308.054329 write 0x028491 thru 0x034b3d (0xc6ad bytes) 000349 1787828308.073401 trunc from 0x02b130 to 0x031531 (0x6402 bytes) 000350 1787828308.116908 mapread 0x013fda thru 0x021942 (0xd969 bytes) 000351 1787828308.141154 write 0x000eea thru 0x00d99f (0xcab6 bytes) 000353 1787828308.150950 mapwrite 0x03694b thru 0x03ffff (0x96b5 bytes) 000354 1787828308.218042 trunc from 0x040000 to 0x029bdf (0x16420 bytes) 000355 1787828308.237553 fallocate 0x036dbd thru 0x03ffff (0x9243 bytes) 000356 1787828308.244514 punch from 0x0020b6 to 0x00657d (0x44c8 bytes) 000357 1787828308.252119 read 0x022e59 thru 0x024641 (0x17e9 bytes) 000358 1787828308.253887 fallocate 0x020b77 thru 0x02c0fc (0xb586 bytes) 000359 1787828308.259362 read 0x0173a4 thru 0x01b3c8 (0x4025 bytes) 000361 1787828308.260603 mapwrite 0x029a97 thru 0x036944 (0xceae bytes) 000362 1787828308.310136 read 0x02fea3 thru 0x03398e (0x3aec bytes) 000363 1787828308.311347 mapread 0x025e2d thru 0x029fb3 (0x4187 bytes) 000364 1787828308.314373 read 0x03632d thru 0x036944 (0x618 bytes) 000365 1787828308.318673 mapread 0x017034 thru 0x01c6ef (0x56bc bytes) 000367 1787828308.320613 punch from 0x0305cd to 0x036944 (0x6378 bytes) 000369 1787828308.327043 punch from 0x0320fd to 0x034690 (0x2594 bytes) 000370 1787828308.331585 fallocate 0x00b281 thru 0x00bb3c (0x8bc bytes) 000371 1787828308.336333 read 0x019a42 thru 0x01f431 (0x59f0 bytes) 000372 1787828308.341080 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36945): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36945): Mirror 4: 0x2d3630d5 (stale) Mirror 5: 0x2d3630d5 (stale) Mirror 8: 0x2d3630d5 (stale) Mirror 9: 0x2d3630d5 (stale) Mirror 10: 0x2d3630d5 (stale) Mirror 11: 0x2d3630d5 (stale) Mirror 12: 0x2d3630d5 (stale) Mirror 13: 0x2d3630d5 (stale) Mirror 14: 0x2d3630d5 (stale) CRC-32 checksum value for chunk [0, 0x36945): Mirror 1: 0x420f85f5 Mirror 15: 0x420f85f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36945: skipped 000373 1787828309.486845 trunc from 0x036945 to 0x01b6e1 (0x1b263 bytes) 000375 1787828309.580743 MIRROR_RESYNC 11 mirrors 000376 1787828312.521614 fallocate 0x03c401 thru 0x03ffff (0x3bff bytes) 000377 1787828312.564889 punch from 0x00a682 to 0x01540e (0xad8d bytes) 000378 1787828312.570827 write 0x026e3c thru 0x03592d (0xeaf2 bytes) 000379 1787828312.573695 trunc from 0x03592e to 0x029c23 (0xbd0a bytes) 000380 1787828312.601043 fallocate 0x00184f thru 0x0019be (0x170 bytes) 000381 1787828312.605793 mapwrite 0x035335 thru 0x03ffff (0xaccb bytes) 000382 1787828312.670553 write 0x01b367 thru 0x01d69a (0x2334 bytes) 000383 1787828312.687499 MIRROR_RESYNC 11 mirrors 000385 1787828316.355363 punch from 0x02ca0f to 0x03a030 (0xd622 bytes) 000387 1787828316.396269 read 0x023e0e thru 0x0316a9 (0xd89c bytes) 000388 1787828316.413372 write 0x022400 thru 0x02ac8e (0x888f bytes) 000389 1787828316.419316 write 0x024881 thru 0x029a5a (0x51da bytes) 000390 1787828316.429540 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf1974552 (stale) Mirror 5: 0xf1974552 (stale) Mirror 8: 0xf1974552 (stale) Mirror 9: 0xf1974552 (stale) Mirror 10: 0xf1974552 (stale) Mirror 11: 0xf1974552 (stale) Mirror 12: 0xf1974552 (stale) Mirror 13: 0xf1974552 (stale) Mirror 14: 0xf1974552 (stale) Mirror 15: 0xf1974552 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf7995e95 Mirror 16: 0xf7995e95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000391 1787828317.577203 read 0x01a802 thru 0x0254cb (0xacca bytes) 000392 1787828317.599457 fallocate 0x036f2e thru 0x03ffff (0x90d2 bytes) 000393 1787828317.657865 fallocate 0x01b696 thru 0x01bf79 (0x8e4 bytes) 000394 1787828317.672364 trunc from 0x02860b to 0x0372a8 (0xec9e bytes) 000395 1787828317.678270 trunc from 0x007590 to 0x01570e (0xe17f bytes) 000396 1787828317.685628 trunc from 0x040000 to 0x037a02 (0x85fd bytes) 000397 1787828317.746369 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x37a03): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a03): Mirror 4: 0x9f88d0b7 (stale) Mirror 5: 0x9f88d0b7 (stale) Mirror 8: 0x9f88d0b7 (stale) Mirror 9: 0x9f88d0b7 (stale) Mirror 10: 0x9f88d0b7 (stale) Mirror 11: 0x9f88d0b7 (stale) Mirror 12: 0x9f88d0b7 (stale) Mirror 13: 0x9f88d0b7 (stale) Mirror 14: 0x9f88d0b7 (stale) Mirror 15: 0x9f88d0b7 (stale) Mirror 16: 0x46f9dcf0 (stale) CRC-32 checksum value for chunk [0, 0x37a03): Mirror 1: 0xb220d55b Mirror 17: 0xb220d55b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a03: skipped 000398 1787828318.987505 mapread 0x02f227 thru 0x0368ee (0x76c8 bytes) 000399 1787828319.015261 trunc from 0x037a03 to 0x025bba (0x11e48 bytes) 000400 1787828319.095057 fallocate 0x001e96 thru 0x00632b (0x4496 bytes) 000401 1787828319.110318 MIRROR_RESYNC 13 mirrors 000402 1787828323.258517 mapread 0x014e87 thru 0x01f655 (0xa7cf bytes) 000403 1787828323.338929 read 0x00a9e8 thru 0x0103f2 (0x5a0b bytes) 000404 1787828323.358865 read 0x00af6b thru 0x00b807 (0x89d bytes) 000405 1787828323.359297 punch from 0x021e9b to 0x025bba (0x3d20 bytes) 000406 1787828323.442301 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x25bbb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25bbb): Mirror 4: 0xce79d890 (stale) Mirror 5: 0xce79d890 (stale) Mirror 8: 0xce79d890 (stale) Mirror 9: 0xce79d890 (stale) Mirror 10: 0xce79d890 (stale) Mirror 11: 0xce79d890 (stale) Mirror 12: 0xce79d890 (stale) Mirror 13: 0xce79d890 (stale) Mirror 14: 0xce79d890 (stale) Mirror 15: 0xce79d890 (stale) Mirror 16: 0xce79d890 (stale) Mirror 17: 0xce79d890 (stale) CRC-32 checksum value for chunk [0, 0x25bbb): Mirror 1: 0x9647659e Mirror 18: 0x9647659e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25bbb: skipped 000408 1787828324.907872 write 0x00a396 thru 0x0146d7 (0xa342 bytes) 000409 1787828324.976322 MIRROR_RESYNC 14 mirrors 000410 1787828329.357560 write 0x0182b1 thru 0x019eb6 (0x1c06 bytes) 000411 1787828329.401594 mapwrite 0x01932b thru 0x027e36 (0xeb0c bytes) 000412 1787828329.483431 write 0x03ea30 thru 0x03ffff (0x15d0 bytes) 000414 1787828329.496911 MIRROR_RESYNC 14 mirrors 000415 1787828333.812415 mapwrite 0x012fc8 thru 0x0171b7 (0x41f0 bytes) 000416 1787828333.959553 mapread 0x023b1f thru 0x0329ca (0xeeac bytes) 000417 1787828334.021470 trunc from 0x0255f4 to 0x02f351 (0x9d5e bytes) 000418 1787828334.027117 read 0x013763 thru 0x0157da (0x2078 bytes) 000419 1787828334.027779 punch from 0x021ce4 to 0x027388 (0x56a5 bytes) 000420 1787828334.041215 read 0x0170a6 thru 0x01b3eb (0x4346 bytes) 000421 1787828334.048252 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb6caccc1 (stale) Mirror 5: 0xb6caccc1 (stale) Mirror 8: 0xb6caccc1 (stale) Mirror 9: 0xb6caccc1 (stale) Mirror 11: 0xb6caccc1 (stale) Mirror 12: 0xb6caccc1 (stale) Mirror 13: 0xb6caccc1 (stale) Mirror 14: 0xb6caccc1 (stale) Mirror 15: 0xb6caccc1 (stale) Mirror 16: 0xb6caccc1 (stale) Mirror 17: 0xb6caccc1 (stale) Mirror 18: 0xb6caccc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb6caccc1 (stale) Mirror 5: 0xb6caccc1 (stale) Mirror 8: 0xb6caccc1 (stale) Mirror 9: 0xb6caccc1 (stale) Mirror 11: 0xb6caccc1 (stale) Mirror 12: 0xb6caccc1 (stale) Mirror 13: 0xb6caccc1 (stale) Mirror 14: 0xb6caccc1 (stale) Mirror 15: 0xb6caccc1 (stale) Mirror 16: 0xb6caccc1 (stale) Mirror 17: 0xb6caccc1 (stale) Mirror 18: 0xb6caccc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000422 1787828335.024941 trunc from 0x040000 to 0x025a5d (0x1a5a2 bytes) 000423 1787828335.050543 MIRROR_RESYNC 13 mirrors 000424 1787828338.350434 trunc from 0x025a5e to 0x017b7f (0xdede bytes) 000425 1787828338.418452 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17b80): Mirror 4: 0xc6a94bdf (stale) Mirror 5: 0xc6a94bdf (stale) Mirror 8: 0xc6a94bdf (stale) Mirror 9: 0xc6a94bdf (stale) Mirror 11: 0xc6a94bdf (stale) Mirror 12: 0xc6a94bdf (stale) Mirror 13: 0xc6a94bdf (stale) Mirror 15: 0xc6a94bdf (stale) Mirror 16: 0xc6a94bdf (stale) Mirror 17: 0xc6a94bdf (stale) Mirror 18: 0xc6a94bdf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b80): Mirror 4: 0xc6a94bdf (stale) Mirror 5: 0xc6a94bdf (stale) Mirror 8: 0xc6a94bdf (stale) Mirror 9: 0xc6a94bdf (stale) Mirror 11: 0xc6a94bdf (stale) Mirror 12: 0xc6a94bdf (stale) Mirror 13: 0xc6a94bdf (stale) Mirror 15: 0xc6a94bdf (stale) Mirror 16: 0xc6a94bdf (stale) Mirror 17: 0xc6a94bdf (stale) Mirror 18: 0xc6a94bdf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b80: skipped 000426 1787828339.064674 mapread 0x0101e3 thru 0x017b7f (0x799d bytes) 000427 1787828339.085372 read 0x013f95 thru 0x017b7f (0x3beb bytes) 000428 1787828339.088361 mapread 0x013e76 thru 0x016cb4 (0x2e3f bytes) 000429 1787828339.091466 read 0x000546 thru 0x0044b7 (0x3f72 bytes) 000430 1787828339.105783 trunc from 0x00c4fb to 0x011bd2 (0x56d8 bytes) 000431 1787828339.112282 write 0x024873 thru 0x02e41e (0x9bac bytes) 000432 1787828339.118399 mapwrite 0x032234 thru 0x0369ad (0x477a bytes) 000433 1787828339.182063 read 0x034de0 thru 0x0369ad (0x1bce bytes) 000434 1787828339.183051 mapread 0x01d239 thru 0x0288ee (0xb6b6 bytes) 000435 1787828339.185489 write 0x0304f1 thru 0x03b0d9 (0xabe9 bytes) 000436 1787828339.194109 MIRROR_RESYNC 12 mirrors 000437 1787828343.529753 mapwrite 0x034996 thru 0x03e095 (0x9700 bytes) 000438 1787828343.619887 mapwrite 0x005c33 thru 0x00de03 (0x81d1 bytes) 000440 1787828343.652776 read 0x03618f thru 0x0365ad (0x41f bytes) 000441 1787828343.653431 punch from 0x03e07d to 0x03e095 (0x19 bytes) 000442 1787828343.664525 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e096): Mirror 4: 0x7b807c81 (stale) Mirror 5: 0x7b807c81 (stale) Mirror 8: 0x7b807c81 (stale) Mirror 9: 0x7b807c81 (stale) Mirror 11: 0x7b807c81 (stale) Mirror 12: 0x7b807c81 (stale) Mirror 13: 0x7b807c81 (stale) Mirror 15: 0x7b807c81 (stale) Mirror 16: 0x7b807c81 (stale) Mirror 17: 0x7b807c81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e096): Mirror 4: 0x7b807c81 (stale) Mirror 5: 0x7b807c81 (stale) Mirror 8: 0x7b807c81 (stale) Mirror 9: 0x7b807c81 (stale) Mirror 11: 0x7b807c81 (stale) Mirror 12: 0x7b807c81 (stale) Mirror 13: 0x7b807c81 (stale) Mirror 15: 0x7b807c81 (stale) Mirror 16: 0x7b807c81 (stale) Mirror 17: 0x7b807c81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e096: skipped 000443 1787828344.444408 mapwrite 0x020405 thru 0x02438e (0x3f8a bytes) 000446 1787828344.487471 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e096): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e096): Mirror 4: 0x7b807c81 (stale) Mirror 5: 0x7b807c81 (stale) Mirror 8: 0x7b807c81 (stale) Mirror 9: 0x7b807c81 (stale) Mirror 11: 0x7b807c81 (stale) Mirror 12: 0x7b807c81 (stale) Mirror 13: 0x7b807c81 (stale) Mirror 15: 0x7b807c81 (stale) Mirror 16: 0x7b807c81 (stale) Mirror 17: 0x7b807c81 (stale) CRC-32 checksum value for chunk [0, 0x3e096): Mirror 1: 0x2c589e8 Mirror 18: 0x2c589e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e096: skipped 000447 1787828345.618925 mapread 0x02cf80 thru 0x02e75d (0x17de bytes) 000448 1787828345.643232 trunc from 0x0097b8 to 0x012f1c (0x9765 bytes) 000449 1787828345.684353 trunc from 0x035b7e to 0x03d6b3 (0x7b36 bytes) 000450 1787828345.691799 fallocate 0x03cfc0 thru 0x03ffff (0x3040 bytes) 000451 1787828345.697010 fallocate 0x023c03 thru 0x025158 (0x1556 bytes) 000452 1787828345.701701 mapwrite 0x02b5b1 thru 0x031a85 (0x64d5 bytes) 000453 1787828345.730143 mapwrite 0x000a0a thru 0x009c12 (0x9209 bytes) 000454 1787828345.755249 punch from 0x011499 to 0x01c315 (0xae7d bytes) 000455 1787828345.760509 fallocate 0x02aa94 thru 0x036226 (0xb793 bytes) 000456 1787828345.772248 MIRROR_RESYNC 12 mirrors 000458 1787828348.946502 trunc from 0x0302a4 to 0x032eac (0x2c09 bytes) 000459 1787828349.024571 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3e096): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e096): Mirror 4: 0xe1bbbef5 (stale) Mirror 5: 0xe1bbbef5 (stale) Mirror 8: 0xe1bbbef5 (stale) Mirror 9: 0xe1bbbef5 (stale) Mirror 11: 0xe1bbbef5 (stale) Mirror 12: 0xe1bbbef5 (stale) Mirror 13: 0xe1bbbef5 (stale) Mirror 15: 0xe1bbbef5 (stale) Mirror 16: 0xe1bbbef5 (stale) Mirror 17: 0xe1bbbef5 (stale) Mirror 18: 0xe1bbbef5 (stale) CRC-32 checksum value for chunk [0, 0x3e096): Mirror 1: 0xff1918e2 Mirror 19: 0xff1918e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e096: skipped 000460 1787828350.710236 fallocate 0x02ac92 thru 0x031827 (0x6b96 bytes) 000461 1787828350.796339 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e096): Mirror 5: 0xe1bbbef5 (stale) Mirror 8: 0xe1bbbef5 (stale) Mirror 9: 0xe1bbbef5 (stale) Mirror 11: 0xe1bbbef5 (stale) Mirror 12: 0xe1bbbef5 (stale) Mirror 13: 0xe1bbbef5 (stale) Mirror 15: 0xe1bbbef5 (stale) Mirror 16: 0xe1bbbef5 (stale) Mirror 17: 0xe1bbbef5 (stale) Mirror 18: 0xe1bbbef5 (stale) Mirror 19: 0xff1918e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e096): Mirror 5: 0xe1bbbef5 (stale) Mirror 8: 0xe1bbbef5 (stale) Mirror 9: 0xe1bbbef5 (stale) Mirror 11: 0xe1bbbef5 (stale) Mirror 12: 0xe1bbbef5 (stale) Mirror 13: 0xe1bbbef5 (stale) Mirror 15: 0xe1bbbef5 (stale) Mirror 16: 0xe1bbbef5 (stale) Mirror 17: 0xe1bbbef5 (stale) Mirror 18: 0xe1bbbef5 (stale) Mirror 19: 0xff1918e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e096: skipped 000462 1787828351.756256 fallocate 0x00214f thru 0x006ab9 (0x496b bytes) 000463 1787828351.761681 read 0x028568 thru 0x0368a2 (0xe33b bytes) 000464 1787828351.778231 mapwrite 0x0143bd thru 0x0151a3 (0xde7 bytes) 000465 1787828351.801155 write 0x008ce1 thru 0x00bb45 (0x2e65 bytes) 000466 1787828351.833501 mapwrite 0x01e620 thru 0x02a180 (0xbb61 bytes) 000467 1787828351.874491 read 0x014178 thru 0x021827 (0xd6b0 bytes) 000468 1787828351.887444 mapwrite 0x018755 thru 0x01b565 (0x2e11 bytes) 000469 1787828351.909330 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3e096): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e096): Mirror 5: 0xe1bbbef5 (stale) Mirror 8: 0xe1bbbef5 (stale) Mirror 9: 0xe1bbbef5 (stale) Mirror 11: 0xe1bbbef5 (stale) Mirror 12: 0xe1bbbef5 (stale) Mirror 13: 0xe1bbbef5 (stale) Mirror 15: 0xe1bbbef5 (stale) Mirror 16: 0xe1bbbef5 (stale) Mirror 17: 0xe1bbbef5 (stale) Mirror 18: 0xe1bbbef5 (stale) Mirror 19: 0xff1918e2 (stale) CRC-32 checksum value for chunk [0, 0x3e096): Mirror 1: 0xe1fa0166 Mirror 20: 0xe1fa0166 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e096: skipped 000470 1787828353.257841 fallocate 0x0135ab thru 0x016fe1 (0x3a37 bytes) 000471 1787828353.313447 mapwrite 0x00135e thru 0x003c94 (0x2937 bytes) 000473 1787828353.347592 punch from 0x01083c to 0x015597 (0x4d5c bytes) 000474 1787828353.354507 write 0x01cd6c thru 0x01d033 (0x2c8 bytes) 000475 1787828353.364491 mapwrite 0x01fb69 thru 0x021d89 (0x2221 bytes) 000476 1787828353.389371 mapwrite 0x02d262 thru 0x02dd6a (0xb09 bytes) 000477 1787828353.409240 trunc from 0x03d281 to 0x03e095 (0xe15 bytes) 000478 1787828353.414859 read 0x01ed5a thru 0x028580 (0x9827 bytes) 000479 1787828353.416081 write 0x032fe2 thru 0x03e622 (0xb641 bytes) 000480 1787828353.422224 mapread 0x037d72 thru 0x03e622 (0x68b1 bytes) 000481 1787828353.427854 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e623): Mirror 5: 0x4309b101 (stale) Mirror 8: 0x4309b101 (stale) Mirror 9: 0x4309b101 (stale) Mirror 11: 0x4309b101 (stale) Mirror 12: 0x4309b101 (stale) Mirror 13: 0x4309b101 (stale) Mirror 15: 0x4309b101 (stale) Mirror 16: 0x4309b101 (stale) Mirror 17: 0x4309b101 (stale) Mirror 18: 0x4309b101 (stale) Mirror 19: 0x1fc09f6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e623): Mirror 5: 0x4309b101 (stale) Mirror 8: 0x4309b101 (stale) Mirror 9: 0x4309b101 (stale) Mirror 11: 0x4309b101 (stale) Mirror 12: 0x4309b101 (stale) Mirror 13: 0x4309b101 (stale) Mirror 15: 0x4309b101 (stale) Mirror 16: 0x4309b101 (stale) Mirror 17: 0x4309b101 (stale) Mirror 18: 0x4309b101 (stale) Mirror 19: 0x1fc09f6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e623: skipped 000482 1787828354.278654 punch from 0x00d935 to 0x017a6f (0xa13b bytes) 000483 1787828354.287535 read 0x02e176 thru 0x034c54 (0x6adf bytes) 000485 1787828354.307649 write 0x022580 thru 0x02d3ed (0xae6e bytes) 000486 1787828354.337516 read 0x00ba24 thru 0x00e695 (0x2c72 bytes) 000487 1787828354.356034 mapwrite 0x03f728 thru 0x03ffff (0x8d8 bytes) 000488 1787828354.401767 trunc from 0x02da53 to 0x039dcf (0xc37d bytes) 000489 1787828354.416866 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9d0eca58 (stale) Mirror 8: 0x9d0eca58 (stale) Mirror 11: 0x9d0eca58 (stale) Mirror 12: 0x9d0eca58 (stale) Mirror 13: 0x9d0eca58 (stale) Mirror 15: 0x9d0eca58 (stale) Mirror 16: 0x9d0eca58 (stale) Mirror 17: 0x9d0eca58 (stale) Mirror 18: 0x9d0eca58 (stale) Mirror 19: 0x9434ec65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9d0eca58 (stale) Mirror 8: 0x9d0eca58 (stale) Mirror 11: 0x9d0eca58 (stale) Mirror 12: 0x9d0eca58 (stale) Mirror 13: 0x9d0eca58 (stale) Mirror 15: 0x9d0eca58 (stale) Mirror 16: 0x9d0eca58 (stale) Mirror 17: 0x9d0eca58 (stale) Mirror 18: 0x9d0eca58 (stale) Mirror 19: 0x9434ec65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000490 1787828355.160556 punch from 0x01861c to 0x023129 (0xab0e bytes) 000491 1787828355.166866 read 0x031523 thru 0x03382d (0x230b bytes) 000492 1787828355.176504 fallocate 0x018410 thru 0x0267c2 (0xe3b3 bytes) 000493 1787828355.182078 punch from 0x000168 to 0x00cb8a (0xca23 bytes) 000494 1787828355.187786 trunc from 0x0099ab to 0x00cff7 (0x364d bytes) 000495 1787828355.192771 punch from 0x030bf5 to 0x03fc39 (0xf045 bytes) 000496 1787828355.198511 punch from 0x02308a to 0x027033 (0x3faa bytes) 000497 1787828355.204080 trunc from 0x040000 to 0x03ff09 (0xf6 bytes) 000498 1787828355.231103 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ff0a): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff0a): Mirror 5: 0xa6c712ed (stale) Mirror 8: 0xa6c712ed (stale) Mirror 11: 0xa6c712ed (stale) Mirror 12: 0xa6c712ed (stale) Mirror 13: 0xa6c712ed (stale) Mirror 15: 0xa6c712ed (stale) Mirror 16: 0xa6c712ed (stale) Mirror 17: 0xa6c712ed (stale) Mirror 18: 0xa6c712ed (stale) Mirror 19: 0xf37ff596 (stale) CRC-32 checksum value for chunk [0, 0x3ff0a): Mirror 1: 0x9cdfba53 Mirror 20: 0x9cdfba53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff0a: skipped 000499 1787828356.475895 fallocate 0x026c85 thru 0x036c7f (0xfffb bytes) 000500 1787828356.514586 trunc from 0x03ff0a to 0x01ab9f (0x2536a bytes) 000501 1787828356.552137 read 0x00e4f9 thru 0x010431 (0x1f39 bytes) 000502 1787828356.557007 trunc from 0x01aba0 to 0x010d04 (0x9e9b bytes) 000503 1787828356.573774 mapwrite 0x033e26 thru 0x034889 (0xa64 bytes) 000504 1787828356.608270 trunc from 0x03488a to 0x02da17 (0x6e72 bytes) 000505 1787828356.623535 mapwrite 0x031765 thru 0x03e603 (0xce9f bytes) 000506 1787828356.689882 write 0x01ff8b thru 0x02a943 (0xa9b9 bytes) 000507 1787828356.700609 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e604): Mirror 5: 0x69e1d63a (stale) Mirror 8: 0x69e1d63a (stale) Mirror 11: 0x69e1d63a (stale) Mirror 13: 0x69e1d63a (stale) Mirror 15: 0x69e1d63a (stale) Mirror 16: 0x69e1d63a (stale) Mirror 17: 0x69e1d63a (stale) Mirror 18: 0x69e1d63a (stale) Mirror 19: 0xadd26568 (stale) Mirror 20: 0x367323e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e604): Mirror 5: 0x69e1d63a (stale) Mirror 8: 0x69e1d63a (stale) Mirror 11: 0x69e1d63a (stale) Mirror 13: 0x69e1d63a (stale) Mirror 15: 0x69e1d63a (stale) Mirror 16: 0x69e1d63a (stale) Mirror 17: 0x69e1d63a (stale) Mirror 18: 0x69e1d63a (stale) Mirror 19: 0xadd26568 (stale) Mirror 20: 0x367323e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e604: skipped 000508 1787828357.259614 write 0x012fb2 thru 0x018c9d (0x5cec bytes) 000509 1787828357.279362 MIRROR_RESYNC 11 mirrors 000510 1787828360.783637 punch from 0x006eac to 0x00883c (0x1991 bytes) 000511 1787828360.831070 punch from 0x02ef02 to 0x03cb8b (0xdc8a bytes) 000512 1787828360.838747 punch from 0x005579 to 0x0089ad (0x3435 bytes) 000513 1787828360.844283 trunc from 0x02c25b to 0x02e6ce (0x2474 bytes) 000514 1787828360.852154 mapread 0x012962 thru 0x0166de (0x3d7d bytes) 000515 1787828360.875166 fallocate 0x039edd thru 0x03ffff (0x6123 bytes) 000516 1787828360.881987 fallocate 0x024444 thru 0x034002 (0xfbbf bytes) 000517 1787828360.887754 fallocate 0x027a54 thru 0x030a0d (0x8fba bytes) 000518 1787828360.893277 write 0x011454 thru 0x017052 (0x5bff bytes) 000519 1787828360.896803 mapwrite 0x004bdf thru 0x00baf2 (0x6f14 bytes) 000520 1787828360.932460 mapwrite 0x01c9ca thru 0x027e33 (0xb46a bytes) 000521 1787828360.983204 fallocate 0x0131d0 thru 0x0153ba (0x21eb bytes) 000522 1787828360.990628 mapread 0x01a752 thru 0x022ff7 (0x88a6 bytes) 000523 1787828360.991805 punch from 0x0222db to 0x02f5f0 (0xd316 bytes) 000524 1787828360.997989 trunc from 0x03e604 to 0x023488 (0x1b17b bytes) 000525 1787828361.028399 write 0x01848e thru 0x01b3cc (0x2f3f bytes) 000526 1787828361.030859 trunc from 0x023489 to 0x008467 (0x1b021 bytes) 000527 1787828361.051562 mapwrite 0x00ef44 thru 0x01b776 (0xc833 bytes) 000528 1787828361.161994 write 0x000d51 thru 0x0053d8 (0x4688 bytes) 000529 1787828361.187624 punch from 0x0127c8 to 0x013dca (0x1603 bytes) 000530 1787828361.198382 mapwrite 0x02e872 thru 0x033630 (0x4dbf bytes) 000531 1787828361.295336 read 0x017226 thru 0x02678b (0xf566 bytes) 000532 1787828361.304462 write 0x0037e1 thru 0x00463c (0xe5c bytes) 000533 1787828361.305599 write 0x01ec85 thru 0x0225a4 (0x3920 bytes) 000534 1787828361.307379 trunc from 0x033631 to 0x000ce1 (0x3294f bytes) 000535 1787828361.325404 trunc from 0x000b45 to 0x000ce1 (0x19d bytes) 000536 1787828361.334716 MIRROR_RESYNC 11 mirrors 000537 1787828362.373756 trunc from 0x0008c1 to 0x000ce1 (0x421 bytes) 000538 1787828362.429149 MIRROR_RESYNC 11 mirrors 000539 1787828363.493298 mapwrite 0x03712a thru 0x03a2c0 (0x3197 bytes) 000540 1787828363.569089 punch from 0x01c37d to 0x029a61 (0xd6e5 bytes) 000541 1787828363.583310 read 0x037910 thru 0x03a2c0 (0x29b1 bytes) 000542 1787828363.584063 trunc from 0x02d53e to 0x039f7f (0xca42 bytes) 000543 1787828363.589680 fallocate 0x037319 thru 0x0380b4 (0xd9c bytes) 000544 1787828363.597083 trunc from 0x03a2c1 to 0x02b045 (0xf27b bytes) 000545 1787828363.626428 punch from 0x023629 to 0x025047 (0x1a1f bytes) 000546 1787828363.633690 write 0x00dfb2 thru 0x0156b4 (0x7703 bytes) 000547 1787828363.644062 mapread 0x00b6d6 thru 0x012daa (0x76d5 bytes) 000548 1787828363.652001 punch from 0x01af39 to 0x02320d (0x82d5 bytes) 000549 1787828363.663886 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2b046): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b046): Mirror 5: 0xca1a4be0 (stale) Mirror 8: 0xca1a4be0 (stale) Mirror 11: 0xca1a4be0 (stale) Mirror 13: 0xca1a4be0 (stale) Mirror 15: 0xca1a4be0 (stale) Mirror 16: 0xca1a4be0 (stale) Mirror 17: 0xca1a4be0 (stale) Mirror 18: 0xca1a4be0 (stale) Mirror 19: 0xca1a4be0 (stale) Mirror 20: 0xca1a4be0 (stale) CRC-32 checksum value for chunk [0, 0x2b046): Mirror 1: 0x4b46799b Mirror 21: 0x4b46799b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b046: skipped 000550 1787828364.733277 mapread 0x007d84 thru 0x014ebb (0xd138 bytes) 000551 1787828364.794933 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2b046): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b046): Mirror 5: 0xca1a4be0 (stale) Mirror 8: 0xca1a4be0 (stale) Mirror 11: 0xca1a4be0 (stale) Mirror 13: 0xca1a4be0 (stale) Mirror 15: 0xca1a4be0 (stale) Mirror 16: 0xca1a4be0 (stale) Mirror 17: 0xca1a4be0 (stale) Mirror 18: 0xca1a4be0 (stale) Mirror 19: 0xca1a4be0 (stale) Mirror 20: 0xca1a4be0 (stale) CRC-32 checksum value for chunk [0, 0x2b046): Mirror 1: 0x4b46799b Mirror 21: 0x4b46799b Mirror 22: 0x4b46799b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b046: skipped 000552 1787828365.991879 read 0x029a67 thru 0x02b045 (0x15df bytes) 000554 1787828365.997358 read 0x018e4f thru 0x027937 (0xeae9 bytes) 000555 1787828366.009340 fallocate 0x03b5b5 thru 0x03ffff (0x4a4b bytes) 000556 1787828366.059117 read 0x0179b6 thru 0x023acf (0xc11a bytes) 000557 1787828366.071298 mapwrite 0x014ceb thru 0x01f555 (0xa86b bytes) 000558 1787828366.091231 punch from 0x02af8e to 0x02daeb (0x2b5e bytes) 000559 1787828366.095904 trunc from 0x025071 to 0x02a603 (0x5593 bytes) 000560 1787828366.100902 mapread 0x012d84 thru 0x021884 (0xeb01 bytes) 000561 1787828366.102204 read 0x017d8c thru 0x01d96a (0x5bdf bytes) 000562 1787828366.103873 mapread 0x0157d7 thru 0x01a190 (0x49ba bytes) 000564 1787828366.106249 mapwrite 0x029c05 thru 0x0332db (0x96d7 bytes) 000565 1787828366.132898 trunc from 0x0387df to 0x03fb7b (0x739d bytes) 000566 1787828366.137872 fallocate 0x00150a thru 0x00c46e (0xaf65 bytes) 000567 1787828366.143049 mapwrite 0x013d76 thru 0x020ceb (0xcf76 bytes) 000568 1787828366.173926 fallocate 0x00e508 thru 0x011402 (0x2efb bytes) 000569 1787828366.180884 mapread 0x02fc3e thru 0x02ff58 (0x31b bytes) 000570 1787828366.184487 fallocate 0x01f5d5 thru 0x02cc7e (0xd6aa bytes) 000571 1787828366.190775 read 0x02efe7 thru 0x03ba40 (0xca5a bytes) 000572 1787828366.205759 punch from 0x028d8d to 0x02d7b3 (0x4a27 bytes) 000573 1787828366.210904 mapwrite 0x023a6c thru 0x024a95 (0x102a bytes) 000574 1787828366.227335 mapread 0x03f119 thru 0x03ffff (0xee7 bytes) 000575 1787828366.234044 trunc from 0x040000 to 0x03a849 (0x57b6 bytes) 000576 1787828366.266892 trunc from 0x03a84a to 0x009738 (0x31111 bytes) 000577 1787828366.288176 mapread 0x005c7d thru 0x009738 (0x3abc bytes) 000578 1787828366.290254 punch from 0x0096e4 to 0x009738 (0x55 bytes) 000579 1787828366.297095 trunc from 0x0086dd to 0x009738 (0x105c bytes) 000580 1787828366.307142 punch from 0x00463b to 0x009738 (0x50fe bytes) 000581 1787828366.312908 trunc from 0x000017 to 0x0049ae (0x4998 bytes) 000582 1787828366.318426 mapwrite 0x01471c thru 0x01dd8a (0x966f bytes) 000583 1787828366.405717 trunc from 0x01dd8b to 0x032604 (0x1487a bytes) 000584 1787828366.420202 punch from 0x006027 to 0x007d64 (0x1d3e bytes) 000585 1787828366.425411 mapread 0x006fc3 thru 0x0095ae (0x25ec bytes) 000586 1787828366.469073 write 0x007163 thru 0x00d36e (0x620c bytes) 000587 1787828366.477049 punch from 0x026738 to 0x02cb97 (0x6460 bytes) 000589 1787828366.484355 write 0x033d27 thru 0x034137 (0x411 bytes) 000590 1787828366.486030 mapread 0x010db6 thru 0x012f55 (0x21a0 bytes) 000591 1787828366.510815 write 0x01515b thru 0x022a23 (0xd8c9 bytes) 000592 1787828366.523107 read 0x00d9d5 thru 0x01bcde (0xe30a bytes) 000593 1787828366.546306 punch from 0x015666 to 0x019696 (0x4031 bytes) 000594 1787828366.572130 read 0x02e9d4 thru 0x034137 (0x5764 bytes) 000595 1787828366.573047 mapwrite 0x01a872 thru 0x026366 (0xbaf5 bytes) 000596 1787828366.621255 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x34138): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34138): Mirror 5: 0x5e5489b2 (stale) Mirror 8: 0x5e5489b2 (stale) Mirror 11: 0x5e5489b2 (stale) Mirror 13: 0x5e5489b2 (stale) Mirror 15: 0x5e5489b2 (stale) Mirror 16: 0x5e5489b2 (stale) Mirror 17: 0x5e5489b2 (stale) Mirror 18: 0x5e5489b2 (stale) Mirror 19: 0x5e5489b2 (stale) Mirror 20: 0x5e5489b2 (stale) Mirror 21: 0x9cb95fab (stale) Mirror 22: 0x9cb95fab (stale) CRC-32 checksum value for chunk [0, 0x34138): Mirror 1: 0x64426641 Mirror 23: 0x64426641 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34138: skipped 000597 1787828367.895761 fallocate 0x02431e thru 0x028896 (0x4579 bytes) 000598 1787828367.939114 mapread 0x01e38b thru 0x024a7b (0x66f1 bytes) 000599 1787828367.955375 write 0x01fcc7 thru 0x023426 (0x3760 bytes) 000600 1787828367.967042 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34138): Mirror 5: 0x5e5489b2 (stale) Mirror 8: 0x5e5489b2 (stale) Mirror 11: 0x5e5489b2 (stale) Mirror 13: 0x5e5489b2 (stale) Mirror 15: 0x5e5489b2 (stale) Mirror 16: 0x5e5489b2 (stale) Mirror 17: 0x5e5489b2 (stale) Mirror 18: 0x5e5489b2 (stale) Mirror 19: 0x5e5489b2 (stale) Mirror 20: 0x5e5489b2 (stale) Mirror 21: 0x9cb95fab (stale) Mirror 22: 0x9cb95fab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34138): Mirror 5: 0x5e5489b2 (stale) Mirror 8: 0x5e5489b2 (stale) Mirror 11: 0x5e5489b2 (stale) Mirror 13: 0x5e5489b2 (stale) Mirror 15: 0x5e5489b2 (stale) Mirror 16: 0x5e5489b2 (stale) Mirror 17: 0x5e5489b2 (stale) Mirror 18: 0x5e5489b2 (stale) Mirror 19: 0x5e5489b2 (stale) Mirror 20: 0x5e5489b2 (stale) Mirror 21: 0x9cb95fab (stale) Mirror 22: 0x9cb95fab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34138: skipped 000601 1787828368.759544 trunc from 0x034138 to 0x01bb83 (0x185b4 bytes) 000602 1787828368.806423 punch from 0x01706f to 0x01bb83 (0x4b15 bytes) 000603 1787828368.817145 mapread 0x017c1e thru 0x01bb83 (0x3f66 bytes) 000604 1787828368.835574 mapwrite 0x017cef thru 0x027aa0 (0xfdb2 bytes) 000605 1787828368.891898 mapwrite 0x02fff9 thru 0x035825 (0x582d bytes) 000606 1787828368.933053 mapread 0x012b74 thru 0x018b04 (0x5f91 bytes) 000607 1787828368.934948 write 0x02e7a7 thru 0x03044b (0x1ca5 bytes) 000608 1787828368.937337 punch from 0x0125dc to 0x01ba33 (0x9458 bytes) 000609 1787828368.943405 write 0x028cbe thru 0x0350b5 (0xc3f8 bytes) 000610 1787828368.946041 read 0x0035e1 thru 0x00d3d0 (0x9df0 bytes) 000611 1787828368.955271 trunc from 0x028429 to 0x035825 (0xd3fd bytes) 000612 1787828368.970048 write 0x036354 thru 0x03ffff (0x9cac bytes) 000613 1787828368.972281 read 0x00fee4 thru 0x01fb43 (0xfc60 bytes) 000614 1787828368.983051 read 0x0321e4 thru 0x03fbfd (0xda1a bytes) 000615 1787828368.983900 mapwrite 0x034ae3 thru 0x039afd (0x501b bytes) 000616 1787828369.009897 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 21: 0x9db1aa6b (stale) Mirror 22: 0x9db1aa6b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26cd6c78 Mirror 23: 0x26cd6c78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000617 1787828370.510377 write 0x021645 thru 0x0241f2 (0x2bae bytes) 000618 1787828370.566418 trunc from 0x040000 to 0x007a76 (0x38589 bytes) 000619 1787828370.592203 read 0x000c73 thru 0x00143e (0x7cc bytes) 000620 1787828370.599348 trunc from 0x001774 to 0x0052da (0x3b67 bytes) 000621 1787828370.604723 trunc from 0x007a77 to 0x005138 (0x293e bytes) 000622 1787828370.623415 mapread 0x002cab thru 0x005138 (0x248e bytes) 000623 1787828370.652341 MIRROR_RESYNC 14 mirrors 000624 1787828372.657426 trunc from 0x0042f3 to 0x005138 (0xe46 bytes) 000625 1787828372.714906 write 0x00dfbd thru 0x019929 (0xb96d bytes) 000626 1787828372.719859 trunc from 0x01992a to 0x005a8f (0x13e9a bytes) 000627 1787828372.740791 write 0x03174f thru 0x035182 (0x3a34 bytes) 000628 1787828372.748125 MIRROR_RESYNC 14 mirrors 000629 1787828375.141138 fallocate 0x0004d5 thru 0x00bfe4 (0xbb10 bytes) 000630 1787828375.196681 mapread 0x017b43 thru 0x0276ec (0xfbaa bytes) 000631 1787828375.241263 trunc from 0x035183 to 0x03e6ae (0x952c bytes) 000632 1787828375.257500 write 0x02124f thru 0x02618a (0x4f3c bytes) 000633 1787828375.258810 trunc from 0x0028d3 to 0x00947b (0x6ba9 bytes) 000634 1787828375.263787 write 0x03d82f thru 0x03ffff (0x27d1 bytes) 000635 1787828375.269207 mapwrite 0x032151 thru 0x03ffff (0xdeaf bytes) 000636 1787828375.301024 fallocate 0x025564 thru 0x02e5b8 (0x9055 bytes) 000637 1787828375.307287 write 0x021d5c thru 0x0288c7 (0x6b6c bytes) 000638 1787828375.310529 trunc from 0x017566 to 0x01c01a (0x4ab5 bytes) 000640 1787828375.324012 write 0x022fc5 thru 0x02abe7 (0x7c23 bytes) 000641 1787828375.331256 fallocate 0x004c59 thru 0x007896 (0x2c3e bytes) 000642 1787828375.336517 trunc from 0x040000 to 0x00bbb7 (0x34448 bytes) 000643 1787828375.358180 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbbb8): Mirror 5: 0x5acaa4bf (stale) Mirror 8: 0x5acaa4bf (stale) Mirror 11: 0x5acaa4bf (stale) Mirror 13: 0x5acaa4bf (stale) Mirror 15: 0x5acaa4bf (stale) Mirror 16: 0x5acaa4bf (stale) Mirror 17: 0x5acaa4bf (stale) Mirror 19: 0x5acaa4bf (stale) Mirror 20: 0x5acaa4bf (stale) Mirror 21: 0x5acaa4bf (stale) Mirror 22: 0x5acaa4bf (stale) Mirror 23: 0x5acaa4bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbbb8): Mirror 5: 0x5acaa4bf (stale) Mirror 8: 0x5acaa4bf (stale) Mirror 11: 0x5acaa4bf (stale) Mirror 13: 0x5acaa4bf (stale) Mirror 15: 0x5acaa4bf (stale) Mirror 16: 0x5acaa4bf (stale) Mirror 17: 0x5acaa4bf (stale) Mirror 19: 0x5acaa4bf (stale) Mirror 20: 0x5acaa4bf (stale) Mirror 21: 0x5acaa4bf (stale) Mirror 22: 0x5acaa4bf (stale) Mirror 23: 0x5acaa4bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbbb8: skipped 000644 1787828375.992906 fallocate 0x00b533 thru 0x015f49 (0xaa17 bytes) 000645 1787828376.000880 fallocate 0x002a28 thru 0x011683 (0xec5c bytes) 000646 1787828376.010784 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15f4a): Mirror 5: 0x3b0b6b3a (stale) Mirror 8: 0x3b0b6b3a (stale) Mirror 11: 0x3b0b6b3a (stale) Mirror 13: 0x3b0b6b3a (stale) Mirror 15: 0x3b0b6b3a (stale) Mirror 16: 0x3b0b6b3a (stale) Mirror 19: 0x3b0b6b3a (stale) Mirror 20: 0x3b0b6b3a (stale) Mirror 21: 0x3b0b6b3a (stale) Mirror 22: 0x3b0b6b3a (stale) Mirror 23: 0x3b0b6b3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15f4a): Mirror 5: 0x3b0b6b3a (stale) Mirror 8: 0x3b0b6b3a (stale) Mirror 11: 0x3b0b6b3a (stale) Mirror 13: 0x3b0b6b3a (stale) Mirror 15: 0x3b0b6b3a (stale) Mirror 16: 0x3b0b6b3a (stale) Mirror 19: 0x3b0b6b3a (stale) Mirror 20: 0x3b0b6b3a (stale) Mirror 21: 0x3b0b6b3a (stale) Mirror 22: 0x3b0b6b3a (stale) Mirror 23: 0x3b0b6b3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15f4a: skipped 000647 1787828376.567767 write 0x03e2bb thru 0x03ffff (0x1d45 bytes) 000648 1787828376.572075 read 0x023043 thru 0x02874a (0x5708 bytes) 000649 1787828376.584131 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x64d9d941 (stale) Mirror 8: 0x64d9d941 (stale) Mirror 11: 0x64d9d941 (stale) Mirror 13: 0x64d9d941 (stale) Mirror 15: 0x64d9d941 (stale) Mirror 16: 0x64d9d941 (stale) Mirror 19: 0x64d9d941 (stale) Mirror 20: 0x64d9d941 (stale) Mirror 21: 0x64d9d941 (stale) Mirror 22: 0x64d9d941 (stale) Mirror 23: 0x64d9d941 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf0fb4d46 Mirror 24: 0xf0fb4d46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000650 1787828377.723951 trunc from 0x040000 to 0x02fb98 (0x10467 bytes) 000652 1787828377.784106 punch from 0x02a355 to 0x02fb98 (0x5844 bytes) 000653 1787828377.788104 trunc from 0x02fb99 to 0x00e884 (0x21314 bytes) 000655 1787828377.808799 mapread 0x000a0a thru 0x00e884 (0xde7b bytes) 000657 1787828377.834784 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe885): Mirror 5: 0xd2006081 (stale) Mirror 8: 0xd2006081 (stale) Mirror 11: 0xd2006081 (stale) Mirror 13: 0xd2006081 (stale) Mirror 15: 0xd2006081 (stale) Mirror 16: 0xd2006081 (stale) Mirror 19: 0xd2006081 (stale) Mirror 20: 0xd2006081 (stale) Mirror 21: 0xd2006081 (stale) Mirror 22: 0xd2006081 (stale) Mirror 23: 0xd2006081 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe885): Mirror 5: 0xd2006081 (stale) Mirror 8: 0xd2006081 (stale) Mirror 11: 0xd2006081 (stale) Mirror 13: 0xd2006081 (stale) Mirror 15: 0xd2006081 (stale) Mirror 16: 0xd2006081 (stale) Mirror 19: 0xd2006081 (stale) Mirror 20: 0xd2006081 (stale) Mirror 21: 0xd2006081 (stale) Mirror 22: 0xd2006081 (stale) Mirror 23: 0xd2006081 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe885: skipped 000659 1787828378.311508 trunc from 0x004b99 to 0x0072d8 (0x2740 bytes) 000660 1787828378.315937 trunc from 0x00e885 to 0x01d1a6 (0xe922 bytes) 000661 1787828378.337520 mapread 0x01b54d thru 0x01d1a6 (0x1c5a bytes) 000662 1787828378.356466 fallocate 0x00291d thru 0x00642b (0x3b0f bytes) 000663 1787828378.362253 trunc from 0x01077e to 0x01ce1f (0xc6a2 bytes) 000664 1787828378.373572 trunc from 0x00a3a2 to 0x00fc72 (0x58d1 bytes) 000665 1787828378.379240 mapwrite 0x014399 thru 0x017db2 (0x3a1a bytes) 000666 1787828378.421431 write 0x034dad thru 0x03f5b9 (0xa80d bytes) 000667 1787828378.423806 trunc from 0x03f5ba to 0x01374f (0x2be6a bytes) 000669 1787828378.453403 trunc from 0x013750 to 0x02224e (0xeaff bytes) 000670 1787828378.470954 write 0x0383b1 thru 0x03c951 (0x45a1 bytes) 000671 1787828378.474745 trunc from 0x03c952 to 0x02f27c (0xd6d5 bytes) 000672 1787828378.508869 mapread 0x00515b thru 0x00f90c (0xa7b2 bytes) 000673 1787828378.546389 read 0x015a39 thru 0x0202c7 (0xa88f bytes) 000674 1787828378.560492 trunc from 0x02f27d to 0x01d7ac (0x11ad0 bytes) 000675 1787828378.605919 fallocate 0x02d7c5 thru 0x02ef50 (0x178c bytes) 000676 1787828378.616549 punch from 0x008e15 to 0x00dcef (0x4edb bytes) 000677 1787828378.634238 punch from 0x019b66 to 0x01d7ac (0x3c47 bytes) 000678 1787828378.641121 mapwrite 0x0115f0 thru 0x0207b0 (0xf1c1 bytes) 000679 1787828378.750595 trunc from 0x0207b1 to 0x013ab5 (0xccfb bytes) 000680 1787828378.776385 write 0x01a13f thru 0x028cdb (0xeb9d bytes) 000681 1787828378.778530 mapread 0x01780d thru 0x022902 (0xb0f6 bytes) 000683 1787828378.792922 trunc from 0x00631c to 0x00b5f4 (0x52d9 bytes) 000684 1787828378.816135 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28cdc): Mirror 8: 0xb16d0faa (stale) Mirror 11: 0xb16d0faa (stale) Mirror 13: 0xb16d0faa (stale) Mirror 15: 0xb16d0faa (stale) Mirror 16: 0xb16d0faa (stale) Mirror 19: 0xb16d0faa (stale) Mirror 20: 0xb16d0faa (stale) Mirror 21: 0xb16d0faa (stale) Mirror 22: 0xb16d0faa (stale) Mirror 23: 0xb16d0faa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28cdc): Mirror 8: 0xb16d0faa (stale) Mirror 11: 0xb16d0faa (stale) Mirror 13: 0xb16d0faa (stale) Mirror 15: 0xb16d0faa (stale) Mirror 16: 0xb16d0faa (stale) Mirror 19: 0xb16d0faa (stale) Mirror 20: 0xb16d0faa (stale) Mirror 21: 0xb16d0faa (stale) Mirror 22: 0xb16d0faa (stale) Mirror 23: 0xb16d0faa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28cdc: skipped 000685 1787828379.492868 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x28cdc): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28cdc): Mirror 8: 0xb16d0faa (stale) Mirror 11: 0xb16d0faa (stale) Mirror 13: 0xb16d0faa (stale) Mirror 15: 0xb16d0faa (stale) Mirror 16: 0xb16d0faa (stale) Mirror 19: 0xb16d0faa (stale) Mirror 20: 0xb16d0faa (stale) Mirror 21: 0xb16d0faa (stale) Mirror 22: 0xb16d0faa (stale) Mirror 23: 0xb16d0faa (stale) CRC-32 checksum value for chunk [0, 0x28cdc): Mirror 1: 0xd150d9ca Mirror 24: 0xd150d9ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28cdc: skipped 000686 1787828380.714802 fallocate 0x0004a9 thru 0x001d81 (0x18d9 bytes) 000688 1787828380.795270 trunc from 0x028cdc to 0x0034e3 (0x257f8 bytes) 000689 1787828380.825745 trunc from 0x0034e4 to 0x0136b8 (0x101d5 bytes) 000690 1787828380.847120 trunc from 0x0136b9 to 0x017000 (0x3948 bytes) 000691 1787828380.868079 trunc from 0x017001 to 0x00eb4c (0x84b4 bytes) 000693 1787828380.885840 trunc from 0x00afce to 0x00eb4c (0x3b7f bytes) 000695 1787828380.891231 punch from 0x006eb6 to 0x00de8d (0x6fd8 bytes) 000696 1787828380.896144 trunc from 0x003c99 to 0x00c5ab (0x8913 bytes) 000697 1787828380.904247 MIRROR_RESYNC 12 mirrors 000698 1787828382.610813 read 0x00160b thru 0x00eb4c (0xd542 bytes) 000699 1787828382.619381 write 0x030a65 thru 0x0311b2 (0x74e bytes) 000700 1787828382.658703 punch from 0x018936 to 0x0191fe (0x8c9 bytes) 000701 1787828382.668735 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x311b3): Mirror 8: 0x88b47275 (stale) Mirror 11: 0x88b47275 (stale) Mirror 13: 0x88b47275 (stale) Mirror 16: 0x88b47275 (stale) Mirror 19: 0x88b47275 (stale) Mirror 20: 0x88b47275 (stale) Mirror 21: 0x88b47275 (stale) Mirror 22: 0x88b47275 (stale) Mirror 23: 0x88b47275 (stale) Mirror 24: 0x88b47275 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x311b3): Mirror 8: 0x88b47275 (stale) Mirror 11: 0x88b47275 (stale) Mirror 13: 0x88b47275 (stale) Mirror 16: 0x88b47275 (stale) Mirror 19: 0x88b47275 (stale) Mirror 20: 0x88b47275 (stale) Mirror 21: 0x88b47275 (stale) Mirror 22: 0x88b47275 (stale) Mirror 23: 0x88b47275 (stale) Mirror 24: 0x88b47275 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x311b3: skipped 000702 1787828383.200204 trunc from 0x02de17 to 0x0311b2 (0x339c bytes) 000703 1787828383.208765 mapwrite 0x0172b8 thru 0x01ce33 (0x5b7c bytes) 000704 1787828383.259201 MIRROR_RESYNC 11 mirrors 000705 1787828385.525807 write 0x038fef thru 0x03ffff (0x7011 bytes) 000706 1787828385.581699 mapwrite 0x03e04b thru 0x03ffff (0x1fb5 bytes) 000707 1787828385.608056 punch from 0x038387 to 0x03cce7 (0x4961 bytes) 000708 1787828385.619075 mapwrite 0x024e4f thru 0x0285e8 (0x379a bytes) 000709 1787828385.684378 mapread 0x02c5c0 thru 0x02d3fd (0xe3e bytes) 000710 1787828385.685733 trunc from 0x02578e to 0x02fe15 (0xa688 bytes) 000711 1787828385.690896 fallocate 0x030a98 thru 0x039ed6 (0x943f bytes) 000712 1787828385.698018 read 0x034ba0 thru 0x03ffff (0xb460 bytes) 000713 1787828385.706794 mapwrite 0x021582 thru 0x0274fe (0x5f7d bytes) 000715 1787828385.753498 trunc from 0x040000 to 0x0200a3 (0x1ff5c bytes) 000716 1787828385.799030 punch from 0x00926c to 0x01694f (0xd6e4 bytes) 000717 1787828385.811646 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x200a4): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200a4): Mirror 8: 0x425f5afb (stale) Mirror 11: 0x425f5afb (stale) Mirror 13: 0x425f5afb (stale) Mirror 16: 0x425f5afb (stale) Mirror 19: 0x425f5afb (stale) Mirror 20: 0x425f5afb (stale) Mirror 21: 0x425f5afb (stale) Mirror 22: 0x425f5afb (stale) Mirror 23: 0x425f5afb (stale) Mirror 24: 0x425f5afb (stale) CRC-32 checksum value for chunk [0, 0x200a4): Mirror 1: 0x425f5afb Mirror 25: 0x425f5afb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200a4: skipped 000718 1787828386.908639 read 0x005b22 thru 0x00924c (0x372b bytes) 000719 1787828386.915497 fallocate 0x039e7c thru 0x03dce6 (0x3e6b bytes) 000720 1787828386.959243 write 0x02a0bf thru 0x02f754 (0x5696 bytes) 000721 1787828386.968871 read 0x002a1c thru 0x00c02f (0x9614 bytes) 000722 1787828386.987244 mapread 0x02cf22 thru 0x02f754 (0x2833 bytes) 000723 1787828386.988691 mapwrite 0x00aed5 thru 0x016fb5 (0xc0e1 bytes) 000725 1787828387.015500 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f755): Mirror 8: 0xe62b8c3f (stale) Mirror 11: 0xe62b8c3f (stale) Mirror 13: 0xe62b8c3f (stale) Mirror 16: 0xe62b8c3f (stale) Mirror 20: 0xe62b8c3f (stale) Mirror 21: 0xe62b8c3f (stale) Mirror 22: 0xe62b8c3f (stale) Mirror 23: 0xe62b8c3f (stale) Mirror 24: 0xe62b8c3f (stale) Mirror 25: 0xe62b8c3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f755): Mirror 8: 0xe62b8c3f (stale) Mirror 11: 0xe62b8c3f (stale) Mirror 13: 0xe62b8c3f (stale) Mirror 16: 0xe62b8c3f (stale) Mirror 20: 0xe62b8c3f (stale) Mirror 21: 0xe62b8c3f (stale) Mirror 22: 0xe62b8c3f (stale) Mirror 23: 0xe62b8c3f (stale) Mirror 24: 0xe62b8c3f (stale) Mirror 25: 0xe62b8c3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f755: skipped 000726 1787828387.664368 trunc from 0x008a94 to 0x00b963 (0x2ed0 bytes) 000727 1787828387.670896 trunc from 0x026de5 to 0x0290cf (0x22eb bytes) 000728 1787828387.676910 trunc from 0x010bc2 to 0x0140de (0x351d bytes) 000729 1787828387.686165 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f755): Mirror 8: 0xe62b8c3f (stale) Mirror 11: 0xe62b8c3f (stale) Mirror 13: 0xe62b8c3f (stale) Mirror 20: 0xe62b8c3f (stale) Mirror 21: 0xe62b8c3f (stale) Mirror 22: 0xe62b8c3f (stale) Mirror 23: 0xe62b8c3f (stale) Mirror 24: 0xe62b8c3f (stale) Mirror 25: 0xe62b8c3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f755): Mirror 8: 0xe62b8c3f (stale) Mirror 11: 0xe62b8c3f (stale) Mirror 13: 0xe62b8c3f (stale) Mirror 20: 0xe62b8c3f (stale) Mirror 21: 0xe62b8c3f (stale) Mirror 22: 0xe62b8c3f (stale) Mirror 23: 0xe62b8c3f (stale) Mirror 24: 0xe62b8c3f (stale) Mirror 25: 0xe62b8c3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f755: skipped 000730 1787828388.335763 read 0x02e3ea thru 0x02f754 (0x136b bytes) 000731 1787828388.344424 read 0x00c042 thru 0x00cf28 (0xee7 bytes) 000732 1787828388.362107 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2f755): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f755): Mirror 8: 0xe62b8c3f (stale) Mirror 11: 0xe62b8c3f (stale) Mirror 13: 0xe62b8c3f (stale) Mirror 20: 0xe62b8c3f (stale) Mirror 21: 0xe62b8c3f (stale) Mirror 22: 0xe62b8c3f (stale) Mirror 23: 0xe62b8c3f (stale) Mirror 24: 0xe62b8c3f (stale) Mirror 25: 0xe62b8c3f (stale) CRC-32 checksum value for chunk [0, 0x2f755): Mirror 1: 0x6ca0eade Mirror 26: 0x6ca0eade lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f755: skipped 000733 1787828389.561048 MIRROR_RESYNC 11 mirrors 000734 1787828392.186095 trunc from 0x02f755 to 0x021e37 (0xd91d bytes) 000736 1787828392.236852 punch from 0x00474e to 0x0082cd (0x3b80 bytes) 000737 1787828392.242735 fallocate 0x01a062 thru 0x01f9e1 (0x5980 bytes) 000738 1787828392.247918 punch from 0x00d0d4 to 0x013e38 (0x6d65 bytes) 000739 1787828392.252679 write 0x02b92a thru 0x0340aa (0x8781 bytes) 000740 1787828392.255763 trunc from 0x0340ab to 0x02ad7a (0x9330 bytes) 000741 1787828392.279654 write 0x039a82 thru 0x03cb75 (0x30f4 bytes) 000742 1787828392.288721 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb76): Mirror 8: 0x400b5e6f (stale) Mirror 13: 0x400b5e6f (stale) Mirror 20: 0x400b5e6f (stale) Mirror 21: 0x400b5e6f (stale) Mirror 22: 0x400b5e6f (stale) Mirror 23: 0x400b5e6f (stale) Mirror 24: 0x400b5e6f (stale) Mirror 25: 0x400b5e6f (stale) Mirror 26: 0x400b5e6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb76): Mirror 8: 0x400b5e6f (stale) Mirror 13: 0x400b5e6f (stale) Mirror 20: 0x400b5e6f (stale) Mirror 21: 0x400b5e6f (stale) Mirror 22: 0x400b5e6f (stale) Mirror 23: 0x400b5e6f (stale) Mirror 24: 0x400b5e6f (stale) Mirror 25: 0x400b5e6f (stale) Mirror 26: 0x400b5e6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb76: skipped 000743 1787828392.894413 fallocate 0x002551 thru 0x0040e1 (0x1b91 bytes) 000744 1787828392.898542 mapread 0x0220db thru 0x0229ec (0x912 bytes) 000745 1787828392.905831 write 0x014f34 thru 0x01cf0c (0x7fd9 bytes) 000747 1787828392.920138 punch from 0x0132d7 to 0x015e4a (0x2b74 bytes) 000748 1787828392.933091 mapread 0x01b4c2 thru 0x02579e (0xa2dd bytes) 000749 1787828392.946269 trunc from 0x00b089 to 0x013526 (0x849e bytes) 000751 1787828392.954552 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3cb76): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb76): Mirror 8: 0x400b5e6f (stale) Mirror 13: 0x400b5e6f (stale) Mirror 20: 0x400b5e6f (stale) Mirror 21: 0x400b5e6f (stale) Mirror 22: 0x400b5e6f (stale) Mirror 23: 0x400b5e6f (stale) Mirror 24: 0x400b5e6f (stale) Mirror 25: 0x400b5e6f (stale) Mirror 26: 0x400b5e6f (stale) CRC-32 checksum value for chunk [0, 0x3cb76): Mirror 1: 0x8559308c Mirror 27: 0x8559308c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb76: skipped 000753 1787828394.233683 write 0x002e99 thru 0x007dc2 (0x4f2a bytes) 000754 1787828394.306278 fallocate 0x00bbd3 thru 0x015d05 (0xa133 bytes) 000756 1787828394.312605 read 0x03534c thru 0x035e79 (0xb2e bytes) 000757 1787828394.318369 fallocate 0x03bfc3 thru 0x03c219 (0x257 bytes) 000758 1787828394.328949 MIRROR_RESYNC 11 mirrors 000759 1787828397.631066 MIRROR_RESYNC 11 mirrors 000760 1787828398.343151 write 0x0024ee thru 0x00d2b2 (0xadc5 bytes) 000761 1787828398.419209 mapwrite 0x023ab7 thru 0x0309b6 (0xcf00 bytes) 000762 1787828398.524112 mapread 0x006e6b thru 0x00e446 (0x75dc bytes) 000763 1787828398.538201 read 0x0029bb thru 0x00e7ec (0xbe32 bytes) 000764 1787828398.540187 read 0x0264e7 thru 0x034e6e (0xe988 bytes) 000765 1787828398.545912 MIRROR_RESYNC 11 mirrors 000766 1787828401.809177 write 0x03cefd thru 0x03d89e (0x9a2 bytes) 000767 1787828401.900936 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d89f): Mirror 8: 0xff13c973 (stale) Mirror 13: 0xff13c973 (stale) Mirror 20: 0xff13c973 (stale) Mirror 21: 0xff13c973 (stale) Mirror 22: 0xff13c973 (stale) Mirror 24: 0xff13c973 (stale) Mirror 25: 0xff13c973 (stale) Mirror 26: 0xff13c973 (stale) Mirror 27: 0xff13c973 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d89f): Mirror 8: 0xff13c973 (stale) Mirror 13: 0xff13c973 (stale) Mirror 20: 0xff13c973 (stale) Mirror 21: 0xff13c973 (stale) Mirror 22: 0xff13c973 (stale) Mirror 24: 0xff13c973 (stale) Mirror 25: 0xff13c973 (stale) Mirror 26: 0xff13c973 (stale) Mirror 27: 0xff13c973 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d89f: skipped 000768 1787828402.736749 trunc from 0x03d89f to 0x01b0dc (0x227c2 bytes) 000769 1787828402.792235 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b0dd): Mirror 8: 0x463d90f1 (stale) Mirror 13: 0x463d90f1 (stale) Mirror 20: 0x463d90f1 (stale) Mirror 21: 0x463d90f1 (stale) Mirror 22: 0x463d90f1 (stale) Mirror 25: 0x463d90f1 (stale) Mirror 26: 0x463d90f1 (stale) Mirror 27: 0x463d90f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b0dd): Mirror 8: 0x463d90f1 (stale) Mirror 13: 0x463d90f1 (stale) Mirror 20: 0x463d90f1 (stale) Mirror 21: 0x463d90f1 (stale) Mirror 22: 0x463d90f1 (stale) Mirror 25: 0x463d90f1 (stale) Mirror 26: 0x463d90f1 (stale) Mirror 27: 0x463d90f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b0dd: skipped 000770 1787828403.367444 write 0x032155 thru 0x03d9fa (0xb8a6 bytes) 000771 1787828403.383106 trunc from 0x03d9fb to 0x02bef3 (0x11b07 bytes) 000772 1787828403.413877 punch from 0x0176c2 to 0x02174c (0xa08b bytes) 000773 1787828403.423123 mapwrite 0x013e8b thru 0x01ba39 (0x7baf bytes) 000774 1787828403.485244 read 0x02602a thru 0x02bef3 (0x5eca bytes) 000775 1787828403.501601 read 0x0055f7 thru 0x00a52f (0x4f39 bytes) 000777 1787828403.527274 trunc from 0x02799e to 0x02bef3 (0x4556 bytes) 000778 1787828403.542299 mapwrite 0x01855a thru 0x01b78c (0x3233 bytes) 000779 1787828403.562112 mapread 0x00441d thru 0x0081fc (0x3de0 bytes) 000780 1787828403.567638 mapread 0x019478 thru 0x01b467 (0x1ff0 bytes) 000781 1787828403.568778 mapwrite 0x0181fd thru 0x019bdf (0x19e3 bytes) 000782 1787828403.604642 fallocate 0x0292a2 thru 0x034cd0 (0xba2f bytes) 000783 1787828403.610692 mapwrite 0x03c24a thru 0x03ce78 (0xc2f bytes) 000784 1787828403.659806 read 0x0190a2 thru 0x024b75 (0xbad4 bytes) 000785 1787828403.662059 mapread 0x026f35 thru 0x035eec (0xefb8 bytes) 000786 1787828403.675098 trunc from 0x012d10 to 0x01cf7e (0xa26f bytes) 000787 1787828403.682688 mapread 0x01b8fb thru 0x02123e (0x5944 bytes) 000788 1787828403.700774 MIRROR_RESYNC 9 mirrors 000789 1787828405.815525 mapwrite 0x001002 thru 0x010146 (0xf145 bytes) 000790 1787828405.965592 punch from 0x0190cd to 0x0251ed (0xc121 bytes) 000791 1787828405.972213 mapread 0x00b0ab thru 0x00de6f (0x2dc5 bytes) 000792 1787828405.975922 mapread 0x03737c thru 0x037cc9 (0x94e bytes) 000793 1787828405.997419 mapwrite 0x012447 thru 0x018e33 (0x69ed bytes) 000794 1787828406.037804 trunc from 0x03ce79 to 0x03bc10 (0x1268 bytes) 000796 1787828406.071587 punch from 0x007076 to 0x00e7c1 (0x774c bytes) 000797 1787828406.076316 read 0x004c07 thru 0x00a8ac (0x5ca6 bytes) 000798 1787828406.085637 read 0x00e55e thru 0x0113fd (0x2ea0 bytes) 000799 1787828406.111278 mapwrite 0x01ffe4 thru 0x026100 (0x611d bytes) 000800 1787828406.129509 trunc from 0x03bc11 to 0x0088ce (0x33342 bytes) 000801 1787828406.144352 trunc from 0x00679a to 0x0088ce (0x2135 bytes) 000802 1787828406.150669 read 0x004b0e thru 0x00621b (0x170e bytes) 000803 1787828406.161548 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x88cf): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x88cf): Mirror 8: 0x510324f0 (stale) Mirror 13: 0x510324f0 (stale) Mirror 20: 0x510324f0 (stale) Mirror 21: 0x510324f0 (stale) Mirror 22: 0x510324f0 (stale) Mirror 25: 0x510324f0 (stale) Mirror 26: 0x510324f0 (stale) Mirror 27: 0x510324f0 (stale) CRC-32 checksum value for chunk [0, 0x88cf): Mirror 1: 0xc436391a Mirror 28: 0xc436391a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x88cf: skipped 000804 1787828407.196907 trunc from 0x0088cf to 0x0288a2 (0x1ffd4 bytes) 000805 1787828407.292684 mapwrite 0x01ec3f thru 0x0228e0 (0x3ca2 bytes) 000806 1787828407.338491 fallocate 0x00c162 thru 0x01a1f3 (0xe092 bytes) 000807 1787828407.346532 trunc from 0x0288a3 to 0x02a68a (0x1de8 bytes) 000808 1787828407.380793 fallocate 0x01e7e5 thru 0x02c086 (0xd8a2 bytes) 000809 1787828407.388067 read 0x016f93 thru 0x026efa (0xff68 bytes) 000810 1787828407.396904 trunc from 0x001a27 to 0x0074c8 (0x5aa2 bytes) 000811 1787828407.403544 trunc from 0x02a68b to 0x028eba (0x17d0 bytes) 000812 1787828407.427520 trunc from 0x028ebb to 0x006b8b (0x2232f bytes) 000813 1787828407.451693 fallocate 0x031225 thru 0x03bc97 (0xaa73 bytes) 000814 1787828407.462602 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 8: 0xbc1f2617 (stale) Mirror 13: 0xbc1f2617 (stale) Mirror 20: 0xbc1f2617 (stale) Mirror 21: 0xbc1f2617 (stale) Mirror 22: 0xbc1f2617 (stale) Mirror 25: 0xbc1f2617 (stale) Mirror 26: 0xbc1f2617 (stale) Mirror 27: 0xbc1f2617 (stale) Mirror 28: 0x309dc92b (stale) CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 1: 0x80515baa Mirror 29: 0x80515baa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b8c: skipped 000816 1787828408.403901 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 8: 0xbc1f2617 (stale) Mirror 13: 0xbc1f2617 (stale) Mirror 20: 0xbc1f2617 (stale) Mirror 21: 0xbc1f2617 (stale) Mirror 22: 0xbc1f2617 (stale) Mirror 25: 0xbc1f2617 (stale) Mirror 27: 0xbc1f2617 (stale) Mirror 28: 0x309dc92b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 8: 0xbc1f2617 (stale) Mirror 13: 0xbc1f2617 (stale) Mirror 20: 0xbc1f2617 (stale) Mirror 21: 0xbc1f2617 (stale) Mirror 22: 0xbc1f2617 (stale) Mirror 25: 0xbc1f2617 (stale) Mirror 27: 0xbc1f2617 (stale) Mirror 28: 0x309dc92b (stale) CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 1: 0x80515baa Mirror 29: 0x80515baa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b8c: skipped 000817 1787828409.035110 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 13: 0xbc1f2617 (stale) Mirror 20: 0xbc1f2617 (stale) Mirror 21: 0xbc1f2617 (stale) Mirror 22: 0xbc1f2617 (stale) Mirror 25: 0xbc1f2617 (stale) Mirror 27: 0xbc1f2617 (stale) Mirror 28: 0x309dc92b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 13: 0xbc1f2617 (stale) Mirror 20: 0xbc1f2617 (stale) Mirror 21: 0xbc1f2617 (stale) Mirror 22: 0xbc1f2617 (stale) Mirror 25: 0xbc1f2617 (stale) Mirror 27: 0xbc1f2617 (stale) Mirror 28: 0x309dc92b (stale) CRC-32 checksum value for chunk [0, 0x6b8c): Mirror 1: 0x80515baa Mirror 29: 0x80515baa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b8c: skipped 000819 1787828409.805812 trunc from 0x006b8c to 0x0134bd (0xc932 bytes) 000820 1787828409.897621 trunc from 0x0134be to 0x019740 (0x6283 bytes) 000821 1787828409.920376 read 0x00a234 thru 0x00b347 (0x1114 bytes) 000822 1787828409.927567 trunc from 0x019741 to 0x02b572 (0x11e32 bytes) 000823 1787828409.942277 trunc from 0x02b573 to 0x019b6c (0x11a06 bytes) 000824 1787828409.966384 read 0x00811d thru 0x01574f (0xd633 bytes) 000825 1787828409.993631 read 0x014ad4 thru 0x019b6c (0x5099 bytes) 000826 1787828409.995384 mapread 0x01344c thru 0x013bec (0x7a1 bytes) 000827 1787828409.998177 mapwrite 0x019d6a thru 0x01d73d (0x39d4 bytes) 000829 1787828410.040467 mapread 0x00e0c0 thru 0x012e6f (0x4db0 bytes) 000830 1787828410.042180 read 0x00e724 thru 0x019a92 (0xb36f bytes) 000831 1787828410.043227 fallocate 0x0294c6 thru 0x036947 (0xd482 bytes) 000832 1787828410.050913 read 0x00ccb5 thru 0x00fa86 (0x2dd2 bytes) 000833 1787828410.051847 trunc from 0x036948 to 0x027970 (0xefd7 bytes) 000834 1787828410.068721 write 0x0207bc thru 0x029c07 (0x944c bytes) 000835 1787828410.078064 trunc from 0x029c08 to 0x00adc5 (0x1ee42 bytes) 000836 1787828410.094784 write 0x028563 thru 0x031efd (0x999b bytes) 000837 1787828410.099833 write 0x03e818 thru 0x03ffff (0x17e8 bytes) 000838 1787828410.102148 trunc from 0x03c59c to 0x03ffff (0x3a64 bytes) 000839 1787828410.124021 write 0x01a3d2 thru 0x01d09d (0x2ccc bytes) 000840 1787828410.142121 trunc from 0x040000 to 0x02b2ca (0x14d35 bytes) 000841 1787828410.159065 read 0x01bcfa thru 0x01e0a5 (0x23ac bytes) 000842 1787828410.175017 trunc from 0x02b2cb to 0x03b721 (0x10457 bytes) 000843 1787828410.202750 read 0x002420 thru 0x00888f (0x6470 bytes) 000844 1787828410.203447 trunc from 0x00ae3e to 0x00d3f9 (0x25bc bytes) 000845 1787828410.211013 fallocate 0x03382f thru 0x03ffff (0xc7d1 bytes) 000846 1787828410.236242 MIRROR_RESYNC 9 mirrors 000847 1787828412.537654 mapread 0x0042b5 thru 0x00ca0d (0x8759 bytes) 000848 1787828412.570192 read 0x00dab3 thru 0x0186db (0xac29 bytes) 000849 1787828412.584109 trunc from 0x002de6 to 0x009dc4 (0x6fdf bytes) 000850 1787828412.622198 punch from 0x0169bd to 0x01f4ea (0x8b2e bytes) 000851 1787828412.626764 mapread 0x0151bb thru 0x01e850 (0x9696 bytes) 000852 1787828412.645385 mapwrite 0x02bdf4 thru 0x0357f3 (0x9a00 bytes) 000853 1787828412.692490 mapwrite 0x00d40b thru 0x013347 (0x5f3d bytes) 000854 1787828412.725648 mapread 0x02f748 thru 0x037c0c (0x84c5 bytes) 000855 1787828412.729644 fallocate 0x033e77 thru 0x03f49e (0xb628 bytes) 000856 1787828412.735693 mapwrite 0x006da5 thru 0x00a051 (0x32ad bytes) 000857 1787828412.761925 fallocate 0x031bb3 thru 0x03fd4d (0xe19b bytes) 000858 1787828412.773115 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf7aee21d (stale) Mirror 20: 0xf7aee21d (stale) Mirror 21: 0xf7aee21d (stale) Mirror 22: 0xf7aee21d (stale) Mirror 25: 0xf7aee21d (stale) Mirror 27: 0xf7aee21d (stale) Mirror 28: 0xf7aee21d (stale) Mirror 29: 0xf7aee21d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa595c2bf Mirror 30: 0xa595c2bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000859 1787828413.987447 trunc from 0x003f59 to 0x00ece9 (0xad91 bytes) 000860 1787828414.050005 fallocate 0x00500b thru 0x008a98 (0x3a8e bytes) 000861 1787828414.056934 trunc from 0x010d9d to 0x019e55 (0x90b9 bytes) 000862 1787828414.062819 mapwrite 0x027653 thru 0x032979 (0xb327 bytes) 000863 1787828414.122525 trunc from 0x040000 to 0x022277 (0x1dd88 bytes) 000864 1787828414.154536 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x22278): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22278): Mirror 13: 0xb5941633 (stale) Mirror 20: 0xb5941633 (stale) Mirror 21: 0xb5941633 (stale) Mirror 22: 0xb5941633 (stale) Mirror 25: 0xb5941633 (stale) Mirror 27: 0xb5941633 (stale) Mirror 28: 0xb5941633 (stale) Mirror 29: 0xb5941633 (stale) Mirror 30: 0xfad19f00 (stale) CRC-32 checksum value for chunk [0, 0x22278): Mirror 1: 0xffdf3401 Mirror 31: 0xffdf3401 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22278: skipped 000865 1787828415.243788 trunc from 0x010cec to 0x015663 (0x4978 bytes) 000867 1787828415.302945 trunc from 0x022278 to 0x016862 (0xba15 bytes) 000868 1787828415.339423 punch from 0x013778 to 0x016862 (0x30eb bytes) 000869 1787828415.344107 fallocate 0x03e69f thru 0x03ffff (0x1961 bytes) 000871 1787828415.350788 trunc from 0x040000 to 0x00381c (0x3c7e3 bytes) 000873 1787828415.366809 mapwrite 0x03bb34 thru 0x03ce78 (0x1345 bytes) 000874 1787828415.403005 read 0x019bec thru 0x021d80 (0x8195 bytes) 000875 1787828415.418714 read 0x00fb6d thru 0x0108e3 (0xd77 bytes) 000876 1787828415.450847 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce79): Mirror 13: 0xbb7482c1 (stale) Mirror 20: 0xbb7482c1 (stale) Mirror 21: 0xbb7482c1 (stale) Mirror 22: 0xbb7482c1 (stale) Mirror 25: 0xbb7482c1 (stale) Mirror 27: 0xbb7482c1 (stale) Mirror 28: 0xbb7482c1 (stale) Mirror 29: 0xbb7482c1 (stale) Mirror 31: 0xdaaab3b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce79): Mirror 13: 0xbb7482c1 (stale) Mirror 20: 0xbb7482c1 (stale) Mirror 21: 0xbb7482c1 (stale) Mirror 22: 0xbb7482c1 (stale) Mirror 25: 0xbb7482c1 (stale) Mirror 27: 0xbb7482c1 (stale) Mirror 28: 0xbb7482c1 (stale) Mirror 29: 0xbb7482c1 (stale) Mirror 31: 0xdaaab3b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce79: skipped 000878 1787828416.337149 punch from 0x00e460 to 0x01d7c0 (0xf361 bytes) 000879 1787828416.342139 mapread 0x02efaa thru 0x03be15 (0xce6c bytes) 000880 1787828416.367125 mapwrite 0x011794 thru 0x0148f7 (0x3164 bytes) 000881 1787828416.393468 punch from 0x028a3d to 0x036e72 (0xe436 bytes) 000882 1787828416.398868 read 0x014a29 thru 0x01f27d (0xa855 bytes) 000883 1787828416.409824 mapwrite 0x033306 thru 0x03ffff (0xccfa bytes) 000884 1787828416.461083 mapread 0x02e311 thru 0x035f0e (0x7bfe bytes) 000885 1787828416.462276 punch from 0x0234a0 to 0x025a8e (0x25ef bytes) 000886 1787828416.470590 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf7aee21d (stale) Mirror 20: 0xf7aee21d (stale) Mirror 21: 0xf7aee21d (stale) Mirror 22: 0xf7aee21d (stale) Mirror 27: 0xf7aee21d (stale) Mirror 28: 0xf7aee21d (stale) Mirror 29: 0xf7aee21d (stale) Mirror 31: 0x156b4128 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf7aee21d (stale) Mirror 20: 0xf7aee21d (stale) Mirror 21: 0xf7aee21d (stale) Mirror 22: 0xf7aee21d (stale) Mirror 27: 0xf7aee21d (stale) Mirror 28: 0xf7aee21d (stale) Mirror 29: 0xf7aee21d (stale) Mirror 31: 0x156b4128 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1787828417.050257 mapread 0x01ecc8 thru 0x024859 (0x5b92 bytes) 000888 1787828417.086835 write 0x00abf2 thru 0x011b37 (0x6f46 bytes) 000889 1787828417.102038 trunc from 0x040000 to 0x0367e0 (0x981f bytes) 000890 1787828417.130082 trunc from 0x0367e1 to 0x00aa19 (0x2bdc7 bytes) 000891 1787828417.155310 mapwrite 0x00fd02 thru 0x016960 (0x6c5f bytes) 000892 1787828417.241560 trunc from 0x01219a to 0x016960 (0x47c7 bytes) 000893 1787828417.250538 trunc from 0x011aeb to 0x013499 (0x19af bytes) 000894 1787828417.261836 punch from 0x003097 to 0x01305a (0xffc4 bytes) 000895 1787828417.267298 fallocate 0x02768e thru 0x02c8b0 (0x5223 bytes) 000896 1787828417.285548 MIRROR_RESYNC 9 mirrors 000897 1787828419.152061 MIRROR_EXTEND to 10 mirrors 000898 1787828420.006118 trunc from 0x02c8b1 to 0x017d7d (0x14b33 bytes) 000899 1787828420.084283 fallocate 0x03c561 thru 0x03ffff (0x3a9f bytes) 000900 1787828420.088668 trunc from 0x014fc5 to 0x017d7d (0x2db9 bytes) 000901 1787828420.093423 read 0x006f53 thru 0x00a4c4 (0x3572 bytes) 000902 1787828420.109348 write 0x0148d8 thru 0x022141 (0xd86a bytes) 000903 1787828420.123884 punch from 0x015cbd to 0x01b61d (0x5961 bytes) 000904 1787828420.158634 trunc from 0x0163ef to 0x0205ae (0xa1c0 bytes) 000905 1787828420.164226 fallocate 0x035750 thru 0x03ffff (0xa8b0 bytes) 000906 1787828420.172218 trunc from 0x022142 to 0x02a2d2 (0x8191 bytes) 000907 1787828420.199913 trunc from 0x013dc0 to 0x01744f (0x3690 bytes) 000908 1787828420.212708 read 0x00bdf9 thru 0x017a76 (0xbc7e bytes) 000909 1787828420.226891 trunc from 0x011b61 to 0x01e43a (0xc8da bytes) 000910 1787828420.232508 mapread 0x019058 thru 0x021fab (0x8f54 bytes) 000911 1787828420.260509 fallocate 0x03de65 thru 0x03ffff (0x219b bytes) 000912 1787828420.265548 fallocate 0x0241d7 thru 0x03411f (0xff49 bytes) 000913 1787828420.271582 read 0x031cde thru 0x03860a (0x692d bytes) 000914 1787828420.281625 mapwrite 0x003912 thru 0x011b54 (0xe243 bytes) 000915 1787828420.320130 write 0x03e87e thru 0x03ffff (0x1782 bytes) 000916 1787828420.330689 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x1a0784fe (stale) Mirror 20: 0x1a0784fe (stale) Mirror 22: 0x1a0784fe (stale) Mirror 27: 0x1a0784fe (stale) Mirror 28: 0x1a0784fe (stale) Mirror 29: 0x1a0784fe (stale) Mirror 31: 0x1a0784fe (stale) Mirror 32: 0x1a0784fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x1a0784fe (stale) Mirror 20: 0x1a0784fe (stale) Mirror 22: 0x1a0784fe (stale) Mirror 27: 0x1a0784fe (stale) Mirror 28: 0x1a0784fe (stale) Mirror 29: 0x1a0784fe (stale) Mirror 31: 0x1a0784fe (stale) Mirror 32: 0x1a0784fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1787828420.926104 fallocate 0x00a20c thru 0x017c23 (0xda18 bytes) 000918 1787828420.931039 punch from 0x037b8c to 0x039fd5 (0x244a bytes) 000919 1787828420.938388 punch from 0x03fcff to 0x03ffff (0x301 bytes) 000920 1787828420.944948 read 0x003b47 thru 0x00af6f (0x7429 bytes) 000921 1787828420.957860 trunc from 0x040000 to 0x02c6a9 (0x13956 bytes) 000922 1787828420.978020 write 0x009d7c thru 0x00c275 (0x24fa bytes) 000923 1787828420.983935 trunc from 0x02c6aa to 0x00188d (0x2ae1c bytes) 000924 1787828420.996757 mapwrite 0x03836f thru 0x03a2fa (0x1f8c bytes) 000925 1787828421.054231 MIRROR_RESYNC 9 mirrors 000926 1787828422.607608 MIRROR_EXTEND to 10 mirrors 000928 1787828423.373344 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) 000929 1787828423.806985 read 0x017aab thru 0x0266c2 (0xec18 bytes) 000930 1787828423.823109 write 0x013c8a thru 0x0183dc (0x4753 bytes) 000931 1787828423.916038 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a2fb): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2fb): Mirror 20: 0xdfd921da (stale) Mirror 22: 0xdfd921da (stale) Mirror 27: 0xdfd921da (stale) Mirror 28: 0xdfd921da (stale) Mirror 29: 0xdfd921da (stale) Mirror 31: 0xdfd921da (stale) Mirror 32: 0xdfd921da (stale) Mirror 33: 0xdfd921da (stale) CRC-32 checksum value for chunk [0, 0x3a2fb): Mirror 1: 0xbb14088 Mirror 34: 0xbb14088 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2fb: skipped 000932 1787828424.966375 mapread 0x02dba9 thru 0x030c05 (0x305d bytes) 000933 1787828424.999999 fallocate 0x02de64 thru 0x033b6f (0x5d0c bytes) 000934 1787828425.085098 read 0x02dae5 thru 0x039ff0 (0xc50c bytes) 000935 1787828425.099326 mapwrite 0x02b07f thru 0x032406 (0x7388 bytes) 000936 1787828425.159274 read 0x012eb9 thru 0x019eb1 (0x6ff9 bytes) 000937 1787828425.177636 write 0x02858f thru 0x028abe (0x530 bytes) 000939 1787828425.182845 mapread 0x001cdf thru 0x003bf1 (0x1f13 bytes) 000940 1787828425.184437 read 0x006420 thru 0x013a21 (0xd602 bytes) 000942 1787828425.186501 write 0x015481 thru 0x0164ab (0x102b bytes) 000943 1787828425.191694 MIRROR_RESYNC 10 mirrors 000944 1787828428.242349 read 0x036d5e thru 0x038e51 (0x20f4 bytes) 000945 1787828428.249485 trunc from 0x03a2fb to 0x03ff60 (0x5c66 bytes) 000946 1787828428.321397 fallocate 0x02f052 thru 0x03913f (0xa0ee bytes) 000947 1787828428.329411 fallocate 0x011748 thru 0x019639 (0x7ef2 bytes) 000948 1787828428.342016 mapwrite 0x01688d thru 0x017ac4 (0x1238 bytes) 000949 1787828428.368597 trunc from 0x03ff61 to 0x007cf1 (0x3826f bytes) 000950 1787828428.389053 mapread 0x003447 thru 0x007cf1 (0x48ab bytes) 000951 1787828428.399101 read 0x00334a thru 0x007471 (0x4128 bytes) 000952 1787828428.399846 write 0x02ec17 thru 0x033c1a (0x5004 bytes) 000953 1787828428.401852 read 0x02fb22 thru 0x033c1a (0x40f9 bytes) 000954 1787828428.402462 mapread 0x017667 thru 0x01cbfb (0x5595 bytes) 000956 1787828428.414264 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c1b): Mirror 20: 0x73fa00ec (stale) Mirror 22: 0x73fa00ec (stale) Mirror 27: 0x73fa00ec (stale) Mirror 28: 0x73fa00ec (stale) Mirror 29: 0x73fa00ec (stale) Mirror 32: 0x73fa00ec (stale) Mirror 33: 0x73fa00ec (stale) Mirror 34: 0x73fa00ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c1b): Mirror 20: 0x73fa00ec (stale) Mirror 22: 0x73fa00ec (stale) Mirror 27: 0x73fa00ec (stale) Mirror 28: 0x73fa00ec (stale) Mirror 29: 0x73fa00ec (stale) Mirror 32: 0x73fa00ec (stale) Mirror 33: 0x73fa00ec (stale) Mirror 34: 0x73fa00ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c1b: skipped 000957 1787828429.024756 trunc from 0x02515e to 0x0278f2 (0x2795 bytes) 000958 1787828429.031571 read 0x010cbb thru 0x01310b (0x2451 bytes) 000959 1787828429.040091 fallocate 0x030595 thru 0x0378c2 (0x732e bytes) 000960 1787828429.049868 read 0x0192c5 thru 0x020885 (0x75c1 bytes) 000961 1787828429.063536 write 0x02d7e6 thru 0x02deaf (0x6ca bytes) 000962 1787828429.069559 punch from 0x0025e2 to 0x008f79 (0x6998 bytes) 000964 1787828429.083559 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33c1b): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c1b): Mirror 20: 0x73fa00ec (stale) Mirror 22: 0x73fa00ec (stale) Mirror 27: 0x73fa00ec (stale) Mirror 28: 0x73fa00ec (stale) Mirror 29: 0x73fa00ec (stale) Mirror 32: 0x73fa00ec (stale) Mirror 33: 0x73fa00ec (stale) Mirror 34: 0x73fa00ec (stale) CRC-32 checksum value for chunk [0, 0x33c1b): Mirror 1: 0x1adcd084 Mirror 35: 0x1adcd084 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c1b: skipped 000965 1787828430.213992 trunc from 0x032686 to 0x0333bc (0xd37 bytes) 000966 1787828430.271403 trunc from 0x033c1b to 0x033e16 (0x1fc bytes) 000967 1787828430.336526 mapread 0x02bbcf thru 0x033e16 (0x8248 bytes) 000968 1787828430.353739 trunc from 0x00e5e5 to 0x010997 (0x23b3 bytes) 000969 1787828430.358577 write 0x00616a thru 0x015db7 (0xfc4e bytes) 000970 1787828430.371533 read 0x01aade thru 0x02586a (0xad8d bytes) 000971 1787828430.383315 mapread 0x02d3f2 thru 0x02e378 (0xf87 bytes) 000972 1787828430.384876 fallocate 0x01a84b thru 0x01cdc0 (0x2576 bytes) 000973 1787828430.392452 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33e17): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e17): Mirror 20: 0xd939f4fb (stale) Mirror 22: 0xd939f4fb (stale) Mirror 27: 0xd939f4fb (stale) Mirror 28: 0xd939f4fb (stale) Mirror 29: 0xd939f4fb (stale) Mirror 32: 0xd939f4fb (stale) Mirror 33: 0xd939f4fb (stale) Mirror 34: 0xd939f4fb (stale) Mirror 35: 0x647f6acd (stale) CRC-32 checksum value for chunk [0, 0x33e17): Mirror 1: 0x7df20742 Mirror 36: 0x7df20742 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e17: skipped 000974 1787828431.291619 mapread 0x0276b2 thru 0x02bb18 (0x4467 bytes) 000975 1787828431.311955 trunc from 0x00efb9 to 0x01caee (0xdb36 bytes) 000976 1787828431.349430 mapwrite 0x01407c thru 0x0234e1 (0xf466 bytes) 000977 1787828431.402434 mapwrite 0x02ed12 thru 0x03645c (0x774b bytes) 000978 1787828431.534031 trunc from 0x0318c7 to 0x03645c (0x4b96 bytes) 000979 1787828431.543907 trunc from 0x02ab50 to 0x03645c (0xb90d bytes) 000980 1787828431.551911 fallocate 0x0080c7 thru 0x00b07d (0x2fb7 bytes) 000981 1787828431.559912 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3645d): Mirror 22: 0xa3f8ad84 (stale) Mirror 27: 0xa3f8ad84 (stale) Mirror 28: 0xa3f8ad84 (stale) Mirror 29: 0xa3f8ad84 (stale) Mirror 32: 0xa3f8ad84 (stale) Mirror 33: 0xa3f8ad84 (stale) Mirror 34: 0xa3f8ad84 (stale) Mirror 35: 0x8448633e (stale) Mirror 36: 0x78d17228 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3645d): Mirror 22: 0xa3f8ad84 (stale) Mirror 27: 0xa3f8ad84 (stale) Mirror 28: 0xa3f8ad84 (stale) Mirror 29: 0xa3f8ad84 (stale) Mirror 32: 0xa3f8ad84 (stale) Mirror 33: 0xa3f8ad84 (stale) Mirror 34: 0xa3f8ad84 (stale) Mirror 35: 0x8448633e (stale) Mirror 36: 0x78d17228 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3645d: skipped 000982 1787828432.178818 trunc from 0x00f773 to 0x01e5fa (0xee88 bytes) 000984 1787828432.188496 trunc from 0x03645d to 0x020b5a (0x15902 bytes) 000985 1787828432.214177 trunc from 0x00f065 to 0x01bc16 (0xcbb2 bytes) 000986 1787828432.228438 mapread 0x01c76a thru 0x020b5a (0x43f1 bytes) 000987 1787828432.254867 trunc from 0x020b5b to 0x02dded (0xd293 bytes) 000988 1787828432.286154 read 0x01795e thru 0x01b8af (0x3f52 bytes) 000989 1787828432.286896 write 0x01f1b8 thru 0x02d9b9 (0xe802 bytes) 000990 1787828432.296669 read 0x018e01 thru 0x020e3f (0x803f bytes) 000991 1787828432.299954 write 0x03738d thru 0x03d019 (0x5c8d bytes) 000992 1787828432.304842 trunc from 0x01bf9f to 0x020c9f (0x4d01 bytes) 000993 1787828432.355427 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d01a): Mirror 22: 0x2a79caf7 (stale) Mirror 27: 0x2a79caf7 (stale) Mirror 28: 0x2a79caf7 (stale) Mirror 32: 0x2a79caf7 (stale) Mirror 33: 0x2a79caf7 (stale) Mirror 34: 0x2a79caf7 (stale) Mirror 35: 0xfbf1ffa5 (stale) Mirror 36: 0x849f6fbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d01a): Mirror 22: 0x2a79caf7 (stale) Mirror 27: 0x2a79caf7 (stale) Mirror 28: 0x2a79caf7 (stale) Mirror 32: 0x2a79caf7 (stale) Mirror 33: 0x2a79caf7 (stale) Mirror 34: 0x2a79caf7 (stale) Mirror 35: 0xfbf1ffa5 (stale) Mirror 36: 0x849f6fbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d01a: skipped 000994 1787828432.794667 read 0x007b94 thru 0x011fce (0xa43b bytes) 000996 1787828432.805139 punch from 0x024c13 to 0x028f8e (0x437c bytes) 000997 1787828432.810383 trunc from 0x02f953 to 0x03c874 (0xcf22 bytes) 000998 1787828432.815534 fallocate 0x003b66 thru 0x00e3bc (0xa857 bytes) 000999 1787828432.823667 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3d01a): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d01a): Mirror 22: 0x2a79caf7 (stale) Mirror 27: 0x2a79caf7 (stale) Mirror 28: 0x2a79caf7 (stale) Mirror 32: 0x2a79caf7 (stale) Mirror 33: 0x2a79caf7 (stale) Mirror 34: 0x2a79caf7 (stale) Mirror 35: 0xfbf1ffa5 (stale) Mirror 36: 0x849f6fbb (stale) CRC-32 checksum value for chunk [0, 0x3d01a): Mirror 1: 0xf0881689 Mirror 37: 0xf0881689 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d01a: skipped 001000 1787828434.097127 read 0x018935 thru 0x019397 (0xa63 bytes) All operations completed A-OK! PASS 70a (168s) == sanity-flr test 71: check mirror extend parent fid ==== 07:00:38 (1787828438) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1f1:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 07:00:47 (1787828447) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283881 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114026 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 07:00:57 (1787828457) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6888896 .. Extending file size to 3769664 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7298624 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8255936 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8882464 .. Extending file size to 9220992 .. Extending file size to 7031264 .. Extending file size to 7206496 .. Extending file size to 7384192 .. Extending file size to 8017184 .. Extending file size to 8775648 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62818 62819 62820 62822 62823 done 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) 192.168.206.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:02:16 (1787828536) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1702944 .. Extending file size to 5607232 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5646208 .. Extending file size to 5739360 .. Extending file size to 5945184 .. Extending file size to 6284384 .. Extending file size to 7102688 .. Extending file size to 7430432 .. Extending file size to 8064576 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8889280 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7641280 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8924608 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8981824 .. Extending file size to 9156224 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64555 64556 64559 64560 64561 done 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) 192.168.206.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:03:36 (1787828616) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2400 1268904 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24672 3582348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18792 3588228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43464 7170576 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2400 1268904 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24672 3582348 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18792 3588228 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43464 7170576 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110001 s, 19.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.200655 s, 10.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x39ca859d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:03:55 (1787828635) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0940253 s, 22.3 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 07:04:01 (1787828641) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:04:08 (1787828648) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0602243 s, 34.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x265:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:04:15 (1787828655) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11119 s, 18.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:04:24 (1787828664) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120778 s, 17.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:04:57 (1787828697) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg617-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113276 s, 9.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x266:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x269:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x268:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x266:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x269:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x268:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x267:0x0] } PASS 204d (50s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:05:47 (1787828747) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg617-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0736225 s, 28.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:06:32 (1787828792) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg617-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg617-server: Pool lustre.test_204f created Adding targets to pool oleg617-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg617-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102459 s, 20.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg617-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg617-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg617-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:07:43 (1787828863) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:07:50 (1787828870) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:07:58 (1787828878) oleg617-server: Pool lustre.test_206 created Adding targets to pool oleg617-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg617-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg617-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg617-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg617-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:08:27 (1787828907) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742254 s, 14.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541126 s, 19.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb8e03754 Mirror 3: 0xb8e03754 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:08:37 (1787828917) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:08:41 (1787828921) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:08:44 (1787828924) Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0787488 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1360000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1360000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1360000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x15c0000 Read this (4 bytes): BBBB 192.168.206.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:09:22 (1787828962) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0266926 s, 307 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:09:29 (1787828969) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370988 s, 28.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 07:09:31.000000000 -0400 Modify: 2026-08-27 07:09:31.000000000 -0400 Change: 2026-08-27 07:09:32.000000000 -0400 Birth: 2026-08-27 07:09:31.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:09:37 (1787828977) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349237 s, 30.0 MB/s oleg617-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg617-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg617-server: O/280000401/d1/141: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:09:58 (1787828998) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.236646 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0208034 s, 197 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 07:10 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 07:10 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:10:06 (1787829006) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:10:14 (1787829014) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1892 sec =========== 07:10:24 (1787829024) === sanity-flr: start cleanup 07:10:26 (1787829026) === === sanity-flr: finish cleanup 07:10:34 (1787829034) ===