-----============= acceptance-small: sanity-flr ============----- Mon Mar 9 20:19:58 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg260-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:20:08 (1773102008) === oleg260-client.virtnet: executing check_config_client /mnt/lustre oleg260-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg260-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88d4455d2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88d4455d2800.idle_timeout=debug disable quota as required oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:20:27 (1773102027) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8164] == sanity-flr test 0a: lfs mirror create with -N option == 20:20:28 (1773102028) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:20:37 (1773102037) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:20:39 (1773102039) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:20:41 (1773102041) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:20:52 (1773102052) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:20:54 (1773102054) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 20:20:56 (1773102056) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88d4455d2800.stats=clear osc.lustre-OST0001-osc-ffff88d4455d2800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:21:00 (1773102060) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:21:05 (1773102065) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:21:09 (1773102069) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:21:11 (1773102071) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 20:21:16 (1773102076) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:21:20 (1773102080) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:21:23 (1773102083) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:21:27 (1773102087) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:21:32 (1773102092) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:21:35 (1773102095) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.495929 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434465 s, 24.1 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:21:39 (1773102099) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.197657 s, 79.6 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:21:43 (1773102103) Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg260-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (39s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:22:22 (1773102142) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.270953 s, 34.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (23s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:22:45 (1773102165) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg260-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (46s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:23:32 (1773102212) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.7658 s, 29.7 MB/s -rw-r--r-- 1 root root 52428800 Mar 9 20:23 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.564s user 0m0.002s sys 0m0.159s Stopping /mnt/lustre-ost1 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88d4455d2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.627s user 0m0.000s sys 0m0.177s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg260-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg260-server oleg260-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.792s user 0m0.002s sys 0m0.161s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg260-server: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg260-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88d4455d2800.ost_server_uuid in FULL state after 0 sec PASS 33b (52s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:24:24 (1773102264) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:24:26 (1773102266) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:24:28 (1773102268) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 20:24:31 (1773102271) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 20:24:36 (1773102276) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.315941 s, 39.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:24:43 (1773102283) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191883 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026654 s, 39.3 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:24:47 (1773102287) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:24:52 (1773102292) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000692018 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000703197 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000901729 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000859288 s, 1.2 MB/s flv=10 olv=10 PASS 36d (6s) == sanity-flr test 37: mirror I/O API verification ======= 20:24:58 (1773102298) create test files with size 15756731 .. 5e1e2eafab3d4846bd6964fb7226989a - 47ba096255bf5f92af6e0c0d84e826db - 882500765b94d1d06b00c208d4e9b277 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre PASS 37 (12s) == sanity-flr test 38: resync ============================ 20:25:10 (1773102310) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:25:18 (1773102318) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:25:22 (1773102322) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411234 s, 25.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268785 s, 39.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x40:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 20:25:26 (1773102326) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0481668 s, 43.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.144805 s, 29.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106484 s, 19.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 20:25:34 (1773102334) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:25:36 (1773102336) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:25:39 (1773102339) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202183 s, 51.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 20:25:44 (1773102344) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.258417 s, 12.2 MB/s PASS 44a (6s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:25:50 (1773102350) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:25:56 (1773102356) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.147238 s, 71.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (6s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:26:02 (1773102362) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88d4454d3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.183113 s, 57.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg260-client.virtnet /mnt/lustre (opts:) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88d447fe5000.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:26:11 (1773102371) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (21s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:26:32 (1773102392) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (21s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:26:53 (1773102413) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 20:26:55 (1773102415) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241384 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0552314 s, 57.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9c051606 (stale) Mirror 2: 0x9c051606 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: d8ad9f3d136b84642feb671ce61899a5 - mirror 1 checksum: 43de4e6b746c9e96f342fca14e612954 - mirror 2 checksum: d8ad9f3d136b84642feb671ce61899a5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 9 20:26:58 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 9 20:26:58 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d8ad9f3d136b84642feb671ce61899a5 - mirror 1 checksum: 43de4e6b746c9e96f342fca14e612954 - mirror 2 checksum: 43de4e6b746c9e96f342fca14e612954 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 9 20:26:58 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 9 20:26:58 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 48 (6s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:27:01 (1773102421) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202757 s, 51.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 399360.. 400383: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (5s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:27:06 (1773102426) Failing mds1 on oleg260-server Stopping /mnt/lustre-mds1 (opts:) on oleg260-server 20:27:16 (1773102436) shut down facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server Failover mds1 to oleg260-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 20:27:34 (1773102454) targets are mounted 20:27:34 (1773102454) facet_failover done oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142395 s, 73.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:27:43 (1773102463) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0140522 s, 1.5 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00836701 s, 2.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (6s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:27:49 (1773102469) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0356119 s, 575 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00439874 s, 4.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:27:53 (1773102473) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00499089 s, 3.3 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:27:57 (1773102477) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0215321 s, 951 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:28:00 (1773102480) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00376147 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0188816 s, 217 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:28:04 (1773102484) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:28:15 (1773102495) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:28:37 (1773102517) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:28:56 (1773102536) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:29:16 (1773102556) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 20:29:20 (1773102560) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55841 mirror split pid 55842 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55841 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55842 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 20:30:23 (1773102623) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 2624 skipping zero size read skipping zero length punch hole skipping zero size read 000007 1773102624.624856 mapwrite 0x0264f2 thru 0x035769 (0xf278 bytes) truncating to largest ever: 0x2f465 000008 1773102624.695598 trunc from 0x03576a to 0x02f464 (0x6305 bytes) 000009 1773102624.715034 read 0x01f753 thru 0x023651 (0x3eff bytes) 000010 1773102624.722256 mapwrite 0x01809e thru 0x020147 (0x80aa bytes) 000011 1773102624.755370 fallocate 0x011468 thru 0x0212d3 (0xfe6c bytes) 000012 1773102624.760586 mapread 0x004a14 thru 0x010469 (0xba56 bytes) 000013 1773102624.769675 mapwrite 0x003c11 thru 0x007e07 (0x41f7 bytes) 000014 1773102624.794853 trunc from 0x02f465 to 0x005727 (0x29d3d bytes) 000015 1773102624.814904 punch from 0x003666 to 0x005727 (0x20c2 bytes) 000016 1773102624.819584 mapwrite 0x0243e2 thru 0x02a71c (0x633b bytes) 000017 1773102624.866932 trunc from 0x02a71d to 0x014d05 (0x15a17 bytes) 000018 1773102624.880570 trunc from 0x004130 to 0x004439 (0x30a bytes) fallocating to largest ever: 0x3a531 000021 1773102624.885244 fallocate 0x035911 thru 0x03a530 (0x4c20 bytes) 000022 1773102624.889087 mapread 0x022b90 thru 0x031260 (0xe6d1 bytes) 000023 1773102624.909819 trunc from 0x01703f to 0x018f4d (0x1f0f bytes) 000025 1773102624.915695 write 0x01506d thru 0x01c414 (0x73a8 bytes) 000026 1773102624.929555 fallocate 0x014311 thru 0x022904 (0xe5f4 bytes) 000027 1773102624.936689 write 0x0351eb thru 0x03e4ec (0x9302 bytes) 000028 1773102624.946761 fallocate 0x002592 thru 0x00693d (0x43ac bytes) 000029 1773102624.950492 punch from 0x0257bb to 0x02aa85 (0x52cb bytes) 000030 1773102624.956986 trunc from 0x03e4ed to 0x016f09 (0x275e3 bytes) 000031 1773102624.973532 fallocate 0x014b72 thru 0x01b83d (0x6ccc bytes) 000033 1773102624.977870 write 0x01d82f thru 0x02a7de (0xcfb0 bytes) 000034 1773102624.979985 trunc from 0x015883 to 0x022c97 (0xd415 bytes) 000035 1773102624.994142 punch from 0x028f38 to 0x029f8b (0x1054 bytes) 000036 1773102624.998437 punch from 0x0288d0 to 0x02a7de (0x1f0f bytes) 000037 1773102625.017109 MIRROR_EXTEND to 2 mirrors 000038 1773102625.496651 trunc from 0x02a7df to 0x025640 (0x519e bytes) 000039 1773102625.515537 trunc from 0x023450 to 0x02400c (0xbbd bytes) 000040 1773102625.517748 read 0x010783 thru 0x013dd0 (0x364e bytes) 000041 1773102625.520898 trunc from 0x0236b3 to 0x02535f (0x1cad bytes) 000042 1773102625.524894 trunc from 0x025641 to 0x01edff (0x6841 bytes) 000043 1773102625.532926 fallocate 0x02f3cc thru 0x02f5ba (0x1ef bytes) 000044 1773102625.535536 trunc from 0x0290e3 to 0x02f5ba (0x64d8 bytes) 000045 1773102625.538892 punch from 0x003c1d to 0x007a44 (0x3e28 bytes) 000046 1773102625.542220 trunc from 0x007d23 to 0x01198b (0x9c69 bytes) 000047 1773102625.546100 trunc from 0x029ce8 to 0x02f5ba (0x58d3 bytes) 000048 1773102625.548657 punch from 0x026547 to 0x027d2d (0x17e7 bytes) 000049 1773102625.552576 punch from 0x01bec5 to 0x01db82 (0x1cbe bytes) 000050 1773102625.556388 punch from 0x01bb3f to 0x024cc3 (0x9185 bytes) 000051 1773102625.558967 fallocate 0x01b45b thru 0x023ca2 (0x8848 bytes) 000052 1773102625.562070 trunc from 0x02f5bb to 0x000fdd (0x2e5dd bytes) 000053 1773102625.570553 write 0x035daa thru 0x03ffff (0xa256 bytes) 000054 1773102625.577044 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5dca5093 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b4d7321 Mirror 3: 0x9b4d7321 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000055 1773102625.930105 fallocate 0x015d17 thru 0x01ce43 (0x712d bytes) 000056 1773102625.945382 read 0x02954d thru 0x036900 (0xd3b4 bytes) 000058 1773102625.950277 trunc from 0x01e7f0 to 0x02ad4e (0xc55f bytes) 000059 1773102625.952975 write 0x01cfd8 thru 0x024755 (0x777e bytes) 000060 1773102625.959576 mapread 0x02f559 thru 0x03b1b6 (0xbc5e bytes) 000061 1773102625.964223 trunc from 0x031594 to 0x0317db (0x248 bytes) fallocating to largest ever: 0x40000 000062 1773102625.968748 fallocate 0x03fcf3 thru 0x03ffff (0x30d bytes) 000063 1773102625.975128 MIRROR_RESYNC 3 mirrors 000064 1773102626.359468 trunc from 0x021099 to 0x024c43 (0x3bab bytes) 000065 1773102626.378719 trunc from 0x040000 to 0x02d530 (0x12acf bytes) 000066 1773102626.392451 mapwrite 0x003e6f thru 0x010f9d (0xd12f bytes) 000067 1773102626.427198 trunc from 0x027e30 to 0x02be56 (0x4027 bytes) 000068 1773102626.430114 punch from 0x00fb70 to 0x01fad4 (0xff65 bytes) 000069 1773102626.434220 read 0x01b351 thru 0x028151 (0xce01 bytes) 000070 1773102626.448270 write 0x00854c thru 0x00fb15 (0x75ca bytes) 000071 1773102626.456356 MIRROR_RESYNC 3 mirrors 000072 1773102626.930158 trunc from 0x02d531 to 0x03da99 (0x10569 bytes) 000073 1773102626.982705 write 0x01022e thru 0x016274 (0x6047 bytes) 000074 1773102626.991110 read 0x02a147 thru 0x034389 (0xa243 bytes) 000075 1773102626.999002 punch from 0x0192a0 to 0x02174c (0x84ad bytes) 000076 1773102627.002578 fallocate 0x00532c thru 0x015066 (0xfd3b bytes) 000077 1773102627.006146 fallocate 0x00d892 thru 0x014608 (0x6d77 bytes) 000078 1773102627.010478 mapwrite 0x02fd65 thru 0x03e902 (0xeb9e bytes) 000080 1773102627.056275 MIRROR_RESYNC 3 mirrors 000081 1773102627.523763 punch from 0x0002f8 to 0x009247 (0x8f50 bytes) 000082 1773102627.546000 read 0x009509 thru 0x01860b (0xf103 bytes) 000083 1773102627.552738 mapwrite 0x0102d7 thru 0x01ce19 (0xcb43 bytes) 000085 1773102627.594393 read 0x00df9b thru 0x0110a1 (0x3107 bytes) 000087 1773102627.600164 MIRROR_RESYNC 3 mirrors 000088 1773102628.289127 mapread 0x03c02d thru 0x03e902 (0x28d6 bytes) 000089 1773102628.312298 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000090 1773102628.479577 trunc from 0x03e903 to 0x0266c6 (0x1823c bytes) 000091 1773102628.539301 trunc from 0x0266c7 to 0x03d859 (0x17193 bytes) 000092 1773102628.554901 punch from 0x03b5cf to 0x03d859 (0x228b bytes) 000093 1773102628.562001 punch from 0x015a36 to 0x024823 (0xedee bytes) 000094 1773102628.566987 trunc from 0x03d85a to 0x00d754 (0x30105 bytes) 000095 1773102628.578561 read 0x00a9ec thru 0x00c0fb (0x1710 bytes) 000096 1773102628.586935 punch from 0x000b20 to 0x00d754 (0xcc35 bytes) 000097 1773102628.591900 read 0x00c85e thru 0x00d754 (0xef7 bytes) 000098 1773102628.604537 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd755): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd755): Mirror 3: 0x850baa22 (stale) CRC-32 checksum value for chunk [0, 0xd755): Mirror 1: 0x740a5b8d Mirror 4: 0x740a5b8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd755: skipped 000099 1773102629.005392 fallocate 0x021570 thru 0x02d929 (0xc3ba bytes) 000100 1773102629.033806 fallocate 0x034a08 thru 0x03639f (0x1998 bytes) 000101 1773102629.037418 punch from 0x00623a to 0x0110ad (0xae74 bytes) 000102 1773102629.040242 fallocate 0x024b40 thru 0x024eba (0x37b bytes) 000103 1773102629.045395 trunc from 0x02d92a to 0x025196 (0x8793 bytes) 000104 1773102629.076815 fallocate 0x01fa02 thru 0x0290ae (0x96ad bytes) 000105 1773102629.080099 mapread 0x01c6d6 thru 0x02347b (0x6da6 bytes) 000106 1773102629.097834 read 0x0073b8 thru 0x00b544 (0x418d bytes) 000107 1773102629.101496 trunc from 0x0012ac to 0x006e12 (0x5b67 bytes) 000108 1773102629.104209 mapwrite 0x0218ad thru 0x0233c0 (0x1b14 bytes) 000109 1773102629.116063 write 0x03e1cf thru 0x03ffff (0x1e31 bytes) 000110 1773102629.117213 trunc from 0x01fa96 to 0x02cfcf (0xd53a bytes) 000111 1773102629.123766 read 0x03acce thru 0x03ffff (0x5332 bytes) 000112 1773102629.135897 mapwrite 0x033b73 thru 0x03c531 (0x89bf bytes) 000113 1773102629.158577 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5527f595 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d6e9f27 Mirror 5: 0x7d6e9f27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000114 1773102629.697129 mapread 0x01052c thru 0x0204dd (0xffb2 bytes) 000115 1773102629.740894 fallocate 0x031a52 thru 0x03ffff (0xe5ae bytes) 000116 1773102629.769477 mapwrite 0x016c9f thru 0x023664 (0xc9c6 bytes) 000117 1773102629.804027 fallocate 0x0003cf thru 0x00f303 (0xef35 bytes) 000118 1773102629.808822 trunc from 0x0079dd to 0x015e7d (0xe4a1 bytes) 000119 1773102629.813558 fallocate 0x02fd9d thru 0x0335c3 (0x3827 bytes) 000121 1773102629.817293 read 0x032027 thru 0x03ffff (0xdfd9 bytes) 000122 1773102629.824337 read 0x0290f7 thru 0x02a597 (0x14a1 bytes) 000123 1773102629.835261 trunc from 0x026d07 to 0x02db92 (0x6e8c bytes) 000124 1773102629.851337 fallocate 0x01ca7c thru 0x01d695 (0xc1a bytes) 000125 1773102629.857217 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5527f595 (stale) Mirror 4: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5527f595 (stale) Mirror 4: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000126 1773102630.044626 punch from 0x037078 to 0x03c17b (0x5104 bytes) 000127 1773102630.049496 mapwrite 0x004d98 thru 0x01064f (0xb8b8 bytes) 000128 1773102630.073304 punch from 0x02b53b to 0x02bc2b (0x6f1 bytes) 000129 1773102630.080359 punch from 0x00d567 to 0x01149d (0x3f37 bytes) 000130 1773102630.085221 trunc from 0x00c001 to 0x014b97 (0x8b97 bytes) 000131 1773102630.090884 punch from 0x03cd65 to 0x03dba5 (0xe41 bytes) 000132 1773102630.096756 write 0x03548e thru 0x039271 (0x3de4 bytes) 000133 1773102630.112252 fallocate 0x0071cd thru 0x013c6c (0xcaa0 bytes) 000134 1773102630.118301 fallocate 0x00b7e8 thru 0x013879 (0x8092 bytes) 000136 1773102630.124657 MIRROR_RESYNC 3 mirrors 000137 1773102630.739582 trunc from 0x040000 to 0x034ada (0xb525 bytes) 000140 1773102630.784388 write 0x020c75 thru 0x024e88 (0x4214 bytes) 000141 1773102630.790922 fallocate 0x0063cf thru 0x0133e1 (0xd013 bytes) 000142 1773102630.794088 fallocate 0x018c11 thru 0x021fb2 (0x93a2 bytes) 000143 1773102630.798063 mapread 0x002584 thru 0x010bc9 (0xe646 bytes) 000144 1773102630.803793 trunc from 0x034adb to 0x030009 (0x4ad1 bytes) 000145 1773102630.817533 write 0x024974 thru 0x02dafe (0x918b bytes) 000146 1773102630.824686 MIRROR_RESYNC 3 mirrors 000147 1773102631.214354 write 0x0281fa thru 0x036a60 (0xe867 bytes) 000148 1773102631.233231 trunc from 0x026416 to 0x027f96 (0x1b81 bytes) 000149 1773102631.237960 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: 0x15762d4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: 0x15762d4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a61: skipped 000150 1773102631.397724 read 0x00a033 thru 0x013402 (0x93d0 bytes) 000151 1773102631.403704 punch from 0x032ed2 to 0x036a60 (0x3b8f bytes) 000152 1773102631.407480 trunc from 0x02c714 to 0x02f50b (0x2df8 bytes) 000153 1773102631.411668 read 0x003d37 thru 0x0118db (0xdba5 bytes) 000154 1773102631.420445 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: 0x15762d4e (stale) CRC-32 checksum value for chunk [0, 0x36a61): Mirror 1: 0xd84ece10 Mirror 5: 0xd84ece10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a61: skipped 000155 1773102631.823280 mapwrite 0x02e16e thru 0x03be7d (0xdd10 bytes) 000156 1773102631.902731 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0x93b6d0bd (stale) Mirror 5: 0x471382e8 (stale) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 1: 0xa8617df Mirror 6: 0xa8617df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be7e: skipped 000157 1773102632.493659 fallocate 0x01dbb0 thru 0x02b4e8 (0xd939 bytes) 000158 1773102632.528585 MIRROR_RESYNC 4 mirrors 000160 1773102633.179519 punch from 0x010fc3 to 0x01896b (0x79a9 bytes) 000161 1773102633.212321 punch from 0x032a6f to 0x035337 (0x28c9 bytes) 000162 1773102633.220279 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0xa8617df (stale) Mirror 5: 0xa8617df (stale) Mirror 6: 0xa8617df (stale) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 1: 0x450d1b1d Mirror 7: 0x450d1b1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be7e: skipped 000163 1773102633.762911 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0xa8617df (stale) Mirror 6: 0xa8617df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0xa8617df (stale) Mirror 6: 0xa8617df (stale) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 1: 0x450d1b1d Mirror 7: 0x450d1b1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be7e: skipped 000164 1773102634.148577 punch from 0x036c9b to 0x038982 (0x1ce8 bytes) 000165 1773102634.201011 mapwrite 0x036813 thru 0x03ffff (0x97ed bytes) 000167 1773102634.273014 trunc from 0x040000 to 0x0066dd (0x39922 bytes) 000168 1773102634.287163 punch from 0x002bd3 to 0x0066dd (0x3b0b bytes) 000169 1773102634.291022 mapwrite 0x008e11 thru 0x011a15 (0x8c05 bytes) 000170 1773102634.325992 fallocate 0x02dccc thru 0x03a099 (0xc3ce bytes) 000172 1773102634.336680 fallocate 0x03d416 thru 0x03db1c (0x707 bytes) 000173 1773102634.341386 trunc from 0x011243 to 0x0200fe (0xeebc bytes) 000174 1773102634.347879 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 4: 0x134ff766 (stale) Mirror 6: 0x134ff766 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 4: 0x134ff766 (stale) Mirror 6: 0x134ff766 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1d: skipped 000175 1773102634.560705 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 6: 0x134ff766 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 6: 0x134ff766 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1d: skipped 000176 1773102634.713621 fallocate 0x022d47 thru 0x02c033 (0x92ed bytes) 000177 1773102634.716926 trunc from 0x03db1d to 0x01741e (0x266fe bytes) 000178 1773102634.730468 trunc from 0x01741f to 0x039474 (0x22056 bytes) 000179 1773102634.741766 punch from 0x0382ef to 0x039474 (0x1186 bytes) 000180 1773102634.746805 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000181 1773102634.826636 trunc from 0x039475 to 0x01afd5 (0x1e49f bytes) 000182 1773102634.841766 trunc from 0x01afd6 to 0x0133b0 (0x7c25 bytes) 000183 1773102634.852127 write 0x0131d8 thru 0x019f4c (0x6d75 bytes) 000184 1773102634.857738 mapread 0x00e504 thru 0x019399 (0xae96 bytes) 000185 1773102634.869266 punch from 0x015f78 to 0x019d8b (0x3e14 bytes) 000186 1773102634.872758 mapread 0x014292 thru 0x019f4c (0x5cbb bytes) 000187 1773102634.882850 trunc from 0x019f4d to 0x00056c (0x199e0 bytes) 000188 1773102634.897334 trunc from 0x00013d to 0x00056c (0x430 bytes) 000189 1773102634.904132 trunc from 0x00054e to 0x00056c (0x1f bytes) 000191 1773102634.908265 trunc from 0x0003b3 to 0x00056c (0x1ba bytes) 000192 1773102634.913130 mapread 0x0004ad thru 0x00056c (0xc0 bytes) 000195 1773102634.919081 punch from 0x000312 to 0x00056c (0x25b bytes) 000196 1773102634.922137 punch from 0x000408 to 0x00056c (0x165 bytes) 000197 1773102634.925966 mapread 0x000553 thru 0x00056c (0x1a bytes) 000198 1773102634.929699 read 0x00016f thru 0x00056c (0x3fe bytes) 000201 1773102634.931299 trunc from 0x00056d to 0x007b09 (0x759d bytes) 000202 1773102634.943278 trunc from 0x007b0a to 0x03c23d (0x34734 bytes) 000204 1773102634.958324 trunc from 0x026770 to 0x027ed1 (0x1762 bytes) 000205 1773102634.961710 mapwrite 0x0328cf thru 0x0349f4 (0x2126 bytes) 000206 1773102634.982472 mapread 0x001efd thru 0x00b759 (0x985d bytes) 000207 1773102634.988510 mapwrite 0x02c201 thru 0x032415 (0x6215 bytes) 000208 1773102635.002569 write 0x008bc7 thru 0x0142e7 (0xb721 bytes) 000209 1773102635.007899 trunc from 0x00c37d to 0x01a8c8 (0xe54c bytes) 000210 1773102635.023285 fallocate 0x02a05c thru 0x02b48c (0x1431 bytes) 000212 1773102635.031463 MIRROR_EXTEND to 2 mirrors 000213 1773102635.398588 read 0x00f727 thru 0x01f466 (0xfd40 bytes) 000214 1773102635.406999 trunc from 0x0340c1 to 0x03c23d (0x817d bytes) 000215 1773102635.442161 fallocate 0x02459b thru 0x033727 (0xf18d bytes) 000216 1773102635.445524 read 0x003871 thru 0x0049c4 (0x1154 bytes) 000217 1773102635.466482 trunc from 0x03c23e to 0x00dbb8 (0x2e685 bytes) 000219 1773102635.498872 trunc from 0x000972 to 0x004b29 (0x41b8 bytes) 000220 1773102635.504200 read 0x0052b5 thru 0x00ad22 (0x5a6e bytes) 000221 1773102635.504956 punch from 0x004f5f to 0x00dbb8 (0x8c5a bytes) 000222 1773102635.508009 trunc from 0x00dbb9 to 0x00b2e3 (0x28d5 bytes) 000223 1773102635.519265 fallocate 0x030e70 thru 0x03ffff (0xf190 bytes) 000224 1773102635.523449 trunc from 0x00b5f7 to 0x00e50d (0x2f17 bytes) 000225 1773102635.526257 mapread 0x015ca0 thru 0x01fa15 (0x9d76 bytes) 000226 1773102635.536546 read 0x02c82b thru 0x0310d3 (0x48a9 bytes) 000227 1773102635.537243 fallocate 0x012b06 thru 0x01ba86 (0x8f81 bytes) 000228 1773102635.546113 MIRROR_RESYNC 2 mirrors 000230 1773102635.778269 trunc from 0x040000 to 0x007501 (0x38afe bytes) 000231 1773102635.817860 MIRROR_RESYNC 2 mirrors 000232 1773102636.062144 write 0x03e5ff thru 0x03f4d0 (0xed2 bytes) 000233 1773102636.085761 trunc from 0x03f4d1 to 0x022017 (0x1d4b9 bytes) 000234 1773102636.102486 mapwrite 0x02b4e3 thru 0x0391f5 (0xdd13 bytes) 000235 1773102636.146511 punch from 0x014ccd to 0x019e42 (0x5176 bytes) 000236 1773102636.150875 punch from 0x00bc14 to 0x017167 (0xb554 bytes) 000237 1773102636.156026 read 0x02c25e thru 0x035684 (0x9427 bytes) 000238 1773102636.156739 trunc from 0x0368d9 to 0x0391f5 (0x291d bytes) 000239 1773102636.162371 fallocate 0x000381 thru 0x00d255 (0xced5 bytes) 000240 1773102636.166150 read 0x01eae6 thru 0x025d88 (0x72a3 bytes) 000241 1773102636.173103 fallocate 0x00c1b2 thru 0x01bfe7 (0xfe36 bytes) 000242 1773102636.176025 write 0x02183e thru 0x026e7c (0x563f bytes) 000243 1773102636.176996 punch from 0x022f9c to 0x023253 (0x2b8 bytes) 000244 1773102636.185302 mapwrite 0x02d672 thru 0x035e9d (0x882c bytes) 000245 1773102636.202260 trunc from 0x0391f6 to 0x02b7bf (0xda36 bytes) 000246 1773102636.215241 MIRROR_RESYNC 2 mirrors 000247 1773102636.518391 punch from 0x018774 to 0x01b0f0 (0x297d bytes) 000248 1773102636.559165 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000249 1773102636.656098 MIRROR_EXTEND to 2 mirrors 000251 1773102637.003456 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000252 1773102637.113080 mapwrite 0x02b5d8 thru 0x036699 (0xb0c2 bytes) 000253 1773102637.186666 trunc from 0x015a1c to 0x01cb05 (0x70ea bytes) 000254 1773102637.190160 read 0x028cd4 thru 0x02e944 (0x5c71 bytes) 000255 1773102637.190689 punch from 0x02bc1d to 0x02dd2a (0x210e bytes) 000256 1773102637.193944 fallocate 0x0341d8 thru 0x03cd98 (0x8bc1 bytes) 000257 1773102637.200108 mapread 0x01c2e6 thru 0x024082 (0x7d9d bytes) 000258 1773102637.205817 punch from 0x00f873 to 0x01ca2d (0xd1bb bytes) 000259 1773102637.209574 read 0x02e644 thru 0x034e27 (0x67e4 bytes) 000260 1773102637.210141 mapwrite 0x01e627 thru 0x025ed3 (0x78ad bytes) 000261 1773102637.231427 trunc from 0x01c2ee to 0x0231f4 (0x6f07 bytes) 000262 1773102637.237576 fallocate 0x0270b1 thru 0x02cda8 (0x5cf8 bytes) 000263 1773102637.240525 punch from 0x0186f5 to 0x022334 (0x9c40 bytes) 000264 1773102637.244545 mapwrite 0x01de68 thru 0x024a6a (0x6c03 bytes) 000265 1773102637.287967 trunc from 0x03cd99 to 0x008307 (0x34a91 bytes) 000266 1773102637.294788 fallocate 0x026e7b thru 0x030ffb (0xa181 bytes) 000267 1773102637.297779 write 0x021e12 thru 0x02cc86 (0xae75 bytes) 000268 1773102637.299607 punch from 0x00c871 to 0x013f96 (0x7726 bytes) 000269 1773102637.302621 read 0x00cf3a thru 0x0123eb (0x54b2 bytes) 000270 1773102637.310243 MIRROR_EXTEND to 2 mirrors 000271 1773102637.629648 fallocate 0x0018f3 thru 0x00bdb1 (0xa4bf bytes) 000272 1773102637.656935 write 0x003832 thru 0x00948c (0x5c5b bytes) 000273 1773102637.670881 mapread 0x004979 thru 0x007cdc (0x3364 bytes) 000274 1773102637.671723 mapread 0x00a112 thru 0x00f702 (0x55f1 bytes) 000275 1773102637.680775 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000277 1773102637.755159 fallocate 0x01a233 thru 0x022104 (0x7ed2 bytes) 000278 1773102637.758146 read 0x00da1b thru 0x01d730 (0xfd16 bytes) 000279 1773102637.767693 trunc from 0x02cc87 to 0x006fc8 (0x25cbe bytes) 000280 1773102637.804211 read 0x00474c thru 0x0061e8 (0x1a9d bytes) 000281 1773102637.810323 write 0x03c2f7 thru 0x03ffff (0x3d09 bytes) 000282 1773102637.811308 fallocate 0x01dbcc thru 0x025340 (0x7775 bytes) 000283 1773102637.815180 mapwrite 0x03f87a thru 0x03ffff (0x786 bytes) 000284 1773102637.823291 write 0x02cc48 thru 0x03ad67 (0xe120 bytes) 000285 1773102637.833316 write 0x000c2d thru 0x00436e (0x3742 bytes) 000286 1773102637.834760 read 0x016213 thru 0x01cc31 (0x6a1f bytes) 000287 1773102637.835344 mapread 0x00358e thru 0x0093ed (0x5e60 bytes) 000288 1773102637.836615 read 0x039d1d thru 0x03ffff (0x62e3 bytes) 000289 1773102637.837175 fallocate 0x00479a thru 0x00afeb (0x6852 bytes) 000290 1773102637.844942 MIRROR_EXTEND to 2 mirrors 000291 1773102638.271639 write 0x01663f thru 0x01a521 (0x3ee3 bytes) 000292 1773102638.306612 punch from 0x01099f to 0x01b201 (0xa863 bytes) 000293 1773102638.315419 fallocate 0x000bd9 thru 0x00b100 (0xa528 bytes) 000294 1773102638.319866 trunc from 0x040000 to 0x017dfb (0x28204 bytes) 000295 1773102638.337148 fallocate 0x0268fc thru 0x027680 (0xd85 bytes) 000296 1773102638.344051 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000297 1773102638.416698 punch from 0x00bfaf to 0x0155c9 (0x961b bytes) 000299 1773102638.421488 trunc from 0x017dfc to 0x0354bf (0x1d6c4 bytes) 000301 1773102638.436641 trunc from 0x00bf39 to 0x00eeee (0x2fb6 bytes) 000302 1773102638.441168 fallocate 0x02a6c9 thru 0x0343da (0x9d12 bytes) 000303 1773102638.444767 fallocate 0x0209a7 thru 0x02180e (0xe68 bytes) 000304 1773102638.447790 fallocate 0x029890 thru 0x038ead (0xf61e bytes) 000305 1773102638.451191 mapread 0x00237b thru 0x00ffa4 (0xdc2a bytes) 000306 1773102638.467980 mapread 0x033d5c thru 0x0354bf (0x1764 bytes) 000307 1773102638.476141 mapwrite 0x025df3 thru 0x035894 (0xfaa2 bytes) 000309 1773102638.518762 mapwrite 0x00e3d1 thru 0x01abe4 (0xc814 bytes) 000310 1773102638.546989 mapwrite 0x004896 thru 0x00887c (0x3fe7 bytes) 000311 1773102638.558219 mapread 0x001fd4 thru 0x00bbed (0x9c1a bytes) 000313 1773102638.559963 mapwrite 0x00e3b1 thru 0x01d762 (0xf3b2 bytes) 000314 1773102638.585122 mapwrite 0x03576c thru 0x03e4b6 (0x8d4b bytes) 000315 1773102638.614204 write 0x006cb4 thru 0x010ba9 (0x9ef6 bytes) 000316 1773102638.615508 punch from 0x03961c to 0x03e4b6 (0x4e9b bytes) 000317 1773102638.620201 mapwrite 0x02f56b thru 0x03d64b (0xe0e1 bytes) 000318 1773102638.643711 mapread 0x0279d4 thru 0x0323f6 (0xaa23 bytes) 000320 1773102638.645436 mapread 0x00c0ba thru 0x019b7d (0xdac4 bytes) 000322 1773102638.647330 punch from 0x015408 to 0x01dd6f (0x8968 bytes) 000324 1773102638.652902 mapread 0x029921 thru 0x02de1e (0x44fe bytes) 000325 1773102638.653734 write 0x022750 thru 0x032408 (0xfcb9 bytes) 000326 1773102638.655295 trunc from 0x03e4b7 to 0x024c1b (0x1989b bytes) 000327 1773102638.673513 trunc from 0x024c1c to 0x015e57 (0xedc4 bytes) 000328 1773102638.685952 write 0x01df38 thru 0x01fb71 (0x1c3a bytes) 000329 1773102638.687085 mapread 0x016b6e thru 0x01740d (0x8a0 bytes) 000330 1773102638.695834 mapread 0x00d8e6 thru 0x013f17 (0x6632 bytes) 000331 1773102638.696766 trunc from 0x001c08 to 0x002bcf (0xfc8 bytes) 000333 1773102638.700255 trunc from 0x01fb72 to 0x00816c (0x17a05 bytes) 000334 1773102638.710267 mapwrite 0x02a1e5 thru 0x038f8f (0xedab bytes) 000335 1773102638.745671 mapread 0x03677c thru 0x038f8f (0x2814 bytes) 000336 1773102638.746696 fallocate 0x036b52 thru 0x039ab1 (0x2f60 bytes) 000337 1773102638.750217 fallocate 0x0302a5 thru 0x03dd6d (0xdac9 bytes) 000338 1773102638.753753 trunc from 0x038f90 to 0x01ca00 (0x1c58f bytes) 000340 1773102638.761869 trunc from 0x01ca01 to 0x010cf9 (0xbd07 bytes) 000341 1773102638.772477 punch from 0x00b879 to 0x00c5a1 (0xd29 bytes) 000343 1773102638.778473 mapread 0x00d053 thru 0x0101da (0x3188 bytes) 000344 1773102638.785330 read 0x000aa8 thru 0x003e0b (0x3364 bytes) 000345 1773102638.789634 trunc from 0x010cfa to 0x03453e (0x23845 bytes) 000346 1773102638.806094 write 0x0271b2 thru 0x02b210 (0x405f bytes) 000347 1773102638.816277 read 0x00a2fd thru 0x00edae (0x4ab2 bytes) 000348 1773102638.816811 read 0x0068ff thru 0x015bb0 (0xf2b2 bytes) 000349 1773102638.823289 mapwrite 0x034993 thru 0x03e165 (0x97d3 bytes) 000350 1773102638.855268 mapwrite 0x0164eb thru 0x01b6c6 (0x51dc bytes) 000351 1773102638.867357 trunc from 0x011f30 to 0x01e23e (0xc30f bytes) 000352 1773102638.870444 trunc from 0x035a32 to 0x037a2f (0x1ffe bytes) 000353 1773102638.874775 trunc from 0x03e166 to 0x01dd70 (0x203f5 bytes) 000354 1773102638.886168 fallocate 0x001e7f thru 0x004d29 (0x2eab bytes) 000356 1773102638.891574 fallocate 0x022210 thru 0x02e668 (0xc459 bytes) 000357 1773102638.898704 trunc from 0x018540 to 0x01b8fc (0x33bd bytes) 000358 1773102638.903216 mapwrite 0x03eb14 thru 0x03ffff (0x14ec bytes) 000359 1773102638.931493 trunc from 0x01ee09 to 0x0213cd (0x25c5 bytes) 000361 1773102638.935289 fallocate 0x0294cf thru 0x0356cb (0xc1fd bytes) 000362 1773102638.938964 read 0x02a3a2 thru 0x038859 (0xe4b8 bytes) 000363 1773102638.946385 mapwrite 0x02be5c thru 0x03403a (0x81df bytes) 000364 1773102638.958620 write 0x0194b3 thru 0x028868 (0xf3b6 bytes) 000365 1773102638.960332 mapread 0x03fed1 thru 0x03ffff (0x12f bytes) 000368 1773102638.963572 mapread 0x02b2ef thru 0x039187 (0xde99 bytes) 000369 1773102638.967391 fallocate 0x0039d3 thru 0x00545f (0x1a8d bytes) 000370 1773102638.971127 trunc from 0x03708c to 0x03ffff (0x8f74 bytes) 000371 1773102638.974276 trunc from 0x024c4a to 0x030f50 (0xc307 bytes) 000372 1773102638.982235 write 0x037abd thru 0x03ffff (0x8543 bytes) 000373 1773102638.985680 write 0x038e00 thru 0x039d0e (0xf0f bytes) 000374 1773102638.987016 punch from 0x03851b to 0x03f2c0 (0x6da6 bytes) 000375 1773102638.995223 fallocate 0x00f8a1 thru 0x01d094 (0xd7f4 bytes) 000376 1773102638.999902 read 0x02b1ee thru 0x02b538 (0x34b bytes) 000378 1773102639.008353 punch from 0x00d053 to 0x012443 (0x53f1 bytes) 000380 1773102639.012817 read 0x010eb6 thru 0x012ef7 (0x2042 bytes) 000381 1773102639.020595 fallocate 0x007c1c thru 0x0101a6 (0x858b bytes) 000382 1773102639.024110 fallocate 0x01c212 thru 0x024168 (0x7f57 bytes) 000383 1773102639.027289 punch from 0x00a596 to 0x00da84 (0x34ef bytes) 000384 1773102639.032678 fallocate 0x02966e thru 0x038486 (0xee19 bytes) 000385 1773102639.036383 fallocate 0x0215b2 thru 0x02c6af (0xb0fe bytes) 000386 1773102639.041032 punch from 0x009ab5 to 0x00f829 (0x5d75 bytes) 000387 1773102639.044557 punch from 0x0396c9 to 0x03ffff (0x6937 bytes) 000388 1773102639.049746 fallocate 0x019426 thru 0x0231e3 (0x9dbe bytes) 000389 1773102639.057599 mapwrite 0x038803 thru 0x03ffff (0x77fd bytes) 000391 1773102639.108887 trunc from 0x0219f1 to 0x02e75c (0xcd6c bytes) 000392 1773102639.119242 read 0x039870 thru 0x03ffff (0x6790 bytes) 000393 1773102639.119967 trunc from 0x024167 to 0x0307f8 (0xc692 bytes) 000394 1773102639.128633 mapread 0x0064fe thru 0x0145bb (0xe0be bytes) 000396 1773102639.136431 read 0x02adce thru 0x02f93e (0x4b71 bytes) 000397 1773102639.136993 read 0x03e764 thru 0x03ffff (0x189c bytes) 000399 1773102639.138353 write 0x0177f1 thru 0x020ea4 (0x96b4 bytes) 000400 1773102639.145848 MIRROR_EXTEND to 2 mirrors 000401 1773102639.576244 read 0x03ac6e thru 0x03ffff (0x5392 bytes) 000402 1773102639.583996 fallocate 0x0144b2 thru 0x01e808 (0xa357 bytes) 000404 1773102639.621397 mapread 0x016909 thru 0x023917 (0xd00f bytes) 000405 1773102639.648287 trunc from 0x040000 to 0x02b161 (0x14e9e bytes) 000406 1773102639.688862 write 0x03fb14 thru 0x03ffff (0x4ec bytes) 000407 1773102639.689857 read 0x008d4a thru 0x00b9c7 (0x2c7e bytes) 000408 1773102639.698353 read 0x03e5d3 thru 0x03ffff (0x1a2d bytes) 000409 1773102639.698838 punch from 0x01a309 to 0x027cd2 (0xd9ca bytes) 000410 1773102639.703865 read 0x00822b thru 0x00d8b8 (0x568e bytes) 000411 1773102639.705113 fallocate 0x03ce2c thru 0x03e5e0 (0x17b5 bytes) 000412 1773102639.709737 mapread 0x018a47 thru 0x01fab5 (0x706f bytes) 000413 1773102639.721562 trunc from 0x03f427 to 0x03ffff (0xbd9 bytes) 000414 1773102639.736646 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc76dc201 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x70c5d14 Mirror 3: 0x70c5d14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000415 1773102640.147658 trunc from 0x029b14 to 0x034a95 (0xaf82 bytes) 000416 1773102640.174712 punch from 0x0079c1 to 0x009145 (0x1785 bytes) 000417 1773102640.178143 mapwrite 0x03742c thru 0x03ffff (0x8bd4 bytes) 000418 1773102640.212959 read 0x027fa1 thru 0x0365d6 (0xe636 bytes) 000420 1773102640.224470 mapwrite 0x018ba5 thru 0x01b691 (0x2aed bytes) 000421 1773102640.249280 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x70c5d14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x70c5d14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000422 1773102640.431730 punch from 0x018612 to 0x022671 (0xa060 bytes) 000425 1773102640.437685 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x70c5d14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x90433c0c Mirror 4: 0x90433c0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000426 1773102640.854763 read 0x008431 thru 0x015531 (0xd101 bytes) 000427 1773102640.867840 fallocate 0x0281f5 thru 0x0308f8 (0x8704 bytes) 000428 1773102640.894613 punch from 0x012f88 to 0x01c2e9 (0x9362 bytes) 000430 1773102640.901047 trunc from 0x040000 to 0x02c340 (0x13cbf bytes) 000431 1773102640.924902 trunc from 0x017c37 to 0x01e26e (0x6638 bytes) 000432 1773102640.928702 mapwrite 0x001995 thru 0x00ea21 (0xd08d bytes) 000433 1773102640.969737 punch from 0x01deb7 to 0x029102 (0xb24c bytes) 000434 1773102640.976321 MIRROR_RESYNC 3 mirrors 000436 1773102641.379118 mapwrite 0x00c519 thru 0x010aac (0x4594 bytes) 000437 1773102641.452652 write 0x03d130 thru 0x03ffff (0x2ed0 bytes) 000438 1773102641.453970 trunc from 0x040000 to 0x0259f7 (0x1a608 bytes) 000440 1773102641.466212 punch from 0x00ccdc to 0x01abe5 (0xdf0a bytes) 000442 1773102641.471180 write 0x02ff2b thru 0x0351bf (0x5295 bytes) 000443 1773102641.472899 read 0x026530 thru 0x02bd7a (0x584b bytes) 000444 1773102641.477425 read 0x0281a6 thru 0x02f57d (0x73d8 bytes) 000445 1773102641.485343 fallocate 0x03867f thru 0x03ffff (0x7981 bytes) 000447 1773102641.493390 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x351c0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351c0): Mirror 3: 0x80d44881 (stale) Mirror 4: 0x80d44881 (stale) CRC-32 checksum value for chunk [0, 0x351c0): Mirror 1: 0x1fe5ed70 Mirror 5: 0x1fe5ed70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351c0: skipped 000448 1773102641.936394 read 0x01a6d0 thru 0x02339b (0x8ccc bytes) 000449 1773102641.948865 write 0x030232 thru 0x03fa3e (0xf80d bytes) 000450 1773102641.976621 read 0x028872 thru 0x02ae5a (0x25e9 bytes) 000452 1773102641.983761 mapread 0x01d67a thru 0x01e9da (0x1361 bytes) 000453 1773102641.984550 trunc from 0x03fa3f to 0x02a284 (0x157ba bytes) 000455 1773102642.003751 read 0x01407e thru 0x020702 (0xc685 bytes) 000456 1773102642.004351 punch from 0x016417 to 0x02120a (0xadf4 bytes) 000457 1773102642.007324 mapread 0x0155f5 thru 0x01b8aa (0x62b6 bytes) 000458 1773102642.013291 read 0x006609 thru 0x00fd62 (0x975a bytes) 000459 1773102642.014003 fallocate 0x01aec6 thru 0x01b336 (0x471 bytes) 000460 1773102642.017176 trunc from 0x02a285 to 0x00994b (0x20939 bytes) 000461 1773102642.025349 fallocate 0x02fddd thru 0x03e863 (0xea87 bytes) 000462 1773102642.030604 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) Mirror 5: 0x9a26f1fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) Mirror 5: 0x9a26f1fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e864: skipped 000463 1773102642.198659 write 0x034960 thru 0x039b48 (0x51e9 bytes) 000464 1773102642.209695 fallocate 0x03c741 thru 0x03e34e (0x1c0e bytes) 000465 1773102642.213303 fallocate 0x00564a thru 0x009c6c (0x4623 bytes) 000466 1773102642.216943 write 0x00db39 thru 0x00dc07 (0xcf bytes) 000467 1773102642.229209 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e864: skipped 000468 1773102642.419155 punch from 0x001d57 to 0x005683 (0x392d bytes) 000469 1773102642.422365 mapwrite 0x01ef64 thru 0x026311 (0x73ae bytes) 000470 1773102642.456299 fallocate 0x03a90c thru 0x03f11b (0x4810 bytes) 000471 1773102642.463809 write 0x00c5b6 thru 0x0102f8 (0x3d43 bytes) 000472 1773102642.477190 trunc from 0x03f11c to 0x03f3ef (0x2d4 bytes) 000473 1773102642.496742 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f3f0): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3f0): Mirror 3: 0x2b006936 (stale) CRC-32 checksum value for chunk [0, 0x3f3f0): Mirror 1: 0x4933f57a Mirror 4: 0x4933f57a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3f0: skipped 000474 1773102643.127210 fallocate 0x03e5d8 thru 0x03ffff (0x1a28 bytes) 000475 1773102643.151517 trunc from 0x03f3f0 to 0x00c0b7 (0x33338 bytes) 000476 1773102643.173353 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc0b8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc0b8): Mirror 3: 0x2e2d8719 (stale) Mirror 4: 0x57266657 (stale) CRC-32 checksum value for chunk [0, 0xc0b8): Mirror 1: 0x57266657 Mirror 5: 0x57266657 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc0b8: skipped 000477 1773102643.690202 write 0x000aa3 thru 0x0033ed (0x294b bytes) 000478 1773102643.713679 read 0x004950 thru 0x00c0b7 (0x7768 bytes) 000480 1773102643.720761 mapwrite 0x02aab5 thru 0x034a08 (0x9f54 bytes) 000481 1773102643.764729 mapwrite 0x03d399 thru 0x03ffff (0x2c67 bytes) 000482 1773102643.794186 mapwrite 0x03f131 thru 0x03ffff (0xecf bytes) 000483 1773102643.804329 mapread 0x02be80 thru 0x02e544 (0x26c5 bytes) 000484 1773102643.805092 read 0x0316b5 thru 0x03d0ac (0xb9f8 bytes) 000486 1773102643.809953 mapread 0x006c8b thru 0x00e695 (0x7a0b bytes) 000487 1773102643.811121 punch from 0x00133f to 0x00faa7 (0xe769 bytes) 000488 1773102643.814628 mapwrite 0x029ef2 thru 0x02af46 (0x1055 bytes) 000489 1773102643.829361 punch from 0x010ca3 to 0x013050 (0x23ae bytes) 000490 1773102643.832664 punch from 0x0200a7 to 0x0298d6 (0x9830 bytes) 000491 1773102643.838147 MIRROR_RESYNC 4 mirrors 000492 1773102644.366500 punch from 0x00528f to 0x00a9a2 (0x5714 bytes) 000493 1773102644.380055 trunc from 0x00322a to 0x00d16c (0x9f43 bytes) 000494 1773102644.384087 read 0x028d46 thru 0x02b79b (0x2a56 bytes) 000495 1773102644.391825 mapread 0x00ac02 thru 0x00b163 (0x562 bytes) 000497 1773102644.403639 mapread 0x018bfb thru 0x021dd0 (0x91d6 bytes) 000499 1773102644.417022 fallocate 0x0320fd thru 0x039a23 (0x7927 bytes) 000500 1773102644.421263 read 0x0016c9 thru 0x00cf31 (0xb869 bytes) 000502 1773102644.442396 trunc from 0x03bd38 to 0x03ffff (0x42c8 bytes) 000503 1773102644.448194 write 0x03f505 thru 0x03ffff (0xafb bytes) 000504 1773102644.455284 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0e2bb14 (stale) Mirror 5: 0xb0e2bb14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0e2bb14 (stale) Mirror 5: 0xb0e2bb14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000505 1773102644.637768 read 0x03cccc thru 0x03ffff (0x3334 bytes) 000506 1773102644.643916 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb0e2bb14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb0e2bb14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000507 1773102644.800197 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb0e2bb14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x290567f9 Mirror 6: 0x290567f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1773102645.287602 write 0x003c5f thru 0x010cb9 (0xd05b bytes) 000509 1773102645.313677 trunc from 0x040000 to 0x034c51 (0xb3ae bytes) 000510 1773102645.325670 punch from 0x00f7b2 to 0x01cd12 (0xd561 bytes) 000511 1773102645.334944 fallocate 0x022ce9 thru 0x02c63b (0x9953 bytes) 000512 1773102645.339190 mapwrite 0x02b0e7 thru 0x038a25 (0xd93f bytes) 000513 1773102645.390310 trunc from 0x0237fa to 0x029142 (0x5949 bytes) 000515 1773102645.394958 fallocate 0x02d5c4 thru 0x03c555 (0xef92 bytes) 000516 1773102645.399681 mapwrite 0x004762 thru 0x00acfa (0x6599 bytes) 000517 1773102645.416119 write 0x025279 thru 0x02ab92 (0x591a bytes) 000518 1773102645.426612 mapwrite 0x01a679 thru 0x021e3d (0x77c5 bytes) 000519 1773102645.455260 read 0x02ad52 thru 0x039dbd (0xf06c bytes) 000520 1773102645.461720 mapwrite 0x016453 thru 0x023ce8 (0xd896 bytes) 000521 1773102645.478076 trunc from 0x006952 to 0x008e2b (0x24da bytes) 000523 1773102645.486496 MIRROR_RESYNC 3 mirrors 000524 1773102646.060143 mapwrite 0x029065 thru 0x02c938 (0x38d4 bytes) 000525 1773102646.124049 mapwrite 0x004153 thru 0x007fb4 (0x3e62 bytes) 000526 1773102646.137888 mapwrite 0x026f9c thru 0x02d0a0 (0x6105 bytes) 000528 1773102646.152203 read 0x028a82 thru 0x031ef0 (0x946f bytes) 000529 1773102646.152685 mapwrite 0x032922 thru 0x034189 (0x1868 bytes) 000530 1773102646.166476 trunc from 0x03c556 to 0x0160ab (0x264aa bytes) 000531 1773102646.198968 fallocate 0x014fca thru 0x01d783 (0x87ba bytes) 000532 1773102646.202910 read 0x00059a thru 0x00dc92 (0xd6f9 bytes) 000533 1773102646.209142 read 0x01d70c thru 0x01d783 (0x78 bytes) 000535 1773102646.210433 fallocate 0x033989 thru 0x03ffff (0xc677 bytes) 000536 1773102646.215690 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d784): Mirror 5: 0xa1008dd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d784): Mirror 5: 0xa1008dd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d784: skipped 000537 1773102646.380338 write 0x02ee99 thru 0x02f4a1 (0x609 bytes) 000538 1773102646.383728 mapwrite 0x02d33a thru 0x034584 (0x724b bytes) 000539 1773102646.421664 mapwrite 0x01686b thru 0x024516 (0xdcac bytes) 000542 1773102646.447564 trunc from 0x032527 to 0x034584 (0x205e bytes) 000543 1773102646.450668 trunc from 0x00acea to 0x00decf (0x31e6 bytes) 000544 1773102646.453520 read 0x00874b thru 0x00c2bf (0x3b75 bytes) 000545 1773102646.458265 read 0x022260 thru 0x0237a3 (0x1544 bytes) 000546 1773102646.458905 trunc from 0x0294b7 to 0x02a5eb (0x1135 bytes) 000547 1773102646.462454 mapwrite 0x001f3b thru 0x00e9bd (0xca83 bytes) 000548 1773102646.483676 trunc from 0x0129f9 to 0x01ad18 (0x8320 bytes) 000549 1773102646.487219 mapwrite 0x02d43e thru 0x03c8d6 (0xf499 bytes) 000550 1773102646.516621 fallocate 0x01063e thru 0x010d4f (0x712 bytes) 000551 1773102646.522781 mapwrite 0x02f5fb thru 0x03054c (0xf52 bytes) 000552 1773102646.530499 trunc from 0x03c8d7 to 0x0066e2 (0x361f4 bytes) 000554 1773102646.544106 trunc from 0x002f87 to 0x006580 (0x35fa bytes) 000555 1773102646.548794 read 0x0058c5 thru 0x005c12 (0x34e bytes) 000556 1773102646.553285 fallocate 0x01203e thru 0x015894 (0x3857 bytes) 000557 1773102646.557852 punch from 0x001e56 to 0x00d14d (0xb2f8 bytes) 000559 1773102646.560446 trunc from 0x0051a6 to 0x00ea09 (0x9864 bytes) 000560 1773102646.563932 write 0x03c00c thru 0x03ffff (0x3ff4 bytes) 000561 1773102646.564806 trunc from 0x022c7f to 0x0313da (0xe75c bytes) 000562 1773102646.567188 read 0x02c682 thru 0x02ea69 (0x23e8 bytes) 000563 1773102646.573905 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000564 1773102646.644065 punch from 0x01a302 to 0x029242 (0xef41 bytes) 000565 1773102646.647223 fallocate 0x0046d4 thru 0x006849 (0x2176 bytes) 000566 1773102646.651645 punch from 0x02da87 to 0x02e253 (0x7cd bytes) 000568 1773102646.655469 mapread 0x00ed44 thru 0x01db89 (0xee46 bytes) 000569 1773102646.678863 mapread 0x0232c1 thru 0x031198 (0xded8 bytes) 000570 1773102646.691429 punch from 0x030dfa to 0x032316 (0x151d bytes) 000571 1773102646.694750 trunc from 0x0095d8 to 0x00df8a (0x49b3 bytes) 000572 1773102646.698805 mapwrite 0x01d68c thru 0x025c9e (0x8613 bytes) 000573 1773102646.714463 trunc from 0x040000 to 0x02c0cc (0x13f33 bytes) 000574 1773102646.724854 mapwrite 0x01edea thru 0x029abf (0xacd6 bytes) 000575 1773102646.739922 trunc from 0x02c0cd to 0x0024f7 (0x29bd5 bytes) 000576 1773102646.750083 mapread 0x000ec8 thru 0x0024f7 (0x1630 bytes) 000577 1773102646.754030 fallocate 0x03e6a5 thru 0x03ffff (0x195b bytes) 000578 1773102646.759224 read 0x03e1d8 thru 0x03ffff (0x1e28 bytes) 000579 1773102646.763530 fallocate 0x010875 thru 0x01dde6 (0xd572 bytes) 000580 1773102646.766815 fallocate 0x01745d thru 0x01a117 (0x2cbb bytes) 000581 1773102646.770259 write 0x00c336 thru 0x0122c0 (0x5f8b bytes) 000582 1773102646.777720 read 0x011ba2 thru 0x01f31f (0xd77e bytes) 000583 1773102646.791347 trunc from 0x040000 to 0x03c351 (0x3cae bytes) 000584 1773102646.802315 mapread 0x01531e thru 0x015ea2 (0xb85 bytes) 000585 1773102646.803214 mapread 0x032e52 thru 0x03c351 (0x9500 bytes) 000586 1773102646.804171 fallocate 0x018914 thru 0x01a254 (0x1941 bytes) 000587 1773102646.806911 write 0x03e328 thru 0x03ffff (0x1cd8 bytes) 000588 1773102646.807935 trunc from 0x040000 to 0x03d483 (0x2b7c bytes) 000590 1773102646.816642 punch from 0x022457 to 0x031f5c (0xfb06 bytes) 000592 1773102646.821227 write 0x015af0 thru 0x0254af (0xf9c0 bytes) 000593 1773102646.829513 mapwrite 0x011235 thru 0x020077 (0xee43 bytes) 000594 1773102646.844321 write 0x010541 thru 0x01e6e4 (0xe1a4 bytes) 000595 1773102646.845869 trunc from 0x001c6f to 0x00601d (0x43af bytes) 000596 1773102646.848811 mapwrite 0x00dedf thru 0x01bb70 (0xdc92 bytes) 000597 1773102646.864598 trunc from 0x03d484 to 0x01544c (0x28037 bytes) 000598 1773102646.877309 mapwrite 0x032197 thru 0x03cd82 (0xabec bytes) 000599 1773102646.941471 mapread 0x02425c thru 0x032efa (0xec9f bytes) 000600 1773102646.983343 trunc from 0x02ba53 to 0x03a070 (0xe61e bytes) 000601 1773102646.988380 write 0x0353c3 thru 0x03ffff (0xac3d bytes) 000602 1773102646.992543 punch from 0x011a46 to 0x01dfda (0xc595 bytes) 000605 1773102646.997915 punch from 0x028d22 to 0x032982 (0x9c61 bytes) 000606 1773102647.000902 trunc from 0x007af3 to 0x00984c (0x1d5a bytes) 000607 1773102647.004683 mapread 0x03d8b8 thru 0x03ffff (0x2748 bytes) 000609 1773102647.006450 punch from 0x00bff7 to 0x00f2b7 (0x32c1 bytes) 000611 1773102647.010987 mapwrite 0x00097a thru 0x005126 (0x47ad bytes) 000612 1773102647.042176 write 0x02bb8d thru 0x03a3d1 (0xe845 bytes) 000613 1773102647.046185 fallocate 0x02eec4 thru 0x02f098 (0x1d5 bytes) 000615 1773102647.050309 punch from 0x008d5c to 0x0105cc (0x7871 bytes) 000616 1773102647.054032 write 0x02ca19 thru 0x039d6f (0xd357 bytes) 000617 1773102647.055458 mapwrite 0x03c93e thru 0x03ffff (0x36c2 bytes) 000619 1773102647.066888 write 0x015d13 thru 0x024c6f (0xef5d bytes) 000620 1773102647.071724 punch from 0x0333bf to 0x03a4bf (0x7101 bytes) 000621 1773102647.075062 write 0x026e2f thru 0x02779d (0x96f bytes) 000622 1773102647.075983 trunc from 0x00962c to 0x018af4 (0xf4c9 bytes) 000624 1773102647.091917 mapwrite 0x003d8f thru 0x01320f (0xf481 bytes) 000625 1773102647.161364 read 0x023acf thru 0x02c49d (0x89cf bytes) 000626 1773102647.167365 read 0x01e84e thru 0x0278e3 (0x9096 bytes) 000627 1773102647.168011 read 0x011efa thru 0x012a8a (0xb91 bytes) 000628 1773102647.168553 mapread 0x01dcdc thru 0x02663c (0x8961 bytes) 000629 1773102647.169323 write 0x034443 thru 0x03ffff (0xbbbd bytes) 000630 1773102647.171361 punch from 0x01c5cc to 0x020474 (0x3ea9 bytes) 000631 1773102647.174384 write 0x03d024 thru 0x03ffff (0x2fdc bytes) 000632 1773102647.175370 trunc from 0x040000 to 0x021688 (0x1e977 bytes) 000633 1773102647.185865 write 0x039a99 thru 0x03f183 (0x56eb bytes) 000634 1773102647.187179 read 0x03caf4 thru 0x03f183 (0x2690 bytes) 000635 1773102647.188077 trunc from 0x03197a to 0x03f183 (0xd80a bytes) 000637 1773102647.197712 fallocate 0x00dbe4 thru 0x016a30 (0x8e4d bytes) 000638 1773102647.200616 read 0x0200c2 thru 0x027a77 (0x79b6 bytes) 000639 1773102647.215282 trunc from 0x00bab8 to 0x011056 (0x559f bytes) 000640 1773102647.219161 write 0x0254b5 thru 0x02d26a (0x7db6 bytes) 000643 1773102647.222283 mapread 0x00f4c3 thru 0x015fc6 (0x6b04 bytes) 000644 1773102647.231930 read 0x01ce0c thru 0x02199e (0x4b93 bytes) 000645 1773102647.232786 punch from 0x02a478 to 0x039109 (0xec92 bytes) 000646 1773102647.241887 trunc from 0x03f184 to 0x03c876 (0x290d bytes) 000647 1773102647.254629 trunc from 0x02b232 to 0x03422d (0x8ffc bytes) 000648 1773102647.259601 trunc from 0x004b44 to 0x01197a (0xce37 bytes) 000649 1773102647.264134 trunc from 0x01e0e1 to 0x01f001 (0xf21 bytes) 000650 1773102647.268052 fallocate 0x034ecb thru 0x034f3c (0x72 bytes) 000653 1773102647.272511 mapread 0x00e3d5 thru 0x01bb99 (0xd7c5 bytes) 000654 1773102647.282971 fallocate 0x03db53 thru 0x03ffff (0x24ad bytes) 000655 1773102647.286071 punch from 0x012cc5 to 0x01e2e6 (0xb622 bytes) 000657 1773102647.295694 trunc from 0x027f6d to 0x037526 (0xf5ba bytes) 000658 1773102647.306133 mapread 0x005f61 thru 0x012d83 (0xce23 bytes) 000659 1773102647.315114 punch from 0x03592c to 0x038f94 (0x3669 bytes) 000660 1773102647.322235 mapwrite 0x01bb2b thru 0x028175 (0xc64b bytes) 000661 1773102647.361135 MIRROR_EXTEND to 2 mirrors 000662 1773102647.710338 read 0x02f876 thru 0x033ab3 (0x423e bytes) 000664 1773102647.716604 fallocate 0x0094ef thru 0x014b5c (0xb66e bytes) 000665 1773102647.742796 read 0x03bd05 thru 0x03ffff (0x42fb bytes) 000666 1773102647.750724 write 0x02e290 thru 0x03cdd5 (0xeb46 bytes) 000668 1773102647.754360 trunc from 0x01ac00 to 0x021f00 (0x7301 bytes) 000669 1773102647.759736 read 0x035d64 thru 0x03ffff (0xa29c bytes) 000670 1773102647.760615 write 0x02f3e5 thru 0x039fa8 (0xabc4 bytes) 000671 1773102647.761584 fallocate 0x018235 thru 0x01fe8b (0x7c57 bytes) 000672 1773102647.764588 write 0x0003e4 thru 0x004c03 (0x4820 bytes) 000673 1773102647.765341 trunc from 0x022fa9 to 0x02690a (0x3962 bytes) 000674 1773102647.769000 trunc from 0x03d564 to 0x03ffff (0x2a9c bytes) 000675 1773102647.774154 MIRROR_RESYNC 2 mirrors 000676 1773102648.281933 write 0x018fa0 thru 0x0228c9 (0x992a bytes) 000677 1773102648.312276 mapread 0x02363b thru 0x0288cd (0x5293 bytes) 000678 1773102648.330975 trunc from 0x040000 to 0x0009ab (0x3f654 bytes) 000679 1773102648.344523 mapread 0x0000e6 thru 0x0009ab (0x8c6 bytes) 000680 1773102648.351475 trunc from 0x0007b4 to 0x0009ab (0x1f8 bytes) 000681 1773102648.360566 punch from 0x000362 to 0x0009ab (0x64a bytes) 000682 1773102648.363787 mapread 0x000314 thru 0x0009ab (0x698 bytes) 000683 1773102648.372194 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000684 1773102648.459100 fallocate 0x009c54 thru 0x00f95c (0x5d09 bytes) 000685 1773102648.462368 mapwrite 0x02271f thru 0x025051 (0x2933 bytes) 000687 1773102648.518516 trunc from 0x0162e5 to 0x01ad21 (0x4a3d bytes) 000688 1773102648.522554 write 0x00c46a thru 0x01b70e (0xf2a5 bytes) 000689 1773102648.532733 read 0x00f5da thru 0x016f03 (0x792a bytes) 000690 1773102648.533379 trunc from 0x025052 to 0x033863 (0xe812 bytes) 000691 1773102648.543274 trunc from 0x00836f to 0x014dd1 (0xca63 bytes) 000692 1773102648.550964 mapread 0x01c992 thru 0x01d230 (0x89f bytes) 000693 1773102648.554798 write 0x00af32 thru 0x00fbe1 (0x4cb0 bytes) 000694 1773102648.562001 write 0x025620 thru 0x02eb6d (0x954e bytes) 000695 1773102648.565427 trunc from 0x033864 to 0x0065f8 (0x2d26b bytes) 000696 1773102648.573536 write 0x03f4b1 thru 0x03ffff (0xb4f bytes) 000697 1773102648.574609 mapread 0x00302f thru 0x00d973 (0xa945 bytes) 000698 1773102648.580325 mapwrite 0x001d18 thru 0x00febb (0xe1a4 bytes) 000699 1773102648.593718 mapwrite 0x03a0a8 thru 0x03ffff (0x5f58 bytes) 000701 1773102648.610391 mapread 0x013289 thru 0x014aaf (0x1827 bytes) 000702 1773102648.620340 punch from 0x022432 to 0x022f7e (0xb4d bytes) 000704 1773102648.623574 mapread 0x00816b thru 0x00d3df (0x5275 bytes) 000705 1773102648.624555 mapwrite 0x0336c0 thru 0x03ba83 (0x83c4 bytes) 000707 1773102648.640111 trunc from 0x040000 to 0x0322fa (0xdd05 bytes) 000709 1773102648.648375 write 0x010d5f thru 0x01a70b (0x99ad bytes) 000710 1773102648.649883 read 0x015618 thru 0x01d581 (0x7f6a bytes) 000711 1773102648.656193 trunc from 0x0090f6 to 0x017813 (0xe71e bytes) 000712 1773102648.671830 punch from 0x011791 to 0x020ecb (0xf73b bytes) 000713 1773102648.680069 trunc from 0x015c11 to 0x024fe4 (0xf3d4 bytes) 000714 1773102648.682834 fallocate 0x00244c thru 0x00c3b5 (0x9f6a bytes) 000715 1773102648.685344 mapwrite 0x00a414 thru 0x0161f7 (0xbde4 bytes) 000716 1773102648.757163 mapwrite 0x016454 thru 0x01c99a (0x6547 bytes) 000717 1773102648.786874 mapread 0x013cfd thru 0x0189bb (0x4cbf bytes) 000718 1773102648.787864 mapread 0x022bdd thru 0x02bc12 (0x9036 bytes) 000719 1773102648.799242 fallocate 0x0365f3 thru 0x03b44e (0x4e5c bytes) 000720 1773102648.802688 read 0x039e92 thru 0x03b44e (0x15bd bytes) 000721 1773102648.807460 punch from 0x02441b to 0x025e96 (0x1a7c bytes) 000723 1773102648.812196 mapread 0x021571 thru 0x023df5 (0x2885 bytes) 000725 1773102648.813672 punch from 0x02ba7f to 0x02fd82 (0x4304 bytes) 000726 1773102648.819014 punch from 0x013806 to 0x01fac3 (0xc2be bytes) 000728 1773102648.827647 read 0x0306f1 thru 0x0366b5 (0x5fc5 bytes) 000729 1773102648.828370 mapread 0x002dff thru 0x00bb42 (0x8d44 bytes) 000732 1773102648.831514 read 0x00e344 thru 0x010d38 (0x29f5 bytes) 000733 1773102648.832869 write 0x00ccb1 thru 0x012b2f (0x5e7f bytes) 000734 1773102648.834270 trunc from 0x03b44f to 0x02596f (0x15adf bytes) 000735 1773102648.844136 mapread 0x01ac8b thru 0x02596f (0xace5 bytes) 000737 1773102648.862973 MIRROR_EXTEND to 2 mirrors 000738 1773102649.296294 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000741 1773102649.385458 mapread 0x01b8c8 thru 0x02596f (0xa0a8 bytes) 000742 1773102649.398416 mapwrite 0x023ffe thru 0x0290a2 (0x50a5 bytes) 000743 1773102649.445469 mapread 0x017c25 thru 0x01d5f4 (0x59d0 bytes) 000744 1773102649.453812 trunc from 0x0290a3 to 0x00e7d5 (0x1a8cd bytes) 000745 1773102649.467294 mapread 0x00d610 thru 0x00e7d5 (0x11c6 bytes) 000746 1773102649.477878 fallocate 0x03cb85 thru 0x03ffff (0x347b bytes) 000749 1773102649.485380 trunc from 0x040000 to 0x00463a (0x3b9c5 bytes) 000750 1773102649.497293 trunc from 0x0041e7 to 0x00463a (0x454 bytes) 000751 1773102649.500575 trunc from 0x0000ef to 0x00463a (0x454c bytes) 000753 1773102649.505260 trunc from 0x00463b to 0x03c23d (0x37c03 bytes) 000754 1773102649.520431 punch from 0x025092 to 0x033056 (0xdfc5 bytes) 000755 1773102649.525346 mapwrite 0x021d84 thru 0x02d14d (0xb3ca bytes) 000756 1773102649.565548 MIRROR_EXTEND to 2 mirrors 000757 1773102649.971717 MIRROR_RESYNC 2 mirrors 000758 1773102650.172227 fallocate 0x01e578 thru 0x021367 (0x2df0 bytes) 000759 1773102650.196983 trunc from 0x03c23e to 0x018de4 (0x23459 bytes) 000760 1773102650.236279 write 0x00fcc4 thru 0x012aef (0x2e2c bytes) 000761 1773102650.245930 punch from 0x005aeb to 0x0138f3 (0xde09 bytes) 000762 1773102650.258721 read 0x004d40 thru 0x005632 (0x8f3 bytes) 000763 1773102650.266036 fallocate 0x03c6f9 thru 0x03ffff (0x3907 bytes) 000764 1773102650.269635 fallocate 0x018780 thru 0x01f026 (0x68a7 bytes) 000765 1773102650.273168 write 0x02f602 thru 0x030df7 (0x17f6 bytes) 000766 1773102650.274575 read 0x0149a0 thru 0x02242c (0xda8d bytes) 000767 1773102650.280870 mapread 0x011761 thru 0x0184e6 (0x6d86 bytes) 000768 1773102650.282009 trunc from 0x030df8 to 0x002c06 (0x2e1f1 bytes) 000769 1773102650.296585 mapwrite 0x008e73 thru 0x01477a (0xb908 bytes) 000770 1773102650.376775 fallocate 0x038862 thru 0x03ffff (0x779e bytes) 000771 1773102650.381221 trunc from 0x040000 to 0x013ecc (0x2c133 bytes) 000772 1773102650.392462 write 0x0245ff thru 0x02a5af (0x5fb1 bytes) 000773 1773102650.393663 mapread 0x01e618 thru 0x023ac0 (0x54a9 bytes) 000774 1773102650.408618 punch from 0x019d41 to 0x01f871 (0x5b31 bytes) 000775 1773102650.410717 fallocate 0x027db1 thru 0x0362f5 (0xe545 bytes) 000776 1773102650.413841 trunc from 0x02a5b0 to 0x03b27a (0x10ccb bytes) 000777 1773102650.423120 trunc from 0x01cca9 to 0x0232f0 (0x6648 bytes) 000778 1773102650.426120 write 0x03e6dd thru 0x03ffff (0x1923 bytes) 000779 1773102650.429059 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3067001d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeea496e Mirror 3: 0xeea496e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1773102650.930946 trunc from 0x00a5ff to 0x00bb3b (0x153d bytes) 000783 1773102650.962175 read 0x03e7be thru 0x03ffff (0x1842 bytes) 000784 1773102650.966280 trunc from 0x0265f7 to 0x029d1e (0x3728 bytes) 000785 1773102650.969532 mapwrite 0x0311b4 thru 0x033b32 (0x297f bytes) 000787 1773102650.990314 trunc from 0x040000 to 0x03df49 (0x20b6 bytes) 000788 1773102651.025327 mapwrite 0x027dc5 thru 0x02ea94 (0x6cd0 bytes) 000789 1773102651.042432 mapwrite 0x027d0d thru 0x032c47 (0xaf3b bytes) 000790 1773102651.061241 trunc from 0x0182b8 to 0x01fa6a (0x77b3 bytes) 000791 1773102651.068599 MIRROR_RESYNC 3 mirrors 000792 1773102651.592008 mapwrite 0x0193fd thru 0x0285d1 (0xf1d5 bytes) 000793 1773102651.680375 trunc from 0x01b76e to 0x026fc0 (0xb853 bytes) 000794 1773102651.685155 trunc from 0x02798c to 0x03695b (0xefd0 bytes) 000795 1773102651.689092 trunc from 0x01b6c0 to 0x01c709 (0x104a bytes) 000796 1773102651.694911 read 0x03984d thru 0x03bb17 (0x22cb bytes) 000797 1773102651.702564 punch from 0x014784 to 0x01a273 (0x5af0 bytes) 000798 1773102651.707635 read 0x01766b thru 0x02745f (0xfdf5 bytes) 000799 1773102651.714748 trunc from 0x03cb2a to 0x03df49 (0x1420 bytes) 000800 1773102651.726097 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df4a): Mirror 3: 0x4655fc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df4a): Mirror 3: 0x4655fc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df4a: skipped 000801 1773102651.925525 fallocate 0x037db1 thru 0x03b822 (0x3a72 bytes) 000802 1773102651.929660 mapread 0x01f20f thru 0x024192 (0x4f84 bytes) 000803 1773102651.940348 mapwrite 0x02147a thru 0x02ed53 (0xd8da bytes) 000804 1773102651.961734 trunc from 0x03df4a to 0x015473 (0x28ad6 bytes) 000805 1773102651.989746 read 0x010c4d thru 0x015473 (0x4827 bytes) 000806 1773102651.996125 read 0x00fb33 thru 0x015473 (0x5941 bytes) 000807 1773102652.004180 trunc from 0x00cdcc to 0x015473 (0x86a8 bytes) 000808 1773102652.007726 mapread 0x0028e0 thru 0x0051fc (0x291d bytes) 000809 1773102652.008845 punch from 0x00c8da to 0x013e21 (0x7548 bytes) 000810 1773102652.014339 punch from 0x000283 to 0x00af31 (0xacaf bytes) 000811 1773102652.021579 MIRROR_RESYNC 2 mirrors 000812 1773102652.371299 mapread 0x000d73 thru 0x00c2d1 (0xb55f bytes) 000813 1773102652.388581 fallocate 0x01a13c thru 0x01ea14 (0x48d9 bytes) 000814 1773102652.412494 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000815 1773102652.497951 MIRROR_EXTEND to 2 mirrors 000816 1773102652.941386 mapwrite 0x01aa6b thru 0x0256ba (0xac50 bytes) 000817 1773102652.991782 trunc from 0x0256bb to 0x012ef0 (0x127ca bytes) 000820 1773102653.006507 mapwrite 0x03c2d1 thru 0x03ffff (0x3d2f bytes) 000821 1773102653.035255 MIRROR_RESYNC 2 mirrors 000822 1773102653.356999 trunc from 0x0020dc to 0x011a9e (0xf9c3 bytes) 000823 1773102653.386322 trunc from 0x040000 to 0x015692 (0x2a96d bytes) 000824 1773102653.415567 read 0x00fe30 thru 0x013893 (0x3a64 bytes) 000826 1773102653.426310 mapread 0x0140a2 thru 0x015692 (0x15f1 bytes) 000830 1773102653.436530 read 0x003f5b thru 0x011e09 (0xdeaf bytes) 000831 1773102653.443484 mapread 0x000d28 thru 0x002818 (0x1af1 bytes) 000833 1773102653.447914 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: 0x10154f75 (stale) CRC-32 checksum value for chunk [0, 0x15693): Mirror 1: 0x73bcede6 Mirror 3: 0x73bcede6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15693: skipped 000834 1773102653.835576 trunc from 0x002ff0 to 0x010d00 (0xdd11 bytes) 000835 1773102653.877045 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: 0x10154f75 (stale) Mirror 3: 0x73bcede6 (stale) CRC-32 checksum value for chunk [0, 0x15693): Mirror 1: 0x73bcede6 Mirror 4: 0x73bcede6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15693: skipped 000836 1773102654.398251 read 0x012dc1 thru 0x015692 (0x28d2 bytes) 000837 1773102654.405654 trunc from 0x015693 to 0x022a16 (0xd384 bytes) 000838 1773102654.440473 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22a17): Mirror 2: 0x90a0e41c (stale) Mirror 4: 0x6cbdb54e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a17): Mirror 2: 0x90a0e41c (stale) Mirror 4: 0x6cbdb54e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a17: skipped 000839 1773102654.587802 fallocate 0x0158d4 thru 0x022c87 (0xd3b4 bytes) 000842 1773102654.591670 punch from 0x013c55 to 0x01d50e (0x98ba bytes) 000843 1773102654.594950 fallocate 0x007e0b thru 0x01752a (0xf720 bytes) 000844 1773102654.597966 mapread 0x00aa7e thru 0x017176 (0xc6f9 bytes) 000845 1773102654.614045 punch from 0x00a1b4 to 0x0177a1 (0xd5ee bytes) 000846 1773102654.616615 mapwrite 0x038e1f thru 0x03c225 (0x3407 bytes) 000847 1773102654.643011 trunc from 0x028eb7 to 0x034e74 (0xbfbe bytes) 000849 1773102654.648141 mapread 0x021bc5 thru 0x02efdf (0xd41b bytes) 000850 1773102654.659264 mapwrite 0x0008d3 thru 0x0029c8 (0x20f6 bytes) 000851 1773102654.673743 trunc from 0x03c226 to 0x031fe1 (0xa244 bytes) 000852 1773102654.679512 trunc from 0x031fe2 to 0x002c07 (0x2f3da bytes) 000853 1773102654.686786 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c08): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c08): Mirror 2: 0x12267976 (stale) Mirror 4: 0x12267976 (stale) CRC-32 checksum value for chunk [0, 0x2c08): Mirror 1: 0xf101c7a2 Mirror 5: 0xf101c7a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c08: skipped 000854 1773102654.995937 mapwrite 0x010e1a thru 0x01a02b (0x9212 bytes) 000855 1773102655.039320 read 0x002af1 thru 0x00c053 (0x9563 bytes) 000856 1773102655.045575 read 0x00125d thru 0x0071a9 (0x5f4d bytes) 000857 1773102655.048395 trunc from 0x015955 to 0x018300 (0x29ac bytes) 000858 1773102655.052076 mapwrite 0x01889a thru 0x02839f (0xfb06 bytes) 000859 1773102655.083042 read 0x025d31 thru 0x02839f (0x266f bytes) 000860 1773102655.083533 write 0x03c353 thru 0x03ffff (0x3cad bytes) 000861 1773102655.084330 write 0x017b71 thru 0x01efb9 (0x7449 bytes) 000863 1773102655.092937 read 0x03d0d6 thru 0x03ffff (0x2f2a bytes) 000864 1773102655.093550 write 0x029123 thru 0x03507b (0xbf59 bytes) 000865 1773102655.095603 read 0x01f1ff thru 0x023227 (0x4029 bytes) 000866 1773102655.096308 punch from 0x014d0a to 0x01658b (0x1882 bytes) 000867 1773102655.100048 punch from 0x032794 to 0x03b3ce (0x8c3b bytes) 000868 1773102655.116378 write 0x0395e5 thru 0x03d285 (0x3ca1 bytes) 000870 1773102655.125010 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfbc9fe67 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0x8e66e952 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x28f2cebc Mirror 6: 0x28f2cebc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000871 1773102655.584981 mapread 0x005416 thru 0x00d9d5 (0x85c0 bytes) 000872 1773102655.599315 write 0x00e82d thru 0x01c791 (0xdf65 bytes) 000873 1773102655.622655 write 0x0301af thru 0x036456 (0x62a8 bytes) 000874 1773102655.630362 read 0x02fdbd thru 0x03e341 (0xe585 bytes) 000875 1773102655.635698 read 0x03e637 thru 0x03ffff (0x19c9 bytes) 000876 1773102655.636342 fallocate 0x02ebde thru 0x034ca0 (0x60c3 bytes) 000877 1773102655.639835 punch from 0x00b9d7 to 0x00c3dc (0xa06 bytes) 000879 1773102655.644336 fallocate 0x036429 thru 0x039504 (0x30dc bytes) 000880 1773102655.653579 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfbc9fe67 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0x28f2cebc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfbc9fe67 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0x28f2cebc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000881 1773102655.864775 trunc from 0x02c72f to 0x03555e (0x8e30 bytes) 000882 1773102655.867288 mapread 0x0384e6 thru 0x03ffff (0x7b1a bytes) 000883 1773102655.880956 punch from 0x037363 to 0x03ffff (0x8c9d bytes) 000884 1773102655.886196 trunc from 0x040000 to 0x0375ea (0x8a15 bytes) 000885 1773102655.907383 trunc from 0x00446e to 0x00b21b (0x6dae bytes) 000886 1773102655.910671 punch from 0x00ec9c to 0x0197aa (0xab0f bytes) 000887 1773102655.916595 trunc from 0x02dcaf to 0x03040d (0x275f bytes) 000889 1773102655.920309 punch from 0x0110d8 to 0x01da26 (0xc94f bytes) 000890 1773102655.923391 mapwrite 0x0111cf thru 0x01c7c5 (0xb5f7 bytes) 000891 1773102655.947801 trunc from 0x0375eb to 0x01d09b (0x1a54f bytes) 000892 1773102655.958574 read 0x002f84 thru 0x00f1f3 (0xc270 bytes) 000893 1773102655.963502 read 0x002ec5 thru 0x00fd82 (0xcebe bytes) 000894 1773102655.964047 trunc from 0x016b75 to 0x01d09b (0x6527 bytes) 000895 1773102655.967049 mapwrite 0x0254f8 thru 0x02af03 (0x5a0c bytes) 000896 1773102655.990871 fallocate 0x01c3c4 thru 0x01e431 (0x206e bytes) 000897 1773102655.994240 write 0x034d40 thru 0x03b7b5 (0x6a76 bytes) 000898 1773102655.995576 trunc from 0x0221ca to 0x02b4f6 (0x932d bytes) 000899 1773102655.999128 read 0x01f039 thru 0x022976 (0x393e bytes) 000900 1773102656.006992 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b7b6): Mirror 4: 0x4883c263 (stale) Mirror 6: 0x7112bdf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7b6): Mirror 4: 0x4883c263 (stale) Mirror 6: 0x7112bdf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7b6: skipped 000901 1773102656.196202 read 0x030dc2 thru 0x035593 (0x47d2 bytes) 000902 1773102656.206721 trunc from 0x03b7b6 to 0x026cba (0x14afb bytes) 000903 1773102656.223425 write 0x01536a thru 0x01dfac (0x8c43 bytes) 000904 1773102656.237212 fallocate 0x01c12a thru 0x02968b (0xd562 bytes) 000905 1773102656.244838 write 0x02a107 thru 0x038dfe (0xecf8 bytes) 000906 1773102656.248193 read 0x02131d thru 0x030049 (0xed2d bytes) 000908 1773102656.258117 mapread 0x0011bf thru 0x00a3d1 (0x9213 bytes) 000909 1773102656.259363 trunc from 0x03322b to 0x038dfe (0x5bd4 bytes) 000910 1773102656.271005 trunc from 0x038dff to 0x029178 (0xfc86 bytes) 000911 1773102656.281226 write 0x02363d thru 0x02a360 (0x6d24 bytes) 000912 1773102656.282719 punch from 0x010977 to 0x01283b (0x1ec5 bytes) 000914 1773102656.288940 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a361): Mirror 6: 0x73f786a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a361): Mirror 6: 0x73f786a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a361: skipped 000916 1773102656.472780 read 0x007058 thru 0x01393d (0xc8e6 bytes) 000917 1773102656.480209 write 0x029ae1 thru 0x0362ed (0xc80d bytes) 000919 1773102656.486527 mapread 0x02914f thru 0x0362ed (0xd19f bytes) 000920 1773102656.487435 trunc from 0x0362ee to 0x03cbe1 (0x68f4 bytes) 000921 1773102656.496739 fallocate 0x034fcf thru 0x03ffff (0xb031 bytes) 000922 1773102656.500088 trunc from 0x040000 to 0x001a98 (0x3e567 bytes) 000923 1773102656.510261 read 0x000426 thru 0x001a98 (0x1673 bytes) 000924 1773102656.514433 fallocate 0x000504 thru 0x00ef62 (0xea5f bytes) 000925 1773102656.520007 fallocate 0x034ee5 thru 0x0374eb (0x2607 bytes) 000927 1773102656.526063 fallocate 0x034be4 thru 0x0383db (0x37f8 bytes) 000928 1773102656.530756 mapwrite 0x02e25e thru 0x03c9e5 (0xe788 bytes) 000929 1773102656.559982 punch from 0x02e10f to 0x03260d (0x44ff bytes) 000930 1773102656.563723 fallocate 0x03f72e thru 0x03ffff (0x8d2 bytes) 000931 1773102656.570254 mapwrite 0x000289 thru 0x003d96 (0x3b0e bytes) 000932 1773102656.585975 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000933 1773102656.652351 mapread 0x01bf8a thru 0x02ba2e (0xfaa5 bytes) 000934 1773102656.671614 mapwrite 0x03105d thru 0x03d591 (0xc535 bytes) 000935 1773102656.704874 trunc from 0x025332 to 0x025bc1 (0x890 bytes) 000937 1773102656.711137 write 0x03251d thru 0x03e9c7 (0xc4ab bytes) 000938 1773102656.712584 punch from 0x02cedc to 0x032ab0 (0x5bd5 bytes) 000940 1773102656.721443 fallocate 0x0333db thru 0x03ffff (0xcc25 bytes) 000941 1773102656.725445 read 0x023593 thru 0x02b212 (0x7c80 bytes) 000942 1773102656.730262 write 0x00771e thru 0x00d0d7 (0x59ba bytes) 000943 1773102656.736655 trunc from 0x0380e4 to 0x03e9c7 (0x68e4 bytes) 000944 1773102656.739606 mapread 0x02a6c6 thru 0x035629 (0xaf64 bytes) 000945 1773102656.746663 write 0x0009f3 thru 0x008dc3 (0x83d1 bytes) 000946 1773102656.754217 trunc from 0x00a1a8 to 0x012aff (0x8958 bytes) 000947 1773102656.762569 fallocate 0x008f20 thru 0x00d5bc (0x469d bytes) 000948 1773102656.765950 mapread 0x020088 thru 0x021524 (0x149d bytes) 000949 1773102656.766713 punch from 0x030807 to 0x03a4bd (0x9cb7 bytes) 000950 1773102656.770864 mapread 0x01fd4c thru 0x022537 (0x27ec bytes) 000951 1773102656.771892 read 0x015039 thru 0x022418 (0xd3e0 bytes) 000952 1773102656.777770 mapwrite 0x03df05 thru 0x03ffff (0x20fb bytes) 000953 1773102656.800895 MIRROR_EXTEND to 2 mirrors 000954 1773102657.236541 trunc from 0x040000 to 0x01d206 (0x22df9 bytes) 000955 1773102657.278290 trunc from 0x003be3 to 0x0069cd (0x2deb bytes) 000956 1773102657.287531 punch from 0x00a537 to 0x00d733 (0x31fd bytes) 000958 1773102657.293567 read 0x00dc0d thru 0x013373 (0x5767 bytes) 000959 1773102657.302478 mapwrite 0x02c44f thru 0x03b537 (0xf0e9 bytes) 000960 1773102657.361471 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000961 1773102657.452198 mapread 0x009659 thru 0x00f64c (0x5ff4 bytes) 000962 1773102657.463712 mapwrite 0x01b796 thru 0x02b685 (0xfef0 bytes) 000963 1773102657.497248 punch from 0x00bfed to 0x01b582 (0xf596 bytes) 000965 1773102657.505762 trunc from 0x03b538 to 0x027561 (0x13fd6 bytes) 000966 1773102657.526820 trunc from 0x027562 to 0x013996 (0x13bcb bytes) 000967 1773102657.547243 mapwrite 0x037e35 thru 0x03f278 (0x7444 bytes) 000969 1773102657.580241 mapwrite 0x019c7c thru 0x023240 (0x95c5 bytes) 000970 1773102657.618146 write 0x02861d thru 0x033836 (0xb21a bytes) 000971 1773102657.619893 trunc from 0x03f279 to 0x010bd6 (0x2e6a2 bytes) 000972 1773102657.631526 trunc from 0x005cd4 to 0x007118 (0x1445 bytes) 000973 1773102657.636057 punch from 0x003297 to 0x006cdd (0x3a47 bytes) 000975 1773102657.643941 punch from 0x00b77e to 0x010bd6 (0x5459 bytes) 000977 1773102657.648803 write 0x014711 thru 0x01e7df (0xa0cf bytes) 000978 1773102657.649958 mapwrite 0x03ed6e thru 0x03ffff (0x1292 bytes) 000979 1773102657.682220 mapread 0x0162f6 thru 0x018a51 (0x275c bytes) 000980 1773102657.683025 punch from 0x02672c to 0x029f78 (0x384d bytes) 000981 1773102657.685490 read 0x02aab7 thru 0x02e9f3 (0x3f3d bytes) 000982 1773102657.698006 MIRROR_EXTEND to 2 mirrors 000983 1773102658.076845 punch from 0x020bd1 to 0x02dd0e (0xd13e bytes) 000984 1773102658.098169 mapread 0x00921b thru 0x0120fb (0x8ee1 bytes) 000985 1773102658.119443 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x306:0x0]: Invalid argument (22) 000986 1773102658.222912 mapwrite 0x02ca0e thru 0x035ab9 (0x90ac bytes) 000988 1773102658.249928 fallocate 0x03cf31 thru 0x03ffff (0x30cf bytes) 000989 1773102658.254517 punch from 0x0383cd to 0x03db15 (0x5749 bytes) 000990 1773102658.258559 trunc from 0x040000 to 0x00fe37 (0x301c8 bytes) 000991 1773102658.279964 mapwrite 0x021c4c thru 0x023e8d (0x2242 bytes) 000992 1773102658.302293 trunc from 0x019e28 to 0x023e8d (0xa066 bytes) 000994 1773102658.310394 mapread 0x014888 thru 0x019374 (0x4aed bytes) 000995 1773102658.321844 mapread 0x015ffe thru 0x0217f1 (0xb7f4 bytes) 000996 1773102658.337285 write 0x00684c thru 0x006de5 (0x59a bytes) 000997 1773102658.345174 punch from 0x00ad46 to 0x015263 (0xa51e bytes) 000998 1773102658.356041 fallocate 0x00f38f thru 0x0123f8 (0x306a bytes) 000999 1773102658.365003 trunc from 0x023e8e to 0x024766 (0x8d9 bytes) 001000 1773102658.383120 trunc from 0x020bf0 to 0x024766 (0x3b77 bytes) All operations completed A-OK! PASS 70a (38s) == sanity-flr test 71: check mirror extend parent fid ==== 20:31:01 (1773102661) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x345:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 72: check append on FLR file ========== 20:31:06 (1773102666) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0943789 s, 22.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554349 s, 18.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 20:31:09 (1773102669) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1101376 .. Extending file size to 400960 .. done Extending file size to 3414912 .. Extending file size to 2357984 .. Extending file size to 6079296 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5071680 .. Extending file size to 7617728 .. done Extending file size to 6898272 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8734720 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7508096 .. Extending file size to 8759968 .. Extending file size to 8471104 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9224672 .. Extending file size to 9017280 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9145952 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9300896 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 60605 60606 60607 60608 60609 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) 192.168.202.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:32:19 (1773102739) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6692224 .. Extending file size to 4058944 .. Extending file size to 7599072 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6061824 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6238240 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6667104 .. Extending file size to 8464736 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8868736 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9105376 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8620384 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8731680 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62757 62759 62760 62762 62764 done 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) 192.168.202.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:33:27 (1773102807) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2328 1268976 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18776 3588244 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24644 3582376 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43420 7170620 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2328 1268976 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18776 3588244 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24644 3582376 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43420 7170620 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0575165 s, 36.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0913732 s, 23.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbf045ea3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:33:36 (1773102816) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0496725 s, 42.2 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 20:33:39 (1773102819) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:33:43 (1773102823) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0541765 s, 38.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x447:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x445:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x446:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:33:47 (1773102827) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0632383 s, 33.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:33:52 (1773102832) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0448064 s, 46.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:34:19 (1773102859) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436964 s, 24.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:34:55 (1773102895) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x448:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x449:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.035273 s, 59.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x448:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x449:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x447:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x448:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x449:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x447:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:35:35 (1773102935) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg260-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg260-server: Pool lustre.test_204f created Adding targets to pool oleg260-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg260-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0374435 s, 56.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg260-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg260-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg260-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:36:25 (1773102985) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:36:28 (1773102988) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:36:32 (1773102992) oleg260-server: Pool lustre.test_206 created Adding targets to pool oleg260-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg260-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg260-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg260-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg260-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:36:48 (1773103008) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424474 s, 24.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307406 s, 34.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8c70a515 Mirror 3: 0x8c70a515 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:36:53 (1773103013) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:36:55 (1773103015) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:36:56 (1773103016) Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0319893 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1f40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2280000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2280000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2280000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2530000 Read this (4 bytes): BBBB 192.168.202.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg260-client.virtnet /mnt/lustre2 (opts:) PASS 209a (21s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:37:17 (1773103037) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0114727 s, 714 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:37:20 (1773103040) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177193 s, 59.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-09 20:37:21.000000000 -0400 Modify: 2026-03-09 20:37:21.000000000 -0400 Change: 2026-03-09 20:37:22.000000000 -0400 Birth: 2026-03-09 20:37:22.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:37:24 (1773103044) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260085 s, 40.3 MB/s oleg260-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg260-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg260-server: O/280000401/d31/bf: File not found by ext2_lookup PASS 210b (15s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:37:39 (1773103059) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.107174 s, 382 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137077 s, 299 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 9 20:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 9 20:37 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:37:43 (1773103063) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:37:46 (1773103066) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1071 sec =========== 20:37:50 (1773103070) === sanity-flr: start cleanup 20:37:51 (1773103071) === === sanity-flr: finish cleanup 20:37:55 (1773103075) ===