-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 13:32:48 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg448-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:33:10 (1788802390) === oleg448-client.virtnet: executing check_config_client /mnt/lustre oleg448-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg448-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa096c38dd800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa096c38dd800.idle_timeout=debug disable quota as required oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:33:55 (1788802435) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8234] == sanity-flr test 0a: lfs mirror create with -N option == 13:33:58 (1788802438) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:34:13 (1788802453) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:34:16 (1788802456) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:34:21 (1788802461) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:34:47 (1788802487) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:34:51 (1788802491) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 13:34:55 (1788802495) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa096c38dd800.stats=clear osc.lustre-OST0001-osc-ffffa096c38dd800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:35:05 (1788802505) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:35:17 (1788802517) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:35:26 (1788802526) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:35:30 (1788802530) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:35:39 (1788802539) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 13:35:49 (1788802549) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:35:56 (1788802556) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:36:05 (1788802565) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:36:14 (1788802574) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:36:27 (1788802587) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:36:37 (1788802597) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.706098 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114873 s, 9.1 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:36:48 (1788802608) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.574019 s, 23.7 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:36:59 (1788802619) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:37:12 (1788802632) Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (86s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:38:37 (1788802717) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.377625 s, 16.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:39:29 (1788802769) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (100s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:41:09 (1788802869) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.1726 s, 12.6 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 13:41 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.300s user 0m0.001s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa096c38dd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.183s user 0m0.001s sys 0m0.200s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.316s user 0m0.001s sys 0m0.171s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa096c38dd800.ost_server_uuid in FULL state after 0 sec PASS 33b (109s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:42:58 (1788802978) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:43:02 (1788802982) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:43:06 (1788802986) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 13:43:11 (1788802991) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 13:43:21 (1788803001) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.798933 s, 15.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:43:37 (1788803017) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706604 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624612 s, 16.8 MB/s PASS 36b (12s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:43:49 (1788803029) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:44:01 (1788803041) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127584 s, 803 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112899 s, 907 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000835103 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0018778 s, 545 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 13:44:14 (1788803054) create test files with size 15759993 .. e8935cab044d5f9ddf1a91209e8f7623 - 29cd043d50dd0a0d683684c477c41cd1 - 2c8160caf281defc96bd9e92229becaa - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 13:44:40 (1788803080) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:44:53 (1788803093) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:45:01 (1788803101) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12688 s, 8.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb2:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb3:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269212 s, 38.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 13:45:10 (1788803110) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106979 s, 19.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.29886 s, 14.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141799 s, 14.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 13:45:24 (1788803124) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 13:45:28 (1788803128) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:45:31 (1788803131) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269393 s, 38.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 13:45:39 (1788803139) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.343153 s, 9.2 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:45:51 (1788803151) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:46:02 (1788803162) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.279701 s, 37.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:46:15 (1788803175) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa096c808b000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.381159 s, 27.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa096d954c000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:46:30 (1788803190) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:47:10 (1788803230) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:47:47 (1788803267) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 13:47:50 (1788803270) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.344183 s, 9.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.182523 s, 17.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7801df86 (nosync,stale) Mirror 2: 0x7801df86 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 491f1620b3cf47a6dc42f48574bada80 - mirror 1 checksum: 72b70fbe043f96ea13ec0cc52bb571db - mirror 2 checksum: 491f1620b3cf47a6dc42f48574bada80 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T13:47:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T13:47:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 491f1620b3cf47a6dc42f48574bada80 - mirror 1 checksum: 72b70fbe043f96ea13ec0cc52bb571db - mirror 2 checksum: 72b70fbe043f96ea13ec0cc52bb571db - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T13:47:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T13:47:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:48:03 (1788803283) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526498 s, 19.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:48:11 (1788803291) Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 13:48:24 (1788803304) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server Failover mds1 to oleg448-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 13:48:48 (1788803328) targets are mounted 13:48:48 (1788803328) facet_failover done oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391456 s, 26.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:49:05 (1788803345) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0360583 s, 568 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0138083 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:49:14 (1788803354) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0635072 s, 322 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0098848 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:49:24 (1788803364) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0087762 s, 1.9 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:49:31 (1788803371) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0360142 s, 569 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:49:38 (1788803378) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00725198 s, 565 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.046743 s, 87.6 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:49:44 (1788803384) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:50:04 (1788803404) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:50:31 (1788803431) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:50:55 (1788803455) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:51:18 (1788803478) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 13:51:25 (1788803485) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57591 mirror split pid 57592 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57591 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57592 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 13:52:32 (1788803552) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 3554 000001 1788803554.648736 write 0x02376e thru 0x025782 (0x2015 bytes) 000002 1788803554.665560 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3364b 000003 1788803555.175603 trunc from 0x025783 to 0x03364a (0xdec8 bytes) 000004 1788803555.251130 mapwrite 0x014fa1 thru 0x0185c7 (0x3627 bytes) 000005 1788803555.308899 read 0x01cb3d thru 0x021a68 (0x4f2c bytes) 000006 1788803555.320909 mapwrite 0x02c904 thru 0x037e47 (0xb544 bytes) 000007 1788803555.382621 trunc from 0x0074be to 0x00e027 (0x6b6a bytes) 000008 1788803555.389113 write 0x026421 thru 0x02f91f (0x94ff bytes) 000009 1788803555.390802 mapread 0x0116fa thru 0x01ec95 (0xd59c bytes) 000011 1788803555.393659 mapwrite 0x02dc4c thru 0x03217a (0x452f bytes) 000012 1788803555.430621 MIRROR_RESYNC 2 mirrors 000013 1788803556.181107 write 0x01193e thru 0x01a89e (0x8f61 bytes) 000014 1788803556.234169 mapread 0x008579 thru 0x010747 (0x81cf bytes) 000015 1788803556.269355 MIRROR_RESYNC 2 mirrors 000017 1788803557.215156 write 0x0066c6 thru 0x010712 (0xa04d bytes) 000018 1788803557.275274 fallocate 0x013aa0 thru 0x0170fa (0x365b bytes) truncating to largest ever: 0x35e9d 000019 1788803557.283655 trunc from 0x037e48 to 0x035e9c (0x1fab bytes) fallocating to largest ever: 0x3921a 000021 1788803557.376047 fallocate 0x02f2fb thru 0x039219 (0x9f1f bytes) 000022 1788803557.388137 write 0x002402 thru 0x0039db (0x15da bytes) 000024 1788803557.419245 read 0x0236bf thru 0x032566 (0xeea8 bytes) 000025 1788803557.462262 mapwrite 0x00c7a7 thru 0x01bf98 (0xf7f2 bytes) 000026 1788803557.582529 trunc from 0x03921a to 0x0376e9 (0x1b30 bytes) 000027 1788803557.615370 write 0x01ae3a thru 0x01da6c (0x2c33 bytes) fallocating to largest ever: 0x40000 000028 1788803557.633245 fallocate 0x03bb09 thru 0x03ffff (0x44f7 bytes) 000029 1788803557.655748 mapwrite 0x011d51 thru 0x01a3cb (0x867b bytes) 000030 1788803557.704837 mapwrite 0x02f3d5 thru 0x03590a (0x6536 bytes) 000031 1788803557.761826 mapwrite 0x025f9f thru 0x0263cf (0x431 bytes) 000032 1788803557.789692 fallocate 0x03026d thru 0x0381cb (0x7f5f bytes) 000034 1788803557.806634 trunc from 0x0376ea to 0x0242ac (0x1343d bytes) 000035 1788803557.850738 write 0x024cb6 thru 0x02a10a (0x5455 bytes) 000036 1788803557.873789 MIRROR_RESYNC 2 mirrors 000037 1788803558.905819 fallocate 0x0120d8 thru 0x02013a (0xe063 bytes) 000038 1788803558.947157 read 0x01f5be thru 0x024706 (0x5149 bytes) 000039 1788803558.960672 fallocate 0x02c764 thru 0x039ec1 (0xd75e bytes) 000040 1788803558.972821 punch from 0x011264 to 0x01d75a (0xc4f7 bytes) 000042 1788803558.983120 read 0x018536 thru 0x018b9d (0x668 bytes) 000043 1788803559.018779 MIRROR_RESYNC 2 mirrors 000044 1788803559.667237 punch from 0x001e36 to 0x0059ef (0x3bba bytes) 000045 1788803559.692356 trunc from 0x02a10b to 0x02ecc5 (0x4bbb bytes) 000046 1788803559.717262 mapwrite 0x00a6ed thru 0x01a587 (0xfe9b bytes) 000047 1788803559.763124 trunc from 0x02ecc6 to 0x035bd1 (0x6f0c bytes) 000048 1788803559.777721 mapwrite 0x0388c8 thru 0x03a80b (0x1f44 bytes) 000049 1788803559.818776 trunc from 0x03a80c to 0x013d4e (0x26abd bytes) 000050 1788803559.835856 mapread 0x00b000 thru 0x013d4e (0x8d4f bytes) 000051 1788803559.836717 trunc from 0x008b0b to 0x013d4e (0xb244 bytes) 000052 1788803559.852632 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000053 1788803559.993991 mapwrite 0x030ed8 thru 0x039bd3 (0x8cfc bytes) 000055 1788803560.053984 mapwrite 0x018293 thru 0x024a3a (0xc7a8 bytes) 000056 1788803560.086184 trunc from 0x039bd4 to 0x014cad (0x24f26 bytes) 000058 1788803560.100222 mapwrite 0x00b9bf thru 0x015ff3 (0xa635 bytes) 000059 1788803560.158317 mapread 0x000c32 thru 0x00f0cf (0xe49e bytes) 000060 1788803560.166980 trunc from 0x00db4b to 0x0113a4 (0x385a bytes) 000061 1788803560.172495 mapread 0x00e19a thru 0x0106a8 (0x250f bytes) 000062 1788803560.180537 mapread 0x0143a0 thru 0x01553c (0x119d bytes) 000064 1788803560.182143 trunc from 0x00a471 to 0x015294 (0xae24 bytes) 000065 1788803560.187181 mapread 0x002d08 thru 0x0097d7 (0x6ad0 bytes) 000066 1788803560.189085 mapread 0x0128dc thru 0x015ff3 (0x3718 bytes) 000068 1788803560.208129 trunc from 0x015ff4 to 0x01a7c7 (0x47d4 bytes) 000069 1788803560.221974 punch from 0x000f0c to 0x00c5ab (0xb6a0 bytes) 000070 1788803560.227112 read 0x00e9b0 thru 0x015c92 (0x72e3 bytes) 000071 1788803560.228300 punch from 0x0190ea to 0x019d1c (0xc33 bytes) 000072 1788803560.234587 read 0x00f561 thru 0x0172ea (0x7d8a bytes) 000073 1788803560.243259 trunc from 0x01a7c8 to 0x016450 (0x4377 bytes) 000074 1788803560.256146 mapwrite 0x03bbbf thru 0x03ffff (0x4441 bytes) 000075 1788803560.291446 punch from 0x02abe6 to 0x0345c3 (0x99de bytes) 000076 1788803560.296254 write 0x03e307 thru 0x03ee87 (0xb81 bytes) 000077 1788803560.298262 punch from 0x0379ec to 0x03ffff (0x8614 bytes) 000078 1788803560.310779 mapread 0x00729e thru 0x0160da (0xee3d bytes) 000079 1788803560.313932 fallocate 0x0328cb thru 0x03929c (0x69d2 bytes) 000080 1788803560.319047 mapread 0x01b1d9 thru 0x01d09c (0x1ec4 bytes) 000081 1788803560.320692 trunc from 0x040000 to 0x02f33f (0x10cc0 bytes) 000084 1788803560.344345 mapread 0x01c6de thru 0x01e85f (0x2182 bytes) 000085 1788803560.346052 mapwrite 0x022d93 thru 0x02e362 (0xb5d0 bytes) 000086 1788803560.434788 punch from 0x004b92 to 0x00e42d (0x989c bytes) 000087 1788803560.441312 punch from 0x0294ae to 0x02b9fd (0x2550 bytes) 000088 1788803560.452505 mapwrite 0x035fd6 thru 0x03ffff (0xa02a bytes) 000089 1788803560.579111 mapread 0x00aa0f thru 0x0103bc (0x59ae bytes) 000091 1788803560.606708 punch from 0x01a7b7 to 0x029881 (0xf0cb bytes) 000092 1788803560.611659 trunc from 0x040000 to 0x03bf7a (0x4085 bytes) 000093 1788803560.639298 MIRROR_EXTEND to 2 mirrors 000094 1788803561.493030 MIRROR_RESYNC 2 mirrors 000095 1788803561.824272 fallocate 0x01aa3a thru 0x02701c (0xc5e3 bytes) 000096 1788803561.875595 punch from 0x022f96 to 0x027588 (0x45f3 bytes) 000097 1788803561.883039 trunc from 0x03bf7b to 0x000dc4 (0x3b1b6 bytes) 000098 1788803561.908894 mapwrite 0x013fc5 thru 0x021a3f (0xda7b bytes) 000099 1788803561.967330 mapwrite 0x035c61 thru 0x03a7af (0x4b4f bytes) 000100 1788803562.007580 fallocate 0x01e978 thru 0x028d27 (0xa3b0 bytes) 000101 1788803562.020207 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a7b0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7b0): Mirror 2: 0x772aadd4 (stale) CRC-32 checksum value for chunk [0, 0x3a7b0): Mirror 1: 0xc73cf97b Mirror 3: 0xc73cf97b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7b0: skipped 000102 1788803562.947141 read 0x011e48 thru 0x017a79 (0x5c32 bytes) 000104 1788803562.982038 write 0x01100d thru 0x011491 (0x485 bytes) 000105 1788803563.038104 mapwrite 0x03e9d9 thru 0x03fe5e (0x1486 bytes) 000106 1788803563.083258 MIRROR_RESYNC 3 mirrors 000107 1788803564.388597 write 0x00b6be thru 0x00dabc (0x23ff bytes) 000108 1788803564.483876 read 0x01ce52 thru 0x024e29 (0x7fd8 bytes) 000109 1788803564.494442 fallocate 0x029a9d thru 0x037e59 (0xe3bd bytes) 000110 1788803564.501970 fallocate 0x034d5b thru 0x03ffff (0xb2a5 bytes) 000111 1788803564.510631 read 0x0312ea thru 0x03d2e8 (0xbfff bytes) 000112 1788803564.532563 trunc from 0x010778 to 0x01f298 (0xeb21 bytes) 000113 1788803564.549896 mapread 0x00e779 thru 0x00feca (0x1752 bytes) 000114 1788803564.551836 fallocate 0x03ad15 thru 0x03b588 (0x874 bytes) 000115 1788803564.559138 trunc from 0x03fe5f to 0x02326d (0x1cbf1 bytes) 000116 1788803564.626926 write 0x02fa42 thru 0x0321f0 (0x27af bytes) 000117 1788803564.628116 write 0x007eb4 thru 0x0148d5 (0xca22 bytes) 000118 1788803564.635217 punch from 0x01eab5 to 0x026160 (0x76ac bytes) 000119 1788803564.640673 trunc from 0x0321f1 to 0x01d7f5 (0x149fb bytes) 000120 1788803564.659423 fallocate 0x0209a1 thru 0x02dee0 (0xd540 bytes) 000121 1788803564.665703 trunc from 0x026fe7 to 0x02dee0 (0x6efa bytes) 000122 1788803564.673486 read 0x01980c thru 0x02245d (0x8c52 bytes) 000123 1788803564.689549 mapwrite 0x00e088 thru 0x0114cc (0x3445 bytes) 000124 1788803564.706369 trunc from 0x015961 to 0x0211a5 (0xb845 bytes) 000125 1788803564.711929 punch from 0x011735 to 0x016dfa (0x56c6 bytes) 000126 1788803564.734627 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dee1): Mirror 2: 0xaa02eb50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dee1): Mirror 2: 0xaa02eb50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dee1: skipped 000127 1788803565.068062 mapread 0x002503 thru 0x00baf6 (0x95f4 bytes) 000128 1788803565.087874 fallocate 0x006b9e thru 0x008f5d (0x23c0 bytes) 000129 1788803565.095537 read 0x0242d0 thru 0x02dee0 (0x9c11 bytes) 000130 1788803565.113938 fallocate 0x00b5cb thru 0x00bf52 (0x988 bytes) 000132 1788803565.119410 fallocate 0x00c81e thru 0x014e61 (0x8644 bytes) 000133 1788803565.125628 write 0x00c805 thru 0x01b841 (0xf03d bytes) 000134 1788803565.141052 read 0x01ab8e thru 0x01b889 (0xcfc bytes) 000135 1788803565.142061 trunc from 0x02abc6 to 0x02dee0 (0x331b bytes) 000136 1788803565.148134 trunc from 0x02dee1 to 0x027325 (0x6bbb bytes) 000137 1788803565.166980 write 0x01b0c9 thru 0x01ce59 (0x1d91 bytes) 000139 1788803565.188613 trunc from 0x027326 to 0x02ed50 (0x7a2b bytes) 000140 1788803565.211839 mapread 0x02b76b thru 0x02e32a (0x2bc0 bytes) 000141 1788803565.221441 punch from 0x0211de to 0x023b14 (0x2937 bytes) 000142 1788803565.225938 trunc from 0x026d8b to 0x02ec43 (0x7eb9 bytes) 000143 1788803565.229967 read 0x02491d thru 0x02ed50 (0xa434 bytes) 000144 1788803565.238878 punch from 0x02dcb4 to 0x02ed50 (0x109d bytes) 000145 1788803565.242626 write 0x02b4bb thru 0x03470d (0x9253 bytes) 000146 1788803565.244940 punch from 0x000647 to 0x00c1d8 (0xbb92 bytes) 000147 1788803565.259565 fallocate 0x032d76 thru 0x03ba1b (0x8ca6 bytes) 000148 1788803565.264028 mapread 0x005f4e thru 0x00f0d7 (0x918a bytes) 000149 1788803565.299482 fallocate 0x021bba thru 0x022dde (0x1225 bytes) 000150 1788803565.306019 mapread 0x013ce4 thru 0x01e66b (0xa988 bytes) 000151 1788803565.311626 MIRROR_RESYNC 2 mirrors 000152 1788803566.057538 mapread 0x0095fd thru 0x01597d (0xc381 bytes) 000153 1788803566.106062 MIRROR_RESYNC 2 mirrors 000154 1788803566.428278 fallocate 0x003dd3 thru 0x00c165 (0x8393 bytes) 000156 1788803566.469676 trunc from 0x029b56 to 0x0352a0 (0xb74b bytes) 000157 1788803566.474619 write 0x0233ab thru 0x02a907 (0x755d bytes) 000158 1788803566.488068 write 0x021798 thru 0x02c48c (0xacf5 bytes) 000159 1788803566.501163 punch from 0x01fec8 to 0x02cc1d (0xcd56 bytes) 000160 1788803566.517257 trunc from 0x02503b to 0x02530a (0x2d0 bytes) 000161 1788803566.528180 fallocate 0x0034b8 thru 0x0064ff (0x3048 bytes) 000162 1788803566.534355 fallocate 0x02022b thru 0x0235f0 (0x33c6 bytes) 000163 1788803566.558244 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba1c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba1c): Mirror 2: 0x7684c74b (stale) CRC-32 checksum value for chunk [0, 0x3ba1c): Mirror 1: 0x8c90d94c Mirror 3: 0x8c90d94c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba1c: skipped 000164 1788803567.537980 mapread 0x030c87 thru 0x03ba1b (0xad95 bytes) 000165 1788803567.607951 read 0x01c5de thru 0x02a0c0 (0xdae3 bytes) 000166 1788803567.638199 mapwrite 0x012b69 thru 0x01397e (0xe16 bytes) 000168 1788803567.743917 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba1c): Mirror 3: 0x8c90d94c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba1c): Mirror 3: 0x8c90d94c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba1c: skipped 000169 1788803567.997004 trunc from 0x03ba1c to 0x005fff (0x35a1c bytes) 000170 1788803568.015961 punch from 0x004fa2 to 0x005fff (0x105e bytes) 000171 1788803568.021315 punch from 0x005e0b to 0x005fff (0x1f5 bytes) 000172 1788803568.029222 MIRROR_RESYNC 2 mirrors 000173 1788803568.539503 trunc from 0x0013d0 to 0x005fff (0x4c30 bytes) 000175 1788803568.564084 mapread 0x004998 thru 0x005fff (0x1668 bytes) 000176 1788803568.573441 trunc from 0x006000 to 0x02960e (0x2360f bytes) 000177 1788803568.592314 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000178 1788803568.722642 mapwrite 0x01cb0c thru 0x02b6b2 (0xeba7 bytes) 000180 1788803568.788894 mapread 0x028ec7 thru 0x02b6b2 (0x27ec bytes) 000181 1788803568.789778 trunc from 0x02b6b3 to 0x01edd8 (0xc8da bytes) 000184 1788803568.807055 mapwrite 0x00928b thru 0x00b703 (0x2479 bytes) 000185 1788803568.836638 read 0x018d09 thru 0x01edd8 (0x60d0 bytes) 000186 1788803568.837890 punch from 0x019889 to 0x01e39e (0x4b16 bytes) 000187 1788803568.845816 mapwrite 0x01f72d thru 0x02f4aa (0xfd7e bytes) 000188 1788803568.919941 punch from 0x007829 to 0x016008 (0xe7e0 bytes) 000189 1788803568.924062 mapwrite 0x014d9f thru 0x01788b (0x2aed bytes) 000190 1788803568.951451 trunc from 0x00e745 to 0x016b89 (0x8445 bytes) 000191 1788803568.958652 write 0x02f521 thru 0x03bfc4 (0xcaa4 bytes) 000192 1788803568.960704 read 0x023c2c thru 0x024f43 (0x1318 bytes) 000193 1788803568.962272 punch from 0x0192d7 to 0x0231aa (0x9ed4 bytes) 000194 1788803568.967429 trunc from 0x03bfc5 to 0x021e5c (0x1a168 bytes) 000195 1788803568.994432 mapread 0x00e0f0 thru 0x01a742 (0xc653 bytes) 000196 1788803569.016268 write 0x017473 thru 0x019083 (0x1c11 bytes) 000197 1788803569.017496 read 0x019942 thru 0x021e5c (0x851b bytes) 000198 1788803569.019354 trunc from 0x00b078 to 0x00cd75 (0x1cfe bytes) 000199 1788803569.028280 write 0x0201a1 thru 0x02f18e (0xefee bytes) 000200 1788803569.030064 write 0x011f39 thru 0x0169ed (0x4ab5 bytes) 000201 1788803569.033186 write 0x03e302 thru 0x03fe9c (0x1b9b bytes) 000202 1788803569.038758 trunc from 0x03fe9d to 0x007e94 (0x38008 bytes) 000203 1788803569.060357 punch from 0x005713 to 0x007e94 (0x2782 bytes) 000204 1788803569.065158 punch from 0x001577 to 0x007e94 (0x691e bytes) 000205 1788803569.069402 punch from 0x001f70 to 0x0069d0 (0x4a61 bytes) 000206 1788803569.075564 write 0x00b774 thru 0x01a79a (0xf027 bytes) 000207 1788803569.078047 punch from 0x019df3 to 0x01a79a (0x9a8 bytes) 000209 1788803569.095424 write 0x00f972 thru 0x01c1e9 (0xc878 bytes) 000210 1788803569.098058 mapread 0x019177 thru 0x01c1e9 (0x3073 bytes) 000211 1788803569.104048 MIRROR_EXTEND to 2 mirrors 000212 1788803569.748365 fallocate 0x007637 thru 0x010cf3 (0x96bd bytes) 000214 1788803569.792242 trunc from 0x01c1ea to 0x036c03 (0x1aa1a bytes) 000215 1788803569.857087 write 0x022c5c thru 0x027aef (0x4e94 bytes) 000217 1788803569.868551 trunc from 0x0009a2 to 0x00ab6c (0xa1cb bytes) 000218 1788803569.880321 MIRROR_RESYNC 2 mirrors 000219 1788803570.582563 read 0x01c7b9 thru 0x022250 (0x5a98 bytes) 000220 1788803570.608696 write 0x02fd14 thru 0x03173c (0x1a29 bytes) 000221 1788803570.643697 mapwrite 0x025639 thru 0x02be19 (0x67e1 bytes) 000222 1788803570.715233 mapwrite 0x02191b thru 0x02d3ca (0xbab0 bytes) 000223 1788803570.771352 read 0x0338d3 thru 0x0346e8 (0xe16 bytes) 000224 1788803570.801242 write 0x00d620 thru 0x01b46c (0xde4d bytes) 000225 1788803570.805189 trunc from 0x00b8ad to 0x019298 (0xd9ec bytes) 000226 1788803570.841488 trunc from 0x0320f3 to 0x032345 (0x253 bytes) 000228 1788803570.852247 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36c04): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c04): Mirror 2: 0xad34c1f7 (stale) CRC-32 checksum value for chunk [0, 0x36c04): Mirror 1: 0x115bf8fd Mirror 3: 0x115bf8fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c04: skipped 000229 1788803572.140024 MIRROR_RESYNC 3 mirrors 000232 1788803572.956998 mapwrite 0x022655 thru 0x02acd3 (0x867f bytes) 000233 1788803573.059052 read 0x0170f6 thru 0x0172bb (0x1c6 bytes) 000235 1788803573.066322 punch from 0x007c19 to 0x009f19 (0x2301 bytes) 000236 1788803573.079021 MIRROR_RESYNC 3 mirrors 000237 1788803574.089902 punch from 0x027ced to 0x02feaf (0x81c3 bytes) 000238 1788803574.130032 punch from 0x00fbfc to 0x01bffc (0xc401 bytes) 000239 1788803574.136952 mapwrite 0x014ca2 thru 0x016bb7 (0x1f16 bytes) 000240 1788803574.188595 fallocate 0x020100 thru 0x0239f9 (0x38fa bytes) 000241 1788803574.206122 trunc from 0x036c04 to 0x010d68 (0x25e9b bytes) 000242 1788803574.273440 punch from 0x00e554 to 0x010d68 (0x2815 bytes) 000243 1788803574.280827 trunc from 0x0064eb to 0x0107f3 (0xa309 bytes) 000244 1788803574.297669 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10d69): Mirror 2: 0x1ad4fced (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d69): Mirror 2: 0x1ad4fced (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d69: skipped 000246 1788803574.700693 punch from 0x009765 to 0x00d51c (0x3db8 bytes) 000247 1788803574.720022 trunc from 0x00f59e to 0x010d68 (0x17cb bytes) 000248 1788803574.727239 fallocate 0x028c2a thru 0x0308bc (0x7c93 bytes) 000249 1788803574.741926 mapwrite 0x019b75 thru 0x028921 (0xedad bytes) 000250 1788803574.855971 mapread 0x00ba8d thru 0x00cae8 (0x105c bytes) 000251 1788803574.867597 trunc from 0x028922 to 0x006932 (0x21fef bytes) 000252 1788803574.909440 fallocate 0x006687 thru 0x013a9b (0xd415 bytes) 000253 1788803574.918347 mapread 0x00c8c1 thru 0x0105a4 (0x3ce4 bytes) 000254 1788803574.945346 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000255 1788803575.070281 mapwrite 0x01f6db thru 0x020794 (0x10ba bytes) 000256 1788803575.109295 punch from 0x01ad19 to 0x01f172 (0x445a bytes) 000258 1788803575.114886 trunc from 0x00cd5e to 0x0134dc (0x677f bytes) 000259 1788803575.122305 read 0x01f1da thru 0x020794 (0x15bb bytes) 000260 1788803575.134006 MIRROR_EXTEND to 2 mirrors 000261 1788803575.819553 trunc from 0x007910 to 0x01293c (0xb02d bytes) 000262 1788803575.863107 read 0x01474c thru 0x019e93 (0x5748 bytes) 000264 1788803575.873525 read 0x002068 thru 0x00dbec (0xbb85 bytes) 000265 1788803575.886946 mapwrite 0x03b8df thru 0x03ffff (0x4721 bytes) 000266 1788803575.954973 trunc from 0x040000 to 0x028b1a (0x174e5 bytes) 000267 1788803575.969877 mapread 0x014b51 thru 0x019268 (0x4718 bytes) 000268 1788803575.971248 trunc from 0x027bb1 to 0x028b1a (0xf6a bytes) 000269 1788803575.975154 fallocate 0x028a57 thru 0x02fa67 (0x7011 bytes) 000270 1788803575.978967 fallocate 0x00e461 thru 0x014d29 (0x68c9 bytes) 000271 1788803575.989133 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000272 1788803576.129315 fallocate 0x017b6d thru 0x01bf1e (0x43b2 bytes) 000273 1788803576.134093 trunc from 0x02fa68 to 0x032b66 (0x30ff bytes) 000274 1788803576.161427 trunc from 0x00ca16 to 0x0107d4 (0x3dbf bytes) 000275 1788803576.168446 fallocate 0x008f0b thru 0x014f11 (0xc007 bytes) 000276 1788803576.177801 mapread 0x009f9a thru 0x00f6b6 (0x571d bytes) 000277 1788803576.209375 write 0x03c95b thru 0x03ffff (0x36a5 bytes) 000278 1788803576.213830 mapread 0x026ec9 thru 0x02860f (0x1747 bytes) 000279 1788803576.234556 mapread 0x026f40 thru 0x02a5ad (0x366e bytes) 000280 1788803576.235495 mapwrite 0x023f5b thru 0x028f73 (0x5019 bytes) 000281 1788803576.254823 write 0x00495b thru 0x00c6c2 (0x7d68 bytes) 000282 1788803576.257281 fallocate 0x028826 thru 0x02c9f6 (0x41d1 bytes) 000283 1788803576.263001 read 0x01aac3 thru 0x02125b (0x6799 bytes) 000284 1788803576.273426 trunc from 0x040000 to 0x00a39c (0x35c63 bytes) 000285 1788803576.294262 read 0x0012cf thru 0x003028 (0x1d5a bytes) 000286 1788803576.302192 read 0x00334d thru 0x00a39c (0x7050 bytes) 000287 1788803576.302815 mapwrite 0x002d0a thru 0x004fd7 (0x22ce bytes) 000288 1788803576.321783 punch from 0x004bb1 to 0x009593 (0x49e3 bytes) 000289 1788803576.327601 mapread 0x007f59 thru 0x00a39c (0x2444 bytes) 000290 1788803576.337210 trunc from 0x00a39d to 0x03771a (0x2d37e bytes) 000294 1788803576.357440 fallocate 0x02f05e thru 0x03134b (0x22ee bytes) 000295 1788803576.363928 write 0x014825 thru 0x021bdb (0xd3b7 bytes) 000296 1788803576.376173 mapread 0x019e6c thru 0x021f57 (0x80ec bytes) 000298 1788803576.378421 fallocate 0x004b7a thru 0x00eafd (0x9f84 bytes) 000299 1788803576.383426 mapread 0x02de92 thru 0x036acd (0x8c3c bytes) 000300 1788803576.384756 mapread 0x00c05b thru 0x01311d (0x70c3 bytes) 000301 1788803576.386426 mapwrite 0x01c123 thru 0x0225a1 (0x647f bytes) 000304 1788803576.404304 fallocate 0x01d54f thru 0x023f44 (0x69f6 bytes) 000305 1788803576.409431 write 0x026753 thru 0x028eaa (0x2758 bytes) 000309 1788803576.414338 trunc from 0x01ec76 to 0x021071 (0x23fc bytes) 000311 1788803576.419938 write 0x0141b1 thru 0x01c4a9 (0x82f9 bytes) 000312 1788803576.421812 read 0x00effe thru 0x01e3dd (0xf3e0 bytes) 000313 1788803576.428210 punch from 0x03287c to 0x03771a (0x4e9f bytes) 000314 1788803576.433309 trunc from 0x03771b to 0x025b6b (0x11baf bytes) 000315 1788803576.451515 mapread 0x019f34 thru 0x01c25a (0x2327 bytes) 000316 1788803576.452890 mapread 0x012fac thru 0x01d5db (0xa630 bytes) 000317 1788803576.456938 mapread 0x024c06 thru 0x025b6b (0xf66 bytes) 000318 1788803576.457936 read 0x001838 thru 0x008daa (0x7573 bytes) 000319 1788803576.458517 read 0x0007cc thru 0x00c0c5 (0xb8fa bytes) 000320 1788803576.459045 fallocate 0x02dc09 thru 0x02e514 (0x90c bytes) 000321 1788803576.464830 fallocate 0x01507d thru 0x01ec51 (0x9bd5 bytes) 000322 1788803576.470596 mapwrite 0x009149 thru 0x014158 (0xb010 bytes) 000323 1788803576.490313 trunc from 0x02e515 to 0x00ede1 (0x1f733 bytes) 000324 1788803576.515035 mapwrite 0x0232e0 thru 0x02421d (0xf3e bytes) 000325 1788803576.622837 trunc from 0x02421e to 0x009744 (0x1aad9 bytes) 000326 1788803576.645302 mapread 0x007af4 thru 0x009744 (0x1c51 bytes) 000327 1788803576.646215 read 0x00451c thru 0x009744 (0x5229 bytes) 000328 1788803576.646794 trunc from 0x0000f4 to 0x009744 (0x9651 bytes) 000329 1788803576.653083 mapwrite 0x03b1fd thru 0x03ca19 (0x181d bytes) 000330 1788803576.704352 MIRROR_EXTEND to 2 mirrors 000331 1788803577.395891 MIRROR_RESYNC 2 mirrors 000332 1788803577.702760 trunc from 0x03b572 to 0x03ca19 (0x14a8 bytes) 000333 1788803577.730846 write 0x01e33b thru 0x020923 (0x25e9 bytes) 000334 1788803577.745936 mapwrite 0x02988d thru 0x02a23a (0x9ae bytes) 000335 1788803577.767977 mapread 0x0016fa thru 0x00b0e4 (0x99eb bytes) 000336 1788803577.777332 read 0x003f85 thru 0x00562e (0x16aa bytes) 000337 1788803577.778068 fallocate 0x017457 thru 0x01efda (0x7b84 bytes) 000338 1788803577.782330 punch from 0x0371a5 to 0x037899 (0x6f5 bytes) 000339 1788803577.786154 mapread 0x0054db thru 0x00b662 (0x6188 bytes) 000340 1788803577.787403 write 0x03fa6a thru 0x03ffff (0x596 bytes) 000341 1788803577.788633 punch from 0x009051 to 0x0104f5 (0x74a5 bytes) 000342 1788803577.794912 trunc from 0x040000 to 0x02b1b0 (0x14e4f bytes) 000343 1788803577.828764 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000344 1788803577.999184 trunc from 0x02b1b1 to 0x0256c1 (0x5aef bytes) 000345 1788803578.021137 trunc from 0x0256c2 to 0x017eb1 (0xd810 bytes) 000347 1788803578.043871 punch from 0x00468e to 0x009bfb (0x556e bytes) 000348 1788803578.048263 mapread 0x0079fd thru 0x007b75 (0x179 bytes) 000349 1788803578.055821 read 0x004434 thru 0x013240 (0xee0d bytes) 000351 1788803578.066201 write 0x024392 thru 0x028ba7 (0x4816 bytes) 000352 1788803578.067444 trunc from 0x017673 to 0x026eb7 (0xf845 bytes) 000353 1788803578.081340 mapwrite 0x00208f thru 0x006d07 (0x4c79 bytes) 000354 1788803578.106364 mapread 0x005509 thru 0x00768b (0x2183 bytes) 000355 1788803578.110324 MIRROR_EXTEND to 2 mirrors 000356 1788803578.813199 trunc from 0x028ba8 to 0x028d07 (0x160 bytes) 000357 1788803578.894921 trunc from 0x028d08 to 0x03e11d (0x15416 bytes) 000358 1788803578.935668 write 0x010dc2 thru 0x019d9c (0x8fdb bytes) 000359 1788803578.955557 trunc from 0x03e11e to 0x004b48 (0x395d5 bytes) 000360 1788803578.977768 trunc from 0x004b49 to 0x000feb (0x3b5d bytes) 000361 1788803578.998334 read 0x000d54 thru 0x000feb (0x298 bytes) 000362 1788803579.004763 fallocate 0x00da25 thru 0x019f11 (0xc4ed bytes) 000363 1788803579.010094 punch from 0x006f1a to 0x010539 (0x9620 bytes) 000364 1788803579.018679 trunc from 0x019f12 to 0x0065b1 (0x13960 bytes) 000365 1788803579.033198 write 0x00e8c9 thru 0x0112df (0x2a17 bytes) 000366 1788803579.034704 mapread 0x002de0 thru 0x010cd2 (0xdef3 bytes) 000367 1788803579.052403 trunc from 0x00a030 to 0x00fa6d (0x5a3e bytes) 000368 1788803579.065636 MIRROR_RESYNC 2 mirrors 000369 1788803579.556015 read 0x0087c2 thru 0x0112df (0x8b1e bytes) 000370 1788803579.572797 trunc from 0x0112e0 to 0x036333 (0x25054 bytes) 000371 1788803579.642034 trunc from 0x00177b to 0x00355e (0x1de4 bytes) 000372 1788803579.649918 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36334): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36334): Mirror 2: 0xa96ae0d2 (stale) CRC-32 checksum value for chunk [0, 0x36334): Mirror 1: 0xa96ae0d2 Mirror 3: 0xa96ae0d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36334: skipped 000373 1788803580.625200 trunc from 0x036334 to 0x02f9fa (0x6939 bytes) 000374 1788803580.664040 trunc from 0x013a39 to 0x015d2e (0x22f6 bytes) 000375 1788803580.670149 punch from 0x000d44 to 0x0104b6 (0xf773 bytes) 000376 1788803580.675746 read 0x0197ca thru 0x01ebaf (0x53e6 bytes) 000377 1788803580.684808 trunc from 0x02f9fb to 0x034502 (0x4b08 bytes) 000378 1788803580.704426 mapread 0x02e23d thru 0x034502 (0x62c6 bytes) 000379 1788803580.721612 write 0x0119b3 thru 0x01634f (0x499d bytes) 000380 1788803580.729915 mapread 0x01aa88 thru 0x02a491 (0xfa0a bytes) 000381 1788803580.741919 trunc from 0x034503 to 0x0362d5 (0x1dd3 bytes) 000382 1788803580.761311 MIRROR_RESYNC 3 mirrors 000383 1788803581.578438 MIRROR_EXTEND to 4 mirrors 000384 1788803582.365619 trunc from 0x02149b to 0x02a9e0 (0x9546 bytes) 000386 1788803582.416954 mapwrite 0x03f321 thru 0x03ffff (0xcdf bytes) 000387 1788803582.463544 mapread 0x0233e3 thru 0x02ee00 (0xba1e bytes) 000388 1788803582.510726 mapwrite 0x036c38 thru 0x03c701 (0x5aca bytes) 000389 1788803582.561818 trunc from 0x015e73 to 0x01f9a8 (0x9b36 bytes) 000390 1788803582.574201 trunc from 0x036c6d to 0x03ffff (0x9393 bytes) 000391 1788803582.582117 punch from 0x004e8f to 0x00f836 (0xa9a8 bytes) 000392 1788803582.593253 read 0x00967a thru 0x0154e9 (0xbe70 bytes) 000393 1788803582.603333 trunc from 0x040000 to 0x03ee59 (0x11a6 bytes) 000394 1788803582.629233 read 0x02428e thru 0x03393e (0xf6b1 bytes) 000395 1788803582.630407 trunc from 0x03ee5a to 0x0248d0 (0x1a589 bytes) 000396 1788803582.644500 punch from 0x01553d to 0x01c049 (0x6b0d bytes) 000397 1788803582.653400 write 0x0100a8 thru 0x01a5c9 (0xa522 bytes) 000399 1788803582.670636 mapread 0x000781 thru 0x00682d (0x60ad bytes) 000400 1788803582.672076 mapwrite 0x006d8d thru 0x00d54e (0x67c2 bytes) 000401 1788803582.701174 trunc from 0x005ba8 to 0x0142b8 (0xe711 bytes) 000402 1788803582.708373 mapwrite 0x00e790 thru 0x016bca (0x843b bytes) 000403 1788803582.824200 write 0x0015ee thru 0x008919 (0x732c bytes) 000404 1788803582.833673 punch from 0x01e006 to 0x0248d0 (0x68cb bytes) 000405 1788803582.845137 read 0x00ebe2 thru 0x013ae6 (0x4f05 bytes) 000406 1788803582.846330 write 0x03579d thru 0x036fe7 (0x184b bytes) 000407 1788803582.850521 mapwrite 0x03c040 thru 0x03d310 (0x12d1 bytes) 000408 1788803582.920385 punch from 0x013329 to 0x0163e6 (0x30be bytes) 000409 1788803582.927723 mapread 0x013051 thru 0x02168c (0xe63c bytes) 000410 1788803582.983242 punch from 0x0364ef to 0x03d310 (0x6e22 bytes) 000411 1788803582.997216 read 0x03bab5 thru 0x03d310 (0x185c bytes) 000412 1788803583.010328 fallocate 0x022763 thru 0x032163 (0xfa01 bytes) 000413 1788803583.017750 read 0x025201 thru 0x031636 (0xc436 bytes) 000414 1788803583.018974 trunc from 0x03d311 to 0x03f37b (0x206b bytes) 000415 1788803583.047206 write 0x0390e0 thru 0x03ffff (0x6f20 bytes) 000416 1788803583.051152 fallocate 0x03020e thru 0x03b251 (0xb044 bytes) 000417 1788803583.060602 trunc from 0x040000 to 0x00868e (0x37971 bytes) 000418 1788803583.090999 mapwrite 0x039a19 thru 0x03ffff (0x65e7 bytes) 000419 1788803583.196278 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9e562cc7 (stale) Mirror 3: 0x9e562cc7 (stale) Mirror 4: 0x9e562cc7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe372d1a8 Mirror 5: 0xe372d1a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000420 1788803584.310909 MIRROR_RESYNC 5 mirrors 000421 1788803585.225083 punch from 0x023f48 to 0x032ae0 (0xeb99 bytes) 000422 1788803585.279205 mapread 0x009c6a thru 0x00d9ef (0x3d86 bytes) 000423 1788803585.300666 trunc from 0x040000 to 0x03a1db (0x5e24 bytes) 000424 1788803585.356063 mapread 0x038fe9 thru 0x03a1db (0x11f3 bytes) 000425 1788803585.365412 mapwrite 0x036611 thru 0x03ffff (0x99ef bytes) 000426 1788803585.405145 punch from 0x01de09 to 0x028e42 (0xb03a bytes) 000427 1788803585.410271 mapwrite 0x01d688 thru 0x02b6dd (0xe056 bytes) 000428 1788803585.453200 mapread 0x006735 thru 0x0114b2 (0xad7e bytes) 000429 1788803585.459631 MIRROR_RESYNC 5 mirrors 000431 1788803586.599918 mapwrite 0x0250d9 thru 0x02bc64 (0x6b8c bytes) 000432 1788803586.724495 write 0x03a023 thru 0x03ffff (0x5fdd bytes) 000433 1788803586.731756 mapwrite 0x00eb07 thru 0x019dc9 (0xb2c3 bytes) 000434 1788803586.809810 trunc from 0x040000 to 0x005a8b (0x3a574 bytes) 000435 1788803586.858863 mapwrite 0x0015d3 thru 0x00437a (0x2da8 bytes) 000436 1788803586.905228 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5a8c): Mirror 3: 0xe0636edc (stale) Mirror 4: 0xe0636edc (stale) Mirror 5: 0xe0636edc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5a8c): Mirror 3: 0xe0636edc (stale) Mirror 4: 0xe0636edc (stale) Mirror 5: 0xe0636edc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5a8c: skipped 000438 1788803587.374258 MIRROR_RESYNC 4 mirrors 000439 1788803587.996080 mapwrite 0x026b62 thru 0x02beb2 (0x5351 bytes) 000440 1788803588.131529 fallocate 0x0245b3 thru 0x026ccc (0x271a bytes) 000441 1788803588.142104 punch from 0x011b4d to 0x0156ed (0x3ba1 bytes) 000442 1788803588.148082 mapread 0x02851b thru 0x02beb2 (0x3998 bytes) 000443 1788803588.153910 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2beb3): Mirror 3: 0x9069d505 (stale) Mirror 5: 0x9069d505 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2beb3): Mirror 3: 0x9069d505 (stale) Mirror 5: 0x9069d505 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2beb3: skipped 000444 1788803588.564110 mapread 0x02868c thru 0x02beb2 (0x3827 bytes) 000445 1788803588.591624 mapwrite 0x035ca7 thru 0x03ffff (0xa359 bytes) 000446 1788803588.687561 read 0x02a990 thru 0x0379ef (0xd060 bytes) 000447 1788803588.698726 trunc from 0x0288fe to 0x02dd8b (0x548e bytes) 000448 1788803588.709540 fallocate 0x00d492 thru 0x0129fa (0x5569 bytes) 000449 1788803588.737729 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf66e382a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf66e382a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000450 1788803589.069641 MIRROR_RESYNC 2 mirrors 000451 1788803589.714432 trunc from 0x040000 to 0x0039fb (0x3c604 bytes) 000452 1788803589.746784 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000453 1788803589.871172 mapwrite 0x029916 thru 0x034524 (0xac0f bytes) 000454 1788803589.935282 mapread 0x016801 thru 0x023396 (0xcb96 bytes) 000455 1788803589.960869 fallocate 0x0002ac thru 0x001587 (0x12dc bytes) 000456 1788803589.965239 trunc from 0x034525 to 0x020421 (0x14103 bytes) 000458 1788803589.980334 punch from 0x002461 to 0x00dce6 (0xb886 bytes) 000459 1788803589.985061 read 0x019173 thru 0x020421 (0x72af bytes) 000460 1788803589.986091 read 0x0181a7 thru 0x0190a6 (0xf00 bytes) 000461 1788803589.986740 fallocate 0x01c719 thru 0x024128 (0x7a10 bytes) 000464 1788803589.995869 fallocate 0x028f2a thru 0x02d47e (0x4555 bytes) 000465 1788803590.000733 mapwrite 0x01912d thru 0x026276 (0xd14a bytes) 000466 1788803590.041045 read 0x00bf14 thru 0x0151d6 (0x92c3 bytes) 000467 1788803590.041913 write 0x01b9e7 thru 0x0211b4 (0x57ce bytes) 000468 1788803590.044221 fallocate 0x005f50 thru 0x00b2fc (0x53ad bytes) 000469 1788803590.053133 trunc from 0x026277 to 0x031969 (0xb6f3 bytes) 000470 1788803590.070817 fallocate 0x01e16e thru 0x029e94 (0xbd27 bytes) 000471 1788803590.075528 trunc from 0x005b41 to 0x005e54 (0x314 bytes) 000472 1788803590.079743 write 0x024995 thru 0x02ec35 (0xa2a1 bytes) 000473 1788803590.087734 mapwrite 0x0112d1 thru 0x015b7c (0x48ac bytes) 000474 1788803590.103525 mapread 0x018a88 thru 0x022ce3 (0xa25c bytes) 000475 1788803590.104651 trunc from 0x03196a to 0x00e46d (0x234fc bytes) 000476 1788803590.118717 write 0x03c69e thru 0x03ffff (0x3962 bytes) 000477 1788803590.120229 trunc from 0x040000 to 0x039fc6 (0x6039 bytes) 000478 1788803590.136893 trunc from 0x039fc7 to 0x03b419 (0x1453 bytes) 000480 1788803590.157416 trunc from 0x031bb2 to 0x031e2f (0x27e bytes) 000481 1788803590.161683 punch from 0x035f6f to 0x03b419 (0x54ab bytes) 000482 1788803590.167685 trunc from 0x03b41a to 0x03fc87 (0x486e bytes) 000483 1788803590.190997 fallocate 0x008816 thru 0x00f4d0 (0x6cbb bytes) 000484 1788803590.201694 trunc from 0x03fc88 to 0x02a858 (0x1542f bytes) 000487 1788803590.216918 trunc from 0x016910 to 0x021fda (0xb6cb bytes) 000488 1788803590.223262 trunc from 0x010942 to 0x01411d (0x37dc bytes) 000491 1788803590.229968 mapwrite 0x01c6a0 thru 0x0225d5 (0x5f36 bytes) 000492 1788803590.305585 mapread 0x0069e8 thru 0x00d7af (0x6dc8 bytes) 000493 1788803590.310103 write 0x020232 thru 0x02811a (0x7ee9 bytes) 000494 1788803590.315234 fallocate 0x02047a thru 0x0287fc (0x8383 bytes) 000495 1788803590.320973 read 0x004693 thru 0x0093e0 (0x4d4e bytes) 000497 1788803590.322568 trunc from 0x0080c3 to 0x00d730 (0x566e bytes) 000498 1788803590.327417 punch from 0x004dbe to 0x00a9f6 (0x5c39 bytes) 000499 1788803590.333905 write 0x03cd97 thru 0x03ffff (0x3269 bytes) 000500 1788803590.335411 mapwrite 0x03068d thru 0x03ffff (0xf973 bytes) 000501 1788803590.368000 punch from 0x012df8 to 0x020ac5 (0xdcce bytes) 000502 1788803590.372651 punch from 0x0146a3 to 0x014843 (0x1a1 bytes) 000504 1788803590.377322 fallocate 0x0175cc thru 0x0203e5 (0x8e1a bytes) 000505 1788803590.381693 punch from 0x018b10 to 0x023095 (0xa586 bytes) 000507 1788803590.387594 write 0x01fe9b thru 0x02e334 (0xe49a bytes) 000508 1788803590.392722 punch from 0x03ca71 to 0x03ffff (0x358f bytes) 000509 1788803590.397969 trunc from 0x02dffb to 0x0393c1 (0xb3c7 bytes) 000510 1788803590.410716 read 0x021cf1 thru 0x026fbd (0x52cd bytes) 000512 1788803590.412009 mapwrite 0x02612f thru 0x030231 (0xa103 bytes) 000513 1788803590.442411 trunc from 0x00ae5d to 0x00f6ef (0x4893 bytes) 000514 1788803590.446716 mapwrite 0x007e24 thru 0x0117d1 (0x99ae bytes) 000515 1788803590.489638 punch from 0x026302 to 0x031313 (0xb012 bytes) 000516 1788803590.494832 mapread 0x007837 thru 0x013ac7 (0xc291 bytes) 000517 1788803590.496020 fallocate 0x010fe5 thru 0x01bed3 (0xaeef bytes) 000520 1788803590.505580 MIRROR_EXTEND to 2 mirrors 000521 1788803591.340160 read 0x039b01 thru 0x03bb72 (0x2072 bytes) 000522 1788803591.358411 trunc from 0x00a6dd to 0x015c74 (0xb598 bytes) 000523 1788803591.401957 mapwrite 0x03b52a thru 0x03ffff (0x4ad6 bytes) 000526 1788803591.452481 punch from 0x028236 to 0x02ed86 (0x6b51 bytes) 000527 1788803591.461140 trunc from 0x040000 to 0x03820f (0x7df0 bytes) 000528 1788803591.544930 trunc from 0x0096e4 to 0x012449 (0x8d66 bytes) 000529 1788803591.552852 punch from 0x01c896 to 0x022e8e (0x65f9 bytes) 000530 1788803591.572041 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38210): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38210): Mirror 2: 0x930449b8 (stale) CRC-32 checksum value for chunk [0, 0x38210): Mirror 1: 0xed97986d Mirror 3: 0xed97986d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38210: skipped 000532 1788803592.503622 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38210): Mirror 2: 0x930449b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38210): Mirror 2: 0x930449b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38210: skipped 000533 1788803592.775828 fallocate 0x001701 thru 0x00fbb0 (0xe4b0 bytes) 000534 1788803592.797370 trunc from 0x038210 to 0x025644 (0x12bcb bytes) 000535 1788803592.828070 punch from 0x0124fa to 0x01721b (0x4d22 bytes) 000536 1788803592.834600 punch from 0x00480a to 0x0053ba (0xbb1 bytes) 000537 1788803592.839570 write 0x013160 thru 0x02234f (0xf1f0 bytes) 000538 1788803592.851288 trunc from 0x025645 to 0x02518f (0x4b5 bytes) 000539 1788803592.866367 read 0x01f4df thru 0x02518f (0x5cb1 bytes) 000540 1788803592.872067 fallocate 0x031cea thru 0x03e7d8 (0xcaef bytes) 000542 1788803592.882406 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000543 1788803593.015782 trunc from 0x025190 to 0x00b8f3 (0x1989c bytes) 000544 1788803593.037413 mapread 0x0035cd thru 0x00b8f3 (0x8327 bytes) 000545 1788803593.055179 mapwrite 0x010972 thru 0x014316 (0x39a5 bytes) 000546 1788803593.102380 mapwrite 0x013ec1 thru 0x014e44 (0xf84 bytes) 000547 1788803593.157722 MIRROR_EXTEND to 2 mirrors 000548 1788803593.949227 trunc from 0x014e45 to 0x02b39a (0x16556 bytes) 000549 1788803594.081427 mapwrite 0x032238 thru 0x037b0a (0x58d3 bytes) 000550 1788803594.208534 fallocate 0x011841 thru 0x01e19f (0xc95f bytes) 000551 1788803594.214739 write 0x03b448 thru 0x03b6c5 (0x27e bytes) 000552 1788803594.215784 trunc from 0x025f65 to 0x02a29e (0x433a bytes) 000553 1788803594.220212 fallocate 0x0157e6 thru 0x01e0bc (0x88d7 bytes) 000554 1788803594.225365 mapwrite 0x02f849 thru 0x039fa6 (0xa75e bytes) 000555 1788803594.247445 write 0x01e722 thru 0x025199 (0x6a78 bytes) 000556 1788803594.261036 punch from 0x024cfc to 0x02ad8d (0x6092 bytes) 000557 1788803594.282988 fallocate 0x004aed thru 0x00c5e3 (0x7af7 bytes) 000558 1788803594.287807 fallocate 0x0163b3 thru 0x023499 (0xd0e7 bytes) 000559 1788803594.292418 mapwrite 0x00abb6 thru 0x00d13a (0x2585 bytes) 000560 1788803594.313833 trunc from 0x021a0d to 0x02f5c1 (0xdbb5 bytes) 000561 1788803594.318966 trunc from 0x017e35 to 0x01da00 (0x5bcc bytes) 000562 1788803594.323520 trunc from 0x03b6c6 to 0x004def (0x368d6 bytes) 000563 1788803594.349167 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000564 1788803594.491983 write 0x00d2ec thru 0x01b095 (0xddaa bytes) 000566 1788803594.498936 mapread 0x0179ab thru 0x01b095 (0x36eb bytes) 000567 1788803594.500261 read 0x00cf94 thru 0x0139a6 (0x6a13 bytes) 000568 1788803594.506154 mapwrite 0x02cd83 thru 0x03c4b7 (0xf735 bytes) 000569 1788803594.568403 mapwrite 0x01da97 thru 0x028c82 (0xb1ec bytes) 000570 1788803594.597868 mapwrite 0x0027d1 thru 0x01071f (0xdf4f bytes) 000571 1788803594.628576 MIRROR_EXTEND to 2 mirrors 000572 1788803595.393433 read 0x01bfdf thru 0x02a169 (0xe18b bytes) 000573 1788803595.438068 write 0x0103a2 thru 0x01c103 (0xbd62 bytes) 000574 1788803595.500938 write 0x01a502 thru 0x023025 (0x8b24 bytes) 000575 1788803595.513535 write 0x039a1d thru 0x03ffff (0x65e3 bytes) 000576 1788803595.521597 punch from 0x01d9d0 to 0x0286b2 (0xace3 bytes) 000577 1788803595.542964 write 0x006177 thru 0x00c542 (0x63cc bytes) 000578 1788803595.559326 mapwrite 0x01266b thru 0x018f2d (0x68c3 bytes) 000579 1788803595.607235 punch from 0x015fb5 to 0x02571d (0xf769 bytes) 000580 1788803595.623975 read 0x006757 thru 0x0151e3 (0xea8d bytes) 000581 1788803595.637292 mapwrite 0x02b7c1 thru 0x03a2f5 (0xeb35 bytes) 000582 1788803595.682354 mapread 0x038527 thru 0x03fdf7 (0x78d1 bytes) 000583 1788803595.683696 trunc from 0x03bea4 to 0x03eb29 (0x2c86 bytes) 000584 1788803595.692840 trunc from 0x018fd8 to 0x027c72 (0xec9b bytes) 000585 1788803595.699144 punch from 0x03ce42 to 0x03ffff (0x31be bytes) 000586 1788803595.710637 fallocate 0x017fab thru 0x01a3e3 (0x2439 bytes) 000587 1788803595.717116 read 0x01c1d8 thru 0x022914 (0x673d bytes) 000588 1788803595.725110 read 0x03de61 thru 0x03dfd6 (0x176 bytes) 000589 1788803595.725879 read 0x02bf46 thru 0x030cb3 (0x4d6e bytes) 000590 1788803595.731540 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6e19fb32 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad20c8aa Mirror 3: 0xad20c8aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000591 1788803597.030340 punch from 0x02c37c to 0x02eacf (0x2754 bytes) 000592 1788803597.088747 fallocate 0x02391c thru 0x028161 (0x4846 bytes) 000593 1788803597.102727 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6e19fb32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6e19fb32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000595 1788803597.446507 punch from 0x012708 to 0x01a233 (0x7b2c bytes) 000596 1788803597.453256 read 0x028bb4 thru 0x031336 (0x8783 bytes) 000597 1788803597.469381 trunc from 0x040000 to 0x002843 (0x3d7bc bytes) 000598 1788803597.497032 trunc from 0x002844 to 0x02276e (0x1ff2b bytes) 000599 1788803597.524326 mapread 0x01d2e1 thru 0x02276e (0x548e bytes) 000600 1788803597.546292 mapread 0x00c738 thru 0x0142f6 (0x7bbf bytes) 000601 1788803597.560724 trunc from 0x02276f to 0x03840d (0x15c9f bytes) 000602 1788803597.586732 write 0x01cc33 thru 0x0299f6 (0xcdc4 bytes) 000603 1788803597.596542 trunc from 0x00958b to 0x0142af (0xad25 bytes) 000605 1788803597.607363 mapread 0x01d813 thru 0x028671 (0xae5f bytes) 000606 1788803597.608985 write 0x011c71 thru 0x01fbd2 (0xdf62 bytes) 000607 1788803597.617983 write 0x02decb thru 0x035356 (0x748c bytes) 000608 1788803597.627928 write 0x02a1e6 thru 0x039633 (0xf44e bytes) 000609 1788803597.634157 mapread 0x030b41 thru 0x038330 (0x77f0 bytes) 000610 1788803597.635096 trunc from 0x039634 to 0x03107d (0x85b6 bytes) 000611 1788803597.649377 trunc from 0x00d15e to 0x01888d (0xb730 bytes) 000612 1788803597.669191 mapwrite 0x0203d9 thru 0x02e946 (0xe56e bytes) 000613 1788803597.692990 read 0x0039d2 thru 0x01357e (0xfbad bytes) 000614 1788803597.709027 write 0x0358b3 thru 0x03ffff (0xa74d bytes) 000615 1788803597.710596 punch from 0x01b39d to 0x01f75a (0x43be bytes) 000616 1788803597.720786 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6e19fb32 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9328d32 Mirror 3: 0xf9328d32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000617 1788803598.745265 mapwrite 0x01fc56 thru 0x0204de (0x889 bytes) 000618 1788803598.878758 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6e19fb32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6e19fb32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000619 1788803599.275476 mapwrite 0x0298c1 thru 0x02d6e3 (0x3e23 bytes) 000620 1788803599.356354 punch from 0x01baa5 to 0x02ba2b (0xff87 bytes) 000621 1788803599.370808 mapread 0x034741 thru 0x03bb11 (0x73d1 bytes) 000622 1788803599.392118 write 0x0335fc thru 0x03ffff (0xca04 bytes) 000623 1788803599.396817 punch from 0x0216c0 to 0x02c30a (0xac4b bytes) 000624 1788803599.436922 punch from 0x030316 to 0x03dde5 (0xdad0 bytes) 000625 1788803599.494913 mapwrite 0x0101be thru 0x011f6a (0x1dad bytes) 000626 1788803599.543519 mapread 0x021885 thru 0x030828 (0xefa4 bytes) 000627 1788803599.591410 fallocate 0x00a175 thru 0x017319 (0xd1a5 bytes) 000628 1788803599.608550 punch from 0x034e8b to 0x03efaa (0xa120 bytes) 000629 1788803599.625076 punch from 0x024dc9 to 0x0333bd (0xe5f5 bytes) 000630 1788803599.640382 mapwrite 0x00924c thru 0x00a126 (0xedb bytes) 000631 1788803599.675070 MIRROR_RESYNC 2 mirrors 000632 1788803600.638956 mapread 0x032394 thru 0x03ffff (0xdc6c bytes) 000633 1788803600.669892 MIRROR_EXTEND to 3 mirrors 000634 1788803601.375262 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000635 1788803601.723063 write 0x029442 thru 0x030e50 (0x7a0f bytes) 000637 1788803601.781090 mapread 0x029756 thru 0x02b850 (0x20fb bytes) 000638 1788803601.782846 mapread 0x03e087 thru 0x03ea70 (0x9ea bytes) 000640 1788803601.788613 read 0x001d06 thru 0x005985 (0x3c80 bytes) 000641 1788803601.799904 trunc from 0x040000 to 0x004ab7 (0x3b548 bytes) 000642 1788803601.821864 trunc from 0x004ab8 to 0x03c30b (0x37854 bytes) 000643 1788803601.837644 write 0x011164 thru 0x011834 (0x6d1 bytes) 000644 1788803601.841886 mapwrite 0x038697 thru 0x03c6ed (0x4057 bytes) 000647 1788803601.882991 write 0x01d5fc thru 0x022656 (0x505b bytes) 000648 1788803601.894885 fallocate 0x0272d8 thru 0x02ab11 (0x383a bytes) 000649 1788803601.900446 punch from 0x008e52 to 0x011587 (0x8736 bytes) 000650 1788803601.907821 mapread 0x008aed thru 0x013956 (0xae6a bytes) 000651 1788803601.929871 trunc from 0x03c6ee to 0x02308f (0x1965e bytes) 000652 1788803601.953814 read 0x005993 thru 0x0127c0 (0xce2e bytes) 000654 1788803601.955509 write 0x02613d thru 0x03363a (0xd4fe bytes) 000655 1788803601.960404 mapwrite 0x019209 thru 0x026fc3 (0xddbb bytes) 000656 1788803601.992761 write 0x028b4c thru 0x035594 (0xca49 bytes) 000657 1788803601.994795 punch from 0x01277c to 0x0137a1 (0x1026 bytes) 000658 1788803602.003638 MIRROR_RESYNC 2 mirrors 000660 1788803602.610103 mapread 0x0151cc thru 0x01a64b (0x5480 bytes) 000662 1788803602.640904 write 0x03c2c6 thru 0x03cfd4 (0xd0f bytes) 000663 1788803602.685138 write 0x035d5a thru 0x03ffff (0xa2a6 bytes) 000664 1788803602.695693 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000665 1788803602.832955 read 0x02e53b thru 0x03e30b (0xfdd1 bytes) 000666 1788803602.878330 trunc from 0x040000 to 0x00c1a0 (0x33e5f bytes) 000667 1788803602.923802 MIRROR_EXTEND to 2 mirrors 000668 1788803603.655425 trunc from 0x00c1a1 to 0x014cc9 (0x8b29 bytes) 000669 1788803603.730450 punch from 0x002f14 to 0x00d8ad (0xa99a bytes) 000670 1788803603.742422 fallocate 0x032133 thru 0x03f3d5 (0xd2a3 bytes) 000671 1788803603.753743 MIRROR_RESYNC 2 mirrors 000672 1788803604.329261 trunc from 0x03f3d6 to 0x00feda (0x2f4fb bytes) 000673 1788803604.365901 trunc from 0x00fedb to 0x03f034 (0x2f15a bytes) 000674 1788803604.389789 MIRROR_RESYNC 2 mirrors 000675 1788803604.899956 write 0x010b2d thru 0x012a04 (0x1ed8 bytes) 000676 1788803604.991284 write 0x02b597 thru 0x02d458 (0x1ec2 bytes) 000677 1788803605.033920 punch from 0x019037 to 0x01d796 (0x4760 bytes) 000678 1788803605.046883 trunc from 0x03f035 to 0x01b315 (0x23d1f bytes) 000679 1788803605.075359 trunc from 0x01b316 to 0x030e9d (0x15b88 bytes) 000681 1788803605.109642 read 0x0103e4 thru 0x016fc7 (0x6be4 bytes) 000682 1788803605.118357 trunc from 0x030e9e to 0x00570a (0x2b793 bytes) 000683 1788803605.139319 punch from 0x0006f8 to 0x00570a (0x5013 bytes) 000684 1788803605.144705 fallocate 0x03dcea thru 0x03ffff (0x2316 bytes) 000685 1788803605.149445 read 0x0034c7 thru 0x00570a (0x2244 bytes) 000686 1788803605.163952 punch from 0x001594 to 0x005118 (0x3b85 bytes) 000687 1788803605.184115 MIRROR_RESYNC 2 mirrors 000688 1788803605.762819 trunc from 0x00570b to 0x034e62 (0x2f758 bytes) 000689 1788803605.824360 mapwrite 0x00e5d9 thru 0x013f59 (0x5981 bytes) 000690 1788803605.864399 mapwrite 0x024f7d thru 0x03255d (0xd5e1 bytes) 000691 1788803605.944196 mapwrite 0x03e052 thru 0x03ffff (0x1fae bytes) 000692 1788803605.987584 trunc from 0x040000 to 0x02add7 (0x15228 bytes) 000694 1788803606.001248 trunc from 0x02add8 to 0x03c5a6 (0x117cf bytes) 000695 1788803606.018126 fallocate 0x021f65 thru 0x024d76 (0x2e12 bytes) 000696 1788803606.023129 mapwrite 0x009713 thru 0x0192d1 (0xfbbf bytes) 000697 1788803606.049517 fallocate 0x00eee8 thru 0x0128a9 (0x39c2 bytes) 000698 1788803606.053647 trunc from 0x03c5a7 to 0x01dacb (0x1eadb bytes) 000699 1788803606.071263 read 0x0146d3 thru 0x0198eb (0x5219 bytes) 000700 1788803606.072902 write 0x00b80c thru 0x00d04d (0x1842 bytes) 000701 1788803606.074740 trunc from 0x00d1e9 to 0x019694 (0xc4ac bytes) 000702 1788803606.084252 trunc from 0x007fc6 to 0x0100f3 (0x812e bytes) 000703 1788803606.089761 punch from 0x0069cc to 0x00b277 (0x48ac bytes) 000704 1788803606.093374 trunc from 0x00b77c to 0x01a3fe (0xec83 bytes) 000705 1788803606.101494 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000707 1788803606.231129 write 0x003c53 thru 0x0138a6 (0xfc54 bytes) 000708 1788803606.249921 trunc from 0x017b67 to 0x01ce57 (0x52f1 bytes) 000709 1788803606.256803 mapread 0x00e070 thru 0x016255 (0x81e6 bytes) 000710 1788803606.312352 mapwrite 0x03a6a5 thru 0x03ffff (0x595b bytes) 000711 1788803606.355227 trunc from 0x003fc8 to 0x005bef (0x1c28 bytes) 000714 1788803606.362249 fallocate 0x013930 thru 0x020176 (0xc847 bytes) 000715 1788803606.366987 read 0x03f7c4 thru 0x03ffff (0x83c bytes) 000716 1788803606.378731 trunc from 0x0149e3 to 0x015f56 (0x1574 bytes) 000717 1788803606.386081 write 0x0012df thru 0x00f24c (0xdf6e bytes) 000718 1788803606.392085 fallocate 0x0039fb thru 0x007623 (0x3c29 bytes) 000719 1788803606.396753 mapread 0x0393c9 thru 0x03ffff (0x6c37 bytes) 000720 1788803606.398158 mapread 0x0182c0 thru 0x01ab08 (0x2849 bytes) 000721 1788803606.399619 trunc from 0x011897 to 0x017efc (0x6666 bytes) 000722 1788803606.404942 read 0x03a2fb thru 0x03ffff (0x5d05 bytes) 000723 1788803606.405722 trunc from 0x040000 to 0x03eabc (0x1543 bytes) 000725 1788803606.428849 mapwrite 0x001140 thru 0x00e4a7 (0xd368 bytes) 000726 1788803606.460051 mapread 0x028960 thru 0x02b0c6 (0x2767 bytes) 000728 1788803606.472233 punch from 0x03d676 to 0x03eabc (0x1447 bytes) 000730 1788803606.478518 fallocate 0x0118b9 thru 0x021228 (0xf970 bytes) 000731 1788803606.483421 fallocate 0x037c75 thru 0x03ffff (0x838b bytes) 000732 1788803606.488367 trunc from 0x040000 to 0x01fb94 (0x2046b bytes) 000733 1788803606.519633 fallocate 0x0316e5 thru 0x038551 (0x6e6d bytes) 000734 1788803606.528320 trunc from 0x013294 to 0x015bc8 (0x2935 bytes) 000735 1788803606.534303 mapwrite 0x0194a2 thru 0x0239ce (0xa52d bytes) 000736 1788803606.626712 read 0x01bcfe thru 0x01ef03 (0x3206 bytes) 000737 1788803606.627361 mapread 0x0053e1 thru 0x012f6a (0xdb8a bytes) 000738 1788803606.628936 punch from 0x00e985 to 0x01ac42 (0xc2be bytes) 000739 1788803606.638225 fallocate 0x03224b thru 0x033767 (0x151d bytes) 000740 1788803606.650584 trunc from 0x0239cf to 0x03bdbc (0x183ee bytes) 000741 1788803606.700045 punch from 0x009119 to 0x00b5c8 (0x24b0 bytes) 000742 1788803606.707421 mapread 0x0345fb thru 0x0361e6 (0x1bec bytes) 000743 1788803606.735635 fallocate 0x01210a thru 0x018751 (0x6648 bytes) 000745 1788803606.751198 mapwrite 0x039d88 thru 0x03ffff (0x6278 bytes) 000746 1788803606.856005 trunc from 0x040000 to 0x03823a (0x7dc5 bytes) 000747 1788803606.897026 MIRROR_EXTEND to 2 mirrors 000749 1788803608.057835 punch from 0x02c699 to 0x0367e8 (0xa150 bytes) 000750 1788803608.093007 trunc from 0x0305c7 to 0x03814d (0x7b87 bytes) 000751 1788803608.099228 trunc from 0x00a914 to 0x00de34 (0x3521 bytes) 000752 1788803608.109830 trunc from 0x03823b to 0x02f9c1 (0x8879 bytes) 000753 1788803608.140128 fallocate 0x019ca8 thru 0x01c986 (0x2cdf bytes) 000755 1788803608.148901 write 0x017772 thru 0x018fb4 (0x1843 bytes) 000756 1788803608.172026 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000757 1788803608.464430 read 0x001082 thru 0x007156 (0x60d5 bytes) 000758 1788803608.482316 trunc from 0x02f9c2 to 0x02cfdc (0x29e5 bytes) 000760 1788803608.567285 trunc from 0x00e9e2 to 0x01541f (0x6a3e bytes) 000761 1788803608.587653 mapwrite 0x03b1e8 thru 0x03ffff (0x4e18 bytes) 000763 1788803608.738680 trunc from 0x030be7 to 0x037970 (0x6d8a bytes) 000764 1788803608.742722 write 0x03411f thru 0x03a373 (0x6255 bytes) 000765 1788803608.747559 write 0x01ad65 thru 0x0292e1 (0xe57d bytes) 000766 1788803608.758091 write 0x00a544 thru 0x0194ac (0xef69 bytes) 000767 1788803608.768036 mapread 0x033ccf thru 0x038d8f (0x50c1 bytes) 000769 1788803608.776163 trunc from 0x038c6c to 0x03ffff (0x7394 bytes) 000770 1788803608.795925 trunc from 0x040000 to 0x039612 (0x69ed bytes) 000771 1788803608.812057 read 0x01352e thru 0x01afc7 (0x7a9a bytes) 000772 1788803608.813122 fallocate 0x02161b thru 0x027580 (0x5f66 bytes) 000773 1788803608.821348 trunc from 0x039613 to 0x0306ad (0x8f65 bytes) 000774 1788803608.837659 mapread 0x0104e7 thru 0x012435 (0x1f4f bytes) 000775 1788803608.838860 fallocate 0x009b1b thru 0x00ed72 (0x5258 bytes) 000776 1788803608.846703 fallocate 0x01ce30 thru 0x02613c (0x930d bytes) 000777 1788803608.857056 write 0x01140a thru 0x01bedf (0xaad6 bytes) 000778 1788803608.863384 mapwrite 0x0026bc thru 0x00b2c5 (0x8c0a bytes) 000780 1788803608.902646 trunc from 0x0306ae to 0x03b226 (0xab79 bytes) 000782 1788803608.922109 MIRROR_EXTEND to 2 mirrors 000783 1788803610.011449 MIRROR_EXTEND to 3 mirrors 000784 1788803610.746704 mapwrite 0x0347c1 thru 0x038657 (0x3e97 bytes) 000785 1788803610.911107 mapread 0x0320f7 thru 0x035d1e (0x3c28 bytes) 000787 1788803610.914583 trunc from 0x018d54 to 0x024141 (0xb3ee bytes) 000788 1788803610.923656 write 0x03fb2f thru 0x03ffff (0x4d1 bytes) 000789 1788803610.930824 mapwrite 0x03cbac thru 0x03ffff (0x3454 bytes) 000790 1788803610.953197 trunc from 0x004800 to 0x012ce1 (0xe4e2 bytes) 000791 1788803610.961771 punch from 0x01032a to 0x01cdd8 (0xcaaf bytes) 000792 1788803610.969728 trunc from 0x016d3f to 0x0170d8 (0x39a bytes) 000793 1788803610.978544 fallocate 0x023ae4 thru 0x02d115 (0x9632 bytes) 000794 1788803610.988054 mapread 0x026ef5 thru 0x029536 (0x2642 bytes) 000795 1788803611.005993 punch from 0x02a3b3 to 0x02da55 (0x36a3 bytes) 000796 1788803611.011979 trunc from 0x02355f to 0x03086f (0xd311 bytes) 000797 1788803611.018012 fallocate 0x007355 thru 0x00fc1e (0x88ca bytes) 000798 1788803611.022872 write 0x02e4e3 thru 0x0396f2 (0xb210 bytes) 000799 1788803611.031840 punch from 0x0198e0 to 0x01a58d (0xcae bytes) 000800 1788803611.040644 MIRROR_RESYNC 3 mirrors 000801 1788803611.961292 trunc from 0x013ca6 to 0x01b02c (0x7387 bytes) 000802 1788803612.002920 mapwrite 0x001d98 thru 0x00708a (0x52f3 bytes) 000803 1788803612.043382 mapread 0x038b60 thru 0x03ffff (0x74a0 bytes) 000804 1788803612.057501 trunc from 0x040000 to 0x01e933 (0x216cc bytes) 000805 1788803612.083117 mapread 0x011767 thru 0x01afd5 (0x986f bytes) 000806 1788803612.104567 mapwrite 0x010020 thru 0x019aaf (0x9a90 bytes) 000807 1788803612.130207 trunc from 0x01e934 to 0x02927b (0xa948 bytes) 000808 1788803612.154649 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2927c): Mirror 3: 0x688cdfcd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2927c): Mirror 3: 0x688cdfcd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2927c: skipped 000809 1788803612.656040 trunc from 0x02927c to 0x002407 (0x26e74 bytes) 000810 1788803612.688105 trunc from 0x002408 to 0x024a40 (0x22639 bytes) 000811 1788803612.714376 mapwrite 0x02c7e3 thru 0x034d98 (0x85b6 bytes) 000812 1788803612.798941 trunc from 0x031324 to 0x034d98 (0x3a75 bytes) 000813 1788803612.806318 read 0x01e310 thru 0x0280c4 (0x9db5 bytes) 000814 1788803612.821067 write 0x025572 thru 0x033bf8 (0xe687 bytes) 000815 1788803612.829296 mapwrite 0x036898 thru 0x0389e6 (0x214f bytes) 000816 1788803612.883159 mapwrite 0x035778 thru 0x037067 (0x18f0 bytes) 000817 1788803612.896145 punch from 0x01ec9c to 0x02cb9c (0xdf01 bytes) 000818 1788803612.902071 fallocate 0x037c06 thru 0x03ffff (0x83fa bytes) 000819 1788803612.906685 trunc from 0x032d4f to 0x035f7c (0x322e bytes) 000820 1788803612.912039 write 0x018d32 thru 0x027615 (0xe8e4 bytes) 000821 1788803612.922930 punch from 0x0240c5 to 0x02d259 (0x9195 bytes) 000822 1788803612.940027 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000825 1788803613.064222 write 0x02e1ba thru 0x03b410 (0xd257 bytes) 000826 1788803613.076694 trunc from 0x0311f5 to 0x03b410 (0xa21c bytes) 000827 1788803613.101377 punch from 0x029222 to 0x02d799 (0x4578 bytes) 000828 1788803613.107323 trunc from 0x03b411 to 0x00aa54 (0x309bc bytes) 000829 1788803613.121904 trunc from 0x00aa55 to 0x01cba6 (0x12152 bytes) 000830 1788803613.139283 fallocate 0x00c490 thru 0x01c298 (0xfe09 bytes) 000831 1788803613.144316 trunc from 0x01b340 to 0x01cba6 (0x1867 bytes) 000832 1788803613.148557 read 0x019d95 thru 0x01cba6 (0x2e12 bytes) 000833 1788803613.158362 read 0x017df2 thru 0x01cba6 (0x4db5 bytes) 000835 1788803613.170640 punch from 0x00f155 to 0x012153 (0x2fff bytes) 000836 1788803613.175784 punch from 0x014448 to 0x01cba6 (0x875f bytes) 000837 1788803613.182780 mapread 0x00b4d7 thru 0x016cc8 (0xb7f2 bytes) 000838 1788803613.226799 trunc from 0x0110b0 to 0x01cba6 (0xbaf7 bytes) 000839 1788803613.232735 mapwrite 0x00dc9d thru 0x014361 (0x66c5 bytes) 000840 1788803613.277950 trunc from 0x01cba7 to 0x006788 (0x1641e bytes) 000841 1788803613.297086 punch from 0x00343f to 0x006788 (0x334a bytes) 000842 1788803613.302581 punch from 0x0032b4 to 0x006788 (0x34d5 bytes) 000843 1788803613.308450 trunc from 0x006789 to 0x024f6c (0x1e7e4 bytes) 000844 1788803613.325829 punch from 0x00f18e to 0x014b3e (0x59b1 bytes) 000845 1788803613.330091 trunc from 0x024f6d to 0x01b644 (0x9928 bytes) 000846 1788803613.343578 fallocate 0x0291fa thru 0x02e03e (0x4e45 bytes) 000847 1788803613.352607 MIRROR_EXTEND to 2 mirrors 000848 1788803613.879174 MIRROR_RESYNC 2 mirrors 000849 1788803614.362397 trunc from 0x0237bb to 0x02b7d5 (0x801b bytes) 000850 1788803614.400758 trunc from 0x01437d to 0x0235c8 (0xf24c bytes) 000851 1788803614.408202 punch from 0x01e354 to 0x01f3a5 (0x1052 bytes) 000852 1788803614.412881 trunc from 0x02e03f to 0x03cb12 (0xead4 bytes) 000853 1788803614.458054 fallocate 0x020ecd thru 0x02e425 (0xd559 bytes) 000854 1788803614.464092 trunc from 0x03cb13 to 0x00cb7c (0x2ff96 bytes) 000855 1788803614.479617 trunc from 0x00cb7d to 0x00d06e (0x4f2 bytes) 000857 1788803614.505225 trunc from 0x00d06f to 0x011297 (0x4229 bytes) 000858 1788803614.526525 trunc from 0x011298 to 0x0123af (0x1118 bytes) 000859 1788803614.552567 MIRROR_RESYNC 2 mirrors 000860 1788803615.235136 mapread 0x0060f1 thru 0x00f09e (0x8fae bytes) 000861 1788803615.262084 MIRROR_RESYNC 2 mirrors 000862 1788803615.531522 read 0x00ab3f thru 0x0123af (0x7871 bytes) 000864 1788803615.537986 write 0x008dc7 thru 0x00e575 (0x57af bytes) 000867 1788803615.569762 mapread 0x011f19 thru 0x0123af (0x497 bytes) 000868 1788803615.583883 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000869 1788803615.716864 trunc from 0x0123b0 to 0x03731c (0x24f6d bytes) 000870 1788803615.741090 trunc from 0x0369cf to 0x03731c (0x94e bytes) 000871 1788803615.748349 fallocate 0x03eab8 thru 0x03ffff (0x1548 bytes) 000872 1788803615.753351 mapwrite 0x03908a thru 0x03ffff (0x6f76 bytes) 000873 1788803615.788884 read 0x034cd1 thru 0x03ffff (0xb32f bytes) 000874 1788803615.789843 trunc from 0x01c59e to 0x0204fd (0x3f60 bytes) 000875 1788803615.794919 punch from 0x0168db to 0x0175eb (0xd11 bytes) 000876 1788803615.800921 trunc from 0x0243bb to 0x032257 (0xde9d bytes) 000877 1788803615.805548 read 0x008828 thru 0x0187a7 (0xff80 bytes) 000878 1788803615.816662 fallocate 0x0072ba thru 0x010cfb (0x9a42 bytes) 000879 1788803615.821372 mapwrite 0x02ba51 thru 0x03135b (0x590b bytes) 000880 1788803615.837879 trunc from 0x02e6c1 to 0x02faec (0x142c bytes) 000881 1788803615.845644 MIRROR_EXTEND to 2 mirrors 000882 1788803616.549912 write 0x0105d0 thru 0x010892 (0x2c3 bytes) 000883 1788803616.599260 punch from 0x019d0c to 0x01e124 (0x4419 bytes) 000884 1788803616.609986 mapread 0x012400 thru 0x013b30 (0x1731 bytes) 000885 1788803616.626942 read 0x0262f0 thru 0x02e82c (0x853d bytes) 000886 1788803616.637184 write 0x00c1dc thru 0x018179 (0xbf9e bytes) 000887 1788803616.653260 fallocate 0x0156e1 thru 0x021d76 (0xc696 bytes) 000889 1788803616.660324 punch from 0x00d7ce to 0x01d582 (0xfdb5 bytes) 000890 1788803616.692123 mapwrite 0x018a13 thru 0x019df8 (0x13e6 bytes) 000893 1788803616.768571 mapwrite 0x021e37 thru 0x02d35c (0xb526 bytes) 000895 1788803616.857013 mapwrite 0x020fb6 thru 0x02ee97 (0xdee2 bytes) 000896 1788803616.916828 read 0x015e3e thru 0x01a5c2 (0x4785 bytes) 000897 1788803616.918426 trunc from 0x040000 to 0x00eb4d (0x314b2 bytes) 000898 1788803616.948695 trunc from 0x00bd15 to 0x00eab0 (0x2d9c bytes) 000899 1788803616.955300 mapwrite 0x03f3fc thru 0x03ffff (0xc04 bytes) 000900 1788803616.996211 write 0x00e868 thru 0x012098 (0x3831 bytes) 000901 1788803616.997999 write 0x016984 thru 0x01c5ed (0x5c6a bytes) 000902 1788803616.999695 read 0x02febd thru 0x030c09 (0xd4d bytes) 000903 1788803617.000384 punch from 0x02f172 to 0x037ff5 (0x8e84 bytes) 000904 1788803617.006560 trunc from 0x025727 to 0x02bef8 (0x67d2 bytes) 000905 1788803617.011031 read 0x019e91 thru 0x01ec15 (0x4d85 bytes) 000906 1788803617.012174 write 0x0085c9 thru 0x00ca0d (0x4445 bytes) 000907 1788803617.016864 write 0x02c718 thru 0x034437 (0x7d20 bytes) 000908 1788803617.026899 read 0x0202f2 thru 0x0217de (0x14ed bytes) 000909 1788803617.027570 mapwrite 0x030d3f thru 0x03c063 (0xb325 bytes) 000910 1788803617.079218 MIRROR_RESYNC 2 mirrors 000911 1788803618.190003 write 0x022027 thru 0x028e47 (0x6e21 bytes) 000912 1788803618.245958 trunc from 0x00193d to 0x006bca (0x528e bytes) 000913 1788803618.250694 fallocate 0x003eda thru 0x00a2b9 (0x63e0 bytes) 000915 1788803618.255480 trunc from 0x040000 to 0x01e87c (0x21783 bytes) 000916 1788803618.303714 punch from 0x014752 to 0x01e131 (0x99e0 bytes) 000917 1788803618.311858 MIRROR_RESYNC 2 mirrors 000918 1788803619.045174 write 0x01babc thru 0x023e78 (0x83bd bytes) 000919 1788803619.083518 trunc from 0x015b61 to 0x019fcb (0x446b bytes) 000920 1788803619.105288 fallocate 0x024d30 thru 0x0320e5 (0xd3b6 bytes) 000922 1788803619.116882 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) 000924 1788803619.314052 read 0x009b2b thru 0x01573b (0xbc11 bytes) 000925 1788803619.363151 MIRROR_EXTEND to 2 mirrors 000926 1788803620.348941 trunc from 0x0208ac to 0x02bc59 (0xb3ae bytes) 000927 1788803620.384148 punch from 0x02dcd2 to 0x0320e5 (0x4414 bytes) 000928 1788803620.389580 mapread 0x0056f7 thru 0x011316 (0xbc20 bytes) 000929 1788803620.414354 write 0x01f1e4 thru 0x028c63 (0x9a80 bytes) 000930 1788803620.430579 fallocate 0x010d05 thru 0x013fdc (0x32d8 bytes) 000931 1788803620.435999 write 0x035f6e thru 0x03ab0a (0x4b9d bytes) 000932 1788803620.441023 mapwrite 0x00cfeb thru 0x012576 (0x558c bytes) 000933 1788803620.458096 fallocate 0x000a30 thru 0x00c3f0 (0xb9c1 bytes) 000934 1788803620.462688 mapread 0x034e30 thru 0x03ab0a (0x5cdb bytes) 000937 1788803620.471591 punch from 0x0375d7 to 0x037b0c (0x536 bytes) 000939 1788803620.477228 write 0x039325 thru 0x03ffff (0x6cdb bytes) 000940 1788803620.478559 trunc from 0x040000 to 0x00c050 (0x33faf bytes) 000941 1788803620.501844 mapwrite 0x037917 thru 0x03ffff (0x86e9 bytes) 000942 1788803620.568244 trunc from 0x040000 to 0x03bd59 (0x42a6 bytes) 000943 1788803620.596964 punch from 0x00a549 to 0x00e052 (0x3b0a bytes) 000944 1788803620.606277 fallocate 0x02daec thru 0x03ab71 (0xd086 bytes) 000945 1788803620.613502 read 0x01bda5 thru 0x0242a9 (0x8505 bytes) 000946 1788803620.641152 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 2: 0xe0962e9e (stale) CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 1: 0x56b7b3ee Mirror 3: 0x56b7b3ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd5a: skipped 000947 1788803621.989991 punch from 0x023d0f to 0x02cd19 (0x900b bytes) 000948 1788803622.045093 trunc from 0x03bd5a to 0x0124cd (0x2988c bytes) 000949 1788803622.071294 mapwrite 0x00a351 thru 0x00efdb (0x4c8b bytes) 000950 1788803622.108509 trunc from 0x005734 to 0x00caeb (0x73b8 bytes) 000951 1788803622.113639 read 0x0071c3 thru 0x00ec44 (0x7a82 bytes) 000952 1788803622.123538 fallocate 0x03cd9d thru 0x03ffff (0x3263 bytes) 000953 1788803622.131381 write 0x007029 thru 0x015f9f (0xef77 bytes) 000954 1788803622.134338 mapwrite 0x01215c thru 0x017d91 (0x5c36 bytes) 000955 1788803622.168607 mapread 0x01376d thru 0x017d91 (0x4625 bytes) 000956 1788803622.170201 fallocate 0x03fbf3 thru 0x03ffff (0x40d bytes) 000957 1788803622.175595 mapread 0x0089eb thru 0x00ff39 (0x754f bytes) 000958 1788803622.176971 punch from 0x014734 to 0x014b27 (0x3f4 bytes) 000959 1788803622.182872 punch from 0x011dc3 to 0x017d91 (0x5fcf bytes) 000960 1788803622.191820 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x17d92): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17d92): Mirror 2: 0x33a56945 (stale) Mirror 3: 0x5cf51fb3 (stale) CRC-32 checksum value for chunk [0, 0x17d92): Mirror 1: 0x97e5a9f4 Mirror 4: 0x97e5a9f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17d92: skipped 000962 1788803622.879027 read 0x016068 thru 0x017d91 (0x1d2a bytes) 000963 1788803622.895909 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17d92): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17d92): Mirror 2: 0x33a56945 (stale) Mirror 3: 0x5cf51fb3 (stale) CRC-32 checksum value for chunk [0, 0x17d92): Mirror 1: 0x97e5a9f4 Mirror 4: 0x97e5a9f4 Mirror 5: 0x97e5a9f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17d92: skipped 000964 1788803623.729158 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17d92): Mirror 3: 0x5cf51fb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17d92): Mirror 3: 0x5cf51fb3 (stale) CRC-32 checksum value for chunk [0, 0x17d92): Mirror 1: 0x97e5a9f4 Mirror 4: 0x97e5a9f4 Mirror 5: 0x97e5a9f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17d92: skipped 000965 1788803624.447661 mapwrite 0x036239 thru 0x03ffff (0x9dc7 bytes) 000966 1788803624.617450 fallocate 0x00e3da thru 0x00fd83 (0x19aa bytes) 000967 1788803624.624233 fallocate 0x037f06 thru 0x03ffff (0x80fa bytes) 000968 1788803624.644289 MIRROR_RESYNC 4 mirrors 000970 1788803625.856919 mapread 0x0375af thru 0x0382a3 (0xcf5 bytes) 000971 1788803625.869517 mapread 0x01a512 thru 0x025697 (0xb186 bytes) 000973 1788803625.890862 fallocate 0x03d61b thru 0x03ffff (0x29e5 bytes) 000974 1788803625.924851 read 0x023793 thru 0x032ea7 (0xf715 bytes) 000976 1788803625.944808 fallocate 0x0222e5 thru 0x02897e (0x669a bytes) 000977 1788803625.950293 fallocate 0x0396d6 thru 0x039826 (0x151 bytes) 000979 1788803625.972219 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6cb53f95 (stale) Mirror 5: 0x6cb53f95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6cb53f95 (stale) Mirror 5: 0x6cb53f95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000980 1788803626.389661 mapread 0x008563 thru 0x00fe09 (0x78a7 bytes) 000983 1788803626.423533 trunc from 0x01a6b9 to 0x01ab9d (0x4e5 bytes) 000984 1788803626.435655 read 0x03622e thru 0x03ffff (0x9dd2 bytes) 000985 1788803626.453306 write 0x025856 thru 0x02bff1 (0x679c bytes) 000986 1788803626.474734 fallocate 0x016fb7 thru 0x01ba1a (0x4a64 bytes) 000987 1788803626.482819 read 0x0346d7 thru 0x03a1a2 (0x5acc bytes) 000988 1788803626.511858 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6cb53f95 (stale) Mirror 5: 0x6cb53f95 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdda214a8 Mirror 6: 0xdda214a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000990 1788803627.736395 trunc from 0x001be9 to 0x009c07 (0x801f bytes) 000991 1788803627.782198 punch from 0x00c0ab to 0x010c69 (0x4bbf bytes) 000992 1788803627.788692 fallocate 0x01a36c thru 0x02286b (0x8500 bytes) 000993 1788803627.796687 mapwrite 0x00b363 thru 0x01258a (0x7228 bytes) 000995 1788803627.842842 fallocate 0x038285 thru 0x03ffff (0x7d7b bytes) 000996 1788803627.847277 mapread 0x0184c6 thru 0x019e13 (0x194e bytes) 000997 1788803627.857367 write 0x02fce6 thru 0x03a024 (0xa33f bytes) 000998 1788803627.871961 punch from 0x00db5c to 0x01d983 (0xfe28 bytes) 000999 1788803627.878254 punch from 0x008953 to 0x0143a4 (0xba52 bytes) 001000 1788803627.883301 mapwrite 0x016d2a thru 0x0184cd (0x17a4 bytes) All operations completed A-OK! PASS 70a (80s) == sanity-flr test 71: check mirror extend parent fid ==== 13:53:53 (1788803633) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1f8:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 13:54:03 (1788803643) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143334 s, 14.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223479 s, 4.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 13:54:14 (1788803654) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4500480 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2547584 .. failed Extending file size to 8196256 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4703296 .. failed Extending file size to 6037504 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8875328 .. Extending file size to 7603104 .. Extending file size to 8384480 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8759552 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9125312 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8975136 .. done Waiting 61363 61365 61367 61369 61371 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) 192.168.204.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:55:32 (1788803732) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4903712 .. Extending file size to 3894784 .. Extending file size to 7694656 .. Extending file size to 8852192 .. Extending file size to 4566304 .. Extending file size to 8798048 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9038976 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9008256 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62791 62792 62795 62796 62798 done 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) 192.168.204.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (87s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:56:59 (1788803819) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2176 1269128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17708 3589312 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25372 3581648 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43080 7170960 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2176 1269128 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17708 3589312 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25372 3581648 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43080 7170960 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161225 s, 13.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245046 s, 8.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbe9a6cb1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:57:21 (1788803841) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105009 s, 20.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 13:57:28 (1788803848) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x39:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x103:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:57:36 (1788803856) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0643629 s, 32.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x247:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x248:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:57:45 (1788803865) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0830125 s, 25.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:57:53 (1788803873) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132323 s, 15.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:58:26 (1788803906) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465306 s, 22.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:59:22 (1788803962) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0827774 s, 25.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:00:09 (1788804009) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg448-server: Pool lustre.test_204f created Adding targets to pool oleg448-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg448-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114694 s, 18.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x249:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x249:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg448-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg448-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg448-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:01:26 (1788804086) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:01:35 (1788804095) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:01:43 (1788804103) oleg448-server: Pool lustre.test_206 created Adding targets to pool oleg448-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg448-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg448-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg448-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg448-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:02:10 (1788804130) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761866 s, 13.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364067 s, 28.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x69dbf996 Mirror 3: 0x69dbf996 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:02:20 (1788804140) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:02:23 (1788804143) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:02:27 (1788804147) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0915175 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1080000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1720000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1720000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1720000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x18a0000 Read this (4 bytes): BBBB 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:03:02 (1788804182) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0169495 s, 483 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:03:09 (1788804189) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837701 s, 12.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611226 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 14:03:11.000000000 -0400 Modify: 2026-09-07 14:03:11.000000000 -0400 Change: 2026-09-07 14:03:12.000000000 -0400 Birth: 2026-09-07 14:03:11.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:03:17 (1788804197) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326488 s, 32.1 MB/s oleg448-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg448-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg448-server: O/280000401/d21/135: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:03:37 (1788804217) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.264994 s, 155 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0171922 s, 238 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 14:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 14:03 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:03:45 (1788804225) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:03:52 (1788804232) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1873 sec =========== 14:04:02 (1788804242) === sanity-flr: start cleanup 14:04:03 (1788804243) === === sanity-flr: finish cleanup 14:04:13 (1788804253) ===