-----============= acceptance-small: sanity-flr ============----- Thu Apr 9 15:42:01 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg311-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:42:19 (1775763739) === oleg311-client.virtnet: executing check_config_client /mnt/lustre oleg311-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg311-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f9087d65000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f9087d65000.idle_timeout=debug disable quota as required oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:42:50 (1775763770) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7559] == sanity-flr test 0a: lfs mirror create with -N option == 15:42:52 (1775763772) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:43:09 (1775763789) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:43:12 (1775763792) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:43:15 (1775763795) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:43:41 (1775763821) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:43:44 (1775763824) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:43:48 (1775763828) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f9087d65000.stats=clear osc.lustre-OST0001-osc-ffff9f9087d65000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:43:56 (1775763836) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:44:05 (1775763845) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:44:13 (1775763853) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:44:16 (1775763856) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 15:44:24 (1775763864) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:44:32 (1775763872) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:44:35 (1775763875) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:44:43 (1775763883) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:44:53 (1775763893) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:44:59 (1775763899) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 2.40384 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173413 s, 6.0 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:45:13 (1775763913) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.329212 s, 38.2 MB/s PASS 22 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:45:23 (1775763923) Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:46:37 (1775763997) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.502977 s, 20.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:47:14 (1775764034) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:48:28 (1775764108) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.79702 s, 9.0 MB/s -rw-r--r-- 1 root root 52428800 Apr 9 15:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.057s user 0m0.001s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9087d65000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.987s user 0m0.002s sys 0m0.193s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.218s user 0m0.000s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9087d65000.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:49:58 (1775764198) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:50:02 (1775764202) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:50:05 (1775764205) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:50:08 (1775764208) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 15:50:15 (1775764215) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.481354 s, 26.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:50:27 (1775764227) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674813 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155142 s, 67.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:50:35 (1775764235) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:50:44 (1775764244) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129845 s, 789 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000917898 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000760443 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000835866 s, 1.2 MB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 15:50:58 (1775764258) create test files with size 15744914 .. b840b0d518bef797dfe026c7e477e2c1 - 627eceec889afbc9306d6902402fd72a - 47673fcf0083ab74d9b1f9fce44dc928 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 15:51:27 (1775764287) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (21s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:51:48 (1775764308) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:51:57 (1775764317) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176513 s, 5.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703042 s, 14.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 15:52:06 (1775764326) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139513 s, 15.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.298878 s, 14.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192268 s, 10.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 15:52:21 (1775764341) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:52:24 (1775764344) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:52:27 (1775764347) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428114 s, 24.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 15:52:34 (1775764354) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:52:37 (1775764357) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:52:46 (1775764366) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.254101 s, 41.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:52:56 (1775764376) llite.lustre-ffff9f908ac3a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.181536 s, 57.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f9082e66000.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:53:07 (1775764387) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:53:44 (1775764424) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:54:19 (1775764459) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 15:54:22 (1775764462) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.234441 s, 13.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.1607 s, 19.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x689e3e9d (stale) Mirror 2: 0x689e3e9d (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: b444a62823421badc48764c831533524 - mirror 1 checksum: 13ef14e6f14852ae4c3a6100efa0469e - mirror 2 checksum: b444a62823421badc48764c831533524 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 9 15:54:24 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 9 15:54:24 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b444a62823421badc48764c831533524 - mirror 1 checksum: 13ef14e6f14852ae4c3a6100efa0469e - mirror 2 checksum: 13ef14e6f14852ae4c3a6100efa0469e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 9 15:54:24 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 9 15:54:24 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:54:33 (1775764473) Failing mds1 on oleg311-server Stopping /mnt/lustre-mds1 (opts:) on oleg311-server 15:54:41 (1775764481) shut down facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server Failover mds1 to oleg311-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 15:55:14 (1775764514) targets are mounted 15:55:14 (1775764514) facet_failover done oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210232 s, 49.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:55:31 (1775764531) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:55:34 (1775764534) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:55:37 (1775764537) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00605919 s, 2.7 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:55:45 (1775764545) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:55:48 (1775764548) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:55:52 (1775764552) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:56:15 (1775764575) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:56:44 (1775764604) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:57:08 (1775764628) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:57:31 (1775764651) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 15:57:38 (1775764658) mirror create pid 52298 mirror split pid 52300 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7721: 52300 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52298 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 15:58:46 (1775764726) Seed set to 4728 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x20924 000002 1775764728.433264 trunc from 00000000 to 0x020923 (0x20924 bytes) 000004 1775764728.474311 MIRROR_EXTEND to 2 mirrors 000005 1775764729.234127 MIRROR_RESYNC 2 mirrors 000006 1775764729.610060 read 0x00091a thru 0x001516 (0xbfd bytes) 000009 1775764729.617047 mapwrite 0x008667 thru 0x0086d6 (0x70 bytes) 000010 1775764729.801478 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20924): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20924): Mirror 2: 0xb47f8127 (stale) CRC-32 checksum value for chunk [0, 0x20924): Mirror 1: 0x62fd1ab4 Mirror 3: 0x62fd1ab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20924: skipped 000011 1775764730.634638 read 0x00b375 thru 0x017ee3 (0xcb6f bytes) 000012 1775764730.647372 mapread 0x01d5c3 thru 0x020923 (0x3361 bytes) 000013 1775764730.673846 write 0x0209e2 thru 0x02bcd1 (0xb2f0 bytes) 000014 1775764730.791263 write 0x01d7a0 thru 0x025526 (0x7d87 bytes) 000015 1775764730.792496 mapread 0x005d28 thru 0x0118ac (0xbb85 bytes) 000016 1775764730.815816 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2bcd2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcd2): Mirror 2: 0xc5891a4d (stale) Mirror 3: 0xa52223bb (stale) CRC-32 checksum value for chunk [0, 0x2bcd2): Mirror 1: 0x84b12938 Mirror 4: 0x84b12938 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcd2: skipped 000021 1775764731.831579 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bcd2): Mirror 2: 0xc5891a4d (stale) Mirror 3: 0xa52223bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcd2): Mirror 2: 0xc5891a4d (stale) Mirror 3: 0xa52223bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcd2: skipped 000023 1775764732.152717 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bcd2): Mirror 3: 0xa52223bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcd2): Mirror 3: 0xa52223bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcd2: skipped 000024 1775764732.403729 read 0x021110 thru 0x02bcd1 (0xabc2 bytes) 000026 1775764732.422246 MIRROR_RESYNC 2 mirrors 000028 1775764733.273246 mapwrite 0x028564 thru 0x02ed6d (0x680a bytes) 000030 1775764733.443411 trunc from 0x02ed6e to 0x00366c (0x2b701 bytes) truncating to largest ever: 0x2bbba 000031 1775764733.477648 trunc from 0x00366d to 0x02bbb9 (0x2854d bytes) 000034 1775764733.506973 mapread 0x01c43b thru 0x0290aa (0xcc70 bytes) 000035 1775764733.547076 write 0x019e7d thru 0x01f78a (0x590e bytes) 000036 1775764733.563094 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bbba): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bbba): Mirror 3: 0xbe7db645 (stale) CRC-32 checksum value for chunk [0, 0x2bbba): Mirror 1: 0xd5a2e83 Mirror 4: 0xd5a2e83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bbba: skipped 000037 1775764734.572228 mapread 0x0006f4 thru 0x001d44 (0x1651 bytes) 000038 1775764734.598171 mapread 0x017155 thru 0x022385 (0xb231 bytes) 000040 1775764734.654493 read 0x015db6 thru 0x024c6d (0xeeb8 bytes) 000041 1775764734.654997 read 0x0274da thru 0x02a794 (0x32bb bytes) 000044 1775764734.677804 read 0x012bc0 thru 0x01f189 (0xc5ca bytes) 000048 1775764734.679284 read 0x0144c0 thru 0x01522b (0xd6c bytes) 000049 1775764734.679704 read 0x01955d thru 0x020a37 (0x74db bytes) 000050 1775764734.680076 mapread 0x013fda thru 0x01fe9c (0xbec3 bytes) 000051 1775764734.681047 mapread 0x0165ea thru 0x01f3a7 (0x8dbe bytes) 000052 1775764734.696370 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2bbba): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bbba): Mirror 3: 0xbe7db645 (stale) CRC-32 checksum value for chunk [0, 0x2bbba): Mirror 1: 0xd5a2e83 Mirror 4: 0xd5a2e83 Mirror 5: 0xd5a2e83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bbba: skipped 000053 1775764735.607914 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000054 1775764736.002233 read 0x027a43 thru 0x02bbb9 (0x4177 bytes) 000056 1775764736.021935 write 0x021cee thru 0x02ff18 (0xe22b bytes) 000057 1775764736.144306 mapwrite 0x030221 thru 0x034e78 (0x4c58 bytes) 000058 1775764736.306250 write 0x01033f thru 0x01b041 (0xad03 bytes) 000059 1775764736.327174 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34e79): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e79): Mirror 4: 0x8738d597 (stale) Mirror 5: 0x8738d597 (stale) CRC-32 checksum value for chunk [0, 0x34e79): Mirror 1: 0x4c974337 Mirror 6: 0x4c974337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e79: skipped 000060 1775764737.078162 mapwrite 0x00b89a thru 0x00ef1a (0x3681 bytes) 000061 1775764737.275470 trunc from 0x034e79 to 0x00b2a4 (0x29bd4 bytes) 000062 1775764737.336185 mapwrite 0x0356b1 thru 0x0360a6 (0x9f6 bytes) truncating to largest ever: 0x3c34e 000064 1775764737.417832 trunc from 0x0360a7 to 0x03c34d (0x62a7 bytes) 000065 1775764737.440111 mapread 0x00edf5 thru 0x01de4b (0xf057 bytes) 000069 1775764737.476001 mapread 0x028acd thru 0x02cfb2 (0x44e6 bytes) 000072 1775764737.491307 mapwrite 0x01e4ca thru 0x02a770 (0xc2a7 bytes) 000074 1775764737.601125 trunc from 0x03c34e to 0x01557d (0x26dd0 bytes) 000076 1775764737.631061 trunc from 0x01557e to 0x0258c7 (0x1034a bytes) 000078 1775764737.666862 trunc from 0x0258c8 to 0x001e0a (0x23abd bytes) 000081 1775764737.691627 write 0x0021bb thru 0x006f94 (0x4dda bytes) 000084 1775764737.695081 mapwrite 0x0255d1 thru 0x02f8be (0xa2ee bytes) 000085 1775764737.822362 mapwrite 0x039ad7 thru 0x03ffff (0x6529 bytes) 000089 1775764737.926308 mapread 0x01b991 thru 0x026f92 (0xb602 bytes) 000090 1775764737.944258 read 0x0118f8 thru 0x021700 (0xfe09 bytes) 000092 1775764737.958350 mapwrite 0x017088 thru 0x019cf8 (0x2c71 bytes) 000094 1775764738.041385 write 0x00cee4 thru 0x00fd85 (0x2ea2 bytes) 000095 1775764738.058938 write 0x03a6f3 thru 0x03b8df (0x11ed bytes) 000097 1775764738.062148 mapwrite 0x016909 thru 0x019b76 (0x326e bytes) 000100 1775764738.145147 read 0x0189b0 thru 0x021fcd (0x961e bytes) 000101 1775764738.152055 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cf5812d (stale) Mirror 5: 0x8cf5812d (stale) Mirror 6: 0x93bd8d7e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x42bb1d4 Mirror 7: 0x42bb1d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000102 1775764738.949021 trunc from 0x040000 to 0x0056a4 (0x3a95b bytes) 000103 1775764739.066157 write 0x01005a thru 0x014064 (0x400b bytes) 000107 1775764739.073888 write 0x0004e7 thru 0x00b399 (0xaeb3 bytes) 000109 1775764739.096323 mapwrite 0x014889 thru 0x01fcf8 (0xb470 bytes) 000110 1775764739.258096 write 0x02d19a thru 0x03af40 (0xdda7 bytes) 000112 1775764739.265916 write 0x0356a8 thru 0x03ffff (0xa958 bytes) 000113 1775764739.271028 mapwrite 0x039129 thru 0x03ffff (0x6ed7 bytes) 000114 1775764739.336182 read 0x01b578 thru 0x020b1d (0x55a6 bytes) 000115 1775764739.348898 trunc from 0x040000 to 0x00ae92 (0x3516d bytes) 000116 1775764739.374516 read 0x002dbe thru 0x00ae92 (0x80d5 bytes) 000117 1775764739.375163 write 0x03adf3 thru 0x03ffff (0x520d bytes) 000119 1775764739.381284 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cf5812d (stale) Mirror 5: 0x8cf5812d (stale) Mirror 6: 0x93bd8d7e (stale) Mirror 7: 0x42bb1d4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79967da9 Mirror 8: 0x79967da9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000120 1775764740.315223 mapwrite 0x029865 thru 0x03621a (0xc9b6 bytes) 000121 1775764740.466516 read 0x0149cd thru 0x0221e8 (0xd81c bytes) truncating to largest ever: 0x3cc45 000122 1775764740.472751 trunc from 0x040000 to 0x03cc44 (0x33bb bytes) 000123 1775764740.511279 read 0x007962 thru 0x00d918 (0x5fb7 bytes) 000124 1775764740.519471 mapwrite 0x03fb4b thru 0x03ffff (0x4b5 bytes) 000125 1775764740.620120 read 0x018d04 thru 0x01efe1 (0x62de bytes) 000126 1775764740.620855 trunc from 0x040000 to 0x00320b (0x3cdf4 bytes) 000127 1775764740.689727 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x320c): Mirror 4: 0xc625a4dc (stale) Mirror 5: 0xc625a4dc (stale) Mirror 7: 0xccf48314 (stale) Mirror 8: 0x394d85ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x320c): Mirror 4: 0xc625a4dc (stale) Mirror 5: 0xc625a4dc (stale) Mirror 7: 0xccf48314 (stale) Mirror 8: 0x394d85ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x320c: skipped 000128 1775764741.115731 read 0x00132f thru 0x00320b (0x1edd bytes) 000130 1775764741.141753 mapwrite 0x0317ef thru 0x03ffff (0xe811 bytes) 000131 1775764741.345404 mapwrite 0x035322 thru 0x03ffff (0xacde bytes) 000132 1775764741.446122 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8cf5812d (stale) Mirror 7: 0x42bb1d4 (stale) Mirror 8: 0x79967da9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8cf5812d (stale) Mirror 7: 0x42bb1d4 (stale) Mirror 8: 0x79967da9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000133 1775764741.799498 mapwrite 0x038031 thru 0x039f54 (0x1f24 bytes) 000134 1775764741.871214 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8cf5812d (stale) Mirror 7: 0x42bb1d4 (stale) Mirror 8: 0x79967da9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x25e2e77d Mirror 9: 0x25e2e77d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000135 1775764742.901569 mapwrite 0x02a489 thru 0x02f3b6 (0x4f2e bytes) 000136 1775764743.163272 mapread 0x0192e0 thru 0x01db6f (0x4890 bytes) 000138 1775764743.183082 mapread 0x01083d thru 0x01c822 (0xbfe6 bytes) 000140 1775764743.206183 trunc from 0x040000 to 0x01202e (0x2dfd1 bytes) 000141 1775764743.247206 mapwrite 0x006214 thru 0x00842a (0x2217 bytes) 000142 1775764743.338223 mapread 0x00ee5b thru 0x011165 (0x230b bytes) 000143 1775764743.339384 mapread 0x00411e thru 0x01202e (0xdf11 bytes) 000144 1775764743.349631 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1202f): Mirror 5: 0xfa96a84f (stale) Mirror 7: 0x9710e90c (stale) Mirror 8: 0xf07c9067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1202f): Mirror 5: 0xfa96a84f (stale) Mirror 7: 0x9710e90c (stale) Mirror 8: 0xf07c9067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1202f: skipped 000145 1775764743.667649 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1202f): Mirror 7: 0x9710e90c (stale) Mirror 8: 0xf07c9067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1202f): Mirror 7: 0x9710e90c (stale) Mirror 8: 0xf07c9067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1202f: skipped 000147 1775764743.939873 read 0x00a332 thru 0x00dfc6 (0x3c95 bytes) 000151 1775764743.956298 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1202f): Mirror 8: 0xf07c9067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1202f): Mirror 8: 0xf07c9067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1202f: skipped 000152 1775764744.281809 mapread 0x00dd3c thru 0x01202e (0x42f3 bytes) 000153 1775764744.330862 mapread 0x00763d thru 0x007e5d (0x821 bytes) 000154 1775764744.360892 trunc from 0x01202f to 0x002471 (0xfbbd bytes) 000157 1775764744.383856 read 0x0003d9 thru 0x002471 (0x2099 bytes) 000159 1775764744.392321 read 0x001067 thru 0x002471 (0x140b bytes) 000163 1775764744.408836 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2472): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2472): Mirror 8: 0x98192c50 (stale) CRC-32 checksum value for chunk [0, 0x2472): Mirror 1: 0x98192c50 Mirror 9: 0x98192c50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2472: skipped 000164 1775764745.180667 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000165 1775764745.546729 write 0x031f59 thru 0x03afbe (0x9066 bytes) 000167 1775764745.618059 write 0x0314f7 thru 0x03ffff (0xeb09 bytes) 000170 1775764745.621669 read 0x00f4d4 thru 0x017ce5 (0x8812 bytes) 000171 1775764745.629349 write 0x00f013 thru 0x01bf86 (0xcf74 bytes) 000172 1775764745.641015 mapwrite 0x02ecdf thru 0x03497e (0x5ca0 bytes) 000173 1775764745.792543 write 0x000f86 thru 0x00ee27 (0xdea2 bytes) 000175 1775764745.807189 trunc from 0x040000 to 0x0098f6 (0x36709 bytes) 000176 1775764745.878168 trunc from 0x0098f7 to 0x036f27 (0x2d631 bytes) 000179 1775764745.909578 mapread 0x02a6fc thru 0x02b914 (0x1219 bytes) 000186 1775764745.929540 write 0x032479 thru 0x03b784 (0x930c bytes) 000187 1775764745.939204 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000195 1775764746.208398 mapread 0x001e2c thru 0x004164 (0x2339 bytes) 000199 1775764746.267108 write 0x02f34f thru 0x03311a (0x3dcc bytes) 000201 1775764746.290262 mapread 0x010f88 thru 0x0115c7 (0x640 bytes) 000203 1775764746.310864 mapread 0x03502b thru 0x03b784 (0x675a bytes) 000206 1775764746.319726 mapread 0x03231a thru 0x038ced (0x69d4 bytes) 000208 1775764746.328151 mapread 0x0073bc thru 0x00cfd9 (0x5c1e bytes) 000211 1775764746.337931 read 0x00c98a thru 0x018e29 (0xc4a0 bytes) 000212 1775764746.341986 mapread 0x0225b5 thru 0x02e88d (0xc2d9 bytes) 000213 1775764746.367914 trunc from 0x03b785 to 0x0027bb (0x38fc9 bytes) 000214 1775764746.390895 mapread 0x00233e thru 0x0027bb (0x47e bytes) 000216 1775764746.392470 write 0x03d265 thru 0x03ffff (0x2d9b bytes) 000220 1775764746.406207 write 0x014f97 thru 0x024f92 (0xfffc bytes) 000221 1775764746.431789 write 0x03c408 thru 0x03e7ab (0x23a4 bytes) 000222 1775764746.444035 write 0x02be1f thru 0x034708 (0x88ea bytes) 000224 1775764746.475103 write 0x032329 thru 0x03bb59 (0x9831 bytes) 000227 1775764746.482248 read 0x0087b5 thru 0x011619 (0x8e65 bytes) 000228 1775764746.500784 mapread 0x035bfa thru 0x037232 (0x1639 bytes) 000229 1775764746.501813 write 0x007de2 thru 0x00843b (0x65a bytes) 000230 1775764746.502749 mapwrite 0x0240ae thru 0x0288bf (0x4812 bytes) 000233 1775764746.593721 mapwrite 0x001090 thru 0x00a6e5 (0x9656 bytes) 000235 1775764746.641240 mapwrite 0x00dc72 thru 0x0166e1 (0x8a70 bytes) 000236 1775764746.687805 read 0x018b38 thru 0x02725c (0xe725 bytes) 000237 1775764746.688860 read 0x03dd81 thru 0x03ffff (0x227f bytes) 000238 1775764746.690089 write 0x018174 thru 0x02264e (0xa4db bytes) 000241 1775764746.692968 write 0x01a096 thru 0x0295ab (0xf516 bytes) 000242 1775764746.695272 mapread 0x00f3ab thru 0x0198f2 (0xa548 bytes) 000244 1775764746.700952 write 0x0346d6 thru 0x03e9ca (0xa2f5 bytes) 000246 1775764746.703836 mapwrite 0x0106e6 thru 0x01334a (0x2c65 bytes) 000247 1775764746.748386 mapwrite 0x001519 thru 0x007e2d (0x6915 bytes) 000248 1775764746.791192 mapwrite 0x03153c thru 0x03a450 (0x8f15 bytes) 000250 1775764746.834602 read 0x020996 thru 0x022c25 (0x2290 bytes) 000251 1775764746.842625 MIRROR_EXTEND to 2 mirrors 000253 1775764747.614387 trunc from 0x040000 to 0x018bd9 (0x27426 bytes) 000257 1775764747.695974 trunc from 0x018bda to 0x0088a6 (0x10333 bytes) 000258 1775764747.714035 trunc from 0x0088a7 to 0x01cddc (0x14536 bytes) 000259 1775764747.752163 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1cddd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cddd): Mirror 2: 0xb094497b (stale) CRC-32 checksum value for chunk [0, 0x1cddd): Mirror 1: 0xdecbbe1a Mirror 3: 0xdecbbe1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cddd: skipped 000261 1775764748.960388 write 0x03e137 thru 0x03ffff (0x1ec9 bytes) 000263 1775764749.061122 read 0x008c0f thru 0x01145c (0x884e bytes) 000264 1775764749.069582 write 0x025607 thru 0x032477 (0xce71 bytes) 000265 1775764749.086790 mapwrite 0x03ced8 thru 0x03ffff (0x3128 bytes) 000267 1775764749.170712 mapwrite 0x023a0a thru 0x028416 (0x4a0d bytes) 000270 1775764749.255772 mapwrite 0x00f263 thru 0x01b5c6 (0xc364 bytes) 000271 1775764749.392792 mapread 0x00ec12 thru 0x0186eb (0x9ada bytes) 000272 1775764749.396066 mapwrite 0x005b3a thru 0x015086 (0xf54d bytes) 000274 1775764749.536958 MIRROR_RESYNC 3 mirrors 000277 1775764750.523758 mapread 0x03f5af thru 0x03ffff (0xa51 bytes) 000278 1775764750.538325 trunc from 0x040000 to 0x035927 (0xa6d8 bytes) 000279 1775764750.669113 mapwrite 0x0093b9 thru 0x011e32 (0x8a7a bytes) 000281 1775764750.768242 mapread 0x00c8d9 thru 0x0165b9 (0x9ce1 bytes) 000282 1775764750.780375 mapwrite 0x00af1f thru 0x00ec2d (0x3d0f bytes) 000284 1775764750.844657 write 0x02a5bf thru 0x034900 (0xa342 bytes) 000285 1775764750.863742 mapread 0x03124d thru 0x035927 (0x46db bytes) 000289 1775764750.883719 mapwrite 0x03a19c thru 0x03ffff (0x5e64 bytes) 000291 1775764751.009124 trunc from 0x040000 to 0x0250f8 (0x1af07 bytes) 000293 1775764751.075964 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x250f9): Mirror 3: 0xa69d8833 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x250f9): Mirror 3: 0xa69d8833 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x250f9: skipped 000295 1775764751.486751 mapread 0x00acec thru 0x016b41 (0xbe56 bytes) 000296 1775764751.518185 write 0x03f465 thru 0x03ffff (0xb9b bytes) 000298 1775764751.521176 trunc from 0x040000 to 0x00d5a9 (0x32a56 bytes) 000299 1775764751.555565 mapwrite 0x01681c thru 0x01e67a (0x7e5f bytes) 000304 1775764751.715273 mapread 0x011949 thru 0x01cc2e (0xb2e6 bytes) 000307 1775764751.722762 trunc from 0x01e67b to 0x01d8c4 (0xdb6 bytes) 000311 1775764751.769868 trunc from 0x01d8c5 to 0x0161b6 (0x770e bytes) 000313 1775764751.801109 mapread 0x00ca6f thru 0x0145f0 (0x7b82 bytes) 000314 1775764751.817404 trunc from 0x0161b7 to 0x01fbdb (0x9a25 bytes) 000318 1775764751.868114 write 0x02f987 thru 0x03adc4 (0xb43e bytes) 000321 1775764751.879796 write 0x00dbc3 thru 0x010ab6 (0x2ef4 bytes) 000323 1775764751.886922 trunc from 0x03adc5 to 0x011ce4 (0x290e0 bytes) 000326 1775764751.922351 read 0x0013f3 thru 0x010774 (0xf382 bytes) 000327 1775764751.939270 mapread 0x0097ec thru 0x011ce4 (0x84f9 bytes) 000328 1775764751.947632 read 0x00953e thru 0x011ce4 (0x87a7 bytes) 000330 1775764751.958264 write 0x0238bd thru 0x02cd85 (0x94c9 bytes) 000331 1775764751.967656 read 0x02cc4c thru 0x02cd85 (0x13a bytes) 000332 1775764751.975631 mapread 0x01b083 thru 0x01dfaa (0x2f28 bytes) 000334 1775764751.992117 read 0x02ca08 thru 0x02cd85 (0x37e bytes) 000335 1775764751.995237 mapread 0x016018 thru 0x02327f (0xd268 bytes) 000336 1775764752.001994 mapwrite 0x005ff6 thru 0x00b79b (0x57a6 bytes) 000337 1775764752.100171 read 0x020bb7 thru 0x02cd85 (0xc1cf bytes) 000340 1775764752.131649 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2cd86): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd86): Mirror 3: 0xe6f7a2a (stale) CRC-32 checksum value for chunk [0, 0x2cd86): Mirror 1: 0xfd0b8cc7 Mirror 4: 0xfd0b8cc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd86: skipped 000341 1775764753.006825 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd86): Mirror 3: 0xe6f7a2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd86): Mirror 3: 0xe6f7a2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd86: skipped 000342 1775764753.369205 read 0x000457 thru 0x00cb41 (0xc6eb bytes) 000343 1775764753.382994 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000345 1775764753.564039 trunc from 0x02cd86 to 0x00212c (0x2ac59 bytes) 000346 1775764753.639579 write 0x0348bc thru 0x038672 (0x3db7 bytes) 000347 1775764753.641919 write 0x01a91d thru 0x020306 (0x59ea bytes) 000348 1775764753.669662 MIRROR_EXTEND to 2 mirrors 000349 1775764754.513633 mapread 0x00711a thru 0x00fabf (0x89a6 bytes) 000351 1775764754.551065 MIRROR_RESYNC 2 mirrors 000352 1775764754.961652 mapwrite 0x031fc5 thru 0x03ca85 (0xaac1 bytes) 000355 1775764755.140909 trunc from 0x03ca86 to 0x028ff8 (0x13a8d bytes) 000357 1775764755.170255 mapread 0x0283e4 thru 0x028ff8 (0xc15 bytes) 000360 1775764755.188925 mapread 0x00b910 thru 0x015b72 (0xa263 bytes) 000361 1775764755.221490 trunc from 0x028ff9 to 0x037ba9 (0xebb1 bytes) 000362 1775764755.265543 trunc from 0x037baa to 0x008a09 (0x2f1a0 bytes) 000364 1775764755.303213 write 0x02a1f9 thru 0x02e91b (0x4723 bytes) 000368 1775764755.316763 read 0x007973 thru 0x0095a1 (0x1c2f bytes) 000369 1775764755.329972 mapread 0x019c92 thru 0x01df68 (0x42d7 bytes) 000371 1775764755.340864 trunc from 0x02e91c to 0x03787d (0x8f62 bytes) 000372 1775764755.379797 trunc from 0x03787e to 0x017ba9 (0x1fcd4 bytes) 000373 1775764755.410381 mapread 0x0125d5 thru 0x017ba9 (0x55d5 bytes) 000375 1775764755.412197 mapwrite 0x0030b2 thru 0x012854 (0xf7a3 bytes) 000378 1775764755.582350 mapwrite 0x007d90 thru 0x016dff (0xf070 bytes) 000381 1775764755.736368 read 0x0148be thru 0x017ba9 (0x32ec bytes) 000382 1775764755.753666 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x17baa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17baa): Mirror 2: 0xce57ef4e (stale) CRC-32 checksum value for chunk [0, 0x17baa): Mirror 1: 0x543e5dd6 Mirror 3: 0x543e5dd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17baa: skipped 000383 1775764756.698104 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000384 1775764757.048771 mapread 0x00caf1 thru 0x00ebe3 (0x20f3 bytes) 000385 1775764757.106135 trunc from 0x017baa to 0x011e80 (0x5d29 bytes) 000387 1775764757.248039 mapwrite 0x00294e thru 0x005e5a (0x350d bytes) 000389 1775764757.304942 MIRROR_RESYNC 2 mirrors 000390 1775764758.027782 trunc from 0x011e81 to 0x026ac1 (0x14c41 bytes) 000391 1775764758.170855 read 0x002d51 thru 0x00a8c3 (0x7b73 bytes) 000393 1775764758.196339 MIRROR_RESYNC 2 mirrors 000394 1775764758.996173 read 0x01c2cf thru 0x01f9f8 (0x372a bytes) 000395 1775764759.007915 mapread 0x003555 thru 0x012b7c (0xf628 bytes) 000398 1775764759.063845 read 0x010190 thru 0x016ef0 (0x6d61 bytes) 000400 1775764759.064924 trunc from 0x026ac2 to 0x00556d (0x21554 bytes) 000403 1775764759.195882 read 0x003344 thru 0x00556d (0x222a bytes) 000404 1775764759.204381 mapread 0x001567 thru 0x00556d (0x4007 bytes) 000406 1775764759.206449 write 0x03fe30 thru 0x03ffff (0x1d0 bytes) 000407 1775764759.208660 read 0x03ed88 thru 0x03ffff (0x1278 bytes) 000410 1775764759.237596 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1f2c040e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf233de20 Mirror 4: 0xf233de20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000411 1775764760.153194 read 0x02e29b thru 0x03980e (0xb574 bytes) 000412 1775764760.161906 write 0x01fc04 thru 0x024acc (0x4ec9 bytes) 000416 1775764760.297461 trunc from 0x040000 to 0x02edcb (0x11234 bytes) 000417 1775764760.374046 read 0x023891 thru 0x026b0f (0x327f bytes) 000418 1775764760.391035 mapread 0x00f01d thru 0x015ddf (0x6dc3 bytes) 000421 1775764760.395196 mapread 0x00b3c6 thru 0x018b3e (0xd779 bytes) 000424 1775764760.400706 trunc from 0x02edcc to 0x0036d8 (0x2b6f3 bytes) 000426 1775764760.427528 write 0x01113d thru 0x015d5d (0x4c21 bytes) 000427 1775764760.428772 read 0x014552 thru 0x015d5d (0x180c bytes) 000428 1775764760.429367 mapread 0x003437 thru 0x00d473 (0xa03d bytes) 000433 1775764760.446723 read 0x015cc5 thru 0x015d5d (0x99 bytes) 000434 1775764760.447723 mapwrite 0x00998b thru 0x017f6f (0xe5e5 bytes) 000435 1775764760.550891 MIRROR_RESYNC 3 mirrors 000436 1775764761.370881 write 0x0228bb thru 0x0285e3 (0x5d29 bytes) 000437 1775764761.514071 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x285e4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285e4): Mirror 3: 0x96ff17da (stale) Mirror 4: 0x96ff17da (stale) CRC-32 checksum value for chunk [0, 0x285e4): Mirror 1: 0xf7681fd3 Mirror 5: 0xf7681fd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285e4: skipped 000439 1775764762.384218 read 0x003514 thru 0x00a114 (0x6c01 bytes) 000444 1775764762.437981 mapwrite 0x0195ce thru 0x01a178 (0xbab bytes) 000445 1775764762.641264 trunc from 0x0285e4 to 0x0157a3 (0x12e40 bytes) 000446 1775764762.671041 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x157a4): Mirror 4: 0xdf7c04f0 (stale) Mirror 5: 0xdf7c04f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x157a4): Mirror 4: 0xdf7c04f0 (stale) Mirror 5: 0xdf7c04f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x157a4: skipped 000447 1775764763.084579 mapread 0x006ffa thru 0x0157a3 (0xe7aa bytes) 000449 1775764763.134136 trunc from 0x0157a4 to 0x0365d7 (0x20e34 bytes) 000453 1775764763.162750 read 0x00c7bc thru 0x016d0a (0xa54f bytes) 000454 1775764763.183339 mapwrite 0x03a708 thru 0x03c39e (0x1c97 bytes) 000455 1775764763.244294 read 0x03acfc thru 0x03c39e (0x16a3 bytes) 000456 1775764763.245014 mapread 0x02baf2 thru 0x032fc1 (0x74d0 bytes) 000459 1775764763.256189 trunc from 0x03c39f to 0x00884f (0x33b4f bytes) 000460 1775764763.276154 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8850): Mirror 5: 0x47954e68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8850): Mirror 5: 0x47954e68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8850: skipped 000464 1775764763.508821 trunc from 0x008850 to 0x039e4b (0x315fc bytes) 000465 1775764763.523935 read 0x00284a thru 0x0052bc (0x2a73 bytes) truncating to largest ever: 0x3d4d6 000466 1775764763.530107 trunc from 0x039e4c to 0x03d4d5 (0x368a bytes) 000469 1775764763.548185 read 0x038c4d thru 0x03b6ec (0x2aa0 bytes) 000470 1775764763.561236 write 0x03c693 thru 0x03ffff (0x396d bytes) 000474 1775764763.565139 read 0x013d29 thru 0x01a4f4 (0x67cc bytes) 000476 1775764763.571353 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000477 1775764763.690772 mapwrite 0x01e7b3 thru 0x02e410 (0xfc5e bytes) 000478 1775764763.763588 write 0x025456 thru 0x030405 (0xafb0 bytes) 000480 1775764763.768985 write 0x02288c thru 0x030028 (0xd79d bytes) 000482 1775764763.772029 trunc from 0x040000 to 0x038468 (0x7b97 bytes) 000487 1775764763.804849 trunc from 0x038469 to 0x020673 (0x17df5 bytes) 000488 1775764763.834628 mapwrite 0x02a0ba thru 0x02d73a (0x3681 bytes) 000490 1775764763.926743 mapwrite 0x02bf93 thru 0x02e22f (0x229d bytes) 000494 1775764763.999882 mapread 0x025041 thru 0x02e22f (0x91ef bytes) 000495 1775764764.001189 mapread 0x02cd6a thru 0x02e22f (0x14c6 bytes) truncating to largest ever: 0x3eadf 000496 1775764764.003534 trunc from 0x02e230 to 0x03eade (0x108af bytes) 000497 1775764764.017870 mapread 0x036b7b thru 0x03eade (0x7f64 bytes) 000499 1775764764.028949 mapwrite 0x032b82 thru 0x03ffff (0xd47e bytes) 000507 1775764764.114060 write 0x016dc3 thru 0x01a96f (0x3bad bytes) 000509 1775764764.126105 trunc from 0x040000 to 0x00e171 (0x31e8e bytes) 000510 1775764764.149194 trunc from 0x00e172 to 0x017a9f (0x992e bytes) 000511 1775764764.178966 MIRROR_EXTEND to 2 mirrors 000512 1775764765.024341 mapwrite 0x00dbdf thru 0x017e16 (0xa238 bytes) 000515 1775764765.229965 trunc from 0x017e17 to 0x0399b1 (0x21b9b bytes) 000517 1775764765.272059 mapwrite 0x0137d2 thru 0x014ac8 (0x12f7 bytes) 000518 1775764765.320613 read 0x01a171 thru 0x025816 (0xb6a6 bytes) 000522 1775764765.332831 mapread 0x028a3a thru 0x0368ff (0xdec6 bytes) 000523 1775764765.355777 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000524 1775764765.577296 write 0x011215 thru 0x01bfda (0xadc6 bytes) 000525 1775764765.589909 mapwrite 0x00402e thru 0x00529a (0x126d bytes) 000526 1775764765.646639 read 0x0321fb thru 0x035b50 (0x3956 bytes) 000528 1775764765.653212 trunc from 0x0399b2 to 0x01c05c (0x1d955 bytes) 000530 1775764765.668838 trunc from 0x01c05d to 0x01f592 (0x3536 bytes) 000534 1775764765.687961 trunc from 0x01f593 to 0x03a9dc (0x1b44a bytes) 000535 1775764765.709689 trunc from 0x03a9dd to 0x019d6b (0x20c71 bytes) 000538 1775764765.746095 mapread 0x00e482 thru 0x019d6b (0xb8ea bytes) 000539 1775764765.748712 read 0x00e348 thru 0x01131e (0x2fd7 bytes) 000541 1775764765.750045 read 0x0176a0 thru 0x019d6b (0x26cc bytes) 000542 1775764765.755528 read 0x0162fa thru 0x019d6b (0x3a72 bytes) 000543 1775764765.760981 read 0x0038a1 thru 0x00fd16 (0xc476 bytes) 000544 1775764765.762332 mapwrite 0x034e8e thru 0x037ebf (0x3032 bytes) 000545 1775764765.913286 mapread 0x01ad83 thru 0x01f74b (0x49c9 bytes) 000547 1775764765.915035 trunc from 0x037ec0 to 0x03e6fa (0x683b bytes) 000549 1775764765.955858 mapread 0x03400c thru 0x03e6fa (0xa6ef bytes) 000552 1775764765.972492 mapwrite 0x03bb0f thru 0x03ffff (0x44f1 bytes) 000553 1775764766.055286 write 0x02476d thru 0x02979d (0x5031 bytes) 000559 1775764766.063536 read 0x0072b7 thru 0x00fa71 (0x87bb bytes) 000561 1775764766.064933 write 0x0230c2 thru 0x031b21 (0xea60 bytes) 000562 1775764766.071961 MIRROR_EXTEND to 2 mirrors 000563 1775764766.742283 read 0x03e9cf thru 0x03ffff (0x1631 bytes) 000565 1775764766.754858 read 0x01ddb9 thru 0x01fd4c (0x1f94 bytes) 000566 1775764766.777262 read 0x00a496 thru 0x00ed2b (0x4896 bytes) 000567 1775764766.800384 MIRROR_RESYNC 2 mirrors 000568 1775764767.104185 MIRROR_RESYNC 2 mirrors 000569 1775764767.394037 trunc from 0x040000 to 0x011502 (0x2eafd bytes) 000570 1775764767.513580 mapread 0x009039 thru 0x00a345 (0x130d bytes) 000571 1775764767.523039 write 0x0023c5 thru 0x00dd11 (0xb94d bytes) 000572 1775764767.531043 mapwrite 0x00cbf2 thru 0x00dc99 (0x10a8 bytes) 000574 1775764767.573735 trunc from 0x011503 to 0x0353ef (0x23eed bytes) 000575 1775764767.602092 mapread 0x02032d thru 0x02d25a (0xcf2e bytes) 000577 1775764767.634716 MIRROR_RESYNC 2 mirrors 000578 1775764768.295127 trunc from 0x0353f0 to 0x025224 (0x101cb bytes) 000580 1775764768.397092 trunc from 0x025225 to 0x0073a4 (0x1de80 bytes) 000581 1775764768.425314 read 0x003ea8 thru 0x0073a4 (0x34fd bytes) 000584 1775764768.449155 mapwrite 0x02881e thru 0x036419 (0xdbfc bytes) 000586 1775764768.547009 mapread 0x021f07 thru 0x0240df (0x21d9 bytes) 000588 1775764768.556020 trunc from 0x03641a to 0x009ea1 (0x2c578 bytes) 000589 1775764768.582859 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000590 1775764768.740788 MIRROR_EXTEND to 2 mirrors 000591 1775764769.565909 trunc from 0x009ea2 to 0x009da5 (0xfc bytes) 000596 1775764769.669199 mapread 0x007fb4 thru 0x009da5 (0x1df2 bytes) 000597 1775764769.697047 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x9da6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9da6): Mirror 2: 0xc22d5c1a (stale) CRC-32 checksum value for chunk [0, 0x9da6): Mirror 1: 0xc22d5c1a Mirror 3: 0xc22d5c1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9da6: skipped 000598 1775764770.685084 mapread 0x004bee thru 0x006668 (0x1a7b bytes) 000599 1775764770.731817 mapread 0x0003db thru 0x003a88 (0x36ae bytes) 000600 1775764770.741526 mapwrite 0x01d6df thru 0x026155 (0x8a77 bytes) 000601 1775764770.972217 trunc from 0x026156 to 0x029727 (0x35d2 bytes) 000602 1775764770.991057 read 0x01b702 thru 0x01f717 (0x4016 bytes) 000605 1775764770.993776 mapwrite 0x02011a thru 0x029934 (0x981b bytes) 000607 1775764771.090724 MIRROR_RESYNC 3 mirrors 000610 1775764771.664725 MIRROR_EXTEND to 4 mirrors 000611 1775764772.186240 write 0x02e299 thru 0x033db4 (0x5b1c bytes) 000614 1775764772.274976 read 0x00976f thru 0x011003 (0x7895 bytes) 000616 1775764772.286731 write 0x03d8f5 thru 0x03ffff (0x270b bytes) 000617 1775764772.295270 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x348bf17b (stale) Mirror 3: 0x348bf17b (stale) Mirror 4: 0x348bf17b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1f6f0ac5 Mirror 5: 0x1f6f0ac5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000618 1775764773.242891 write 0x00d14b thru 0x0131a6 (0x605c bytes) 000619 1775764773.333681 mapread 0x004b29 thru 0x0089cb (0x3ea3 bytes) 000620 1775764773.345095 mapwrite 0x01e814 thru 0x01f498 (0xc85 bytes) 000621 1775764773.400086 read 0x00d4a2 thru 0x00fcb9 (0x2818 bytes) 000622 1775764773.401240 mapwrite 0x01509c thru 0x01c1a4 (0x7109 bytes) 000625 1775764773.479815 mapread 0x02ff34 thru 0x0364d7 (0x65a4 bytes) 000627 1775764773.501860 mapread 0x029d30 thru 0x03450b (0xa7dc bytes) 000628 1775764773.504058 mapwrite 0x004fd6 thru 0x00cc87 (0x7cb2 bytes) 000630 1775764773.612495 read 0x0396ed thru 0x03ee7d (0x5791 bytes) 000632 1775764773.627851 read 0x007a0a thru 0x00f49e (0x7a95 bytes) 000633 1775764773.632903 mapwrite 0x02ed44 thru 0x03610e (0x73cb bytes) 000635 1775764773.736014 read 0x03f65d thru 0x03ffff (0x9a3 bytes) 000636 1775764773.738803 mapread 0x03bbb2 thru 0x03ffff (0x444e bytes) 000643 1775764773.746067 read 0x0249a4 thru 0x03431a (0xf977 bytes) 000644 1775764773.752199 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x348bf17b (stale) Mirror 3: 0x348bf17b (stale) Mirror 4: 0x348bf17b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x348bf17b (stale) Mirror 3: 0x348bf17b (stale) Mirror 4: 0x348bf17b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000645 1775764774.138756 mapread 0x0030b4 thru 0x01141d (0xe36a bytes) 000646 1775764774.164723 mapread 0x0259c3 thru 0x031b01 (0xc13f bytes) 000647 1775764774.185697 mapwrite 0x01b28e thru 0x02b04c (0xfdbf bytes) 000648 1775764774.271660 MIRROR_RESYNC 4 mirrors 000650 1775764775.082723 trunc from 0x040000 to 0x011b06 (0x2e4f9 bytes) 000652 1775764775.214758 MIRROR_RESYNC 4 mirrors 000653 1775764776.175940 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000654 1775764776.546540 mapread 0x0097ae thru 0x011b06 (0x8359 bytes) 000657 1775764776.568997 write 0x024b6d thru 0x025423 (0x8b7 bytes) 000658 1775764776.635948 write 0x013527 thru 0x01a2a9 (0x6d83 bytes) 000659 1775764776.645661 read 0x024d98 thru 0x025423 (0x68c bytes) 000660 1775764776.646271 mapread 0x0241a4 thru 0x025423 (0x1280 bytes) 000662 1775764776.649022 write 0x01dc77 thru 0x0258e9 (0x7c73 bytes) 000665 1775764776.657815 write 0x029679 thru 0x037218 (0xdba0 bytes) 000669 1775764776.661849 trunc from 0x037219 to 0x015abc (0x2175c bytes) 000670 1775764776.684491 mapread 0x00087a thru 0x00a2e0 (0x9a67 bytes) 000671 1775764776.695210 mapread 0x011d3c thru 0x015abc (0x3d81 bytes) 000672 1775764776.696749 mapwrite 0x029697 thru 0x037115 (0xda7f bytes) 000674 1775764776.789534 write 0x013c20 thru 0x01ac8a (0x706b bytes) 000675 1775764776.796522 write 0x03077d thru 0x038c74 (0x84f8 bytes) 000676 1775764776.798671 mapwrite 0x01950f thru 0x01bf93 (0x2a85 bytes) 000677 1775764776.851383 mapread 0x019d15 thru 0x01b0f8 (0x13e4 bytes) 000678 1775764776.853497 trunc from 0x038c75 to 0x03844b (0x829 bytes) 000681 1775764776.882419 mapread 0x00f681 thru 0x011be1 (0x2561 bytes) 000682 1775764776.883670 read 0x00cfab thru 0x01c371 (0xf3c7 bytes) 000683 1775764776.884408 trunc from 0x03844c to 0x03a6c4 (0x2279 bytes) 000684 1775764776.924753 read 0x01dd1e thru 0x01e8ee (0xbd1 bytes) 000687 1775764776.928321 mapwrite 0x013b64 thru 0x023909 (0xfda6 bytes) 000689 1775764776.991755 mapwrite 0x03dd7b thru 0x03ffff (0x2285 bytes) 000690 1775764777.051793 mapwrite 0x00c284 thru 0x010c98 (0x4a15 bytes) 000693 1775764777.115944 trunc from 0x040000 to 0x022895 (0x1d76a bytes) 000695 1775764777.140009 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22896): Mirror 2: 0xa55c77b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22896): Mirror 2: 0xa55c77b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22896: skipped 000696 1775764777.464773 read 0x00cc3a thru 0x0111d2 (0x4599 bytes) 000697 1775764777.477859 mapwrite 0x0249ea thru 0x03052b (0xbb42 bytes) 000699 1775764777.671755 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3052c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3052c): Mirror 2: 0x71b8bf15 (stale) CRC-32 checksum value for chunk [0, 0x3052c): Mirror 1: 0x1cfb38ac Mirror 3: 0x1cfb38ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3052c: skipped 000702 1775764778.825562 write 0x010d13 thru 0x01779f (0x6a8d bytes) 000703 1775764778.901211 write 0x00ea70 thru 0x0127b1 (0x3d42 bytes) 000704 1775764778.909275 write 0x01777c thru 0x023b9a (0xc41f bytes) 000705 1775764778.915903 mapwrite 0x01a4cf thru 0x01e140 (0x3c72 bytes) 000706 1775764779.003771 trunc from 0x03052c to 0x031655 (0x112a bytes) 000707 1775764779.022865 write 0x01f776 thru 0x0201df (0xa6a bytes) 000708 1775764779.023862 trunc from 0x031656 to 0x01db5e (0x13af7 bytes) 000711 1775764779.055558 read 0x00e6f5 thru 0x01529f (0x6bab bytes) 000712 1775764779.058167 mapread 0x003a92 thru 0x0067e1 (0x2d50 bytes) 000714 1775764779.071705 trunc from 0x01db5f to 0x0081b1 (0x159ad bytes) 000716 1775764779.086886 read 0x0069d0 thru 0x0081b1 (0x17e2 bytes) 000717 1775764779.087718 trunc from 0x0081b2 to 0x02dd13 (0x25b62 bytes) 000719 1775764779.102140 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2dd14): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd14): Mirror 2: 0x4d670ea2 (stale) Mirror 3: 0x9eae6920 (stale) CRC-32 checksum value for chunk [0, 0x2dd14): Mirror 1: 0xd6028cfd Mirror 4: 0xd6028cfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd14: skipped 000720 1775764780.038204 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2dd14): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd14): Mirror 2: 0x4d670ea2 (stale) Mirror 3: 0x9eae6920 (stale) CRC-32 checksum value for chunk [0, 0x2dd14): Mirror 1: 0xd6028cfd Mirror 4: 0xd6028cfd Mirror 5: 0xd6028cfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd14: skipped 000721 1775764781.061949 mapwrite 0x03963c thru 0x03b31d (0x1ce2 bytes) 000723 1775764781.250408 read 0x001068 thru 0x00742e (0x63c7 bytes) 000724 1775764781.258942 mapread 0x034b2d thru 0x037376 (0x284a bytes) 000725 1775764781.260125 read 0x035445 thru 0x03b31d (0x5ed9 bytes) 000728 1775764781.265592 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b31e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b31e): Mirror 2: 0x3a08e25 (stale) Mirror 3: 0x392bc5ba (stale) Mirror 4: 0xebba13d2 (stale) Mirror 5: 0xebba13d2 (stale) CRC-32 checksum value for chunk [0, 0x3b31e): Mirror 1: 0x23521ca6 Mirror 6: 0x23521ca6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b31e: skipped 000729 1775764782.227184 mapwrite 0x038463 thru 0x03bdc3 (0x3961 bytes) 000733 1775764782.456333 mapread 0x00be4e thru 0x0161f3 (0xa3a6 bytes) 000735 1775764782.501284 mapread 0x010b82 thru 0x012455 (0x18d4 bytes) 000736 1775764782.502924 read 0x023c3a thru 0x024cb1 (0x1078 bytes) 000737 1775764782.511895 trunc from 0x03bdc4 to 0x01ecf9 (0x1d0ca bytes) 000739 1775764782.549722 read 0x00b54c thru 0x011aef (0x65a4 bytes) 000743 1775764782.555711 mapwrite 0x02117a thru 0x0250be (0x3f45 bytes) 000744 1775764782.630003 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x250bf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x250bf): Mirror 2: 0x68e5536d (stale) Mirror 3: 0x906508c3 (stale) Mirror 4: 0x2ca92ac9 (stale) Mirror 5: 0x2ca92ac9 (stale) Mirror 6: 0x2ca92ac9 (stale) CRC-32 checksum value for chunk [0, 0x250bf): Mirror 1: 0x2ac7ac71 Mirror 7: 0x2ac7ac71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x250bf: skipped 000747 1775764783.771552 trunc from 0x0250bf to 0x029206 (0x4148 bytes) 000748 1775764783.876636 write 0x028285 thru 0x03568f (0xd40b bytes) 000749 1775764783.890002 write 0x028bf5 thru 0x029f6b (0x1377 bytes) 000752 1775764783.897523 read 0x02bfa0 thru 0x02c655 (0x6b6 bytes) 000753 1775764783.898233 mapwrite 0x000701 thru 0x009697 (0x8f97 bytes) 000755 1775764784.050253 mapwrite 0x01d79b thru 0x02bd50 (0xe5b6 bytes) 000757 1775764784.249612 read 0x02b454 thru 0x03038e (0x4f3b bytes) 000758 1775764784.269180 MIRROR_RESYNC 7 mirrors 000759 1775764785.735993 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000761 1775764786.303401 trunc from 0x035690 to 0x03743c (0x1dad bytes) 000763 1775764786.435231 trunc from 0x03743d to 0x01d3b7 (0x1a085 bytes) 000765 1775764786.469981 read 0x00947e thru 0x01261a (0x919d bytes) 000766 1775764786.492160 mapwrite 0x02b978 thru 0x02caef (0x1178 bytes) 000767 1775764786.670960 write 0x004aec thru 0x009b3b (0x5050 bytes) 000769 1775764786.682046 mapread 0x004d09 thru 0x011bf8 (0xcef0 bytes) 000770 1775764786.684967 trunc from 0x02caf0 to 0x0172e0 (0x1580f bytes) 000772 1775764786.733612 mapwrite 0x02947f thru 0x02c309 (0x2e8b bytes) 000774 1775764786.825365 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c30a): Mirror 2: 0xaf2e6d08 (stale) Mirror 3: 0xaf2e6d08 (stale) Mirror 4: 0xaf2e6d08 (stale) Mirror 7: 0xaf2e6d08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c30a): Mirror 2: 0xaf2e6d08 (stale) Mirror 3: 0xaf2e6d08 (stale) Mirror 4: 0xaf2e6d08 (stale) Mirror 7: 0xaf2e6d08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c30a: skipped 000775 1775764787.383516 mapwrite 0x01c7e7 thru 0x02c4cc (0xfce6 bytes) 000776 1775764787.496636 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c4cd): Mirror 2: 0xe0ef5ce4 (stale) Mirror 4: 0xe0ef5ce4 (stale) Mirror 7: 0xe0ef5ce4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4cd): Mirror 2: 0xe0ef5ce4 (stale) Mirror 4: 0xe0ef5ce4 (stale) Mirror 7: 0xe0ef5ce4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4cd: skipped 000780 1775764787.963627 mapread 0x027b7e thru 0x02c4cc (0x494f bytes) 000781 1775764787.997999 read 0x00bebd thru 0x0175a6 (0xb6ea bytes) 000783 1775764788.013477 write 0x0239a3 thru 0x03010e (0xc76c bytes) 000785 1775764788.021083 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: 0xa4e97ffe (stale) Mirror 4: 0xa4e97ffe (stale) Mirror 7: 0xa4e97ffe (stale) CRC-32 checksum value for chunk [0, 0x3010f): Mirror 1: 0x6d0d9f5 Mirror 8: 0x6d0d9f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3010f: skipped 000786 1775764789.055992 mapread 0x02d726 thru 0x03010e (0x29e9 bytes) 000789 1775764789.099178 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: 0xa4e97ffe (stale) Mirror 4: 0xa4e97ffe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: 0xa4e97ffe (stale) Mirror 4: 0xa4e97ffe (stale) CRC-32 checksum value for chunk [0, 0x3010f): Mirror 1: 0x6d0d9f5 Mirror 8: 0x6d0d9f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3010f: skipped 000791 1775764789.928313 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: 0xa4e97ffe (stale) Mirror 4: 0xa4e97ffe (stale) CRC-32 checksum value for chunk [0, 0x3010f): Mirror 1: 0x6d0d9f5 Mirror 8: 0x6d0d9f5 Mirror 9: 0x6d0d9f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3010f: skipped 000795 1775764791.115099 write 0x004632 thru 0x00cb9d (0x856c bytes) 000796 1775764791.296586 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: 0xa4e97ffe (stale) Mirror 8: 0x6d0d9f5 (stale) Mirror 9: 0x6d0d9f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3010f): Mirror 2: 0xa4e97ffe (stale) Mirror 8: 0x6d0d9f5 (stale) Mirror 9: 0x6d0d9f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3010f: skipped 000797 1775764791.679999 mapwrite 0x035546 thru 0x03b99c (0x6457 bytes) 000798 1775764791.880766 mapwrite 0x037b31 thru 0x03ffff (0x84cf bytes) 000799 1775764791.965832 mapwrite 0x008c8f thru 0x00f8c3 (0x6c35 bytes) 000800 1775764792.063815 MIRROR_RESYNC 4 mirrors 000801 1775764792.924030 read 0x02e8b3 thru 0x03d762 (0xeeb0 bytes) 000807 1775764792.939259 MIRROR_EXTEND to 5 mirrors 000808 1775764793.699970 MIRROR_RESYNC 5 mirrors 000810 1775764794.139753 trunc from 0x040000 to 0x0136c1 (0x2c93e bytes) 000811 1775764794.294720 MIRROR_RESYNC 5 mirrors 000812 1775764795.264647 mapread 0x011bdc thru 0x0136c1 (0x1ae6 bytes) 000813 1775764795.282325 mapwrite 0x021704 thru 0x029c45 (0x8542 bytes) 000814 1775764795.408551 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29c46): Mirror 8: 0xde4c92e0 (stale) Mirror 9: 0xde4c92e0 (stale) Mirror 10: 0xde4c92e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c46): Mirror 8: 0xde4c92e0 (stale) Mirror 9: 0xde4c92e0 (stale) Mirror 10: 0xde4c92e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c46: skipped 000816 1775764795.847535 read 0x00cda2 thru 0x00e4f6 (0x1755 bytes) 000818 1775764795.857897 mapwrite 0x01d184 thru 0x01f413 (0x2290 bytes) 000819 1775764795.933103 read 0x021a94 thru 0x0224aa (0xa17 bytes) 000822 1775764795.937545 mapread 0x00a4c4 thru 0x00b9b7 (0x14f4 bytes) 000823 1775764795.949401 read 0x025a27 thru 0x027e79 (0x2453 bytes) 000824 1775764795.950413 write 0x002fd7 thru 0x00d41e (0xa448 bytes) 000825 1775764795.952718 mapread 0x00d054 thru 0x010234 (0x31e1 bytes) 000827 1775764795.954582 trunc from 0x029c46 to 0x02350d (0x6738 bytes) 000829 1775764795.979476 mapwrite 0x020471 thru 0x02fc50 (0xf7e0 bytes) 000830 1775764796.104505 write 0x03f725 thru 0x03ffff (0x8db bytes) 000831 1775764796.108692 trunc from 0x040000 to 0x01226b (0x2dd94 bytes) 000832 1775764796.141639 mapread 0x00ba0f thru 0x01226b (0x685d bytes) 000835 1775764796.145912 mapread 0x000e17 thru 0x00afdb (0xa1c5 bytes) 000836 1775764796.147360 write 0x014b95 thru 0x019a9b (0x4f07 bytes) 000839 1775764796.151722 write 0x03aad9 thru 0x03adb4 (0x2dc bytes) 000840 1775764796.154270 read 0x007221 thru 0x012f3e (0xbd1e bytes) 000841 1775764796.158557 mapwrite 0x03f06b thru 0x03ffff (0xf95 bytes) 000842 1775764796.271883 trunc from 0x040000 to 0x0316bc (0xe943 bytes) 000843 1775764796.296276 mapwrite 0x004fa7 thru 0x00ba08 (0x6a62 bytes) 000844 1775764796.360839 trunc from 0x0316bd to 0x01e8fa (0x12dc2 bytes) 000845 1775764796.385718 mapread 0x01a3d4 thru 0x01d65d (0x328a bytes) 000846 1775764796.387159 read 0x0096d6 thru 0x00c482 (0x2dad bytes) 000847 1775764796.393175 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 8: 0x51639a42 (stale) Mirror 9: 0x51639a42 (stale) Mirror 10: 0x51639a42 (stale) CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 1: 0xf580f3c9 Mirror 11: 0xf580f3c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e8fb: skipped 000852 1775764797.350261 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 8: 0x51639a42 (stale) Mirror 9: 0x51639a42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 8: 0x51639a42 (stale) Mirror 9: 0x51639a42 (stale) CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 1: 0xf580f3c9 Mirror 11: 0xf580f3c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e8fb: skipped 000853 1775764797.911827 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 8: 0x51639a42 (stale) Mirror 9: 0x51639a42 (stale) CRC-32 checksum value for chunk [0, 0x1e8fb): Mirror 1: 0xf580f3c9 Mirror 11: 0xf580f3c9 Mirror 12: 0xf580f3c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e8fb: skipped 000854 1775764798.964198 mapread 0x0147e2 thru 0x01e8fa (0xa119 bytes) 000855 1775764798.996303 write 0x035c19 thru 0x03b705 (0x5aed bytes) 000856 1775764799.155748 mapwrite 0x00c55a thru 0x018d84 (0xc82b bytes) 000857 1775764799.292853 MIRROR_RESYNC 5 mirrors 000860 1775764800.312524 trunc from 0x03b706 to 0x034eb8 (0x684d bytes) 000862 1775764800.434675 read 0x0307c9 thru 0x034eb8 (0x46f0 bytes) 000864 1775764800.448664 read 0x003cbd thru 0x00e670 (0xa9b4 bytes) 000865 1775764800.461726 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34eb9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34eb9): Mirror 8: 0xca25b5dc (stale) Mirror 9: 0xca25b5dc (stale) Mirror 11: 0xca25b5dc (stale) Mirror 12: 0xca25b5dc (stale) CRC-32 checksum value for chunk [0, 0x34eb9): Mirror 1: 0xca25b5dc Mirror 13: 0xca25b5dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34eb9: skipped 000866 1775764801.241051 trunc from 0x034eb9 to 0x028438 (0xca80 bytes) 000867 1775764801.333822 read 0x015fb6 thru 0x020fe9 (0xb034 bytes) 000869 1775764801.344198 mapread 0x0255f3 thru 0x028438 (0x2e46 bytes) 000871 1775764801.359790 write 0x01b6c0 thru 0x01eaa3 (0x33e4 bytes) 000872 1775764801.361079 read 0x0188eb thru 0x022e0d (0xa523 bytes) 000873 1775764801.362697 read 0x018910 thru 0x025c91 (0xd382 bytes) 000874 1775764801.366764 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x28439): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28439): Mirror 8: 0xd0c959a (stale) Mirror 9: 0xd0c959a (stale) Mirror 11: 0xd0c959a (stale) Mirror 12: 0xd0c959a (stale) Mirror 13: 0xd0c959a (stale) CRC-32 checksum value for chunk [0, 0x28439): Mirror 1: 0x868d0d6a Mirror 14: 0x868d0d6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28439: skipped 000875 1775764802.281845 read 0x026aae thru 0x028438 (0x198b bytes) 000876 1775764802.292869 trunc from 0x028439 to 0x01a720 (0xdd18 bytes) 000877 1775764802.427182 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a721): Mirror 8: 0x43fbee1a (stale) Mirror 9: 0x43fbee1a (stale) Mirror 12: 0x43fbee1a (stale) Mirror 13: 0x43fbee1a (stale) Mirror 14: 0x43fbee1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a721): Mirror 8: 0x43fbee1a (stale) Mirror 9: 0x43fbee1a (stale) Mirror 12: 0x43fbee1a (stale) Mirror 13: 0x43fbee1a (stale) Mirror 14: 0x43fbee1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a721: skipped 000878 1775764802.862820 mapwrite 0x015b84 thru 0x01b703 (0x5b80 bytes) 000879 1775764802.968488 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b704): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b704): Mirror 8: 0x40d888e4 (stale) Mirror 9: 0x40d888e4 (stale) Mirror 12: 0x40d888e4 (stale) Mirror 13: 0x40d888e4 (stale) Mirror 14: 0x70510d29 (stale) CRC-32 checksum value for chunk [0, 0x1b704): Mirror 1: 0xc3557f93 Mirror 15: 0xc3557f93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b704: skipped 000880 1775764803.989123 write 0x0044b0 thru 0x00ba91 (0x75e2 bytes) 000882 1775764804.124793 write 0x02799d thru 0x0326a2 (0xad06 bytes) 000883 1775764804.137803 MIRROR_RESYNC 7 mirrors 000884 1775764805.451809 mapwrite 0x007d7a thru 0x010841 (0x8ac8 bytes) 000885 1775764805.614204 write 0x012260 thru 0x01a1e9 (0x7f8a bytes) 000892 1775764805.638406 mapread 0x01247b thru 0x02036f (0xdef5 bytes) 000893 1775764805.652732 MIRROR_RESYNC 7 mirrors 000895 1775764806.864524 read 0x011aec thru 0x01e9c3 (0xced8 bytes) 000896 1775764806.873960 mapread 0x02b9b5 thru 0x0317e2 (0x5e2e bytes) 000897 1775764806.900321 trunc from 0x0326a3 to 0x022d10 (0xf992 bytes) 000900 1775764806.997302 write 0x01b68b thru 0x01bcc1 (0x637 bytes) 000901 1775764807.004472 read 0x003c30 thru 0x005493 (0x1864 bytes) 000902 1775764807.013261 write 0x024f75 thru 0x025798 (0x824 bytes) 000904 1775764807.015121 read 0x01c2f8 thru 0x01f4a5 (0x31ae bytes) 000905 1775764807.021732 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x25799): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25799): Mirror 8: 0xe4378c78 (stale) Mirror 9: 0xe4378c78 (stale) Mirror 12: 0xe4378c78 (stale) Mirror 13: 0xe4378c78 (stale) Mirror 14: 0xe4378c78 (stale) Mirror 15: 0xe4378c78 (stale) CRC-32 checksum value for chunk [0, 0x25799): Mirror 1: 0x3b80072c Mirror 16: 0x3b80072c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25799: skipped 000907 1775764807.814750 mapread 0x006f8b thru 0x00ca8c (0x5b02 bytes) 000909 1775764807.848992 write 0x02f480 thru 0x03bfcc (0xcb4d bytes) 000910 1775764808.010137 mapread 0x0065a0 thru 0x0081eb (0x1c4c bytes) 000911 1775764808.025871 mapwrite 0x01e728 thru 0x0287c9 (0xa0a2 bytes) 000913 1775764808.184759 read 0x026e6c thru 0x032b92 (0xbd27 bytes) 000914 1775764808.199591 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3bfcd): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfcd): Mirror 8: 0x1bfe9e1c (stale) Mirror 9: 0x1bfe9e1c (stale) Mirror 12: 0x1bfe9e1c (stale) Mirror 13: 0x1bfe9e1c (stale) Mirror 14: 0x1bfe9e1c (stale) Mirror 15: 0x1bfe9e1c (stale) Mirror 16: 0xfdd3b6f0 (stale) CRC-32 checksum value for chunk [0, 0x3bfcd): Mirror 1: 0xaf412997 Mirror 17: 0xaf412997 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfcd: skipped 000916 1775764809.191633 read 0x0222ef thru 0x028e2f (0x6b41 bytes) 000918 1775764809.204059 mapread 0x0303ac thru 0x031437 (0x108c bytes) 000919 1775764809.221675 read 0x0087fb thru 0x016c4d (0xe453 bytes) 000920 1775764809.237582 write 0x0303e0 thru 0x037807 (0x7428 bytes) 000923 1775764809.411703 mapread 0x0021de thru 0x010ae3 (0xe906 bytes) 000924 1775764809.525804 trunc from 0x03bfcd to 0x033299 (0x8d33 bytes) 000925 1775764809.599645 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3329a): Mirror 8: 0x2ddff6a5 (stale) Mirror 9: 0x2ddff6a5 (stale) Mirror 12: 0x2ddff6a5 (stale) Mirror 13: 0x2ddff6a5 (stale) Mirror 14: 0x2ddff6a5 (stale) Mirror 16: 0x36e58c3c (stale) Mirror 17: 0x86c68a1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3329a): Mirror 8: 0x2ddff6a5 (stale) Mirror 9: 0x2ddff6a5 (stale) Mirror 12: 0x2ddff6a5 (stale) Mirror 13: 0x2ddff6a5 (stale) Mirror 14: 0x2ddff6a5 (stale) Mirror 16: 0x36e58c3c (stale) Mirror 17: 0x86c68a1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3329a: skipped 000928 1775764810.195153 mapread 0x02d8f0 thru 0x03205b (0x476c bytes) 000929 1775764810.224579 write 0x03856c thru 0x03d7ac (0x5241 bytes) 000930 1775764810.240903 MIRROR_RESYNC 8 mirrors 000933 1775764812.009008 write 0x017826 thru 0x01f52c (0x7d07 bytes) 000934 1775764812.180258 mapwrite 0x027070 thru 0x031239 (0xa1ca bytes) 000935 1775764812.337795 trunc from 0x03d7ad to 0x03d42b (0x381 bytes) 000936 1775764812.377519 trunc from 0x03d42c to 0x003fa6 (0x39485 bytes) 000940 1775764812.409827 mapread 0x002e8b thru 0x003fa6 (0x111c bytes) 000941 1775764812.426194 trunc from 0x003fa7 to 0x0163ab (0x12405 bytes) 000943 1775764812.463755 trunc from 0x0163ac to 0x019b43 (0x3798 bytes) 000944 1775764812.486772 write 0x0083ff thru 0x01209d (0x9c9f bytes) 000945 1775764812.507874 mapread 0x002b46 thru 0x006161 (0x361c bytes) 000947 1775764812.521967 read 0x0122a6 thru 0x019b43 (0x789e bytes) 000949 1775764812.537149 mapwrite 0x01af73 thru 0x025792 (0xa820 bytes) 000950 1775764812.625706 mapwrite 0x036488 thru 0x03ffff (0x9b78 bytes) 000951 1775764812.735582 mapread 0x0024fd thru 0x0029c3 (0x4c7 bytes) 000952 1775764812.736795 read 0x014ec0 thru 0x022060 (0xd1a1 bytes) 000953 1775764812.739540 mapwrite 0x016d1c thru 0x01957f (0x2864 bytes) 000954 1775764812.808920 MIRROR_RESYNC 8 mirrors 000957 1775764814.139764 trunc from 0x040000 to 0x0132bf (0x2cd40 bytes) 000959 1775764814.261202 mapread 0x01088d thru 0x0132bf (0x2a33 bytes) 000960 1775764814.291334 mapread 0x002f49 thru 0x004755 (0x180d bytes) 000961 1775764814.307135 MIRROR_RESYNC 8 mirrors 000963 1775764815.861601 write 0x01a7a2 thru 0x01c0b4 (0x1913 bytes) 000964 1775764816.006414 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c0b5): Mirror 8: 0xc0ad373e (stale) Mirror 9: 0xc0ad373e (stale) Mirror 13: 0xc0ad373e (stale) Mirror 14: 0xc0ad373e (stale) Mirror 16: 0xc0ad373e (stale) Mirror 17: 0xc0ad373e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c0b5): Mirror 8: 0xc0ad373e (stale) Mirror 9: 0xc0ad373e (stale) Mirror 13: 0xc0ad373e (stale) Mirror 14: 0xc0ad373e (stale) Mirror 16: 0xc0ad373e (stale) Mirror 17: 0xc0ad373e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c0b5: skipped 000965 1775764816.696985 trunc from 0x01c0b5 to 0x036cf4 (0x1ac40 bytes) 000966 1775764816.714687 write 0x01af51 thru 0x026479 (0xb529 bytes) 000967 1775764816.726619 mapwrite 0x02ed4c thru 0x02fcfa (0xfaf bytes) 000968 1775764816.835294 mapread 0x01e2b4 thru 0x020012 (0x1d5f bytes) 000969 1775764816.836594 mapread 0x032cf4 thru 0x036cf4 (0x4001 bytes) 000970 1775764816.850130 mapread 0x00664e thru 0x00c8cd (0x6280 bytes) 000971 1775764816.890390 trunc from 0x036cf5 to 0x001435 (0x358bf bytes) 000975 1775764816.931004 write 0x00c561 thru 0x01ab3c (0xe5dc bytes) 000977 1775764816.941560 write 0x01918c thru 0x028925 (0xf79a bytes) 000980 1775764816.947651 trunc from 0x028926 to 0x0099f4 (0x1ef31 bytes) 000983 1775764816.976568 read 0x00328c thru 0x0099f4 (0x6769 bytes) 000984 1775764816.986181 read 0x00219b thru 0x002c44 (0xaaa bytes) 000985 1775764817.003250 mapwrite 0x033d8f thru 0x0347f9 (0xa6b bytes) 000986 1775764817.200033 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x347fa): Mirror 9: 0xf74e942d (stale) Mirror 13: 0xf74e942d (stale) Mirror 14: 0xf74e942d (stale) Mirror 16: 0xf74e942d (stale) Mirror 17: 0xf74e942d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347fa): Mirror 9: 0xf74e942d (stale) Mirror 13: 0xf74e942d (stale) Mirror 14: 0xf74e942d (stale) Mirror 16: 0xf74e942d (stale) Mirror 17: 0xf74e942d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347fa: skipped 000987 1775764817.926955 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x347fa): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347fa): Mirror 9: 0xf74e942d (stale) Mirror 13: 0xf74e942d (stale) Mirror 14: 0xf74e942d (stale) Mirror 16: 0xf74e942d (stale) Mirror 17: 0xf74e942d (stale) CRC-32 checksum value for chunk [0, 0x347fa): Mirror 1: 0xf4eee3e3 Mirror 18: 0xf4eee3e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347fa: skipped 000988 1775764818.794925 MIRROR_RESYNC 7 mirrors 000989 1775764819.896328 read 0x026d92 thru 0x02c313 (0x5582 bytes) 000990 1775764819.917098 MIRROR_RESYNC 7 mirrors 000991 1775764820.634034 MIRROR_RESYNC 7 mirrors 000992 1775764821.267373 MIRROR_EXTEND to 8 mirrors 000995 1775764822.090010 mapwrite 0x01a7a6 thru 0x027006 (0xc861 bytes) 000996 1775764822.531809 read 0x02ccd0 thru 0x0347f9 (0x7b2a bytes) 000999 1775764822.551786 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x347fa): Mirror 9: 0xf4eee3e3 (stale) Mirror 13: 0xf4eee3e3 (stale) Mirror 16: 0xf4eee3e3 (stale) Mirror 17: 0xf4eee3e3 (stale) Mirror 18: 0xf4eee3e3 (stale) Mirror 19: 0xf4eee3e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347fa): Mirror 9: 0xf4eee3e3 (stale) Mirror 13: 0xf4eee3e3 (stale) Mirror 16: 0xf4eee3e3 (stale) Mirror 17: 0xf4eee3e3 (stale) Mirror 18: 0xf4eee3e3 (stale) Mirror 19: 0xf4eee3e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347fa: skipped 001000 1775764823.359706 write 0x019f7b thru 0x022df4 (0x8e7a bytes) All operations completed A-OK! PASS 70a (102s) == sanity-flr test 71: check mirror extend parent fid ==== 16:00:28 (1775764828) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 16:00:32 (1775764832) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x243:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197509 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x243:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195285 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x244:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x243:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x244:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x243:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x249:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 16:00:40 (1775764840) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2969312 .. Extending file size to 3569888 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5163168 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8094656 .. failed Extending file size to 8649376 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4571776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6719232 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7837376 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8231424 .. Extending file size to 8724608 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8713344 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9018048 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 55432 55433 55434 55435 55438 done 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) 192.168.203.111@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:02:02 (1775764922) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7954016 .. Extending file size to 9265760 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9323232 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8413760 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56830 56831 56832 56836 56837 done 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) 192.168.203.111@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:03:23 (1775765003) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154657 s, 13.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.379799 s, 5.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1581be3f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:03:45 (1775765025) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0886191 s, 23.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:03:52 (1775765032) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:03:59 (1775765039) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218684 s, 9.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x260:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:04:07 (1775765047) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211881 s, 9.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x261:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x261:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:04:15 (1775765055) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195156 s, 10.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:04:48 (1775765088) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114386 s, 9.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:05:46 (1775765146) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.249387 s, 8.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:06:45 (1775765205) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. Creating new pool test_204f oleg311-server: Pool lustre.test_204f created Adding targets to pool oleg311-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg311-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31061 s, 6.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x268:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x268:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg311-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg311-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg311-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:07:58 (1775765278) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:08:06 (1775765286) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:08:13 (1775765293) oleg311-server: Pool lustre.test_206 created Adding targets to pool oleg311-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg311-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg311-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg311-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg311-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:08:39 (1775765319) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971828 s, 10.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321424 s, 32.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6e79a345 Mirror 3: 0x6e79a345 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:08:50 (1775765330) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:08:54 (1775765334) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:08:58 (1775765338) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.158707 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1230000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb90000 Read this (4 bytes): BBBB 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:09:33 (1775765373) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0202488 s, 405 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:09:40 (1775765380) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028444 s, 36.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20a lmm_fid: [0x200000404:0x20a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20a lmm_fid: [0x200000404:0x20a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834442 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-09 16:09:41.000000000 -0400 Modify: 2026-04-09 16:09:41.000000000 -0400 Change: 2026-04-09 16:09:42.000000000 -0400 Birth: 2026-04-09 16:09:41.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:09:48 (1775765388) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:09:51 (1775765391) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.794972 s, 51.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0825185 s, 49.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 9 16:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 9 16:09 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:10:00 (1775765400) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:10:08 (1775765408) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1694 sec =========== 16:10:17 (1775765417) === sanity-flr: start cleanup 16:10:19 (1775765419) === === sanity-flr: finish cleanup 16:10:27 (1775765427) ===