-----============= acceptance-small: sanity-flr ============----- Sat Apr 11 01:09:53 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg329-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:10:01 (1775884201) === oleg329-client.virtnet: executing check_config_client /mnt/lustre oleg329-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg329-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c168bccd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c168bccd000.idle_timeout=debug disable quota as required oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:10:13 (1775884213) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7573] == sanity-flr test 0a: lfs mirror create with -N option == 01:10:14 (1775884214) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:10:21 (1775884221) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:10:22 (1775884222) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:10:24 (1775884224) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:10:34 (1775884234) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:10:35 (1775884235) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 01:10:36 (1775884236) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c168bccd000.stats=clear osc.lustre-OST0001-osc-ffff9c168bccd000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:10:39 (1775884239) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:10:43 (1775884243) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:10:47 (1775884247) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:10:48 (1775884248) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:10:51 (1775884251) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 01:10:55 (1775884255) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:10:59 (1775884259) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:11:00 (1775884260) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:11:03 (1775884263) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:11:07 (1775884267) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:11:09 (1775884269) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.297237 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530783 s, 19.8 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:11:13 (1775884273) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.181794 s, 104 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:11:17 (1775884277) Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:11:48 (1775884308) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.072306 s, 43.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:12:04 (1775884324) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (31s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:12:35 (1775884355) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.72646 s, 30.4 MB/s -rw-r--r-- 1 root root 52428800 Apr 11 01:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.366s user 0m0.001s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c168bccd000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.423s user 0m0.000s sys 0m0.162s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.473s user 0m0.001s sys 0m0.163s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c168bccd000.ost_server_uuid in FULL state after 0 sec PASS 33b (39s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:13:14 (1775884394) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:13:15 (1775884395) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:13:16 (1775884396) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:13:17 (1775884397) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 01:13:20 (1775884400) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.221138 s, 56.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:13:25 (1775884405) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014094 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158473 s, 66.2 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:13:28 (1775884408) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:13:32 (1775884412) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000708269 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000819095 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000788966 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000981332 s, 1.0 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 01:13:36 (1775884416) create test files with size 15737668 .. 56aa0ddb6a11a466fcc21c885a632d58 - 1a5bd9aec0fea67fe5666152ce79d605 - 6253e5ced02bb1127e0956cc565c9d87 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 01:13:44 (1775884424) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:13:49 (1775884429) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:13:52 (1775884432) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569356 s, 18.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154941 s, 67.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 01:13:56 (1775884436) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0677802 s, 30.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.184653 s, 22.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123816 s, 16.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 01:14:02 (1775884442) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 01:14:03 (1775884443) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:14:05 (1775884445) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146792 s, 71.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 01:14:08 (1775884448) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:14:09 (1775884449) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:14:13 (1775884453) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118254 s, 88.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:14:30 (1775884470) llite.lustre-ffff9c1683e86800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.123405 s, 85.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c169013c000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:14:35 (1775884475) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (16s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:14:51 (1775884491) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (16s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:15:07 (1775884507) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:15:08 (1775884508) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0915499 s, 34.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0714892 s, 44.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x35ea8c3 (stale) Mirror 2: 0x35ea8c3 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 2f808e5d3d9f1cb45ea5ce241439f429 - mirror 1 checksum: 5f886c276adfdf6cd0b07a717524c569 - mirror 2 checksum: 2f808e5d3d9f1cb45ea5ce241439f429 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Apr 11 01:15:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Apr 11 01:15:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2f808e5d3d9f1cb45ea5ce241439f429 - mirror 1 checksum: 5f886c276adfdf6cd0b07a717524c569 - mirror 2 checksum: 5f886c276adfdf6cd0b07a717524c569 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Apr 11 01:15:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Apr 11 01:15:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:15:13 (1775884513) Failing mds1 on oleg329-server Stopping /mnt/lustre-mds1 (opts:) on oleg329-server 01:15:16 (1775884516) shut down facet: mds1 facet_host: oleg329-server facet_failover_host: oleg329-server Failover mds1 to oleg329-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 01:15:31 (1775884531) targets are mounted 01:15:31 (1775884531) facet_failover done oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155728 s, 67.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:15:37 (1775884537) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:15:39 (1775884539) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:15:40 (1775884540) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00336079 s, 4.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:15:43 (1775884543) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:15:44 (1775884544) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:15:45 (1775884545) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:15:53 (1775884553) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:16:13 (1775884573) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:16:31 (1775884591) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:16:49 (1775884609) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 01:16:52 (1775884612) mirror create pid 52855 mirror split pid 52857 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52855 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52857 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 01:17:54 (1775884674) Seed set to 4675 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1775884675.573440 mapwrite 0x029333 thru 0x02e964 (0x5632 bytes) 000002 1775884675.618290 read 0x00c808 thru 0x00ebc8 (0x23c1 bytes) truncating to largest ever: 0x22a22 000003 1775884675.623373 trunc from 0x02e965 to 0x022a21 (0xbf43 bytes) 000004 1775884675.633431 trunc from 0x022a22 to 0x009f64 (0x18abd bytes) 000005 1775884675.645951 mapread 0x009d0f thru 0x009f64 (0x256 bytes) 000006 1775884675.651868 mapwrite 0x02b15b thru 0x031e09 (0x6caf bytes) 000007 1775884675.690764 mapwrite 0x00b719 thru 0x016cae (0xb596 bytes) 000008 1775884675.723659 mapread 0x00c565 thru 0x0121d4 (0x5c70 bytes) 000010 1775884675.724969 write 0x01e704 thru 0x02223d (0x3b3a bytes) 000014 1775884675.731339 trunc from 0x031e0a to 0x017e3e (0x19fcb bytes) 000017 1775884675.738746 mapread 0x00f921 thru 0x0133d2 (0x3ab2 bytes) 000018 1775884675.739624 read 0x0032a2 thru 0x0066fb (0x345a bytes) 000019 1775884675.740233 mapread 0x01795a thru 0x017e3e (0x4e5 bytes) 000020 1775884675.741084 read 0x001eac thru 0x0079bc (0x5b11 bytes) 000021 1775884675.746550 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3e730 000022 1775884675.942567 trunc from 0x017e3f to 0x03e72f (0x268f1 bytes) 000023 1775884675.978694 trunc from 0x03e730 to 0x0377ee (0x6f41 bytes) 000024 1775884675.987831 mapread 0x00fa4a thru 0x0163ed (0x69a4 bytes) 000025 1775884675.997955 mapread 0x005099 thru 0x00aa85 (0x59ed bytes) 000029 1775884676.008343 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) 000030 1775884676.071770 mapread 0x000234 thru 0x00a2fc (0xa0c9 bytes) 000031 1775884676.084224 trunc from 0x0377ef to 0x0040c5 (0x33729 bytes) 000032 1775884676.093104 mapwrite 0x02bbca thru 0x035a97 (0x9ece bytes) 000033 1775884676.134473 mapwrite 0x01d66e thru 0x01f1a7 (0x1b3a bytes) 000038 1775884676.163283 write 0x004b11 thru 0x00fe52 (0xb342 bytes) 000039 1775884676.169355 mapread 0x00e60a thru 0x017fb2 (0x99a9 bytes) 000040 1775884676.176348 read 0x00989f thru 0x0149cf (0xb131 bytes) 000045 1775884676.178573 write 0x030a9e thru 0x03e18c (0xd6ef bytes) 000046 1775884676.179951 mapread 0x012195 thru 0x01eca7 (0xcb13 bytes) 000049 1775884676.182176 mapread 0x00dd47 thru 0x016f69 (0x9223 bytes) 000050 1775884676.183146 write 0x009031 thru 0x00c1fa (0x31ca bytes) 000051 1775884676.184036 trunc from 0x03e18d to 0x01d769 (0x20a23 bytes) 000054 1775884676.192883 trunc from 0x01d76a to 0x0294ca (0xbd61 bytes) 000055 1775884676.200415 trunc from 0x0294cb to 0x017cdb (0x117ef bytes) 000057 1775884676.207825 read 0x0054e0 thru 0x007261 (0x1d82 bytes) 000058 1775884676.208350 write 0x01b5a0 thru 0x023b1c (0x857d bytes) 000062 1775884676.211157 write 0x0058b8 thru 0x00e664 (0x8dad bytes) 000063 1775884676.212328 mapread 0x00a438 thru 0x018bd3 (0xe79c bytes) 000064 1775884676.216704 mapread 0x0045fc thru 0x006536 (0x1f3b bytes) 000066 1775884676.218441 mapread 0x0218e5 thru 0x023b1c (0x2238 bytes) 000069 1775884676.220584 mapread 0x01f5ac thru 0x023b1c (0x4571 bytes) 000071 1775884676.222229 mapwrite 0x02b29d thru 0x02de42 (0x2ba6 bytes) 000072 1775884676.255003 mapwrite 0x031fe8 thru 0x03aa81 (0x8a9a bytes) 000073 1775884676.289703 mapread 0x00d5ec thru 0x016930 (0x9345 bytes) 000075 1775884676.291371 trunc from 0x03aa82 to 0x024db9 (0x15cc8 bytes) 000076 1775884676.298304 trunc from 0x024dba to 0x039610 (0x14857 bytes) 000077 1775884676.305925 read 0x032971 thru 0x0382c1 (0x5951 bytes) 000078 1775884676.311165 write 0x017332 thru 0x018075 (0xd44 bytes) 000080 1775884676.312568 mapread 0x000e41 thru 0x0059cf (0x4b8f bytes) 000081 1775884676.313431 mapwrite 0x00cb61 thru 0x0126e6 (0x5b86 bytes) 000082 1775884676.335006 write 0x027570 thru 0x0306ea (0x917b bytes) 000085 1775884676.337219 read 0x02a009 thru 0x02c5d1 (0x25c9 bytes) 000090 1775884676.339962 trunc from 0x039611 to 0x02d2de (0xc332 bytes) 000091 1775884676.350217 mapwrite 0x006f70 thru 0x00bc92 (0x4d23 bytes) 000092 1775884676.371916 write 0x03c75d thru 0x03ffff (0x38a3 bytes) 000093 1775884676.373011 write 0x00ea5f thru 0x019bc9 (0xb16b bytes) 000094 1775884676.374059 trunc from 0x040000 to 0x01c7b9 (0x23846 bytes) 000095 1775884676.386542 MIRROR_EXTEND to 2 mirrors 000096 1775884676.643558 read 0x000fbd thru 0x00dd4a (0xcd8e bytes) 000100 1775884676.664474 mapread 0x012ff5 thru 0x018f7a (0x5f86 bytes) 000102 1775884676.685070 trunc from 0x01c7ba to 0x034d4c (0x18593 bytes) 000103 1775884676.724034 trunc from 0x034d4d to 0x032606 (0x2746 bytes) 000106 1775884676.734348 write 0x032926 thru 0x03ffff (0xd6da bytes) 000107 1775884676.740411 write 0x02073a thru 0x02e4da (0xdda1 bytes) 000111 1775884676.748209 mapwrite 0x00f7cf thru 0x011fd5 (0x2807 bytes) 000113 1775884676.777907 mapread 0x039309 thru 0x03ffff (0x6cf7 bytes) 000114 1775884676.778874 mapread 0x03abf5 thru 0x03f54a (0x4956 bytes) 000115 1775884676.779721 write 0x039588 thru 0x039e91 (0x90a bytes) 000116 1775884676.780656 write 0x00e3d1 thru 0x01ac08 (0xc838 bytes) 000117 1775884676.781891 mapwrite 0x000461 thru 0x007b71 (0x7711 bytes) 000118 1775884676.807881 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x716c2900 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x49a75aa9 Mirror 3: 0x49a75aa9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000119 1775884677.135351 mapread 0x021a6b thru 0x029978 (0x7f0e bytes) 000120 1775884677.152429 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x716c2900 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x49a75aa9 Mirror 3: 0x49a75aa9 Mirror 4: 0x49a75aa9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000122 1775884677.530191 mapwrite 0x0341d1 thru 0x039634 (0x5464 bytes) 000125 1775884677.601269 read 0x014719 thru 0x02382d (0xf115 bytes) 000126 1775884677.606014 trunc from 0x040000 to 0x02f4d2 (0x10b2d bytes) 000127 1775884677.618828 read 0x02b422 thru 0x02bab4 (0x693 bytes) 000128 1775884677.624356 mapwrite 0x027e42 thru 0x03573e (0xd8fd bytes) 000132 1775884677.674603 read 0x00f4c2 thru 0x01cc96 (0xd7d5 bytes) 000133 1775884677.675571 mapwrite 0x026f06 thru 0x028e20 (0x1f1b bytes) 000134 1775884677.697264 trunc from 0x03573f to 0x0119f3 (0x23d4b bytes) truncating to largest ever: 0x3fac9 000135 1775884677.709786 trunc from 0x0119f4 to 0x03fac8 (0x2e0d5 bytes) 000136 1775884677.719462 mapread 0x02e46f thru 0x02fbc4 (0x1756 bytes) 000137 1775884677.726925 mapwrite 0x01e6c1 thru 0x02cf20 (0xe860 bytes) 000139 1775884677.753422 read 0x039607 thru 0x03b792 (0x218c bytes) 000140 1775884677.754120 read 0x00a37d thru 0x013110 (0x8d94 bytes) 000141 1775884677.754777 write 0x030786 thru 0x03b3fe (0xac79 bytes) 000147 1775884677.762288 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fac9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fac9): Mirror 2: 0x9e464089 (stale) Mirror 3: 0xd3a73d6d (stale) Mirror 4: 0xd3a73d6d (stale) CRC-32 checksum value for chunk [0, 0x3fac9): Mirror 1: 0xa63795b8 Mirror 5: 0xa63795b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fac9: skipped 000148 1775884678.067668 mapwrite 0x037d20 thru 0x03b8e8 (0x3bc9 bytes) 000153 1775884678.143629 trunc from 0x03fac9 to 0x021fa5 (0x1db23 bytes) 000155 1775884678.167746 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21fa6): Mirror 2: 0xf64d071a (stale) Mirror 3: 0x32c06da5 (stale) Mirror 5: 0x31718c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21fa6): Mirror 2: 0xf64d071a (stale) Mirror 3: 0x32c06da5 (stale) Mirror 5: 0x31718c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21fa6: skipped 000156 1775884678.300190 trunc from 0x021fa6 to 0x025328 (0x3383 bytes) 000160 1775884678.310706 read 0x017f6a thru 0x025328 (0xd3bf bytes) 000162 1775884678.315531 write 0x008f76 thru 0x0126aa (0x9735 bytes) 000164 1775884678.323066 write 0x039e42 thru 0x03ffff (0x61be bytes) 000165 1775884678.324726 read 0x01fed8 thru 0x02118c (0x12b5 bytes) 000167 1775884678.325742 write 0x0002e3 thru 0x008d57 (0x8a75 bytes) 000168 1775884678.331717 mapwrite 0x03139c thru 0x03bda6 (0xaa0b bytes) 000169 1775884678.356088 trunc from 0x040000 to 0x035af3 (0xa50c bytes) 000172 1775884678.367057 write 0x009bff thru 0x0119de (0x7de0 bytes) 000173 1775884678.368429 mapread 0x028bc4 thru 0x02d162 (0x459f bytes) 000176 1775884678.370584 mapread 0x02f126 thru 0x035af3 (0x69ce bytes) 000178 1775884678.372260 mapread 0x02425b thru 0x032813 (0xe5b9 bytes) 000179 1775884678.373763 write 0x000fbc thru 0x001b50 (0xb95 bytes) 000180 1775884678.374824 trunc from 0x035af4 to 0x00514d (0x309a6 bytes) 000181 1775884678.383926 mapread 0x002d78 thru 0x00514d (0x23d6 bytes) 000183 1775884678.385288 trunc from 0x00514e to 0x02c82b (0x276de bytes) 000185 1775884678.396780 write 0x032949 thru 0x0390e0 (0x6798 bytes) 000188 1775884678.401226 MIRROR_RESYNC 4 mirrors 000190 1775884678.715458 mapwrite 0x002f1f thru 0x010140 (0xd222 bytes) 000191 1775884678.792089 write 0x03de68 thru 0x03ffff (0x2198 bytes) 000192 1775884678.793170 read 0x00f6b7 thru 0x012d1c (0x3666 bytes) 000193 1775884678.796001 read 0x00945b thru 0x00a667 (0x120d bytes) 000196 1775884678.797658 mapwrite 0x005c47 thru 0x01557c (0xf936 bytes) 000197 1775884678.823300 trunc from 0x040000 to 0x0386f0 (0x790f bytes) 000198 1775884678.840227 read 0x027a7e thru 0x028574 (0xaf7 bytes) 000199 1775884678.840738 read 0x00f8a9 thru 0x0132f3 (0x3a4b bytes) 000201 1775884678.841850 mapread 0x037394 thru 0x0386f0 (0x135d bytes) 000202 1775884678.842838 mapread 0x036431 thru 0x037a40 (0x1610 bytes) 000204 1775884678.844531 read 0x0289cb thru 0x02d124 (0x475a bytes) 000205 1775884678.845186 mapread 0x01b414 thru 0x028d1b (0xd908 bytes) 000207 1775884678.846594 mapread 0x003ad4 thru 0x009757 (0x5c84 bytes) 000210 1775884678.848224 mapread 0x007de0 thru 0x00aa0e (0x2c2f bytes) 000211 1775884678.848983 write 0x02a115 thru 0x03285d (0x8749 bytes) 000214 1775884678.851124 write 0x000839 thru 0x0101e5 (0xf9ad bytes) 000215 1775884678.852442 mapwrite 0x035dfe thru 0x037858 (0x1a5b bytes) 000216 1775884678.883418 write 0x031705 thru 0x03ffff (0xe8fb bytes) 000218 1775884678.885430 mapread 0x03136f thru 0x03a504 (0x9196 bytes) 000219 1775884678.886466 read 0x00ae3c thru 0x01654f (0xb714 bytes) 000222 1775884678.888095 write 0x03120e thru 0x031cf1 (0xae4 bytes) 000223 1775884678.888732 mapread 0x02d93e thru 0x03d086 (0xf749 bytes) 000225 1775884678.890266 trunc from 0x040000 to 0x01deb2 (0x2214d bytes) 000226 1775884678.901360 trunc from 0x01deb3 to 0x02e2b9 (0x10407 bytes) 000228 1775884678.911932 read 0x01c706 thru 0x0218bb (0x51b6 bytes) 000229 1775884678.916854 write 0x03a3a9 thru 0x03ffff (0x5c57 bytes) 000230 1775884678.918064 mapwrite 0x009f07 thru 0x00dd6f (0x3e69 bytes) 000231 1775884678.946790 trunc from 0x040000 to 0x03bc2b (0x43d4 bytes) 000233 1775884678.958519 read 0x02e1f6 thru 0x033b17 (0x5922 bytes) 000234 1775884678.963723 mapread 0x01fc7f thru 0x0281a9 (0x852b bytes) 000236 1775884678.965393 read 0x02dda5 thru 0x034fc5 (0x7221 bytes) 000237 1775884678.968910 MIRROR_RESYNC 4 mirrors 000238 1775884679.267250 write 0x003ae5 thru 0x0137ab (0xfcc7 bytes) 000239 1775884679.299858 mapread 0x021660 thru 0x02b2ce (0x9c6f bytes) 000240 1775884679.311859 trunc from 0x03bc2c to 0x03c17e (0x553 bytes) 000242 1775884679.321816 trunc from 0x03c17f to 0x00f244 (0x2cf3a bytes) 000243 1775884679.328995 mapwrite 0x02b8e9 thru 0x02ebbd (0x32d5 bytes) 000244 1775884679.360296 mapread 0x02e608 thru 0x02ebbd (0x5b6 bytes) 000246 1775884679.361600 read 0x0005d2 thru 0x009f7f (0x99ae bytes) 000249 1775884679.366035 write 0x0350cb thru 0x03e34c (0x9282 bytes) 000251 1775884679.367954 write 0x00cb72 thru 0x00e0bb (0x154a bytes) 000252 1775884679.368908 trunc from 0x03e34d to 0x016f59 (0x273f3 bytes) 000253 1775884679.377587 mapread 0x007505 thru 0x00acc1 (0x37bd bytes) 000254 1775884679.378488 mapread 0x00fa7f thru 0x013ccb (0x424d bytes) 000256 1775884679.384297 mapread 0x006467 thru 0x015e89 (0xfa23 bytes) 000257 1775884679.385430 read 0x0155aa thru 0x016f59 (0x19b0 bytes) 000258 1775884679.386070 trunc from 0x016f5a to 0x0214e7 (0xa58e bytes) 000259 1775884679.396793 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x214e8): Mirror 3: 0x3ffe9a2f (stale) Mirror 5: 0x3ffe9a2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214e8): Mirror 3: 0x3ffe9a2f (stale) Mirror 5: 0x3ffe9a2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214e8: skipped 000262 1775884679.546195 read 0x017d9d thru 0x0214e7 (0x974b bytes) 000263 1775884679.550877 trunc from 0x0214e8 to 0x01f874 (0x1c73 bytes) 000264 1775884679.557462 read 0x011074 thru 0x01619a (0x5127 bytes) 000265 1775884679.561316 trunc from 0x01f875 to 0x01dd52 (0x1b22 bytes) 000266 1775884679.571578 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dd53): Mirror 3: 0xdb6386be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dd53): Mirror 3: 0xdb6386be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dd53: skipped 000267 1775884679.681397 MIRROR_RESYNC 2 mirrors 000268 1775884679.880558 trunc from 0x01dd53 to 0x03a687 (0x1c935 bytes) 000270 1775884679.911875 mapwrite 0x00b727 thru 0x0197f7 (0xe0d1 bytes) 000271 1775884679.939933 trunc from 0x03a688 to 0x020d45 (0x19942 bytes) 000273 1775884679.948446 trunc from 0x020d46 to 0x02309a (0x2355 bytes) 000274 1775884679.954940 read 0x009c27 thru 0x00c5a6 (0x2980 bytes) 000277 1775884679.956434 write 0x023d6b thru 0x0310c5 (0xd35b bytes) 000278 1775884679.962214 mapread 0x01134d thru 0x012a51 (0x1705 bytes) 000280 1775884679.963609 write 0x03bec9 thru 0x03dd04 (0x1e3c bytes) 000281 1775884679.964635 mapwrite 0x011635 thru 0x0212e9 (0xfcb5 bytes) 000282 1775884679.989254 mapwrite 0x015454 thru 0x019018 (0x3bc5 bytes) 000284 1775884680.015050 trunc from 0x03dd05 to 0x006229 (0x37adb bytes) 000285 1775884680.025362 read 0x00368c thru 0x006229 (0x2b9e bytes) 000289 1775884680.034480 mapwrite 0x02740e thru 0x029b8c (0x277f bytes) 000293 1775884680.080648 mapread 0x00b7b3 thru 0x017000 (0xb84e bytes) 000294 1775884680.081805 read 0x026811 thru 0x029b8c (0x337c bytes) 000295 1775884680.082418 mapwrite 0x038f6c thru 0x03ffff (0x7094 bytes) 000296 1775884680.120740 mapwrite 0x02dccb thru 0x033387 (0x56bd bytes) 000298 1775884680.151508 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6299a738 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xea54fbbd Mirror 4: 0xea54fbbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000299 1775884680.439992 read 0x014500 thru 0x019a47 (0x5548 bytes) 000300 1775884680.444845 read 0x02c20f thru 0x030ca4 (0x4a96 bytes) 000302 1775884680.453088 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6299a738 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xea54fbbd Mirror 4: 0xea54fbbd Mirror 5: 0xea54fbbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000303 1775884680.748254 read 0x024b1e thru 0x033099 (0xe57c bytes) 000305 1775884680.757611 mapread 0x01195a thru 0x01406d (0x2714 bytes) 000307 1775884680.770193 trunc from 0x040000 to 0x0004fa (0x3fb05 bytes) 000308 1775884680.801994 write 0x00f410 thru 0x01de68 (0xea59 bytes) 000311 1775884680.804477 mapwrite 0x024ad3 thru 0x028377 (0x38a5 bytes) 000312 1775884680.846516 MIRROR_RESYNC 4 mirrors 000313 1775884681.157558 trunc from 0x028378 to 0x03afb2 (0x12c3b bytes) 000315 1775884681.183682 mapwrite 0x02f532 thru 0x036945 (0x7414 bytes) 000317 1775884681.216408 read 0x02b6ce thru 0x03735f (0xbc92 bytes) 000318 1775884681.216942 read 0x03666d thru 0x037d38 (0x16cc bytes) 000321 1775884681.220472 MIRROR_RESYNC 4 mirrors 000322 1775884681.474808 write 0x01855c thru 0x01bd91 (0x3836 bytes) 000323 1775884681.498331 read 0x017016 thru 0x0183c8 (0x13b3 bytes) 000325 1775884681.503424 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3afb3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afb3): Mirror 3: 0x39b58380 (stale) Mirror 4: 0x39b58380 (stale) Mirror 5: 0x39b58380 (stale) CRC-32 checksum value for chunk [0, 0x3afb3): Mirror 1: 0x849e2520 Mirror 6: 0x849e2520 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afb3: skipped 000328 1775884681.783455 read 0x02b95d thru 0x035b02 (0xa1a6 bytes) 000331 1775884681.797123 mapread 0x00d3b6 thru 0x00d615 (0x260 bytes) 000332 1775884681.805118 read 0x035200 thru 0x03afb2 (0x5db3 bytes) 000333 1775884681.812615 mapread 0x001ac6 thru 0x00fdfb (0xe336 bytes) 000334 1775884681.817500 read 0x02a8df thru 0x03787e (0xcfa0 bytes) 000338 1775884681.824023 mapread 0x01896e thru 0x023fb5 (0xb648 bytes) 000340 1775884681.831448 trunc from 0x03afb3 to 0x00a2d6 (0x30cdc bytes) 000341 1775884681.878668 mapread 0x005883 thru 0x0075a4 (0x1d22 bytes) 000343 1775884681.885540 write 0x0200d9 thru 0x02947d (0x93a5 bytes) 000345 1775884681.887630 read 0x00d133 thru 0x019ed7 (0xcda5 bytes) 000346 1775884681.898979 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2947e): Mirror 4: 0xdb78885e (stale) Mirror 5: 0xdb78885e (stale) Mirror 6: 0xed4557a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2947e): Mirror 4: 0xdb78885e (stale) Mirror 5: 0xdb78885e (stale) Mirror 6: 0xed4557a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2947e: skipped 000348 1775884682.065362 mapread 0x01cc6b thru 0x01e830 (0x1bc6 bytes) 000349 1775884682.076522 mapread 0x00b954 thru 0x00f93f (0x3fec bytes) 000351 1775884682.082440 write 0x00c7d7 thru 0x0187f9 (0xc023 bytes) 000353 1775884682.084037 write 0x0002fe thru 0x008ce5 (0x89e8 bytes) 000355 1775884682.090481 write 0x03bbc7 thru 0x03c6b0 (0xaea bytes) 000356 1775884682.091842 mapread 0x01d39c thru 0x01ec83 (0x18e8 bytes) 000358 1775884682.096307 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c6b1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6b1): Mirror 4: 0xc8bf81b8 (stale) Mirror 5: 0xc8bf81b8 (stale) Mirror 6: 0x9233c8b9 (stale) CRC-32 checksum value for chunk [0, 0x3c6b1): Mirror 1: 0xaccc8517 Mirror 7: 0xaccc8517 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6b1: skipped 000359 1775884682.432446 mapread 0x00a21f thru 0x00c391 (0x2173 bytes) 000360 1775884682.453451 MIRROR_RESYNC 5 mirrors 000362 1775884682.791219 MIRROR_RESYNC 5 mirrors 000364 1775884682.971022 mapwrite 0x00e2b3 thru 0x016c88 (0x89d6 bytes) 000365 1775884683.062624 trunc from 0x03c6b1 to 0x035a97 (0x6c19 bytes) 000369 1775884683.087214 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35a98): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a98): Mirror 4: 0xde1e3a75 (stale) Mirror 5: 0xde1e3a75 (stale) Mirror 6: 0xde1e3a75 (stale) Mirror 7: 0xde1e3a75 (stale) CRC-32 checksum value for chunk [0, 0x35a98): Mirror 1: 0x49eb7083 Mirror 8: 0x49eb7083 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a98: skipped 000371 1775884683.517482 mapwrite 0x001a89 thru 0x00a354 (0x88cc bytes) 000372 1775884683.607026 read 0x02c6b8 thru 0x035a97 (0x93e0 bytes) 000374 1775884683.614841 trunc from 0x035a98 to 0x0090b2 (0x2c9e5 bytes) 000375 1775884683.628261 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x90b3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x90b3): Mirror 4: 0xcc2e6f49 (stale) Mirror 5: 0xcc2e6f49 (stale) Mirror 6: 0xcc2e6f49 (stale) Mirror 7: 0xcc2e6f49 (stale) Mirror 8: 0xcc2e6f49 (stale) CRC-32 checksum value for chunk [0, 0x90b3): Mirror 1: 0xfab5e475 Mirror 9: 0xfab5e475 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x90b3: skipped 000376 1775884683.948536 write 0x03446b thru 0x034b27 (0x6bd bytes) 000377 1775884683.976888 trunc from 0x034b28 to 0x03397d (0x11aa bytes) 000380 1775884684.000636 trunc from 0x03397e to 0x00f9e8 (0x23f95 bytes) 000381 1775884684.017657 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xf9e9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf9e9): Mirror 4: 0x2c38a4a9 (stale) Mirror 5: 0x2c38a4a9 (stale) Mirror 6: 0x2c38a4a9 (stale) Mirror 7: 0x2c38a4a9 (stale) Mirror 8: 0xcd83ba87 (stale) Mirror 9: 0xb9fa23d8 (stale) CRC-32 checksum value for chunk [0, 0xf9e9): Mirror 1: 0xb9fa23d8 Mirror 10: 0xb9fa23d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf9e9: skipped 000382 1775884684.375811 mapread 0x0060c2 thru 0x008ff9 (0x2f38 bytes) 000384 1775884684.394913 mapwrite 0x020e29 thru 0x0233dc (0x25b4 bytes) 000386 1775884684.461833 mapread 0x00ab14 thru 0x0192f8 (0xe7e5 bytes) 000387 1775884684.471448 trunc from 0x0233dd to 0x01fb58 (0x3884 bytes) 000390 1775884684.483997 trunc from 0x01fb59 to 0x0166c2 (0x9496 bytes) 000391 1775884684.490350 write 0x038191 thru 0x03d3de (0x524e bytes) 000392 1775884684.491538 mapread 0x02324d thru 0x025912 (0x26c6 bytes) 000394 1775884684.496272 trunc from 0x03d3df to 0x0089cd (0x34a11 bytes) 000396 1775884684.503287 mapwrite 0x00f57d thru 0x011ccf (0x2753 bytes) 000397 1775884684.529446 write 0x01939b thru 0x0237fb (0xa461 bytes) 000398 1775884684.530912 trunc from 0x0237fc to 0x037c95 (0x1449a bytes) 000399 1775884684.538496 write 0x026643 thru 0x034b7d (0xe53b bytes) 000400 1775884684.546068 read 0x02f6c1 thru 0x0348d7 (0x5217 bytes) 000403 1775884684.547915 trunc from 0x037c96 to 0x0321b6 (0x5adf bytes) 000409 1775884684.562430 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x321b7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321b7): Mirror 4: 0xc5f3fc1c (stale) Mirror 5: 0xc5f3fc1c (stale) Mirror 6: 0xc5f3fc1c (stale) Mirror 7: 0xc5f3fc1c (stale) Mirror 8: 0x5f89f70a (stale) Mirror 9: 0x26fbfa7 (stale) Mirror 10: 0x26fbfa7 (stale) CRC-32 checksum value for chunk [0, 0x321b7): Mirror 1: 0x10fb8555 Mirror 11: 0x10fb8555 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321b7: skipped 000411 1775884684.956313 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x321b7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321b7): Mirror 4: 0xc5f3fc1c (stale) Mirror 5: 0xc5f3fc1c (stale) Mirror 6: 0xc5f3fc1c (stale) Mirror 7: 0xc5f3fc1c (stale) Mirror 8: 0x5f89f70a (stale) Mirror 9: 0x26fbfa7 (stale) Mirror 10: 0x26fbfa7 (stale) CRC-32 checksum value for chunk [0, 0x321b7): Mirror 1: 0x10fb8555 Mirror 11: 0x10fb8555 Mirror 12: 0x10fb8555 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321b7: skipped 000413 1775884685.282624 read 0x02b83a thru 0x02eec3 (0x368a bytes) 000414 1775884685.285308 mapwrite 0x036231 thru 0x03dc68 (0x7a38 bytes) 000415 1775884685.351421 mapwrite 0x037a5c thru 0x03ca34 (0x4fd9 bytes) 000416 1775884685.369770 mapwrite 0x00296c thru 0x010042 (0xd6d7 bytes) 000417 1775884685.392430 mapwrite 0x004d19 thru 0x013ab9 (0xeda1 bytes) 000418 1775884685.416281 trunc from 0x03dc69 to 0x02149f (0x1c7c9 bytes) 000419 1775884685.425821 mapread 0x015a33 thru 0x01c85a (0x6e28 bytes) 000420 1775884685.431548 mapread 0x015b91 thru 0x015cdf (0x14f bytes) 000422 1775884685.435107 MIRROR_RESYNC 10 mirrors 000426 1775884685.929321 trunc from 0x0214a0 to 0x00b578 (0x15f27 bytes) 000429 1775884685.972997 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb579): Mirror 4: 0x9e998931 (stale) Mirror 5: 0x9e998931 (stale) Mirror 6: 0x9e998931 (stale) Mirror 7: 0x9e998931 (stale) Mirror 8: 0x9e998931 (stale) Mirror 9: 0x9e998931 (stale) Mirror 10: 0x9e998931 (stale) Mirror 12: 0x9e998931 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb579): Mirror 4: 0x9e998931 (stale) Mirror 5: 0x9e998931 (stale) Mirror 6: 0x9e998931 (stale) Mirror 7: 0x9e998931 (stale) Mirror 8: 0x9e998931 (stale) Mirror 9: 0x9e998931 (stale) Mirror 10: 0x9e998931 (stale) Mirror 12: 0x9e998931 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb579: skipped 000430 1775884686.123853 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xb579): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb579): Mirror 4: 0x9e998931 (stale) Mirror 5: 0x9e998931 (stale) Mirror 6: 0x9e998931 (stale) Mirror 7: 0x9e998931 (stale) Mirror 8: 0x9e998931 (stale) Mirror 9: 0x9e998931 (stale) Mirror 10: 0x9e998931 (stale) Mirror 12: 0x9e998931 (stale) CRC-32 checksum value for chunk [0, 0xb579): Mirror 1: 0x9e998931 Mirror 13: 0x9e998931 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb579: skipped 000435 1775884686.456595 mapread 0x002bce thru 0x00b578 (0x89ab bytes) 000437 1775884686.479845 write 0x032bde thru 0x03efa1 (0xc3c4 bytes) 000439 1775884686.507844 read 0x03dc0a thru 0x03efa1 (0x1398 bytes) 000444 1775884686.509810 trunc from 0x03efa2 to 0x00f306 (0x2fc9b bytes) 000445 1775884686.518158 trunc from 0x00f307 to 0x03bc5d (0x2c957 bytes) 000449 1775884686.525772 mapread 0x0182da thru 0x01f8b4 (0x75db bytes) 000450 1775884686.530801 mapwrite 0x01f215 thru 0x02c930 (0xd71c bytes) 000452 1775884686.558443 mapwrite 0x01d365 thru 0x02aa0c (0xd6a8 bytes) 000453 1775884686.575015 trunc from 0x03bc5e to 0x00bf1a (0x2fd43 bytes) 000456 1775884686.582122 mapwrite 0x01d8c5 thru 0x02c62f (0xed6b bytes) 000457 1775884686.612279 write 0x02f845 thru 0x030075 (0x831 bytes) 000458 1775884686.613337 mapread 0x014c20 thru 0x024677 (0xfa58 bytes) 000460 1775884686.618711 mapread 0x01cae2 thru 0x020700 (0x3c1f bytes) 000461 1775884686.619502 read 0x00966e thru 0x01599d (0xc330 bytes) 000462 1775884686.623230 mapwrite 0x018592 thru 0x01a6b2 (0x2121 bytes) 000463 1775884686.638557 trunc from 0x030076 to 0x005f6c (0x2a109 bytes) 000466 1775884686.646231 mapread 0x005c32 thru 0x005f6c (0x33b bytes) 000467 1775884686.647189 mapread 0x00173c thru 0x005f6c (0x4831 bytes) 000470 1775884686.650411 MIRROR_RESYNC 10 mirrors 000471 1775884687.103300 MIRROR_EXTEND to 11 mirrors 000473 1775884687.353710 MIRROR_EXTEND to 12 mirrors 000474 1775884687.596649 read 0x00223e thru 0x005f6c (0x3d2f bytes) 000475 1775884687.599431 read 0x0054a7 thru 0x005f6c (0xac6 bytes) 000476 1775884687.599721 write 0x01a19b thru 0x01ee4f (0x4cb5 bytes) 000477 1775884687.631024 MIRROR_RESYNC 12 mirrors 000478 1775884688.098830 read 0x015292 thru 0x01ee4f (0x9bbe bytes) 000479 1775884688.102280 read 0x013194 thru 0x01c7a1 (0x960e bytes) 000480 1775884688.105785 mapwrite 0x0189ab thru 0x028928 (0xff7e bytes) 000482 1775884688.165065 write 0x03821c thru 0x03ffff (0x7de4 bytes) 000483 1775884688.166151 mapread 0x03e8a4 thru 0x03ffff (0x175c bytes) 000484 1775884688.166809 mapread 0x0397b6 thru 0x03b215 (0x1a60 bytes) 000485 1775884688.167438 write 0x03b527 thru 0x03ffff (0x4ad9 bytes) 000486 1775884688.168052 mapwrite 0x035aec thru 0x03b196 (0x56ab bytes) 000487 1775884688.184789 mapread 0x00f1a7 thru 0x017a61 (0x88bb bytes) 000488 1775884688.197937 write 0x03c55a thru 0x03ffff (0x3aa6 bytes) 000489 1775884688.198710 write 0x004abc thru 0x00d7a3 (0x8ce8 bytes) 000490 1775884688.202259 trunc from 0x040000 to 0x02b3ce (0x14c31 bytes) 000491 1775884688.208620 trunc from 0x02b3cf to 0x036ecb (0xbafd bytes) 000492 1775884688.213268 mapread 0x02f3e2 thru 0x036ecb (0x7aea bytes) 000493 1775884688.227650 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x36ecc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ecc): Mirror 4: 0x6543603 (stale) Mirror 5: 0x6543603 (stale) Mirror 6: 0x6543603 (stale) Mirror 7: 0x6543603 (stale) Mirror 8: 0x6543603 (stale) Mirror 9: 0x6543603 (stale) Mirror 10: 0x6543603 (stale) Mirror 12: 0x6543603 (stale) Mirror 13: 0x6543603 (stale) Mirror 14: 0x6543603 (stale) Mirror 15: 0x6543603 (stale) CRC-32 checksum value for chunk [0, 0x36ecc): Mirror 1: 0x9dee8845 Mirror 16: 0x9dee8845 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ecc: skipped 000494 1775884688.530043 write 0x035063 thru 0x03ffff (0xaf9d bytes) 000495 1775884688.553192 trunc from 0x040000 to 0x03f22a (0xdd5 bytes) 000496 1775884688.564529 trunc from 0x03f22b to 0x01e091 (0x21199 bytes) 000498 1775884688.570017 write 0x03d1df thru 0x03ffff (0x2e21 bytes) 000501 1775884688.571987 write 0x025a41 thru 0x02d858 (0x7e18 bytes) 000502 1775884688.575904 trunc from 0x040000 to 0x01e697 (0x21968 bytes) 000503 1775884688.581583 trunc from 0x01e698 to 0x030c19 (0x12582 bytes) 000505 1775884688.587242 write 0x01e7e9 thru 0x02be69 (0xd681 bytes) 000506 1775884688.592587 mapread 0x020265 thru 0x02de2e (0xdbca bytes) 000510 1775884688.597259 trunc from 0x030c1a to 0x0377a4 (0x6b8b bytes) 000512 1775884688.603624 mapwrite 0x016ba9 thru 0x02100d (0xa465 bytes) 000514 1775884688.624424 mapread 0x00b6d1 thru 0x00ecfc (0x362c bytes) 000516 1775884688.628456 mapwrite 0x011e53 thru 0x01f17d (0xd32b bytes) 000517 1775884688.642012 trunc from 0x0377a5 to 0x036117 (0x168d bytes) 000519 1775884688.646936 read 0x00478f thru 0x006290 (0x1b02 bytes) 000521 1775884688.647767 trunc from 0x036118 to 0x012a65 (0x236b2 bytes) 000522 1775884688.651962 trunc from 0x012a66 to 0x030129 (0x1d6c4 bytes) 000523 1775884688.655709 mapwrite 0x030c3d thru 0x03b441 (0xa805 bytes) 000525 1775884688.678025 mapwrite 0x0377b5 thru 0x03a664 (0x2eb0 bytes) 000526 1775884688.691764 write 0x022de4 thru 0x023524 (0x741 bytes) 000530 1775884688.696857 trunc from 0x03b442 to 0x0359c4 (0x5a7d bytes) 000532 1775884688.702809 mapread 0x017fce thru 0x01cbdc (0x4c0f bytes) 000533 1775884688.706941 trunc from 0x0359c5 to 0x020130 (0x15894 bytes) 000534 1775884688.713898 read 0x00fc1e thru 0x0190a4 (0x9487 bytes) 000537 1775884688.715548 trunc from 0x020131 to 0x008d66 (0x173ca bytes) 000538 1775884688.722420 mapwrite 0x038e3d thru 0x03ffff (0x71c3 bytes) 000539 1775884688.746414 read 0x0061c5 thru 0x013730 (0xd56c bytes) 000541 1775884688.747696 read 0x00aafe thru 0x017d45 (0xd248 bytes) 000542 1775884688.748348 write 0x018f40 thru 0x0212c1 (0x8382 bytes) 000546 1775884688.750338 read 0x038a2e thru 0x03ffff (0x75d2 bytes) 000548 1775884688.751452 read 0x0072d7 thru 0x0114b0 (0xa1da bytes) 000550 1775884688.752632 mapread 0x038079 thru 0x03ffff (0x7f87 bytes) 000554 1775884688.757028 MIRROR_RESYNC 13 mirrors 000556 1775884689.245819 write 0x01a51d thru 0x02952d (0xf011 bytes) 000558 1775884689.270316 write 0x00ec5e thru 0x00fd37 (0x10da bytes) 000565 1775884689.276651 read 0x03f0c0 thru 0x03ffff (0xf40 bytes) 000567 1775884689.281907 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc8b4d4be (stale) Mirror 6: 0xc8b4d4be (stale) Mirror 7: 0xc8b4d4be (stale) Mirror 8: 0xc8b4d4be (stale) Mirror 9: 0xc8b4d4be (stale) Mirror 10: 0xc8b4d4be (stale) Mirror 12: 0xc8b4d4be (stale) Mirror 13: 0xc8b4d4be (stale) Mirror 14: 0xc8b4d4be (stale) Mirror 15: 0xc8b4d4be (stale) Mirror 16: 0xc8b4d4be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc8b4d4be (stale) Mirror 6: 0xc8b4d4be (stale) Mirror 7: 0xc8b4d4be (stale) Mirror 8: 0xc8b4d4be (stale) Mirror 9: 0xc8b4d4be (stale) Mirror 10: 0xc8b4d4be (stale) Mirror 12: 0xc8b4d4be (stale) Mirror 13: 0xc8b4d4be (stale) Mirror 14: 0xc8b4d4be (stale) Mirror 15: 0xc8b4d4be (stale) Mirror 16: 0xc8b4d4be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000569 1775884689.495266 trunc from 0x040000 to 0x026765 (0x1989a bytes) 000570 1775884689.500456 mapread 0x00151b thru 0x005782 (0x4268 bytes) 000571 1775884689.507335 mapwrite 0x0153ef thru 0x02300e (0xdc20 bytes) 000572 1775884689.533546 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x26766): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26766): Mirror 5: 0x6e2aa4fc (stale) Mirror 6: 0x6e2aa4fc (stale) Mirror 7: 0x6e2aa4fc (stale) Mirror 8: 0x6e2aa4fc (stale) Mirror 9: 0x6e2aa4fc (stale) Mirror 10: 0x6e2aa4fc (stale) Mirror 12: 0x6e2aa4fc (stale) Mirror 13: 0x6e2aa4fc (stale) Mirror 14: 0x6e2aa4fc (stale) Mirror 15: 0x6e2aa4fc (stale) Mirror 16: 0x6e2aa4fc (stale) CRC-32 checksum value for chunk [0, 0x26766): Mirror 1: 0x358728e8 Mirror 17: 0x358728e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26766: skipped 000573 1775884689.835504 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x26766): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26766): Mirror 5: 0x6e2aa4fc (stale) Mirror 6: 0x6e2aa4fc (stale) Mirror 7: 0x6e2aa4fc (stale) Mirror 8: 0x6e2aa4fc (stale) Mirror 9: 0x6e2aa4fc (stale) Mirror 10: 0x6e2aa4fc (stale) Mirror 12: 0x6e2aa4fc (stale) Mirror 13: 0x6e2aa4fc (stale) Mirror 14: 0x6e2aa4fc (stale) Mirror 15: 0x6e2aa4fc (stale) Mirror 16: 0x6e2aa4fc (stale) CRC-32 checksum value for chunk [0, 0x26766): Mirror 1: 0x358728e8 Mirror 17: 0x358728e8 Mirror 18: 0x358728e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26766: skipped 000574 1775884690.110035 trunc from 0x026766 to 0x02c031 (0x58cc bytes) 000577 1775884690.138566 write 0x02404f thru 0x02d93e (0x98f0 bytes) 000578 1775884690.140890 trunc from 0x02d93f to 0x0245e9 (0x9355 bytes) 000583 1775884690.148400 write 0x03edea thru 0x03ffff (0x1216 bytes) 000584 1775884690.150705 MIRROR_RESYNC 14 mirrors 000585 1775884690.666082 read 0x017944 thru 0x024fd1 (0xd68e bytes) 000591 1775884690.669945 read 0x03b29b thru 0x03ffff (0x4d65 bytes) 000592 1775884690.673356 trunc from 0x040000 to 0x01d19d (0x22e62 bytes) 000593 1775884690.695470 write 0x00f648 thru 0x01988e (0xa247 bytes) 000594 1775884690.701003 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1d19e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d19e): Mirror 5: 0xc2b1d75e (stale) Mirror 6: 0xc2b1d75e (stale) Mirror 7: 0xc2b1d75e (stale) Mirror 8: 0xc2b1d75e (stale) Mirror 9: 0xc2b1d75e (stale) Mirror 10: 0xc2b1d75e (stale) Mirror 12: 0xc2b1d75e (stale) Mirror 13: 0xc2b1d75e (stale) Mirror 14: 0xc2b1d75e (stale) Mirror 15: 0xc2b1d75e (stale) Mirror 16: 0xc2b1d75e (stale) Mirror 17: 0xc2b1d75e (stale) Mirror 18: 0xc2b1d75e (stale) CRC-32 checksum value for chunk [0, 0x1d19e): Mirror 1: 0xa2a42251 Mirror 19: 0xa2a42251 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d19e: skipped 000595 1775884691.047105 read 0x01583d thru 0x01c4e1 (0x6ca5 bytes) 000596 1775884691.051708 mapwrite 0x018ea3 thru 0x025034 (0xc192 bytes) 000597 1775884691.105605 write 0x00dcd6 thru 0x013702 (0x5a2d bytes) 000598 1775884691.109760 read 0x01dfca thru 0x01f6a8 (0x16df bytes) 000599 1775884691.111902 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25035): Mirror 5: 0x45693595 (stale) Mirror 7: 0x45693595 (stale) Mirror 8: 0x45693595 (stale) Mirror 9: 0x45693595 (stale) Mirror 10: 0x45693595 (stale) Mirror 12: 0x45693595 (stale) Mirror 13: 0x45693595 (stale) Mirror 14: 0x45693595 (stale) Mirror 15: 0x45693595 (stale) Mirror 16: 0x45693595 (stale) Mirror 17: 0x45693595 (stale) Mirror 18: 0x45693595 (stale) Mirror 19: 0xfae3eaa3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25035): Mirror 5: 0x45693595 (stale) Mirror 7: 0x45693595 (stale) Mirror 8: 0x45693595 (stale) Mirror 9: 0x45693595 (stale) Mirror 10: 0x45693595 (stale) Mirror 12: 0x45693595 (stale) Mirror 13: 0x45693595 (stale) Mirror 14: 0x45693595 (stale) Mirror 15: 0x45693595 (stale) Mirror 16: 0x45693595 (stale) Mirror 17: 0x45693595 (stale) Mirror 18: 0x45693595 (stale) Mirror 19: 0xfae3eaa3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25035: skipped 000600 1775884691.304043 mapwrite 0x015312 thru 0x02459f (0xf28e bytes) 000601 1775884691.333527 mapwrite 0x001c43 thru 0x00a392 (0x8750 bytes) 000602 1775884691.353748 mapwrite 0x00b6e4 thru 0x011d8e (0x66ab bytes) 000604 1775884691.369239 read 0x010a8a thru 0x01469d (0x3c14 bytes) 000605 1775884691.369798 trunc from 0x025035 to 0x0095ea (0x1ba4a bytes) 000607 1775884691.377622 MIRROR_RESYNC 14 mirrors 000608 1775884691.879627 MIRROR_RESYNC 14 mirrors 000614 1775884692.073003 read 0x00659b thru 0x0095ea (0x3050 bytes) 000616 1775884692.075388 read 0x0012a1 thru 0x0095ea (0x834a bytes) 000617 1775884692.077845 mapread 0x007279 thru 0x007e99 (0xc21 bytes) 000619 1775884692.078514 mapread 0x0094d8 thru 0x0095ea (0x113 bytes) 000620 1775884692.078814 read 0x005cd9 thru 0x0095ea (0x3912 bytes) 000622 1775884692.079273 write 0x026190 thru 0x0357cc (0xf63d bytes) 000623 1775884692.100187 mapwrite 0x01835a thru 0x020387 (0x802e bytes) 000625 1775884692.123451 mapread 0x02dccd thru 0x0357cc (0x7b00 bytes) 000626 1775884692.124323 read 0x015d8a thru 0x018169 (0x23e0 bytes) 000628 1775884692.125329 trunc from 0x0357cd to 0x036a6a (0x129e bytes) 000629 1775884692.132537 mapread 0x02ade0 thru 0x032266 (0x7487 bytes) 000631 1775884692.133953 read 0x01b93e thru 0x01dcef (0x23b2 bytes) 000632 1775884692.134495 read 0x02f7a3 thru 0x030858 (0x10b6 bytes) 000634 1775884692.137277 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x36a6b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a6b): Mirror 5: 0x83ba8cfd (stale) Mirror 7: 0x83ba8cfd (stale) Mirror 8: 0x83ba8cfd (stale) Mirror 9: 0x83ba8cfd (stale) Mirror 10: 0x83ba8cfd (stale) Mirror 12: 0x83ba8cfd (stale) Mirror 13: 0x83ba8cfd (stale) Mirror 14: 0x83ba8cfd (stale) Mirror 15: 0x83ba8cfd (stale) Mirror 16: 0x83ba8cfd (stale) Mirror 17: 0x83ba8cfd (stale) Mirror 18: 0x83ba8cfd (stale) Mirror 19: 0x83ba8cfd (stale) CRC-32 checksum value for chunk [0, 0x36a6b): Mirror 1: 0x6193c9ea Mirror 20: 0x6193c9ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a6b: skipped 000636 1775884692.418046 write 0x03b3c5 thru 0x03ffff (0x4c3b bytes) 000637 1775884692.441998 read 0x01955d thru 0x025bce (0xc672 bytes) 000640 1775884692.446720 mapwrite 0x000d05 thru 0x001cb4 (0xfb0 bytes) 000642 1775884692.468545 write 0x030f21 thru 0x031c4c (0xd2c bytes) 000644 1775884692.474963 read 0x005969 thru 0x00f934 (0x9fcc bytes) 000645 1775884692.478914 trunc from 0x040000 to 0x03a75e (0x58a1 bytes) 000649 1775884692.491671 mapwrite 0x00c1bb thru 0x012829 (0x666f bytes) 000650 1775884692.505378 mapwrite 0x019fea thru 0x01a845 (0x85c bytes) 000653 1775884692.517302 trunc from 0x03a75f to 0x0271b3 (0x135ab bytes) 000655 1775884692.524345 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x271b4): Mirror 5: 0x8c8df76e (stale) Mirror 7: 0x8c8df76e (stale) Mirror 8: 0x8c8df76e (stale) Mirror 10: 0x8c8df76e (stale) Mirror 12: 0x8c8df76e (stale) Mirror 13: 0x8c8df76e (stale) Mirror 14: 0x8c8df76e (stale) Mirror 15: 0x8c8df76e (stale) Mirror 16: 0x8c8df76e (stale) Mirror 17: 0x8c8df76e (stale) Mirror 18: 0x8c8df76e (stale) Mirror 19: 0x8c8df76e (stale) Mirror 20: 0x1865dc6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x271b4): Mirror 5: 0x8c8df76e (stale) Mirror 7: 0x8c8df76e (stale) Mirror 8: 0x8c8df76e (stale) Mirror 10: 0x8c8df76e (stale) Mirror 12: 0x8c8df76e (stale) Mirror 13: 0x8c8df76e (stale) Mirror 14: 0x8c8df76e (stale) Mirror 15: 0x8c8df76e (stale) Mirror 16: 0x8c8df76e (stale) Mirror 17: 0x8c8df76e (stale) Mirror 18: 0x8c8df76e (stale) Mirror 19: 0x8c8df76e (stale) Mirror 20: 0x1865dc6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x271b4: skipped 000656 1775884692.737905 mapread 0x00748f thru 0x0086be (0x1230 bytes) 000658 1775884692.742875 trunc from 0x0271b4 to 0x030c19 (0x9a66 bytes) 000659 1775884692.748546 trunc from 0x030c1a to 0x0249e8 (0xc231 bytes) 000660 1775884692.753887 trunc from 0x0249e9 to 0x02921a (0x4832 bytes) 000661 1775884692.758698 mapread 0x0184c0 thru 0x01eaed (0x662e bytes) 000663 1775884692.762835 read 0x027caa thru 0x02921a (0x1571 bytes) 000664 1775884692.765266 read 0x0277f7 thru 0x02921a (0x1a24 bytes) 000665 1775884692.765897 trunc from 0x02921b to 0x01a61b (0xebff bytes) 000668 1775884692.772273 mapread 0x0134cf thru 0x01a61b (0x714d bytes) 000669 1775884692.773393 mapwrite 0x00c8fc thru 0x011e9e (0x55a3 bytes) 000670 1775884692.790097 trunc from 0x01a61c to 0x00749e (0x1317d bytes) 000672 1775884692.796635 read 0x00046b thru 0x00749e (0x7034 bytes) 000674 1775884692.799055 MIRROR_RESYNC 14 mirrors 000678 1775884693.303658 read 0x002698 thru 0x00749e (0x4e07 bytes) 000680 1775884693.306644 trunc from 0x00749f to 0x02a3b5 (0x22f17 bytes) 000683 1775884693.330680 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a3b6): Mirror 7: 0x40eb3c39 (stale) Mirror 8: 0x40eb3c39 (stale) Mirror 10: 0x40eb3c39 (stale) Mirror 12: 0x40eb3c39 (stale) Mirror 13: 0x40eb3c39 (stale) Mirror 14: 0x40eb3c39 (stale) Mirror 15: 0x40eb3c39 (stale) Mirror 16: 0x40eb3c39 (stale) Mirror 17: 0x40eb3c39 (stale) Mirror 18: 0x40eb3c39 (stale) Mirror 19: 0x40eb3c39 (stale) Mirror 20: 0x40eb3c39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a3b6): Mirror 7: 0x40eb3c39 (stale) Mirror 8: 0x40eb3c39 (stale) Mirror 10: 0x40eb3c39 (stale) Mirror 12: 0x40eb3c39 (stale) Mirror 13: 0x40eb3c39 (stale) Mirror 14: 0x40eb3c39 (stale) Mirror 15: 0x40eb3c39 (stale) Mirror 16: 0x40eb3c39 (stale) Mirror 17: 0x40eb3c39 (stale) Mirror 18: 0x40eb3c39 (stale) Mirror 19: 0x40eb3c39 (stale) Mirror 20: 0x40eb3c39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a3b6: skipped 000684 1775884693.512709 mapread 0x012ccf thru 0x014810 (0x1b42 bytes) 000686 1775884693.520682 trunc from 0x02a3b6 to 0x00bd98 (0x1e61d bytes) 000687 1775884693.526928 trunc from 0x00bd99 to 0x01ee0f (0x13077 bytes) 000689 1775884693.533158 mapread 0x01cff4 thru 0x01ee0f (0x1e1c bytes) 000691 1775884693.537825 read 0x006794 thru 0x00d08b (0x68f8 bytes) 000694 1775884693.542866 read 0x01df67 thru 0x01ee0f (0xea9 bytes) 000695 1775884693.543355 mapread 0x00e196 thru 0x01ca6c (0xe8d7 bytes) 000701 1775884693.549419 write 0x020df7 thru 0x023445 (0x264f bytes) 000702 1775884693.550368 mapread 0x015f0b thru 0x023445 (0xd53b bytes) 000704 1775884693.554040 write 0x0033c9 thru 0x004aab (0x16e3 bytes) 000705 1775884693.554812 mapread 0x01ba2b thru 0x020511 (0x4ae7 bytes) 000706 1775884693.555535 read 0x01ff32 thru 0x023445 (0x3514 bytes) 000707 1775884693.556084 mapwrite 0x02318d thru 0x03244b (0xf2bf bytes) 000708 1775884693.586807 read 0x013ead thru 0x01f725 (0xb879 bytes) 000711 1775884693.588600 trunc from 0x03244c to 0x00539c (0x2d0af bytes) 000712 1775884693.595895 trunc from 0x00539d to 0x03cd2e (0x37992 bytes) 000713 1775884693.601857 trunc from 0x03cd2f to 0x00782f (0x354ff bytes) 000714 1775884693.607209 read 0x006cd7 thru 0x00782f (0xb59 bytes) 000719 1775884693.612998 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x7830): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7830): Mirror 7: 0x9c9742dd (stale) Mirror 8: 0x9c9742dd (stale) Mirror 10: 0x9c9742dd (stale) Mirror 12: 0x9c9742dd (stale) Mirror 13: 0x9c9742dd (stale) Mirror 14: 0x9c9742dd (stale) Mirror 15: 0x9c9742dd (stale) Mirror 16: 0x9c9742dd (stale) Mirror 17: 0x9c9742dd (stale) Mirror 18: 0x9c9742dd (stale) Mirror 19: 0x9c9742dd (stale) Mirror 20: 0x9c9742dd (stale) CRC-32 checksum value for chunk [0, 0x7830): Mirror 1: 0x58245328 Mirror 21: 0x58245328 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7830: skipped 000720 1775884693.886263 read 0x0066dc thru 0x00782f (0x1154 bytes) 000722 1775884693.890961 write 0x032753 thru 0x03ffff (0xd8ad bytes) 000723 1775884693.918534 trunc from 0x040000 to 0x031070 (0xef8f bytes) 000725 1775884693.931719 write 0x025b27 thru 0x0273b8 (0x1892 bytes) 000728 1775884693.937013 write 0x02250a thru 0x02e99d (0xc494 bytes) 000729 1775884693.943325 write 0x01a437 thru 0x01f5af (0x5179 bytes) 000731 1775884693.947195 trunc from 0x031071 to 0x02c9a1 (0x46cf bytes) 000733 1775884693.953345 write 0x02a1fe thru 0x0359ad (0xb7b0 bytes) 000736 1775884693.957348 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x359ae): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x359ae): Mirror 7: 0x49ae6181 (stale) Mirror 8: 0x49ae6181 (stale) Mirror 10: 0x49ae6181 (stale) Mirror 12: 0x49ae6181 (stale) Mirror 13: 0x49ae6181 (stale) Mirror 14: 0x49ae6181 (stale) Mirror 15: 0x49ae6181 (stale) Mirror 16: 0x49ae6181 (stale) Mirror 17: 0x49ae6181 (stale) Mirror 18: 0x49ae6181 (stale) Mirror 19: 0x49ae6181 (stale) Mirror 20: 0x49ae6181 (stale) Mirror 21: 0x225ac7ba (stale) CRC-32 checksum value for chunk [0, 0x359ae): Mirror 1: 0x2a38518b Mirror 22: 0x2a38518b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x359ae: skipped 000739 1775884694.301737 write 0x0372d7 thru 0x03ab17 (0x3841 bytes) 000740 1775884694.325757 mapread 0x0017d8 thru 0x00e10f (0xc938 bytes) 000741 1775884694.336044 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3ab18): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab18): Mirror 7: 0x2535b3c7 (stale) Mirror 8: 0x2535b3c7 (stale) Mirror 10: 0x2535b3c7 (stale) Mirror 12: 0x2535b3c7 (stale) Mirror 13: 0x2535b3c7 (stale) Mirror 14: 0x2535b3c7 (stale) Mirror 15: 0x2535b3c7 (stale) Mirror 16: 0x2535b3c7 (stale) Mirror 17: 0x2535b3c7 (stale) Mirror 18: 0x2535b3c7 (stale) Mirror 19: 0x2535b3c7 (stale) Mirror 20: 0x2535b3c7 (stale) Mirror 21: 0x3c15c3d9 (stale) Mirror 22: 0x906551f8 (stale) CRC-32 checksum value for chunk [0, 0x3ab18): Mirror 1: 0x690fa4a1 Mirror 23: 0x690fa4a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab18: skipped 000743 1775884694.654993 trunc from 0x03ab18 to 0x0269e8 (0x1412f bytes) 000744 1775884694.687252 write 0x016c2a thru 0x01d7c4 (0x6b9b bytes) 000745 1775884694.690800 read 0x01d377 thru 0x0269e8 (0x9672 bytes) 000746 1775884694.694470 trunc from 0x0269e9 to 0x0260d5 (0x913 bytes) 000747 1775884694.705401 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x260d6): Mirror 7: 0x2b49afd6 (stale) Mirror 8: 0x2b49afd6 (stale) Mirror 10: 0x2b49afd6 (stale) Mirror 12: 0x2b49afd6 (stale) Mirror 13: 0x2b49afd6 (stale) Mirror 14: 0x2b49afd6 (stale) Mirror 15: 0x2b49afd6 (stale) Mirror 16: 0x2b49afd6 (stale) Mirror 18: 0x2b49afd6 (stale) Mirror 19: 0x2b49afd6 (stale) Mirror 20: 0x2b49afd6 (stale) Mirror 21: 0x21077ca7 (stale) Mirror 22: 0x63d2df8d (stale) Mirror 23: 0x63d2df8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x260d6): Mirror 7: 0x2b49afd6 (stale) Mirror 8: 0x2b49afd6 (stale) Mirror 10: 0x2b49afd6 (stale) Mirror 12: 0x2b49afd6 (stale) Mirror 13: 0x2b49afd6 (stale) Mirror 14: 0x2b49afd6 (stale) Mirror 15: 0x2b49afd6 (stale) Mirror 16: 0x2b49afd6 (stale) Mirror 18: 0x2b49afd6 (stale) Mirror 19: 0x2b49afd6 (stale) Mirror 20: 0x2b49afd6 (stale) Mirror 21: 0x21077ca7 (stale) Mirror 22: 0x63d2df8d (stale) Mirror 23: 0x63d2df8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x260d6: skipped 000748 1775884694.931261 mapwrite 0x00a2d7 thru 0x01074b (0x6475 bytes) 000751 1775884694.955398 trunc from 0x0260d6 to 0x0004eb (0x25bea bytes) 000753 1775884694.961559 trunc from 0x0004ec to 0x02772b (0x27240 bytes) 000754 1775884694.966028 mapread 0x021d0f thru 0x02772b (0x5a1d bytes) 000755 1775884694.970048 mapwrite 0x031a99 thru 0x03ffff (0xe567 bytes) 000756 1775884694.996974 MIRROR_RESYNC 15 mirrors 000757 1775884695.557958 mapread 0x006ba7 thru 0x013f2d (0xd387 bytes) 000758 1775884695.567492 mapread 0x01acf4 thru 0x020042 (0x534f bytes) 000759 1775884695.571555 mapwrite 0x02e6b5 thru 0x03ac2e (0xc57a bytes) 000761 1775884695.638480 trunc from 0x040000 to 0x033068 (0xcf97 bytes) 000763 1775884695.650327 mapread 0x01f26f thru 0x0246d7 (0x5469 bytes) 000764 1775884695.654558 trunc from 0x033069 to 0x002776 (0x308f2 bytes) 000766 1775884695.661045 read 0x0016e6 thru 0x002776 (0x1091 bytes) 000767 1775884695.664011 mapread 0x000b63 thru 0x002776 (0x1c14 bytes) 000768 1775884695.666847 mapwrite 0x03540b thru 0x03ffff (0xabf5 bytes) 000769 1775884695.701007 mapread 0x02fc09 thru 0x0367c9 (0x6bc1 bytes) 000770 1775884695.701916 write 0x004b7b thru 0x0091af (0x4635 bytes) 000771 1775884695.706503 mapwrite 0x02bb17 thru 0x030e3a (0x5324 bytes) 000772 1775884695.726238 write 0x029c21 thru 0x0327af (0x8b8f bytes) 000773 1775884695.727342 mapread 0x03c599 thru 0x03d463 (0xecb bytes) 000774 1775884695.730259 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9688b0c2 (stale) Mirror 8: 0x9688b0c2 (stale) Mirror 10: 0x9688b0c2 (stale) Mirror 12: 0x9688b0c2 (stale) Mirror 13: 0x9688b0c2 (stale) Mirror 14: 0x9688b0c2 (stale) Mirror 15: 0x9688b0c2 (stale) Mirror 16: 0x9688b0c2 (stale) Mirror 18: 0x9688b0c2 (stale) Mirror 19: 0x9688b0c2 (stale) Mirror 20: 0x9688b0c2 (stale) Mirror 21: 0x9688b0c2 (stale) Mirror 22: 0x9688b0c2 (stale) Mirror 23: 0x9688b0c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x47222ffe Mirror 24: 0x47222ffe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000775 1775884696.035732 MIRROR_SPLIT mirror 5 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9688b0c2 (stale) Mirror 8: 0x9688b0c2 (stale) Mirror 10: 0x9688b0c2 (stale) Mirror 12: 0x9688b0c2 (stale) Mirror 14: 0x9688b0c2 (stale) Mirror 15: 0x9688b0c2 (stale) Mirror 16: 0x9688b0c2 (stale) Mirror 18: 0x9688b0c2 (stale) Mirror 19: 0x9688b0c2 (stale) Mirror 20: 0x9688b0c2 (stale) Mirror 21: 0x9688b0c2 (stale) Mirror 22: 0x9688b0c2 (stale) Mirror 23: 0x9688b0c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9688b0c2 (stale) Mirror 8: 0x9688b0c2 (stale) Mirror 10: 0x9688b0c2 (stale) Mirror 12: 0x9688b0c2 (stale) Mirror 14: 0x9688b0c2 (stale) Mirror 15: 0x9688b0c2 (stale) Mirror 16: 0x9688b0c2 (stale) Mirror 18: 0x9688b0c2 (stale) Mirror 19: 0x9688b0c2 (stale) Mirror 20: 0x9688b0c2 (stale) Mirror 21: 0x9688b0c2 (stale) Mirror 22: 0x9688b0c2 (stale) Mirror 23: 0x9688b0c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x47222ffe Mirror 24: 0x47222ffe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000776 1775884696.382105 write 0x011cb0 thru 0x012ac9 (0xe1a bytes) 000777 1775884696.409666 trunc from 0x040000 to 0x0329df (0xd620 bytes) 000778 1775884696.424458 MIRROR_RESYNC 15 mirrors 000779 1775884697.017419 trunc from 0x0329e0 to 0x000d7e (0x31c61 bytes) 000780 1775884697.045089 mapread 0x0008ef thru 0x000d7e (0x490 bytes) 000781 1775884697.048565 mapwrite 0x027591 thru 0x030d0b (0x977b bytes) 000783 1775884697.089434 write 0x020ad9 thru 0x02d86b (0xcd93 bytes) 000785 1775884697.094566 trunc from 0x030d0c to 0x016bbb (0x1a150 bytes) 000786 1775884697.101248 mapwrite 0x01a2c2 thru 0x02425e (0x9f9d bytes) 000788 1775884697.133590 write 0x01e4aa thru 0x01f0f7 (0xc4e bytes) 000790 1775884697.137019 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2425f): Mirror 7: 0x60c85e3e (stale) Mirror 8: 0x60c85e3e (stale) Mirror 10: 0x60c85e3e (stale) Mirror 12: 0x60c85e3e (stale) Mirror 14: 0x60c85e3e (stale) Mirror 16: 0x60c85e3e (stale) Mirror 18: 0x60c85e3e (stale) Mirror 19: 0x60c85e3e (stale) Mirror 20: 0x60c85e3e (stale) Mirror 21: 0x60c85e3e (stale) Mirror 22: 0x60c85e3e (stale) Mirror 23: 0x60c85e3e (stale) Mirror 24: 0x60c85e3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2425f): Mirror 7: 0x60c85e3e (stale) Mirror 8: 0x60c85e3e (stale) Mirror 10: 0x60c85e3e (stale) Mirror 12: 0x60c85e3e (stale) Mirror 14: 0x60c85e3e (stale) Mirror 16: 0x60c85e3e (stale) Mirror 18: 0x60c85e3e (stale) Mirror 19: 0x60c85e3e (stale) Mirror 20: 0x60c85e3e (stale) Mirror 21: 0x60c85e3e (stale) Mirror 22: 0x60c85e3e (stale) Mirror 23: 0x60c85e3e (stale) Mirror 24: 0x60c85e3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2425f: skipped 000791 1775884697.344096 mapwrite 0x004de8 thru 0x008583 (0x379c bytes) 000792 1775884697.369021 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2425f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2425f): Mirror 7: 0x60c85e3e (stale) Mirror 8: 0x60c85e3e (stale) Mirror 10: 0x60c85e3e (stale) Mirror 12: 0x60c85e3e (stale) Mirror 14: 0x60c85e3e (stale) Mirror 16: 0x60c85e3e (stale) Mirror 18: 0x60c85e3e (stale) Mirror 19: 0x60c85e3e (stale) Mirror 20: 0x60c85e3e (stale) Mirror 21: 0x60c85e3e (stale) Mirror 22: 0x60c85e3e (stale) Mirror 23: 0x60c85e3e (stale) Mirror 24: 0x60c85e3e (stale) CRC-32 checksum value for chunk [0, 0x2425f): Mirror 1: 0x8e8eb66e Mirror 25: 0x8e8eb66e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2425f: skipped 000795 1775884697.708222 trunc from 0x02425f to 0x030b81 (0xc923 bytes) 000798 1775884697.742116 trunc from 0x030b82 to 0x01fc2f (0x10f52 bytes) 000800 1775884697.749816 trunc from 0x01fc30 to 0x007ecc (0x17d63 bytes) 000801 1775884697.755250 mapread 0x0061e9 thru 0x007ecc (0x1ce4 bytes) 000803 1775884697.760664 trunc from 0x007ecd to 0x017075 (0xf1a9 bytes) 000805 1775884697.767260 write 0x000a8f thru 0x00e5ab (0xdb1d bytes) 000811 1775884697.775163 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x17076): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17076): Mirror 7: 0x7e01622c (stale) Mirror 8: 0x7e01622c (stale) Mirror 10: 0x7e01622c (stale) Mirror 12: 0x7e01622c (stale) Mirror 14: 0x7e01622c (stale) Mirror 16: 0x7e01622c (stale) Mirror 18: 0x7e01622c (stale) Mirror 19: 0x7e01622c (stale) Mirror 20: 0x7e01622c (stale) Mirror 21: 0x7e01622c (stale) Mirror 22: 0x7e01622c (stale) Mirror 23: 0x7e01622c (stale) Mirror 24: 0x7e01622c (stale) Mirror 25: 0x172fc06c (stale) CRC-32 checksum value for chunk [0, 0x17076): Mirror 1: 0x40b2b1fc Mirror 26: 0x40b2b1fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17076: skipped 000814 1775884698.093616 read 0x006a70 thru 0x0151de (0xe76f bytes) 000817 1775884698.104941 write 0x00a4ac thru 0x00ac78 (0x7cd bytes) 000818 1775884698.134111 mapread 0x014edb thru 0x017075 (0x219b bytes) 000819 1775884698.141027 read 0x0025ad thru 0x008572 (0x5fc6 bytes) 000820 1775884698.144509 mapread 0x014f90 thru 0x017075 (0x20e6 bytes) 000822 1775884698.145676 mapread 0x00cf88 thru 0x016138 (0x91b1 bytes) 000824 1775884698.149676 MIRROR_RESYNC 16 mirrors 000827 1775884698.807341 write 0x022d09 thru 0x02ce50 (0xa148 bytes) 000828 1775884698.834285 mapwrite 0x024a7c thru 0x0302d0 (0xb855 bytes) 000832 1775884698.881483 mapread 0x0038a5 thru 0x00f170 (0xb8cc bytes) 000833 1775884698.886375 read 0x005b9c thru 0x00f885 (0x9cea bytes) 000834 1775884698.889066 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302d1): Mirror 7: 0x34752935 (stale) Mirror 8: 0x34752935 (stale) Mirror 10: 0x34752935 (stale) Mirror 12: 0x34752935 (stale) Mirror 14: 0x34752935 (stale) Mirror 16: 0x34752935 (stale) Mirror 19: 0x34752935 (stale) Mirror 20: 0x34752935 (stale) Mirror 21: 0x34752935 (stale) Mirror 22: 0x34752935 (stale) Mirror 23: 0x34752935 (stale) Mirror 24: 0x34752935 (stale) Mirror 25: 0x34752935 (stale) Mirror 26: 0x34752935 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302d1): Mirror 7: 0x34752935 (stale) Mirror 8: 0x34752935 (stale) Mirror 10: 0x34752935 (stale) Mirror 12: 0x34752935 (stale) Mirror 14: 0x34752935 (stale) Mirror 16: 0x34752935 (stale) Mirror 19: 0x34752935 (stale) Mirror 20: 0x34752935 (stale) Mirror 21: 0x34752935 (stale) Mirror 22: 0x34752935 (stale) Mirror 23: 0x34752935 (stale) Mirror 24: 0x34752935 (stale) Mirror 25: 0x34752935 (stale) Mirror 26: 0x34752935 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302d1: skipped 000835 1775884699.115038 MIRROR_RESYNC 15 mirrors 000836 1775884699.769603 write 0x035701 thru 0x03a580 (0x4e80 bytes) 000837 1775884699.794325 mapread 0x0137b9 thru 0x0191ab (0x59f3 bytes) 000840 1775884699.804446 mapwrite 0x0072a5 thru 0x008d13 (0x1a6f bytes) 000841 1775884699.831871 mapread 0x007c8e thru 0x01099b (0x8d0e bytes) 000842 1775884699.833058 trunc from 0x03a581 to 0x01ea27 (0x1bb59 bytes) 000844 1775884699.846054 read 0x00a777 thru 0x0176b0 (0xcf3a bytes) 000845 1775884699.846803 mapwrite 0x0129e4 thru 0x01b1af (0x87cc bytes) 000846 1775884699.867226 write 0x036be4 thru 0x03ae5c (0x4279 bytes) 000849 1775884699.869414 trunc from 0x03ae5d to 0x03cf3b (0x20df bytes) 000850 1775884699.879885 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf3c): Mirror 8: 0xd1eb21ce (stale) Mirror 10: 0xd1eb21ce (stale) Mirror 12: 0xd1eb21ce (stale) Mirror 14: 0xd1eb21ce (stale) Mirror 16: 0xd1eb21ce (stale) Mirror 19: 0xd1eb21ce (stale) Mirror 20: 0xd1eb21ce (stale) Mirror 21: 0xd1eb21ce (stale) Mirror 22: 0xd1eb21ce (stale) Mirror 23: 0xd1eb21ce (stale) Mirror 24: 0xd1eb21ce (stale) Mirror 25: 0xd1eb21ce (stale) Mirror 26: 0xd1eb21ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf3c): Mirror 8: 0xd1eb21ce (stale) Mirror 10: 0xd1eb21ce (stale) Mirror 12: 0xd1eb21ce (stale) Mirror 14: 0xd1eb21ce (stale) Mirror 16: 0xd1eb21ce (stale) Mirror 19: 0xd1eb21ce (stale) Mirror 20: 0xd1eb21ce (stale) Mirror 21: 0xd1eb21ce (stale) Mirror 22: 0xd1eb21ce (stale) Mirror 23: 0xd1eb21ce (stale) Mirror 24: 0xd1eb21ce (stale) Mirror 25: 0xd1eb21ce (stale) Mirror 26: 0xd1eb21ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf3c: skipped 000851 1775884700.147466 write 0x02404c thru 0x02772d (0x36e2 bytes) 000852 1775884700.151279 mapread 0x01b243 thru 0x023d54 (0x8b12 bytes) 000854 1775884700.156679 read 0x010e8d thru 0x011215 (0x389 bytes) 000855 1775884700.162416 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3cf3c): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf3c): Mirror 8: 0xd1eb21ce (stale) Mirror 10: 0xd1eb21ce (stale) Mirror 12: 0xd1eb21ce (stale) Mirror 14: 0xd1eb21ce (stale) Mirror 16: 0xd1eb21ce (stale) Mirror 19: 0xd1eb21ce (stale) Mirror 20: 0xd1eb21ce (stale) Mirror 21: 0xd1eb21ce (stale) Mirror 22: 0xd1eb21ce (stale) Mirror 23: 0xd1eb21ce (stale) Mirror 24: 0xd1eb21ce (stale) Mirror 25: 0xd1eb21ce (stale) Mirror 26: 0xd1eb21ce (stale) CRC-32 checksum value for chunk [0, 0x3cf3c): Mirror 1: 0x9181cd Mirror 27: 0x9181cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf3c: skipped 000857 1775884700.555640 write 0x00a3f0 thru 0x014dd8 (0xa9e9 bytes) 000858 1775884700.593770 mapread 0x0157d5 thru 0x0202a5 (0xaad1 bytes) 000861 1775884700.603395 read 0x013438 thru 0x01ed8d (0xb956 bytes) 000864 1775884700.604975 mapread 0x0302f8 thru 0x035d5f (0x5a68 bytes) 000865 1775884700.610582 trunc from 0x03cf3c to 0x004f5e (0x37fdd bytes) 000867 1775884700.630338 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4f5f): Mirror 8: 0xb55fb214 (stale) Mirror 10: 0xb55fb214 (stale) Mirror 12: 0xb55fb214 (stale) Mirror 14: 0xb55fb214 (stale) Mirror 19: 0xb55fb214 (stale) Mirror 20: 0xb55fb214 (stale) Mirror 21: 0xb55fb214 (stale) Mirror 22: 0xb55fb214 (stale) Mirror 23: 0xb55fb214 (stale) Mirror 24: 0xb55fb214 (stale) Mirror 25: 0xb55fb214 (stale) Mirror 26: 0xb55fb214 (stale) Mirror 27: 0xb55fb214 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4f5f): Mirror 8: 0xb55fb214 (stale) Mirror 10: 0xb55fb214 (stale) Mirror 12: 0xb55fb214 (stale) Mirror 14: 0xb55fb214 (stale) Mirror 19: 0xb55fb214 (stale) Mirror 20: 0xb55fb214 (stale) Mirror 21: 0xb55fb214 (stale) Mirror 22: 0xb55fb214 (stale) Mirror 23: 0xb55fb214 (stale) Mirror 24: 0xb55fb214 (stale) Mirror 25: 0xb55fb214 (stale) Mirror 26: 0xb55fb214 (stale) Mirror 27: 0xb55fb214 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4f5f: skipped 000868 1775884700.877512 mapwrite 0x031157 thru 0x03832b (0x71d5 bytes) 000869 1775884700.925634 read 0x00ffb0 thru 0x01f5bd (0xf60e bytes) 000871 1775884700.932095 read 0x008477 thru 0x01587f (0xd409 bytes) 000872 1775884700.937424 read 0x025eb1 thru 0x033825 (0xd975 bytes) 000873 1775884700.938097 mapwrite 0x01461f thru 0x01a2d2 (0x5cb4 bytes) 000875 1775884700.961652 trunc from 0x03832c to 0x000fcc (0x3735f bytes) 000876 1775884700.972717 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0xfcd): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfcd): Mirror 8: 0x49d9a708 (stale) Mirror 10: 0x49d9a708 (stale) Mirror 12: 0x49d9a708 (stale) Mirror 14: 0x49d9a708 (stale) Mirror 19: 0x49d9a708 (stale) Mirror 20: 0x49d9a708 (stale) Mirror 21: 0x49d9a708 (stale) Mirror 22: 0x49d9a708 (stale) Mirror 23: 0x49d9a708 (stale) Mirror 24: 0x49d9a708 (stale) Mirror 25: 0x49d9a708 (stale) Mirror 26: 0x49d9a708 (stale) Mirror 27: 0x49d9a708 (stale) CRC-32 checksum value for chunk [0, 0xfcd): Mirror 1: 0x49d9a708 Mirror 28: 0x49d9a708 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfcd: skipped 000878 1775884701.359446 mapread 0x0006e9 thru 0x000fcc (0x8e4 bytes) 000879 1775884701.370971 read 0x000653 thru 0x000fcc (0x97a bytes) 000884 1775884701.388524 mapwrite 0x00832f thru 0x008ac3 (0x795 bytes) 000885 1775884701.447258 trunc from 0x008ac4 to 0x0080ec (0x9d7 bytes) 000886 1775884701.455714 mapread 0x0010b5 thru 0x0080ec (0x7038 bytes) 000887 1775884701.462717 mapread 0x005031 thru 0x0080ec (0x30bc bytes) 000888 1775884701.463750 trunc from 0x0080ed to 0x01df2d (0x15e41 bytes) 000891 1775884701.472632 write 0x034270 thru 0x03dc42 (0x99d3 bytes) 000893 1775884701.476890 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3dc43): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc43): Mirror 8: 0x2ddbb846 (stale) Mirror 10: 0x2ddbb846 (stale) Mirror 12: 0x2ddbb846 (stale) Mirror 14: 0x2ddbb846 (stale) Mirror 19: 0x2ddbb846 (stale) Mirror 20: 0x2ddbb846 (stale) Mirror 21: 0x2ddbb846 (stale) Mirror 22: 0x2ddbb846 (stale) Mirror 23: 0x2ddbb846 (stale) Mirror 24: 0x2ddbb846 (stale) Mirror 25: 0x2ddbb846 (stale) Mirror 26: 0x2ddbb846 (stale) Mirror 27: 0x9a9240b7 (stale) Mirror 28: 0x97229124 (stale) CRC-32 checksum value for chunk [0, 0x3dc43): Mirror 1: 0xc958db19 Mirror 29: 0xc958db19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc43: skipped 000894 1775884701.942202 write 0x0281b2 thru 0x032531 (0xa380 bytes) 000895 1775884701.983853 read 0x0289ff thru 0x02ed11 (0x6313 bytes) 000897 1775884701.984874 read 0x01cfb8 thru 0x01d98b (0x9d4 bytes) 000899 1775884701.991557 trunc from 0x03dc43 to 0x037bc1 (0x6081 bytes) 000900 1775884701.999920 read 0x0280a4 thru 0x03637d (0xe2da bytes) 000904 1775884702.002287 mapwrite 0x02222a thru 0x030ffd (0xedd4 bytes) 000905 1775884702.028325 mapread 0x029461 thru 0x02c1ec (0x2d8c bytes) 000906 1775884702.033065 MIRROR_RESYNC 16 mirrors 000907 1775884702.845917 trunc from 0x037bc2 to 0x02afe2 (0xcbdf bytes) 000908 1775884702.886450 mapread 0x004777 thru 0x010238 (0xbac2 bytes) 000910 1775884702.896436 trunc from 0x02afe3 to 0x0021ea (0x28df8 bytes) 000911 1775884702.905548 mapread 0x001433 thru 0x0021ea (0xdb8 bytes) 000912 1775884702.906374 mapwrite 0x010cd6 thru 0x01872d (0x7a58 bytes) 000915 1775884702.946165 mapwrite 0x037e31 thru 0x03ffff (0x81cf bytes) 000918 1775884702.983824 mapread 0x016572 thru 0x01c6c3 (0x6152 bytes) 000919 1775884702.989425 trunc from 0x040000 to 0x03a658 (0x59a7 bytes) 000920 1775884702.997099 write 0x023bf9 thru 0x02c82b (0x8c33 bytes) 000921 1775884703.003226 trunc from 0x03a659 to 0x022449 (0x1820f bytes) 000922 1775884703.009859 mapread 0x01e4f5 thru 0x022449 (0x3f55 bytes) 000924 1775884703.011272 read 0x00e45e thru 0x0126fa (0x429d bytes) 000926 1775884703.012289 mapread 0x018e55 thru 0x01f0b2 (0x625e bytes) 000927 1775884703.013122 mapread 0x000a1a thru 0x003c31 (0x3218 bytes) 000928 1775884703.019053 mapread 0x012cb2 thru 0x01a3a4 (0x76f3 bytes) 000929 1775884703.022965 MIRROR_RESYNC 16 mirrors 000930 1775884703.762745 trunc from 0x02244a to 0x002606 (0x1fe43 bytes) 000931 1775884703.801665 read 0x001202 thru 0x002606 (0x1405 bytes) 000932 1775884703.804984 trunc from 0x002607 to 0x026ac5 (0x244bf bytes) 000933 1775884703.817001 mapread 0x00ac7d thru 0x00cc1a (0x1f9e bytes) 000941 1775884703.831061 mapread 0x010029 thru 0x01ba4d (0xba25 bytes) 000945 1775884703.838995 write 0x0190ea thru 0x02536b (0xc282 bytes) 000948 1775884703.845244 trunc from 0x026ac6 to 0x00bb5f (0x1af66 bytes) 000949 1775884703.855042 trunc from 0x00bb60 to 0x01d612 (0x11ab3 bytes) 000951 1775884703.863616 mapwrite 0x008456 thru 0x0090bf (0xc6a bytes) 000955 1775884703.895192 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d613): Mirror 8: 0xe2bdb366 (stale) Mirror 10: 0xe2bdb366 (stale) Mirror 12: 0xe2bdb366 (stale) Mirror 14: 0xe2bdb366 (stale) Mirror 19: 0xe2bdb366 (stale) Mirror 21: 0xe2bdb366 (stale) Mirror 22: 0xe2bdb366 (stale) Mirror 23: 0xe2bdb366 (stale) Mirror 24: 0xe2bdb366 (stale) Mirror 25: 0xe2bdb366 (stale) Mirror 26: 0xe2bdb366 (stale) Mirror 27: 0xe2bdb366 (stale) Mirror 28: 0xe2bdb366 (stale) Mirror 29: 0xe2bdb366 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d613): Mirror 8: 0xe2bdb366 (stale) Mirror 10: 0xe2bdb366 (stale) Mirror 12: 0xe2bdb366 (stale) Mirror 14: 0xe2bdb366 (stale) Mirror 19: 0xe2bdb366 (stale) Mirror 21: 0xe2bdb366 (stale) Mirror 22: 0xe2bdb366 (stale) Mirror 23: 0xe2bdb366 (stale) Mirror 24: 0xe2bdb366 (stale) Mirror 25: 0xe2bdb366 (stale) Mirror 26: 0xe2bdb366 (stale) Mirror 27: 0xe2bdb366 (stale) Mirror 28: 0xe2bdb366 (stale) Mirror 29: 0xe2bdb366 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d613: skipped 000957 1775884704.164989 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1d613): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d613): Mirror 8: 0xe2bdb366 (stale) Mirror 10: 0xe2bdb366 (stale) Mirror 12: 0xe2bdb366 (stale) Mirror 14: 0xe2bdb366 (stale) Mirror 19: 0xe2bdb366 (stale) Mirror 21: 0xe2bdb366 (stale) Mirror 22: 0xe2bdb366 (stale) Mirror 23: 0xe2bdb366 (stale) Mirror 24: 0xe2bdb366 (stale) Mirror 25: 0xe2bdb366 (stale) Mirror 26: 0xe2bdb366 (stale) Mirror 27: 0xe2bdb366 (stale) Mirror 28: 0xe2bdb366 (stale) Mirror 29: 0xe2bdb366 (stale) CRC-32 checksum value for chunk [0, 0x1d613): Mirror 1: 0xd5e2e9d2 Mirror 30: 0xd5e2e9d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d613: skipped 000958 1775884704.574600 write 0x03fefd thru 0x03ffff (0x103 bytes) 000960 1775884704.605557 mapread 0x017032 thru 0x01e26e (0x723d bytes) 000961 1775884704.613105 mapwrite 0x00423d thru 0x004d0d (0xad1 bytes) 000963 1775884704.632877 write 0x03e01b thru 0x03ffff (0x1fe5 bytes) 000965 1775884704.639874 write 0x0084fa thru 0x00d0c8 (0x4bcf bytes) 000969 1775884704.643060 mapwrite 0x03599d thru 0x03ffff (0xa663 bytes) 000970 1775884704.663043 mapread 0x01d613 thru 0x0204b3 (0x2ea1 bytes) 000971 1775884704.663965 trunc from 0x040000 to 0x01dc24 (0x223db bytes) 000972 1775884704.672398 read 0x016a4e thru 0x01dc24 (0x71d7 bytes) 000973 1775884704.673085 trunc from 0x01dc25 to 0x01089d (0xd387 bytes) 000976 1775884704.680473 read 0x00b010 thru 0x01089d (0x588e bytes) 000977 1775884704.681091 read 0x00dc99 thru 0x01089d (0x2c05 bytes) 000983 1775884704.684051 write 0x03f8b0 thru 0x03ffff (0x750 bytes) 000984 1775884704.685146 mapwrite 0x002d5b thru 0x010abd (0xdd63 bytes) 000986 1775884704.705477 read 0x037c0a thru 0x03ffff (0x83f6 bytes) 000987 1775884704.713471 MIRROR_SPLIT mirror 4 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x497:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb1013468 (stale) Mirror 10: 0xb1013468 (stale) Mirror 12: 0xb1013468 (stale) Mirror 19: 0xb1013468 (stale) Mirror 21: 0xb1013468 (stale) Mirror 22: 0xb1013468 (stale) Mirror 23: 0xb1013468 (stale) Mirror 24: 0xb1013468 (stale) Mirror 25: 0xb1013468 (stale) Mirror 26: 0xb1013468 (stale) Mirror 27: 0xb1013468 (stale) Mirror 28: 0xb1013468 (stale) Mirror 29: 0xb1013468 (stale) Mirror 30: 0x3ee25d6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb1013468 (stale) Mirror 10: 0xb1013468 (stale) Mirror 12: 0xb1013468 (stale) Mirror 19: 0xb1013468 (stale) Mirror 21: 0xb1013468 (stale) Mirror 22: 0xb1013468 (stale) Mirror 23: 0xb1013468 (stale) Mirror 24: 0xb1013468 (stale) Mirror 25: 0xb1013468 (stale) Mirror 26: 0xb1013468 (stale) Mirror 27: 0xb1013468 (stale) Mirror 28: 0xb1013468 (stale) Mirror 29: 0xb1013468 (stale) Mirror 30: 0x3ee25d6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1775884704.997398 trunc from 0x040000 to 0x008378 (0x37c87 bytes) 000990 1775884705.007267 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x8379): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8379): Mirror 8: 0x8bb81835 (stale) Mirror 10: 0x8bb81835 (stale) Mirror 12: 0x8bb81835 (stale) Mirror 19: 0x8bb81835 (stale) Mirror 21: 0x8bb81835 (stale) Mirror 22: 0x8bb81835 (stale) Mirror 23: 0x8bb81835 (stale) Mirror 24: 0x8bb81835 (stale) Mirror 25: 0x8bb81835 (stale) Mirror 26: 0x8bb81835 (stale) Mirror 27: 0x8bb81835 (stale) Mirror 28: 0x8bb81835 (stale) Mirror 29: 0x8bb81835 (stale) Mirror 30: 0x8bb81835 (stale) CRC-32 checksum value for chunk [0, 0x8379): Mirror 1: 0x90fad4f1 Mirror 31: 0x90fad4f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8379: skipped 000991 1775884705.373432 trunc from 0x008379 to 0x024ce5 (0x1c96d bytes) 000992 1775884705.410650 mapread 0x0218cf thru 0x0242b4 (0x29e6 bytes) 000993 1775884705.420344 trunc from 0x024ce6 to 0x02e8e2 (0x9bfd bytes) 000994 1775884705.429351 mapread 0x01c5ec thru 0x02ae4c (0xe861 bytes) 000995 1775884705.434475 write 0x026f75 thru 0x0297bc (0x2848 bytes) 000997 1775884705.435771 mapwrite 0x028e03 thru 0x032d60 (0x9f5e bytes) 000998 1775884705.467233 read 0x025a8d thru 0x0281d0 (0x2744 bytes) 000999 1775884705.467793 read 0x025e24 thru 0x029132 (0x330f bytes) 001000 1775884705.468425 mapwrite 0x019456 thru 0x020ba0 (0x774b bytes) All operations completed A-OK! PASS 70a (33s) == sanity-flr test 71: check mirror extend parent fid ==== 01:18:27 (1775884707) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 01:18:28 (1775884708) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0856313 s, 24.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643442 s, 16.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ae:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 01:18:31 (1775884711) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 335296 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6356352 .. Extending file size to 7996000 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8274400 .. Extending file size to 8507936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8728544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8392256 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8461536 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9177920 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8868384 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9227424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58102 58103 58105 58106 58109 failed 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:19:38 (1775884778) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7265088 .. Extending file size to 3112544 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9000832 .. failed Extending file size to 6716512 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7869888 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8744448 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9018848 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9139552 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9160608 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 61384 61385 61387 61389 61391 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:20:42 (1775884842) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3456 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3456 2204800 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0353669 s, 59.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0855277 s, 24.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcea49d3c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:20:48 (1775884848) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0261215 s, 80.3 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 01:20:50 (1775884850) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4db:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c8:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x6c3:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:20:52 (1775884852) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0446252 s, 47.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:20:54 (1775884854) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0476692 s, 44.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c7:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:20:56 (1775884856) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0365466 s, 57.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6c9:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:21:20 (1775884880) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203996 s, 51.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6ce:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:22:04 (1775884924) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0397784 s, 52.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:22:50 (1775884970) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. Creating new pool test_204f oleg329-server: Pool lustre.test_204f created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0487665 s, 43.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg329-server: Pool lustre.test_204f destroyed PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:23:47 (1775885027) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:23:49 (1775885029) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:23:52 (1775885032) oleg329-server: Pool lustre.test_206 created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg329-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:24:05 (1775885045) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259182 s, 40.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00900215 s, 116 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdf02c192 Mirror 3: 0xdf02c192 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:24:08 (1775885048) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:24:09 (1775885049) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:24:10 (1775885050) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0189168 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1560000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x11b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x11b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x11b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2020000 Read this (4 bytes): BBBB 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:24:26 (1775885066) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00344295 s, 2.4 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:24:29 (1775885069) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183935 s, 57.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x500 lmm_fid: [0x200000404:0x500:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x500 lmm_fid: [0x200000404:0x500:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6da:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835200 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-11 01:24:29.000000000 -0400 Modify: 2026-04-11 01:24:29.000000000 -0400 Change: 2026-04-11 01:24:30.000000000 -0400 Birth: 2026-04-11 01:24:30.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:24:32 (1775885072) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:24:33 (1775885073) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.160181 s, 256 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.01817 s, 225 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 11 01:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x502:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 11 01:24 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:24:36 (1775885076) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:24:38 (1775885078) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 888 sec ============ 01:24:41 (1775885081) === sanity-flr: start cleanup 01:24:42 (1775885082) === === sanity-flr: finish cleanup 01:24:43 (1775885083) ===