-----============= acceptance-small: sanity-flr ============----- Wed Mar 11 08:40:25 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg354-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:40:31 (1773232831) === oleg354-client.virtnet: executing check_config_client /mnt/lustre oleg354-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg354-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e8982ca3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e8982ca3000.idle_timeout=debug disable quota as required oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:40:43 (1773232843) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8184] == sanity-flr test 0a: lfs mirror create with -N option == 08:40:44 (1773232844) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:40:48 (1773232848) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:40:49 (1773232849) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:40:50 (1773232850) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:40:57 (1773232857) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:40:58 (1773232858) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 08:40:59 (1773232859) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e8982ca3000.stats=clear osc.lustre-OST0001-osc-ffff9e8982ca3000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:41:02 (1773232862) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:41:05 (1773232865) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:41:08 (1773232868) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:41:09 (1773232869) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 08:41:12 (1773232872) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:41:14 (1773232874) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:41:17 (1773232877) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:41:19 (1773232879) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:41:22 (1773232882) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:41:25 (1773232885) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.388422 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177768 s, 59.0 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:41:27 (1773232887) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0305084 s, 103 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:41:30 (1773232890) Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid in DISCONN state after 3 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg354-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:42:01 (1773232921) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.233798 s, 49.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:42:17 (1773232937) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg354-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:42:43 (1773232963) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.01496 s, 51.7 MB/s -rw-r--r-- 1 root root 52428800 Mar 11 08:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.336s user 0m0.000s sys 0m0.197s Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e8982ca3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.305s user 0m0.001s sys 0m0.156s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.376s user 0m0.000s sys 0m0.160s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg354-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e8982ca3000.ost_server_uuid in FULL state after 0 sec PASS 33b (29s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:43:12 (1773232992) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:43:13 (1773232993) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:43:14 (1773232994) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 08:43:16 (1773232996) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 08:43:18 (1773232998) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.161216 s, 78.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:43:22 (1773233002) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127715 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111464 s, 94.1 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:43:24 (1773233004) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:43:28 (1773233008) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000720786 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000716952 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000834941 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000865875 s, 1.2 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 08:43:31 (1773233011) create test files with size 15742757 .. aa8961c3ce18afef67b9ccdfd48960e0 - 22445dc4352e4148bf367fb7c9b8179c - fa45aa1bbb711b7d0bc17408a10784af - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg354-client.virtnet /mnt/lustre (opts:) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 08:43:37 (1773233017) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:43:41 (1773233021) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:43:43 (1773233023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385168 s, 27.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110328 s, 95.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x42:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 08:43:46 (1773233026) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0286522 s, 73.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.117312 s, 35.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0521363 s, 40.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 08:43:51 (1773233031) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 08:43:52 (1773233032) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:43:53 (1773233033) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132052 s, 79.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 08:43:56 (1773233036) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.087612 s, 35.9 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:43:59 (1773233039) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:44:02 (1773233042) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.106676 s, 98.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg354-client.virtnet /mnt/lustre (opts:) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:44:20 (1773233060) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9e898616f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0953977 s, 110 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg354-client.virtnet /mnt/lustre (opts:) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9e8983fbc800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:44:25 (1773233065) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:44:37 (1773233077) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:44:50 (1773233090) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 08:44:51 (1773233091) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0845189 s, 37.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0594724 s, 52.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x840f257b (stale) Mirror 2: 0x840f257b (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 5d80e24fc1a9dece6909a47aeecdf561 - mirror 1 checksum: 2b61a58c558316a06d8b72a894a15c8c - mirror 2 checksum: 5d80e24fc1a9dece6909a47aeecdf561 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Mar 11 08:44:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Mar 11 08:44:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5d80e24fc1a9dece6909a47aeecdf561 - mirror 1 checksum: 2b61a58c558316a06d8b72a894a15c8c - mirror 2 checksum: 2b61a58c558316a06d8b72a894a15c8c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Mar 11 08:44:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Mar 11 08:44:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:44:54 (1773233094) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014145 s, 74.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:44:56 (1773233096) Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 08:45:04 (1773233104) shut down facet: mds1 facet_host: oleg354-server facet_failover_host: oleg354-server Failover mds1 to oleg354-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 08:45:18 (1773233118) targets are mounted 08:45:18 (1773233118) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149167 s, 70.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:45:26 (1773233126) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0118716 s, 1.7 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00553688 s, 3.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:45:29 (1773233129) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0123382 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00360808 s, 5.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:45:32 (1773233132) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0056713 s, 2.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:45:35 (1773233135) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0106447 s, 1.9 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:45:37 (1773233137) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00188821 s, 2.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105322 s, 389 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:45:40 (1773233140) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:45:46 (1773233146) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:46:04 (1773233164) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:46:22 (1773233182) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:46:39 (1773233199) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 08:46:41 (1773233201) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55846 mirror split pid 55847 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55846 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55847 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 08:47:44 (1773233264) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3264 skipping zero length zero range 000002 1773233264.730587 mapwrite 0x019cca thru 0x01fd97 (0x60ce bytes) 000003 1773233264.749083 read 0x002d02 thru 0x00cd53 (0xa052 bytes) 000004 1773233264.752068 read 0x011820 thru 0x019df3 (0x85d4 bytes) 000006 1773233264.756672 MIRROR_EXTEND to 2 mirrors 000007 1773233264.911752 trunc from 0x00cb75 to 0x013672 (0x6afe bytes) truncating to largest ever: 0x3277a 000008 1773233264.920981 trunc from 0x01fd98 to 0x032779 (0x129e2 bytes) 000009 1773233264.932499 read 0x02b1c7 thru 0x032779 (0x75b3 bytes) 000010 1773233264.937258 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) fallocating to largest ever: 0x40000 000011 1773233264.975210 fallocate 0x03d732 thru 0x03ffff (0x28ce bytes) 000012 1773233264.978466 MIRROR_EXTEND to 2 mirrors 000013 1773233265.154204 trunc from 0x03121e to 0x03fbd8 (0xe9bb bytes) 000014 1773233265.164342 write 0x037a69 thru 0x03ffff (0x8597 bytes) 000015 1773233265.169247 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xeb0b3cfc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcd544a5b Mirror 3: 0xcd544a5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000016 1773233265.406971 write 0x021e1f thru 0x02bbbd (0x9d9f bytes) 000017 1773233265.420011 trunc from 0x040000 to 0x0358a9 (0xa756 bytes) 000018 1773233265.425804 trunc from 0x00ee21 to 0x010ceb (0x1ecb bytes) 000019 1773233265.427726 trunc from 0x0358aa to 0x001315 (0x34594 bytes) 000020 1773233265.432293 trunc from 0x001316 to 0x006f9c (0x5c87 bytes) 000021 1773233265.437711 trunc from 0x002167 to 0x0047bf (0x2659 bytes) 000022 1773233265.439621 mapwrite 0x027177 thru 0x029001 (0x1e8b bytes) 000023 1773233265.450768 read 0x0125c3 thru 0x021355 (0xed93 bytes) 000024 1773233265.455112 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29002): Mirror 2: 0xdbb9cce4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29002): Mirror 2: 0xdbb9cce4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29002: skipped 000025 1773233265.524573 read 0x0253c5 thru 0x029001 (0x3c3d bytes) 000026 1773233265.527167 fallocate 0x00a730 thru 0x013b6f (0x9440 bytes) 000027 1773233265.529053 read 0x005de4 thru 0x011576 (0xb793 bytes) 000028 1773233265.532184 mapread 0x0000e2 thru 0x00f875 (0xf794 bytes) 000029 1773233265.533233 read 0x001d3f thru 0x011040 (0xf302 bytes) 000031 1773233265.534023 trunc from 0x006c85 to 0x00b979 (0x4cf5 bytes) 000032 1773233265.536198 mapread 0x018b41 thru 0x0208da (0x7d9a bytes) 000033 1773233265.537290 read 0x004b65 thru 0x0116d7 (0xcb73 bytes) 000034 1773233265.542090 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000036 1773233265.579714 MIRROR_EXTEND to 2 mirrors 000037 1773233265.729033 write 0x03bec9 thru 0x03ffff (0x4137 bytes) 000038 1773233265.739914 trunc from 0x040000 to 0x0312e5 (0xed1a bytes) 000039 1773233265.748765 mapwrite 0x01de9f thru 0x029d91 (0xbef3 bytes) 000040 1773233265.764824 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x312e6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x312e6): Mirror 2: 0x238157c3 (stale) CRC-32 checksum value for chunk [0, 0x312e6): Mirror 1: 0x1c13c7bf Mirror 3: 0x1c13c7bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x312e6: skipped 000041 1773233265.982857 trunc from 0x014848 to 0x015350 (0xb09 bytes) 000042 1773233265.993353 punch from 0x025308 to 0x02da43 (0x873c bytes) 000043 1773233265.995902 trunc from 0x022ffd to 0x0312e5 (0xe2e9 bytes) 000044 1773233265.997962 fallocate 0x0159d1 thru 0x023a2c (0xe05c bytes) 000045 1773233266.000130 trunc from 0x0312e6 to 0x02dfdd (0x3308 bytes) 000046 1773233266.011820 read 0x01307e thru 0x01cd2b (0x9cae bytes) 000047 1773233266.014702 fallocate 0x03a008 thru 0x03ffff (0x5ff8 bytes) 000048 1773233266.016923 mapwrite 0x012e46 thru 0x01ef8f (0xc14a bytes) 000049 1773233266.030662 mapwrite 0x00a264 thru 0x00c79d (0x253a bytes) 000050 1773233266.039365 trunc from 0x021d01 to 0x02fc58 (0xdf58 bytes) 000051 1773233266.041560 punch from 0x03a733 to 0x03ffff (0x58cd bytes) 000052 1773233266.043304 mapwrite 0x0232c1 thru 0x03200e (0xed4e bytes) 000053 1773233266.057466 trunc from 0x040000 to 0x0193c3 (0x26c3c bytes) 000054 1773233266.062333 mapwrite 0x033ee4 thru 0x03977a (0x5897 bytes) 000055 1773233266.074416 punch from 0x00221c to 0x007f1d (0x5d02 bytes) 000056 1773233266.076225 mapread 0x00dc58 thru 0x0181ec (0xa595 bytes) 000057 1773233266.078682 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3977b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3977b): Mirror 2: 0x67a17ab0 (stale) Mirror 3: 0x767dc238 (stale) CRC-32 checksum value for chunk [0, 0x3977b): Mirror 1: 0xb4e3396b Mirror 4: 0xb4e3396b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3977b: skipped 000058 1773233266.304781 punch from 0x039217 to 0x03977a (0x564 bytes) 000059 1773233266.315784 trunc from 0x03977b to 0x015fc8 (0x237b2 bytes) 000060 1773233266.328962 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x15fc9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15fc9): Mirror 2: 0x4458a336 (stale) Mirror 3: 0x4458a336 (stale) Mirror 4: 0x2c9f7237 (stale) CRC-32 checksum value for chunk [0, 0x15fc9): Mirror 1: 0x2c9f7237 Mirror 5: 0x2c9f7237 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15fc9: skipped 000062 1773233266.549581 read 0x00b6a6 thru 0x015fc8 (0xa923 bytes) 000063 1773233266.553903 mapread 0x00cb05 thru 0x01149a (0x4996 bytes) 000064 1773233266.558082 punch from 0x00883d to 0x00cefb (0x46bf bytes) 000065 1773233266.567774 read 0x00f457 thru 0x01516b (0x5d15 bytes) 000066 1773233266.572071 MIRROR_RESYNC 5 mirrors 000067 1773233266.784103 write 0x0029a0 thru 0x00de29 (0xb48a bytes) 000068 1773233266.796308 trunc from 0x015fc9 to 0x00b6af (0xa919 bytes) 000069 1773233266.807513 trunc from 0x00b6b0 to 0x011fa8 (0x68f9 bytes) 000070 1773233266.811920 fallocate 0x02257a thru 0x029b41 (0x75c8 bytes) 000072 1773233266.815563 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29b42): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b42): Mirror 2: 0x46a7485d (stale) Mirror 3: 0x46a7485d (stale) Mirror 4: 0x46a7485d (stale) Mirror 5: 0x46a7485d (stale) CRC-32 checksum value for chunk [0, 0x29b42): Mirror 1: 0xb7033aac Mirror 6: 0xb7033aac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b42: skipped 000073 1773233267.044912 mapwrite 0x022e0f thru 0x028c5a (0x5e4c bytes) 000074 1773233267.083069 mapread 0x0018fc thru 0x0035fe (0x1d03 bytes) 000075 1773233267.086533 read 0x00503f thru 0x006114 (0x10d6 bytes) 000076 1773233267.087164 write 0x01e0a1 thru 0x027359 (0x92b9 bytes) 000078 1773233267.090288 trunc from 0x026c01 to 0x029b41 (0x2f41 bytes) 000079 1773233267.094806 fallocate 0x00931f thru 0x017513 (0xe1f5 bytes) 000080 1773233267.096553 trunc from 0x020607 to 0x02255a (0x1f54 bytes) 000082 1773233267.099629 MIRROR_RESYNC 6 mirrors 000083 1773233267.499017 read 0x001ab0 thru 0x00b5b1 (0x9b02 bytes) 000084 1773233267.502503 MIRROR_RESYNC 6 mirrors 000085 1773233267.635841 fallocate 0x013a11 thru 0x020332 (0xc922 bytes) 000086 1773233267.647110 MIRROR_RESYNC 6 mirrors 000087 1773233268.030348 mapwrite 0x005530 thru 0x00c347 (0x6e18 bytes) 000088 1773233268.065861 mapread 0x017721 thru 0x01b928 (0x4208 bytes) 000089 1773233268.069700 fallocate 0x03b5e1 thru 0x03ffff (0x4a1f bytes) 000090 1773233268.073453 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x25e5af45 (stale) Mirror 3: 0x25e5af45 (stale) Mirror 4: 0x25e5af45 (stale) Mirror 6: 0x25e5af45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x25e5af45 (stale) Mirror 3: 0x25e5af45 (stale) Mirror 4: 0x25e5af45 (stale) Mirror 6: 0x25e5af45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000091 1773233268.180028 mapwrite 0x036c8c thru 0x03ffff (0x9374 bytes) 000092 1773233268.193052 MIRROR_RESYNC 5 mirrors 000093 1773233268.586982 write 0x000766 thru 0x00b56c (0xae07 bytes) 000094 1773233268.598934 write 0x00fc52 thru 0x0127e4 (0x2b93 bytes) 000095 1773233268.603572 mapwrite 0x02c6e6 thru 0x030962 (0x427d bytes) 000096 1773233268.612830 write 0x014ad6 thru 0x016c22 (0x214d bytes) 000097 1773233268.615905 mapread 0x00e201 thru 0x0137bc (0x55bc bytes) 000098 1773233268.618294 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x832f29a1 (stale) Mirror 3: 0x832f29a1 (stale) Mirror 4: 0x832f29a1 (stale) Mirror 6: 0x832f29a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8d5ee481 Mirror 7: 0x8d5ee481 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000099 1773233268.893987 read 0x01b2a2 thru 0x01c758 (0x14b7 bytes) 000100 1773233268.898320 MIRROR_RESYNC 6 mirrors 000102 1773233269.316699 trunc from 0x039207 to 0x03cf58 (0x3d52 bytes) 000103 1773233269.328040 trunc from 0x01bbc4 to 0x027304 (0xb741 bytes) 000104 1773233269.330019 mapwrite 0x029f24 thru 0x038d08 (0xede5 bytes) 000105 1773233269.348738 trunc from 0x040000 to 0x037a5b (0x85a4 bytes) 000106 1773233269.360763 fallocate 0x021744 thru 0x0267e1 (0x509e bytes) 000107 1773233269.363110 write 0x032445 thru 0x032480 (0x3c bytes) 000108 1773233269.364136 mapwrite 0x017178 thru 0x01f258 (0x80e1 bytes) 000109 1773233269.374716 punch from 0x016c32 to 0x02360c (0xc9db bytes) 000110 1773233269.376958 punch from 0x034647 to 0x037a5b (0x3415 bytes) 000112 1773233269.379499 mapread 0x0044f7 thru 0x00ea13 (0xa51d bytes) 000113 1773233269.382919 trunc from 0x037a5c to 0x01748c (0x205cf bytes) 000114 1773233269.389189 MIRROR_RESYNC 6 mirrors 000115 1773233269.642073 write 0x020f35 thru 0x02a54c (0x9618 bytes) 000117 1773233269.652113 trunc from 0x02a54d to 0x0026ae (0x27e9e bytes) 000118 1773233269.658058 trunc from 0x0005c4 to 0x0015cb (0x1008 bytes) 000119 1773233269.662275 write 0x02edd5 thru 0x03c720 (0xd94c bytes) 000120 1773233269.663398 punch from 0x0015b7 to 0x0019a5 (0x3ef bytes) 000121 1773233269.665227 mapread 0x021412 thru 0x026708 (0x52f7 bytes) 000122 1773233269.670257 mapwrite 0x022674 thru 0x0261f3 (0x3b80 bytes) 000123 1773233269.677752 MIRROR_RESYNC 6 mirrors 000124 1773233270.001927 mapwrite 0x014234 thru 0x01d804 (0x95d1 bytes) 000125 1773233270.038136 read 0x01edb9 thru 0x02a512 (0xb75a bytes) 000126 1773233270.040621 trunc from 0x03c721 to 0x038435 (0x42eb bytes) 000127 1773233270.049569 read 0x01e8c8 thru 0x0209c8 (0x2101 bytes) 000128 1773233270.049967 mapwrite 0x02d653 thru 0x038de9 (0xb797 bytes) 000129 1773233270.062037 trunc from 0x038dea to 0x01f2e7 (0x19b02 bytes) 000130 1773233270.065987 fallocate 0x015c34 thru 0x021db7 (0xc184 bytes) 000131 1773233270.067547 write 0x0160ad thru 0x025e31 (0xfd85 bytes) 000133 1773233270.068990 read 0x022cd5 thru 0x025e31 (0x315d bytes) 000134 1773233270.069583 write 0x01e4a5 thru 0x024a17 (0x6573 bytes) 000135 1773233270.070466 trunc from 0x025e32 to 0x037553 (0x11722 bytes) 000136 1773233270.074723 mapread 0x016a91 thru 0x01b11c (0x468c bytes) 000137 1773233270.077042 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37554): Mirror 2: 0xf63f515 (stale) Mirror 4: 0xf63f515 (stale) Mirror 6: 0xf63f515 (stale) Mirror 7: 0xf63f515 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37554): Mirror 2: 0xf63f515 (stale) Mirror 4: 0xf63f515 (stale) Mirror 6: 0xf63f515 (stale) Mirror 7: 0xf63f515 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37554: skipped 000140 1773233270.201700 mapwrite 0x028db8 thru 0x030117 (0x7360 bytes) 000141 1773233270.215440 mapwrite 0x027f4b thru 0x03531d (0xd3d3 bytes) 000142 1773233270.226932 punch from 0x01f20c to 0x022aee (0x38e3 bytes) 000143 1773233270.229090 mapread 0x01eea1 thru 0x02aec9 (0xc029 bytes) 000145 1773233270.233529 mapread 0x024069 thru 0x032e06 (0xed9e bytes) 000146 1773233270.234628 punch from 0x02ad60 to 0x032f57 (0x81f8 bytes) 000147 1773233270.236761 write 0x016b9e thru 0x023c10 (0xd073 bytes) 000148 1773233270.241511 trunc from 0x015e55 to 0x0256fa (0xf8a6 bytes) 000149 1773233270.246432 trunc from 0x037554 to 0x0087d4 (0x2ed7f bytes) 000150 1773233270.251335 trunc from 0x005603 to 0x0087d4 (0x31d2 bytes) 000151 1773233270.253311 write 0x008774 thru 0x00cb58 (0x43e5 bytes) 000152 1773233270.256624 punch from 0x0073fc to 0x00cb58 (0x575d bytes) 000153 1773233270.261085 trunc from 0x001e88 to 0x0057bc (0x3935 bytes) 000154 1773233270.264957 MIRROR_RESYNC 5 mirrors 000155 1773233270.485629 read 0x005a8a thru 0x00cb58 (0x70cf bytes) 000156 1773233270.488242 read 0x00691a thru 0x00cb58 (0x623f bytes) 000157 1773233270.488588 trunc from 0x006697 to 0x007d6a (0x16d4 bytes) 000158 1773233270.498999 punch from 0x006515 to 0x00cb58 (0x6644 bytes) 000159 1773233270.500853 trunc from 0x00cb59 to 0x032b27 (0x25fcf bytes) 000160 1773233270.510170 trunc from 0x0028da to 0x00811c (0x5843 bytes) 000161 1773233270.511960 mapwrite 0x00dce0 thru 0x01774b (0x9a6c bytes) 000162 1773233270.523842 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32b28): Mirror 2: 0x4ac1c165 (stale) Mirror 4: 0x4ac1c165 (stale) Mirror 6: 0x4ac1c165 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b28): Mirror 2: 0x4ac1c165 (stale) Mirror 4: 0x4ac1c165 (stale) Mirror 6: 0x4ac1c165 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b28: skipped 000163 1773233270.610523 fallocate 0x015779 thru 0x024cbd (0xf545 bytes) 000164 1773233270.612277 trunc from 0x02efa1 to 0x032b27 (0x3b87 bytes) 000165 1773233270.613557 trunc from 0x01e76c to 0x029ae6 (0xb37b bytes) 000166 1773233270.615030 mapwrite 0x008174 thru 0x015bd1 (0xda5e bytes) 000167 1773233270.629677 MIRROR_RESYNC 4 mirrors 000168 1773233270.870090 MIRROR_RESYNC 4 mirrors 000169 1773233271.001893 trunc from 0x032b28 to 0x01d94a (0x151dd bytes) 000171 1773233271.016978 mapwrite 0x02aeda thru 0x0353c9 (0xa4f0 bytes) 000172 1773233271.036665 write 0x0061d9 thru 0x00bd50 (0x5b78 bytes) 000173 1773233271.040122 write 0x031b52 thru 0x03ffff (0xe4ae bytes) 000175 1773233271.041504 write 0x024277 thru 0x02b9e0 (0x776a bytes) 000176 1773233271.044723 read 0x013413 thru 0x01362f (0x21d bytes) 000177 1773233271.046413 mapread 0x00b11b thru 0x014bc3 (0x9aa9 bytes) 000178 1773233271.049108 fallocate 0x0314a1 thru 0x0396a9 (0x8209 bytes) 000179 1773233271.050935 fallocate 0x02eeb1 thru 0x03b3ea (0xc53a bytes) 000180 1773233271.052567 punch from 0x015e9d to 0x020b82 (0xace6 bytes) 000181 1773233271.054314 write 0x03e132 thru 0x03ffff (0x1ece bytes) 000182 1773233271.055030 read 0x006b0c thru 0x006b64 (0x59 bytes) 000183 1773233271.055563 fallocate 0x008f1c thru 0x00996b (0xa50 bytes) 000184 1773233271.057880 write 0x0226ff thru 0x0236fe (0x1000 bytes) 000185 1773233271.058677 trunc from 0x024fb7 to 0x031d8e (0xcdd8 bytes) 000186 1773233271.063481 trunc from 0x040000 to 0x020c3d (0x1f3c2 bytes) 000187 1773233271.067833 mapwrite 0x01478c thru 0x018fd5 (0x484a bytes) 000188 1773233271.077772 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20c3e): Mirror 4: 0x7dd097a8 (stale) Mirror 6: 0x7dd097a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c3e): Mirror 4: 0x7dd097a8 (stale) Mirror 6: 0x7dd097a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c3e: skipped 000189 1773233271.157840 trunc from 0x020c3e to 0x03b1af (0x1a572 bytes) 000190 1773233271.162881 read 0x000aa7 thru 0x00660e (0x5b68 bytes) 000191 1773233271.165724 trunc from 0x03b1b0 to 0x031e2e (0x9381 bytes) 000192 1773233271.170067 read 0x02f2c2 thru 0x031e2e (0x2b6d bytes) 000193 1773233271.173457 trunc from 0x031e2f to 0x008a8c (0x293a2 bytes) 000195 1773233271.178826 write 0x03d5fd thru 0x03f373 (0x1d77 bytes) 000196 1773233271.179539 read 0x01c41b thru 0x024f52 (0x8b38 bytes) 000197 1773233271.182870 trunc from 0x018ef4 to 0x0244ef (0xb5fc bytes) 000198 1773233271.184654 mapwrite 0x026b78 thru 0x02856a (0x19f3 bytes) 000199 1773233271.190128 mapread 0x01cff3 thru 0x02c11b (0xf129 bytes) 000200 1773233271.200773 trunc from 0x029805 to 0x0300fb (0x68f7 bytes) 000201 1773233271.202470 fallocate 0x006e75 thru 0x007ac5 (0xc51 bytes) 000202 1773233271.204336 read 0x038a34 thru 0x03f373 (0x6940 bytes) 000203 1773233271.204805 trunc from 0x01ef57 to 0x02aea4 (0xbf4e bytes) 000204 1773233271.206609 write 0x00ef85 thru 0x018851 (0x98cd bytes) 000205 1773233271.209023 read 0x003d83 thru 0x013082 (0xf300 bytes) 000206 1773233271.209639 read 0x015355 thru 0x01f52c (0xa1d8 bytes) 000207 1773233271.212951 mapwrite 0x03fd3f thru 0x03ffff (0x2c1 bytes) 000208 1773233271.222451 mapread 0x0167d0 thru 0x024230 (0xda61 bytes) 000209 1773233271.223431 fallocate 0x03051e thru 0x0374bc (0x6f9f bytes) 000210 1773233271.225570 write 0x01b773 thru 0x022430 (0x6cbe bytes) 000211 1773233271.226392 mapread 0x038cf4 thru 0x03c11c (0x3429 bytes) 000212 1773233271.227329 mapread 0x008e14 thru 0x00da5b (0x4c48 bytes) 000213 1773233271.228055 read 0x00000a thru 0x00517b (0x5172 bytes) 000214 1773233271.230149 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5d52acfd (stale) Mirror 6: 0x5d52acfd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf545ea8d Mirror 7: 0xf545ea8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000215 1773233271.485458 MIRROR_RESYNC 4 mirrors 000216 1773233271.728023 read 0x009b61 thru 0x013e71 (0xa311 bytes) 000217 1773233271.730574 read 0x0089c9 thru 0x011c50 (0x9288 bytes) 000218 1773233271.733579 mapwrite 0x0334ca thru 0x0341c0 (0xcf7 bytes) 000220 1773233271.761310 write 0x030d82 thru 0x031dfd (0x107c bytes) 000221 1773233271.764444 trunc from 0x01b2c1 to 0x0210c3 (0x5e03 bytes) 000222 1773233271.765979 fallocate 0x025e82 thru 0x02eab2 (0x8c31 bytes) 000223 1773233271.767529 mapread 0x011fa1 thru 0x01cab6 (0xab16 bytes) 000224 1773233271.780443 trunc from 0x040000 to 0x0356c2 (0xa93d bytes) 000225 1773233271.789444 trunc from 0x0356c3 to 0x02e424 (0x729e bytes) 000226 1773233271.793328 trunc from 0x02e425 to 0x031246 (0x2e22 bytes) 000227 1773233271.796900 punch from 0x01fb81 to 0x028e08 (0x9288 bytes) 000228 1773233271.798289 mapread 0x02f7ad thru 0x031095 (0x18e9 bytes) 000229 1773233271.802492 mapwrite 0x02008d thru 0x02baec (0xba60 bytes) 000230 1773233271.822533 trunc from 0x031247 to 0x03ed24 (0xdade bytes) 000231 1773233271.826842 read 0x01b240 thru 0x026b7a (0xb93b bytes) 000232 1773233271.830151 trunc from 0x026ba4 to 0x033048 (0xc4a5 bytes) 000233 1773233271.833453 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed25): Mirror 6: 0xae67adcc (stale) Mirror 7: 0xae67adcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed25): Mirror 6: 0xae67adcc (stale) Mirror 7: 0xae67adcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed25: skipped 000234 1773233271.909643 fallocate 0x023504 thru 0x025632 (0x212f bytes) 000235 1773233271.911489 write 0x00c6a2 thru 0x018405 (0xbd64 bytes) 000236 1773233271.916312 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed25): Mirror 6: 0xae67adcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed25): Mirror 6: 0xae67adcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed25: skipped 000237 1773233271.988254 mapread 0x018b69 thru 0x027215 (0xe6ad bytes) 000238 1773233271.995749 fallocate 0x029e26 thru 0x02a51c (0x6f7 bytes) 000239 1773233271.997845 write 0x008d18 thru 0x00b833 (0x2b1c bytes) 000240 1773233272.004021 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000242 1773233272.041855 write 0x0388a3 thru 0x03ffff (0x775d bytes) 000243 1773233272.045000 trunc from 0x040000 to 0x01def9 (0x22106 bytes) 000244 1773233272.049055 fallocate 0x011d0a thru 0x018763 (0x6a5a bytes) 000245 1773233272.051015 fallocate 0x03f1dc thru 0x03ffff (0xe24 bytes) 000246 1773233272.052699 mapread 0x0003c8 thru 0x003acc (0x3705 bytes) 000247 1773233272.056514 write 0x0231f4 thru 0x03315b (0xff68 bytes) 000248 1773233272.057561 trunc from 0x02b613 to 0x031df4 (0x67e2 bytes) 000249 1773233272.061327 fallocate 0x03ea9e thru 0x03ffff (0x1562 bytes) 000250 1773233272.063029 fallocate 0x02df92 thru 0x037882 (0x98f1 bytes) 000251 1773233272.064639 trunc from 0x02a454 to 0x0326f0 (0x829d bytes) 000252 1773233272.066080 punch from 0x0204ce to 0x02ce9d (0xc9d0 bytes) 000253 1773233272.067653 trunc from 0x040000 to 0x0322ec (0xdd13 bytes) 000254 1773233272.071686 trunc from 0x0322ed to 0x01e315 (0x13fd7 bytes) 000255 1773233272.075153 trunc from 0x01e316 to 0x01e439 (0x124 bytes) 000257 1773233272.079464 fallocate 0x03fb97 thru 0x03ffff (0x469 bytes) 000258 1773233272.081165 trunc from 0x01e43a to 0x00d696 (0x10da3 bytes) 000259 1773233272.085090 fallocate 0x032884 thru 0x035e58 (0x35d5 bytes) 000261 1773233272.087402 trunc from 0x00d697 to 0x005714 (0x7f82 bytes) 000262 1773233272.091427 trunc from 0x0019b2 to 0x003529 (0x1b78 bytes) 000263 1773233272.092865 fallocate 0x00f944 thru 0x01bd38 (0xc3f5 bytes) 000264 1773233272.094539 mapread 0x015e16 thru 0x019b2f (0x3d1a bytes) 000267 1773233272.098533 mapread 0x014984 thru 0x01bd38 (0x73b5 bytes) 000268 1773233272.099146 trunc from 0x01bd39 to 0x02274e (0x6a16 bytes) 000269 1773233272.103215 write 0x016a49 thru 0x025300 (0xe8b8 bytes) 000270 1773233272.104327 mapread 0x009e8f thru 0x00f10e (0x5280 bytes) 000271 1773233272.107457 trunc from 0x0030cf to 0x00a549 (0x747b bytes) 000272 1773233272.109103 punch from 0x013c11 to 0x018027 (0x4417 bytes) 000273 1773233272.112668 trunc from 0x0129f2 to 0x01f6fd (0xcd0c bytes) 000274 1773233272.114166 mapwrite 0x01aebc thru 0x0238f9 (0x8a3e bytes) 000276 1773233272.123001 write 0x014a33 thru 0x022367 (0xd935 bytes) 000277 1773233272.124360 trunc from 0x006d5f to 0x0070ed (0x38f bytes) 000279 1773233272.126623 read 0x008c9a thru 0x01199a (0x8d01 bytes) 000280 1773233272.129670 MIRROR_EXTEND to 2 mirrors 000282 1773233272.285688 punch from 0x021494 to 0x025300 (0x3e6d bytes) 000284 1773233272.297094 trunc from 0x0207d7 to 0x0237b7 (0x2fe1 bytes) 000285 1773233272.298885 punch from 0x01ad70 to 0x01dcff (0x2f90 bytes) 000286 1773233272.300410 mapread 0x01ca23 thru 0x02002d (0x360b bytes) 000287 1773233272.306254 punch from 0x008149 to 0x008fb7 (0xe6f bytes) 000288 1773233272.308182 mapread 0x00a100 thru 0x00c988 (0x2889 bytes) 000289 1773233272.311313 mapwrite 0x0368b5 thru 0x03ffff (0x974b bytes) 000290 1773233272.331971 fallocate 0x029168 thru 0x034cc1 (0xbb5a bytes) 000291 1773233272.333784 trunc from 0x040000 to 0x0331e4 (0xce1b bytes) 000292 1773233272.338938 MIRROR_RESYNC 2 mirrors 000293 1773233272.539745 MIRROR_RESYNC 2 mirrors 000294 1773233272.657084 write 0x0046de thru 0x005bb9 (0x14dc bytes) 000295 1773233272.668468 write 0x017287 thru 0x01b306 (0x4080 bytes) 000296 1773233272.671912 mapwrite 0x0131ba thru 0x013aca (0x911 bytes) 000297 1773233272.677404 mapwrite 0x0056b1 thru 0x00b851 (0x61a1 bytes) 000298 1773233272.685431 mapwrite 0x02f3c8 thru 0x03d165 (0xdd9e bytes) 000299 1773233272.707407 punch from 0x00bf6d to 0x01459a (0x862e bytes) 000300 1773233272.709664 trunc from 0x03d166 to 0x00666b (0x36afa bytes) 000301 1773233272.714382 trunc from 0x00666c to 0x00e6d6 (0x806b bytes) 000302 1773233272.718572 trunc from 0x00e6d7 to 0x03c1b7 (0x2dae1 bytes) 000303 1773233272.722639 mapwrite 0x01512d thru 0x015ddc (0xcb0 bytes) 000304 1773233272.729074 mapread 0x030579 thru 0x03b9c9 (0xb451 bytes) 000305 1773233272.733385 trunc from 0x03c1b8 to 0x0096d5 (0x32ae2 bytes) 000306 1773233272.737893 write 0x007e88 thru 0x00f621 (0x779a bytes) 000307 1773233272.740033 write 0x00864c thru 0x0123d2 (0x9d87 bytes) 000308 1773233272.741041 trunc from 0x00091f to 0x002044 (0x1726 bytes) 000309 1773233272.742932 trunc from 0x008047 to 0x00eb17 (0x6ad1 bytes) 000310 1773233272.746811 trunc from 0x0123d3 to 0x007e39 (0xa599 bytes) 000311 1773233272.752140 MIRROR_RESYNC 2 mirrors 000312 1773233272.905975 MIRROR_RESYNC 2 mirrors 000313 1773233273.011239 punch from 0x001b29 to 0x007e39 (0x6311 bytes) 000314 1773233273.022038 trunc from 0x0060b7 to 0x007e39 (0x1d83 bytes) 000315 1773233273.023970 trunc from 0x001b3e to 0x005dd3 (0x4296 bytes) 000316 1773233273.025631 mapread 0x001360 thru 0x0014bc (0x15d bytes) 000317 1773233273.027891 trunc from 0x007e3a to 0x03e2ea (0x364b1 bytes) 000318 1773233273.037534 mapwrite 0x014a99 thru 0x021266 (0xc7ce bytes) 000319 1773233273.051969 fallocate 0x001c02 thru 0x00c1ff (0xa5fe bytes) 000320 1773233273.054167 read 0x03679b thru 0x03e2ea (0x7b50 bytes) 000321 1773233273.057387 write 0x019578 thru 0x024389 (0xae12 bytes) 000322 1773233273.058359 mapwrite 0x008a9c thru 0x009cd4 (0x1239 bytes) 000323 1773233273.067036 mapread 0x006580 thru 0x00ade5 (0x4866 bytes) 000324 1773233273.068143 mapwrite 0x0115af thru 0x01a452 (0x8ea4 bytes) 000325 1773233273.077576 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000326 1773233273.116524 fallocate 0x03e59c thru 0x03ffff (0x1a64 bytes) 000329 1773233273.120312 mapwrite 0x016d73 thru 0x01ab33 (0x3dc1 bytes) 000330 1773233273.132578 write 0x03c253 thru 0x03ffff (0x3dad bytes) 000331 1773233273.137132 read 0x02a1a8 thru 0x02d792 (0x35eb bytes) 000332 1773233273.139296 mapread 0x031baf thru 0x039660 (0x7ab2 bytes) 000333 1773233273.142795 trunc from 0x0372b9 to 0x038e2c (0x1b74 bytes) 000334 1773233273.146484 MIRROR_EXTEND to 2 mirrors 000336 1773233273.325938 fallocate 0x01baa6 thru 0x024ee6 (0x9441 bytes) 000337 1773233273.334936 trunc from 0x036890 to 0x0378f6 (0x1067 bytes) 000338 1773233273.336496 read 0x032f4c thru 0x03ffff (0xd0b4 bytes) 000340 1773233273.339521 trunc from 0x040000 to 0x03a428 (0x5bd7 bytes) 000341 1773233273.349776 write 0x0377de thru 0x038d16 (0x1539 bytes) 000342 1773233273.350576 fallocate 0x00e3ea thru 0x00fe12 (0x1a29 bytes) 000343 1773233273.352278 mapread 0x006bf7 thru 0x01329b (0xc6a5 bytes) 000344 1773233273.360208 fallocate 0x03c7fe thru 0x03ffff (0x3802 bytes) 000345 1773233273.362252 trunc from 0x03a429 to 0x00ab9f (0x2f889 bytes) 000346 1773233273.366510 trunc from 0x003d29 to 0x00ab9f (0x6e77 bytes) 000347 1773233273.368221 mapread 0x0017fd thru 0x00ab9f (0x93a3 bytes) 000348 1773233273.371629 mapread 0x000c49 thru 0x008da3 (0x815b bytes) 000349 1773233273.373906 trunc from 0x00aba0 to 0x015e1a (0xb27b bytes) 000350 1773233273.378119 trunc from 0x015e1b to 0x01d7d2 (0x79b8 bytes) 000352 1773233273.382468 mapwrite 0x02719c thru 0x02a7d5 (0x363a bytes) 000353 1773233273.393529 fallocate 0x013dff thru 0x01a4f1 (0x66f3 bytes) 000354 1773233273.395408 read 0x01ec6e thru 0x025256 (0x65e9 bytes) 000355 1773233273.398274 trunc from 0x013670 to 0x0194ad (0x5e3e bytes) 000356 1773233273.400061 fallocate 0x02ab55 thru 0x034025 (0x94d1 bytes) 000357 1773233273.402020 fallocate 0x02197f thru 0x02f031 (0xd6b3 bytes) 000358 1773233273.403864 mapread 0x002b21 thru 0x00a61b (0x7afb bytes) 000359 1773233273.404649 fallocate 0x0290c5 thru 0x02e246 (0x5182 bytes) 000360 1773233273.406427 mapread 0x01ed1e thru 0x01f957 (0xc3a bytes) 000361 1773233273.407258 trunc from 0x034026 to 0x019fd3 (0x1a052 bytes) 000362 1773233273.411366 punch from 0x00711a to 0x013108 (0xbfef bytes) 000363 1773233273.412953 mapwrite 0x031183 thru 0x0362b6 (0x5134 bytes) 000364 1773233273.424106 trunc from 0x0362b7 to 0x009bc7 (0x2c6ef bytes) 000365 1773233273.428375 mapwrite 0x03ce90 thru 0x03ffff (0x3170 bytes) 000366 1773233273.438161 read 0x01b67e thru 0x01fc74 (0x45f7 bytes) 000367 1773233273.441363 fallocate 0x03d25c thru 0x03ffff (0x2da4 bytes) 000368 1773233273.443421 mapread 0x020b92 thru 0x026e29 (0x6298 bytes) 000369 1773233273.444195 mapread 0x026a6a thru 0x031f26 (0xb4bd bytes) 000370 1773233273.445194 trunc from 0x040000 to 0x028c5f (0x173a0 bytes) 000371 1773233273.451178 MIRROR_RESYNC 2 mirrors 000372 1773233273.602306 read 0x01da5b thru 0x0248d4 (0x6e7a bytes) 000373 1773233273.604805 mapread 0x027062 thru 0x028c5f (0x1bfe bytes) 000375 1773233273.609683 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000377 1773233273.643711 trunc from 0x0087ea to 0x00dbfa (0x5411 bytes) 000378 1773233273.647461 fallocate 0x02e5a3 thru 0x039f8b (0xb9e9 bytes) 000379 1773233273.649263 fallocate 0x016e95 thru 0x019dd7 (0x2f43 bytes) 000380 1773233273.651196 write 0x017eb7 thru 0x0197c6 (0x1910 bytes) 000381 1773233273.655484 trunc from 0x039f8c to 0x0295ba (0x109d1 bytes) 000382 1773233273.664186 trunc from 0x016417 to 0x0205ba (0xa1a4 bytes) 000383 1773233273.667438 mapread 0x025164 thru 0x0295ba (0x4457 bytes) 000384 1773233273.670203 read 0x0284a7 thru 0x0295ba (0x1114 bytes) 000385 1773233273.670749 punch from 0x00152a to 0x010dbb (0xf892 bytes) 000387 1773233273.673134 punch from 0x01f47e to 0x0295ba (0xa13d bytes) 000388 1773233273.674830 read 0x025ea1 thru 0x0295ba (0x371a bytes) 000389 1773233273.678281 read 0x01f417 thru 0x0295ba (0xa1a4 bytes) 000391 1773233273.679485 trunc from 0x0295bb to 0x025050 (0x456a bytes) 000392 1773233273.684379 trunc from 0x025051 to 0x00f65a (0x159f6 bytes) 000393 1773233273.688718 trunc from 0x00f65b to 0x02375d (0x14103 bytes) 000394 1773233273.692969 write 0x03fc17 thru 0x03ffff (0x3e9 bytes) 000395 1773233273.693699 punch from 0x02e849 to 0x035491 (0x6c49 bytes) 000397 1773233273.696086 trunc from 0x040000 to 0x02384f (0x1c7b0 bytes) 000398 1773233273.700282 trunc from 0x00bc14 to 0x00cffd (0x13ea bytes) 000399 1773233273.701955 trunc from 0x01808a to 0x0234d2 (0xb449 bytes) 000402 1773233273.704329 write 0x01d30c thru 0x02cead (0xfba2 bytes) 000403 1773233273.707919 punch from 0x026b11 to 0x02cead (0x639d bytes) 000404 1773233273.711990 mapwrite 0x035793 thru 0x03aaa5 (0x5313 bytes) 000405 1773233273.724372 mapwrite 0x018d29 thru 0x01f998 (0x6c70 bytes) 000407 1773233273.730973 trunc from 0x01f8d5 to 0x023fea (0x4716 bytes) 000408 1773233273.732950 read 0x01f5ac thru 0x029285 (0x9cda bytes) 000409 1773233273.735931 read 0x0368b6 thru 0x03aaa5 (0x41f0 bytes) 000411 1773233273.737102 mapread 0x0176ce thru 0x0231db (0xbb0e bytes) 000412 1773233273.738239 mapread 0x02e785 thru 0x03a726 (0xbfa2 bytes) 000413 1773233273.739329 trunc from 0x021484 to 0x030a9e (0xf61b bytes) 000414 1773233273.741587 punch from 0x021b8a to 0x02b2e5 (0x975c bytes) 000415 1773233273.743270 trunc from 0x03aaa6 to 0x00966e (0x31437 bytes) 000416 1773233273.748031 read 0x0086ed thru 0x00966e (0xf82 bytes) 000417 1773233273.748536 punch from 0x00812e to 0x00966e (0x1541 bytes) 000418 1773233273.750298 fallocate 0x0330df thru 0x039c43 (0x6b65 bytes) 000419 1773233273.752194 write 0x003524 thru 0x0117bb (0xe298 bytes) 000420 1773233273.755085 trunc from 0x00e73e to 0x0114fd (0x2dc0 bytes) 000421 1773233273.759058 trunc from 0x039c44 to 0x006101 (0x33b42 bytes) 000422 1773233273.763584 write 0x02bf4e thru 0x039a90 (0xdb43 bytes) 000423 1773233273.764942 trunc from 0x039a91 to 0x011b44 (0x27f4c bytes) 000424 1773233273.769141 punch from 0x00c532 to 0x0100bf (0x3b8e bytes) 000425 1773233273.770648 mapwrite 0x02eda8 thru 0x03d759 (0xe9b2 bytes) 000426 1773233273.784615 write 0x03c986 thru 0x03eac5 (0x2140 bytes) 000427 1773233273.785353 mapread 0x029f5c thru 0x039da4 (0xfe49 bytes) 000429 1773233273.786704 write 0x00ab5c thru 0x01785d (0xcd02 bytes) 000430 1773233273.787991 trunc from 0x014dab to 0x01b566 (0x67bc bytes) 000431 1773233273.792161 trunc from 0x02a2c8 to 0x02e147 (0x3e80 bytes) 000432 1773233273.793642 fallocate 0x024325 thru 0x02778b (0x3467 bytes) 000434 1773233273.795667 mapwrite 0x0239b4 thru 0x025362 (0x19af bytes) 000436 1773233273.800551 mapwrite 0x025cef thru 0x02e66e (0x8980 bytes) 000438 1773233273.815069 write 0x017f13 thru 0x0246ff (0xc7ed bytes) 000439 1773233273.817484 trunc from 0x01c2b3 to 0x0263a4 (0xa0f2 bytes) 000440 1773233273.822061 punch from 0x00f8e6 to 0x014e89 (0x55a4 bytes) 000441 1773233273.823748 mapwrite 0x020538 thru 0x02c004 (0xbacd bytes) 000442 1773233273.839761 trunc from 0x03686d to 0x03e29f (0x7a33 bytes) 000443 1773233273.841718 trunc from 0x02d09b to 0x039718 (0xc67e bytes) 000445 1773233273.844120 fallocate 0x018995 thru 0x022bd3 (0xa23f bytes) 000446 1773233273.846288 trunc from 0x009734 to 0x00a462 (0xd2f bytes) 000448 1773233273.848769 read 0x01be40 thru 0x020f5e (0x511f bytes) 000449 1773233273.852347 trunc from 0x01f7a7 to 0x0205c3 (0xe1d bytes) 000450 1773233273.854254 trunc from 0x03eac6 to 0x03d3b9 (0x170c bytes) 000451 1773233273.858855 trunc from 0x00c76f to 0x01326d (0x6aff bytes) 000452 1773233273.861119 write 0x03e8aa thru 0x03ffff (0x1756 bytes) 000453 1773233273.862085 punch from 0x013fdc to 0x0210ac (0xd0d1 bytes) 000455 1773233273.864510 write 0x021d8d thru 0x0237a6 (0x1a1a bytes) 000456 1773233273.867049 mapwrite 0x02e1da thru 0x02ef33 (0xd5a bytes) 000457 1773233273.872618 fallocate 0x01c93a thru 0x02914c (0xc813 bytes) 000459 1773233273.876367 MIRROR_EXTEND to 2 mirrors 000460 1773233274.063484 MIRROR_EXTEND to 3 mirrors 000461 1773233274.259811 punch from 0x010e23 to 0x01a3ce (0x95ac bytes) 000462 1773233274.269190 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x714a9795 (stale) Mirror 3: 0x714a9795 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x714a9795 Mirror 4: 0x714a9795 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000463 1773233274.540165 read 0x02d6c1 thru 0x033fe1 (0x6921 bytes) 000464 1773233274.544845 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x714a9795 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x714a9795 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x714a9795 Mirror 4: 0x714a9795 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000465 1773233274.718414 trunc from 0x0303aa to 0x03ec94 (0xe8eb bytes) 000466 1773233274.728215 read 0x000063 thru 0x00d47b (0xd419 bytes) 000467 1773233274.730835 trunc from 0x040000 to 0x009160 (0x36e9f bytes) 000468 1773233274.736886 read 0x0025f7 thru 0x0078dd (0x52e7 bytes) 000469 1773233274.737508 mapread 0x002deb thru 0x0050d4 (0x22ea bytes) 000470 1773233274.738257 read 0x0080a2 thru 0x009160 (0x10bf bytes) 000471 1773233274.738851 mapread 0x002fd7 thru 0x009160 (0x618a bytes) 000472 1773233274.739679 mapread 0x00480f thru 0x007df2 (0x35e4 bytes) 000473 1773233274.740511 trunc from 0x009161 to 0x0118d8 (0x8778 bytes) 000475 1773233274.745505 mapread 0x004536 thru 0x0118d8 (0xd3a3 bytes) 000476 1773233274.748744 mapwrite 0x009682 thru 0x017ee7 (0xe866 bytes) 000477 1773233274.762685 fallocate 0x038c1c thru 0x0394c3 (0x8a8 bytes) 000478 1773233274.764571 punch from 0x010945 to 0x017ee7 (0x75a3 bytes) 000479 1773233274.766213 fallocate 0x02a7a3 thru 0x02b2c6 (0xb24 bytes) 000480 1773233274.768078 trunc from 0x017ee8 to 0x0365ea (0x1e703 bytes) 000481 1773233274.773047 mapread 0x00b12e thru 0x00c7ee (0x16c1 bytes) 000482 1773233274.773703 mapwrite 0x03e042 thru 0x03ffff (0x1fbe bytes) 000483 1773233274.787115 trunc from 0x02efda to 0x030f53 (0x1f7a bytes) 000484 1773233274.789043 mapwrite 0x02990b thru 0x035511 (0xbc07 bytes) 000485 1773233274.801041 mapwrite 0x02215c thru 0x03209a (0xff3f bytes) 000486 1773233274.809855 trunc from 0x01cda5 to 0x0255b0 (0x880c bytes) 000487 1773233274.811572 trunc from 0x02804c to 0x03337c (0xb331 bytes) 000488 1773233274.814716 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x714a9795 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x714a9795 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000489 1773233274.889486 read 0x02f260 thru 0x0386dc (0x947d bytes) 000490 1773233274.893820 MIRROR_RESYNC 2 mirrors 000491 1773233275.126469 trunc from 0x030b81 to 0x03d91f (0xcd9f bytes) 000492 1773233275.135419 trunc from 0x040000 to 0x0398b8 (0x6747 bytes) 000493 1773233275.140508 trunc from 0x0398b9 to 0x036437 (0x3481 bytes) 000494 1773233275.144567 trunc from 0x020a33 to 0x02f46c (0xea3a bytes) 000495 1773233275.146190 trunc from 0x02e8f9 to 0x036437 (0x7b3f bytes) 000496 1773233275.147796 mapwrite 0x031452 thru 0x033607 (0x21b6 bytes) 000497 1773233275.156800 read 0x016bf9 thru 0x01d74b (0x6b53 bytes) 000498 1773233275.159726 trunc from 0x01812f to 0x025a13 (0xd8e5 bytes) 000499 1773233275.162735 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000500 1773233275.196981 mapwrite 0x01eed5 thru 0x01f2c1 (0x3ed bytes) 000501 1773233275.203661 trunc from 0x012b24 to 0x014c86 (0x2163 bytes) 000502 1773233275.205199 write 0x03b4d3 thru 0x03ffff (0x4b2d bytes) 000503 1773233275.207050 fallocate 0x019e44 thru 0x029098 (0xf255 bytes) 000504 1773233275.208839 mapread 0x023595 thru 0x026da9 (0x3815 bytes) 000505 1773233275.211874 mapread 0x03a77d thru 0x03a9ad (0x231 bytes) 000506 1773233275.214655 fallocate 0x02a0f9 thru 0x031b51 (0x7a59 bytes) 000507 1773233275.216440 punch from 0x037ca9 to 0x03ffff (0x8357 bytes) 000508 1773233275.220179 read 0x0007e9 thru 0x00e095 (0xd8ad bytes) 000509 1773233275.222796 trunc from 0x0317e8 to 0x033294 (0x1aad bytes) 000511 1773233275.224495 punch from 0x02c3a2 to 0x036f3a (0xab99 bytes) 000512 1773233275.226184 mapread 0x0090d5 thru 0x0121c2 (0x90ee bytes) 000513 1773233275.229187 read 0x038e4e thru 0x03ffff (0x71b2 bytes) 000515 1773233275.233256 trunc from 0x040000 to 0x00227c (0x3dd83 bytes) 000516 1773233275.238126 write 0x00ded6 thru 0x01d320 (0xf44b bytes) 000517 1773233275.239297 read 0x00423b thru 0x00f1e2 (0xafa8 bytes) 000518 1773233275.243108 MIRROR_EXTEND to 2 mirrors 000519 1773233275.400805 punch from 0x00a06e to 0x01838c (0xe31f bytes) 000521 1773233275.410975 trunc from 0x007cf6 to 0x00e8d0 (0x6bdb bytes) 000523 1773233275.413231 mapread 0x0107c8 thru 0x014724 (0x3f5d bytes) 000526 1773233275.419461 write 0x02776f thru 0x034d79 (0xd60b bytes) 000527 1773233275.423187 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34d7a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d7a): Mirror 2: 0x3d5650b5 (stale) CRC-32 checksum value for chunk [0, 0x34d7a): Mirror 1: 0xc555320 Mirror 3: 0xc555320 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d7a: skipped 000528 1773233275.643264 trunc from 0x034d7a to 0x02d551 (0x7828 bytes) 000529 1773233275.659003 mapwrite 0x03ddec thru 0x03ffff (0x2214 bytes) 000531 1773233275.673342 trunc from 0x006bb2 to 0x009a05 (0x2e54 bytes) 000532 1773233275.674974 punch from 0x01e97d to 0x01f5cf (0xc53 bytes) 000533 1773233275.677572 mapread 0x03ba50 thru 0x03ffff (0x45b0 bytes) 000534 1773233275.678573 read 0x01f9c6 thru 0x02277d (0x2db8 bytes) 000535 1773233275.680784 mapwrite 0x01c72a thru 0x0211f4 (0x4acb bytes) 000536 1773233275.688099 mapwrite 0x02ebed thru 0x03e263 (0xf677 bytes) 000537 1773233275.700108 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xab390736 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xab390736 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000538 1773233275.764660 trunc from 0x01d6e0 to 0x029b08 (0xc429 bytes) 000539 1773233275.766504 trunc from 0x040000 to 0x02513b (0x1aec4 bytes) 000540 1773233275.771563 trunc from 0x01af3b to 0x022779 (0x783f bytes) 000541 1773233275.773142 read 0x0203c0 thru 0x02513b (0x4d7c bytes) 000542 1773233275.775691 punch from 0x01fac8 to 0x02513b (0x5674 bytes) 000543 1773233275.777234 write 0x03e3d7 thru 0x03ffff (0x1c29 bytes) 000544 1773233275.777981 trunc from 0x02c4ae to 0x0388d2 (0xc425 bytes) 000545 1773233275.779374 fallocate 0x013955 thru 0x01c94a (0x8ff6 bytes) 000546 1773233275.782182 MIRROR_RESYNC 2 mirrors 000547 1773233275.943393 read 0x0128ef thru 0x01c0c2 (0x97d4 bytes) 000549 1773233275.946671 mapread 0x03f589 thru 0x03ffff (0xa77 bytes) 000550 1773233275.948704 write 0x030236 thru 0x03c834 (0xc5ff bytes) 000551 1773233275.959557 mapread 0x014982 thru 0x020aff (0xc17e bytes) 000552 1773233275.964790 write 0x000303 thru 0x0093fb (0x90f9 bytes) 000553 1773233275.967623 mapread 0x02d643 thru 0x03615c (0x8b1a bytes) 000554 1773233275.970250 write 0x001a10 thru 0x00d20a (0xb7fb bytes) 000555 1773233275.971117 fallocate 0x032fee thru 0x03ffff (0xd012 bytes) 000556 1773233275.972903 trunc from 0x040000 to 0x0124f4 (0x2db0b bytes) 000557 1773233275.983234 mapwrite 0x03e169 thru 0x03ffff (0x1e97 bytes) 000560 1773233275.995553 punch from 0x032744 to 0x03289f (0x15c bytes) 000561 1773233275.997249 write 0x020245 thru 0x02dc76 (0xda32 bytes) 000562 1773233276.001426 trunc from 0x040000 to 0x01dea3 (0x2215c bytes) 000563 1773233276.005690 trunc from 0x00f6a0 to 0x01dc59 (0xe5ba bytes) 000564 1773233276.007167 write 0x0128ca thru 0x014ce7 (0x241e bytes) 000565 1773233276.009734 mapwrite 0x0205f2 thru 0x020866 (0x275 bytes) 000566 1773233276.020229 write 0x01d13f thru 0x01dc30 (0xaf2 bytes) 000567 1773233276.020879 trunc from 0x020867 to 0x02f1db (0xe975 bytes) 000569 1773233276.024755 punch from 0x0108b0 to 0x014203 (0x3954 bytes) 000570 1773233276.026574 write 0x026746 thru 0x02cafa (0x63b5 bytes) 000571 1773233276.030580 MIRROR_RESYNC 2 mirrors 000572 1773233276.226192 MIRROR_EXTEND to 3 mirrors 000573 1773233276.422446 read 0x002f59 thru 0x006b7c (0x3c24 bytes) 000574 1773233276.424819 trunc from 0x02f1dc to 0x02a651 (0x4b8a bytes) 000575 1773233276.436039 read 0x025e09 thru 0x02a651 (0x4849 bytes) 000576 1773233276.439587 fallocate 0x021008 thru 0x02fdf2 (0xedeb bytes) 000577 1773233276.441397 trunc from 0x0140ce to 0x016a1c (0x294f bytes) 000578 1773233276.443015 mapwrite 0x010acd thru 0x01f56a (0xea9e bytes) 000579 1773233276.454474 mapwrite 0x030f1b thru 0x03c7ce (0xb8b4 bytes) 000580 1773233276.469290 read 0x02d982 thru 0x031fdd (0x465c bytes) 000581 1773233276.469762 trunc from 0x03c7cf to 0x021c14 (0x1abba bytes) 000582 1773233276.474039 mapwrite 0x01dd56 thru 0x025307 (0x75b2 bytes) 000584 1773233276.486870 fallocate 0x0261b1 thru 0x02798b (0x17db bytes) 000585 1773233276.489156 mapwrite 0x03a7e2 thru 0x03ffff (0x581e bytes) 000586 1773233276.501576 read 0x005afc thru 0x00ece3 (0x91e8 bytes) 000587 1773233276.502138 mapwrite 0x0346b4 thru 0x03c321 (0x7c6e bytes) 000588 1773233276.508432 write 0x011f8f thru 0x021637 (0xf6a9 bytes) 000590 1773233276.510255 trunc from 0x040000 to 0x004913 (0x3b6ec bytes) 000591 1773233276.515386 mapwrite 0x03afc0 thru 0x03ffff (0x5040 bytes) 000593 1773233276.528274 mapwrite 0x0090ac thru 0x00b2a2 (0x21f7 bytes) 000594 1773233276.532398 write 0x00994f thru 0x010eb2 (0x7564 bytes) 000595 1773233276.533284 read 0x03a94d thru 0x03ffff (0x56b3 bytes) 000596 1773233276.533843 punch from 0x03a8e9 to 0x03ffff (0x5717 bytes) 000597 1773233276.537021 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x803e6414 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x803e6414 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000598 1773233276.607425 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000599 1773233276.640116 mapwrite 0x029e59 thru 0x02d55a (0x3702 bytes) 000600 1773233276.649809 trunc from 0x015f4a to 0x019202 (0x32b9 bytes) 000601 1773233276.651406 write 0x018a1d thru 0x01e9bb (0x5f9f bytes) 000602 1773233276.655556 read 0x01d29b thru 0x02904a (0xbdb0 bytes) 000603 1773233276.657666 write 0x03e01c thru 0x03ffff (0x1fe4 bytes) 000605 1773233276.660744 punch from 0x00d6e6 to 0x0156d7 (0x7ff2 bytes) 000606 1773233276.662548 punch from 0x003fec to 0x00c869 (0x887e bytes) 000607 1773233276.664229 trunc from 0x040000 to 0x02e16f (0x11e90 bytes) 000608 1773233276.669061 mapwrite 0x005701 thru 0x00ef5c (0x985c bytes) 000609 1773233276.679932 write 0x00e1bc thru 0x010136 (0x1f7b bytes) 000610 1773233276.680658 read 0x000b84 thru 0x003561 (0x29de bytes) 000611 1773233276.683804 fallocate 0x028c9b thru 0x0305ed (0x7953 bytes) 000612 1773233276.685868 fallocate 0x02906b thru 0x03113e (0x80d4 bytes) 000613 1773233276.688034 trunc from 0x015e9b to 0x01baa6 (0x5c0c bytes) 000614 1773233276.689888 mapread 0x0190f5 thru 0x023d22 (0xac2e bytes) 000616 1773233276.694816 fallocate 0x01caf3 thru 0x01db25 (0x1033 bytes) 000617 1773233276.696645 read 0x018ed8 thru 0x019f10 (0x1039 bytes) 000618 1773233276.698696 trunc from 0x01ac26 to 0x0213b1 (0x678c bytes) 000620 1773233276.700809 fallocate 0x01ddc0 thru 0x02caf1 (0xed32 bytes) 000621 1773233276.702534 trunc from 0x02e170 to 0x03263d (0x44ce bytes) 000622 1773233276.706740 fallocate 0x02d609 thru 0x039660 (0xc058 bytes) 000623 1773233276.708237 fallocate 0x026cc3 thru 0x028c94 (0x1fd2 bytes) 000624 1773233276.709840 fallocate 0x035ffe thru 0x03ffff (0xa002 bytes) 000626 1773233276.711858 read 0x00618b thru 0x00d86f (0x76e5 bytes) 000627 1773233276.712221 mapread 0x03951b thru 0x03ca35 (0x351b bytes) 000628 1773233276.715535 trunc from 0x007364 to 0x007bcf (0x86c bytes) 000629 1773233276.717021 write 0x012f56 thru 0x015018 (0x20c3 bytes) 000630 1773233276.718889 MIRROR_EXTEND to 2 mirrors 000631 1773233276.895739 read 0x0057c6 thru 0x01027f (0xaaba bytes) 000632 1773233276.898256 punch from 0x02f9e5 to 0x0399f1 (0xa00d bytes) 000633 1773233276.909052 fallocate 0x02f5d1 thru 0x03c96e (0xd39e bytes) 000634 1773233276.910707 fallocate 0x02b62e thru 0x03a492 (0xee65 bytes) 000635 1773233276.912622 trunc from 0x02997a to 0x039465 (0xfaec bytes) 000636 1773233276.914444 read 0x016ba8 thru 0x01b416 (0x486f bytes) 000637 1773233276.918269 trunc from 0x03f347 to 0x03ffff (0xcb9 bytes) 000638 1773233276.919988 trunc from 0x01817e to 0x027b45 (0xf9c8 bytes) 000639 1773233276.922127 write 0x009b12 thru 0x01616b (0xc65a bytes) 000640 1773233276.924411 trunc from 0x040000 to 0x0374de (0x8b21 bytes) 000641 1773233276.934728 mapread 0x01a17e thru 0x01ea05 (0x4888 bytes) 000642 1773233276.941101 mapread 0x009aeb thru 0x00f27e (0x5794 bytes) 000643 1773233276.941789 punch from 0x0054b0 to 0x00f012 (0x9b63 bytes) 000644 1773233276.946995 mapread 0x012b98 thru 0x013f8e (0x13f7 bytes) 000645 1773233276.947669 read 0x01ccea thru 0x026b99 (0x9eb0 bytes) 000646 1773233276.952173 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000647 1773233276.993469 trunc from 0x0374df to 0x00e610 (0x28ece bytes) 000648 1773233276.998653 trunc from 0x00e611 to 0x011471 (0x2e61 bytes) 000649 1773233277.002574 punch from 0x00ece3 to 0x011471 (0x278f bytes) 000650 1773233277.004163 trunc from 0x0103bc to 0x011471 (0x10b6 bytes) 000651 1773233277.005823 write 0x0196e9 thru 0x0278ce (0xe1e6 bytes) 000652 1773233277.007037 trunc from 0x000635 to 0x003918 (0x32e4 bytes) 000653 1773233277.008982 read 0x02149c thru 0x025917 (0x447c bytes) 000654 1773233277.009368 trunc from 0x00d31d to 0x00dfe2 (0xcc6 bytes) 000655 1773233277.010729 write 0x02b85f thru 0x032473 (0x6c15 bytes) 000656 1773233277.011493 read 0x02b05d thru 0x02d311 (0x22b5 bytes) 000657 1773233277.011921 fallocate 0x02d7a5 thru 0x030ffc (0x3858 bytes) 000658 1773233277.013522 mapread 0x006129 thru 0x00f62f (0x9507 bytes) 000659 1773233277.017001 read 0x0002f0 thru 0x00fea9 (0xfbba bytes) 000660 1773233277.017452 mapwrite 0x006073 thru 0x0075d2 (0x1560 bytes) 000661 1773233277.022965 read 0x004431 thru 0x00f2d3 (0xaea3 bytes) 000662 1773233277.023369 punch from 0x013748 to 0x021e4c (0xe705 bytes) 000663 1773233277.025286 write 0x038011 thru 0x03ffff (0x7fef bytes) 000664 1773233277.026232 punch from 0x011a2b to 0x0122ad (0x883 bytes) 000665 1773233277.028045 fallocate 0x0132c0 thru 0x01cbf7 (0x9938 bytes) 000667 1773233277.030759 trunc from 0x040000 to 0x03938b (0x6c74 bytes) 000668 1773233277.035138 mapwrite 0x014347 thru 0x01c063 (0x7d1d bytes) 000669 1773233277.044277 trunc from 0x03938c to 0x00906f (0x3031c bytes) 000670 1773233277.050170 MIRROR_EXTEND to 2 mirrors 000671 1773233277.223991 punch from 0x006a84 to 0x008e8b (0x2408 bytes) 000672 1773233277.234580 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x9070): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9070): Mirror 2: 0xa0e5f571 (stale) CRC-32 checksum value for chunk [0, 0x9070): Mirror 1: 0x2f5c5461 Mirror 3: 0x2f5c5461 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9070: skipped 000673 1773233277.468769 fallocate 0x009ca3 thru 0x013c3d (0x9f9b bytes) 000674 1773233277.477529 trunc from 0x013c3e to 0x02176c (0xdb2f bytes) 000675 1773233277.483043 write 0x0251e4 thru 0x02a2dd (0x50fa bytes) 000676 1773233277.484039 fallocate 0x0337a9 thru 0x03ffff (0xc857 bytes) 000677 1773233277.485626 trunc from 0x00fb05 to 0x013264 (0x3760 bytes) 000678 1773233277.488363 MIRROR_RESYNC 3 mirrors 000679 1773233277.714530 trunc from 0x023843 to 0x02a2dd (0x6a9b bytes) 000680 1773233277.721946 read 0x0296e5 thru 0x02a2dd (0xbf9 bytes) 000681 1773233277.723871 read 0x0019c8 thru 0x009479 (0x7ab2 bytes) 000682 1773233277.726641 fallocate 0x020575 thru 0x02f813 (0xf29f bytes) 000683 1773233277.729484 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f814): Mirror 3: 0x30af46c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f814): Mirror 3: 0x30af46c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f814: skipped 000685 1773233277.800189 punch from 0x021279 to 0x02eeb4 (0xdc3c bytes) 000686 1773233277.803027 MIRROR_RESYNC 2 mirrors 000687 1773233277.979075 trunc from 0x015fa9 to 0x01e59a (0x85f2 bytes) 000689 1773233277.989103 fallocate 0x02922c thru 0x0371df (0xdfb4 bytes) 000690 1773233277.990881 fallocate 0x00cd22 thru 0x01a191 (0xd470 bytes) 000691 1773233277.992586 mapread 0x01fb7e thru 0x02b6af (0xbb32 bytes) 000692 1773233278.001816 MIRROR_RESYNC 2 mirrors 000693 1773233278.177818 read 0x00ea0f thru 0x00f6a4 (0xc96 bytes) 000694 1773233278.179400 read 0x0217c5 thru 0x02cc9c (0xb4d8 bytes) 000695 1773233278.182355 punch from 0x0070d6 to 0x00f39d (0x82c8 bytes) 000696 1773233278.192068 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x371e0): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x371e0): Mirror 3: 0x7274bc86 (stale) CRC-32 checksum value for chunk [0, 0x371e0): Mirror 1: 0x7274bc86 Mirror 4: 0x7274bc86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x371e0: skipped 000698 1773233278.424657 fallocate 0x01c4dd thru 0x029e74 (0xd998 bytes) 000699 1773233278.433369 mapwrite 0x00337b thru 0x00de6a (0xaaf0 bytes) 000700 1773233278.445165 trunc from 0x0159cb to 0x01bff1 (0x6627 bytes) 000701 1773233278.446892 trunc from 0x0371e0 to 0x012af4 (0x246eb bytes) 000702 1773233278.452732 trunc from 0x003922 to 0x010dc9 (0xd4a8 bytes) 000703 1773233278.454533 write 0x03b9ba thru 0x03ffff (0x4646 bytes) 000704 1773233278.456710 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x70170d8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x70170d8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000706 1773233278.525862 write 0x0392ef thru 0x03eac3 (0x57d5 bytes) 000707 1773233278.529903 fallocate 0x005865 thru 0x0093a9 (0x3b45 bytes) 000708 1773233278.531844 read 0x03434c thru 0x03ffff (0xbcb4 bytes) 000709 1773233278.534218 read 0x017214 thru 0x01b6b3 (0x44a0 bytes) 000710 1773233278.537466 punch from 0x029cb2 to 0x032b1f (0x8e6e bytes) 000711 1773233278.539124 mapread 0x00714d thru 0x00b2eb (0x419f bytes) 000712 1773233278.539886 read 0x02ebfc thru 0x03aa92 (0xbe97 bytes) 000713 1773233278.542551 trunc from 0x039f93 to 0x03ffff (0x606d bytes) 000715 1773233278.546788 read 0x03b245 thru 0x03ffff (0x4dbb bytes) 000716 1773233278.549412 mapwrite 0x00c644 thru 0x0124e2 (0x5e9f bytes) 000717 1773233278.555054 punch from 0x030038 to 0x03477a (0x4743 bytes) 000718 1773233278.556668 fallocate 0x01c5a3 thru 0x02777b (0xb1d9 bytes) 000719 1773233278.558246 mapwrite 0x03e307 thru 0x03ffff (0x1cf9 bytes) 000720 1773233278.562216 mapwrite 0x03173d thru 0x03ffff (0xe8c3 bytes) 000721 1773233278.574801 mapread 0x039a9e thru 0x03bdbb (0x231e bytes) 000722 1773233278.575475 trunc from 0x040000 to 0x00a73d (0x358c2 bytes) 000723 1773233278.579973 write 0x019c3b thru 0x02544a (0xb810 bytes) 000724 1773233278.580932 read 0x02028f thru 0x0246a9 (0x441b bytes) 000725 1773233278.581432 mapread 0x0149c4 thru 0x01c768 (0x7da5 bytes) 000726 1773233278.584267 write 0x00ccfe thru 0x00d3a8 (0x6ab bytes) 000727 1773233278.584891 punch from 0x01ae74 to 0x02110f (0x629c bytes) 000728 1773233278.588772 fallocate 0x0219b3 thru 0x02d75b (0xbda9 bytes) 000729 1773233278.590509 punch from 0x020643 to 0x02544a (0x4e08 bytes) 000730 1773233278.593496 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000731 1773233278.623903 read 0x0051c6 thru 0x00dcdb (0x8b16 bytes) 000732 1773233278.627157 trunc from 0x00b14d to 0x00d6ed (0x25a1 bytes) 000733 1773233278.629051 mapread 0x01ea83 thru 0x0244af (0x5a2d bytes) 000734 1773233278.634065 mapread 0x012642 thru 0x01aba4 (0x8563 bytes) 000735 1773233278.636935 mapwrite 0x031ce0 thru 0x0353ab (0x36cc bytes) 000736 1773233278.649492 trunc from 0x0353ac to 0x00a6f7 (0x2acb4 bytes) 000737 1773233278.653638 trunc from 0x00a6f8 to 0x03c5d1 (0x31eda bytes) 000738 1773233278.657410 write 0x0084e6 thru 0x014606 (0xc121 bytes) 000739 1773233278.659579 mapread 0x006c6d thru 0x00aa20 (0x3db4 bytes) 000740 1773233278.660272 write 0x007f40 thru 0x015ecd (0xdf8e bytes) 000741 1773233278.663307 mapread 0x024bf4 thru 0x032a94 (0xdea1 bytes) 000743 1773233278.669210 mapwrite 0x03647c thru 0x03ffff (0x9b84 bytes) 000744 1773233278.682402 mapwrite 0x0072e4 thru 0x00d7b8 (0x64d5 bytes) 000745 1773233278.687406 fallocate 0x016c0a thru 0x01c492 (0x5889 bytes) 000747 1773233278.690034 trunc from 0x002e7c to 0x009c24 (0x6da9 bytes) 000748 1773233278.693077 MIRROR_EXTEND to 2 mirrors 000749 1773233278.861739 mapread 0x0249ee thru 0x032160 (0xd773 bytes) 000750 1773233278.868958 trunc from 0x017621 to 0x01a3ab (0x2d8b bytes) 000751 1773233278.878369 fallocate 0x0117c3 thru 0x01a638 (0x8e76 bytes) 000752 1773233278.880051 punch from 0x0346c8 to 0x037b3a (0x3473 bytes) 000753 1773233278.881526 punch from 0x00c96f to 0x01634a (0x99dc bytes) 000754 1773233278.883168 mapwrite 0x01e9a7 thru 0x021ef6 (0x3550 bytes) 000755 1773233278.890097 mapread 0x01d3b3 thru 0x01f265 (0x1eb3 bytes) 000757 1773233278.891043 trunc from 0x040000 to 0x02af7c (0x15083 bytes) 000758 1773233278.900821 punch from 0x014105 to 0x016341 (0x223d bytes) 000760 1773233278.902526 fallocate 0x03ccdf thru 0x03ffff (0x3321 bytes) 000761 1773233278.904412 mapread 0x003985 thru 0x004153 (0x7cf bytes) 000762 1773233278.907436 fallocate 0x026e46 thru 0x03480d (0xd9c8 bytes) 000763 1773233278.909277 write 0x02b558 thru 0x03a91b (0xf3c4 bytes) 000764 1773233278.914039 mapwrite 0x009ed5 thru 0x00ff15 (0x6041 bytes) 000765 1773233278.920312 fallocate 0x00a9a5 thru 0x01326c (0x88c8 bytes) 000766 1773233278.923134 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1eb2d94e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x21222ecb Mirror 3: 0x21222ecb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000767 1773233279.163763 mapread 0x02ce03 thru 0x03ad0f (0xdf0d bytes) 000769 1773233279.176459 MIRROR_RESYNC 3 mirrors 000770 1773233279.365240 mapread 0x03f09d thru 0x03ffff (0xf63 bytes) 000772 1773233279.367332 mapread 0x01df20 thru 0x022841 (0x4922 bytes) 000773 1773233279.371007 trunc from 0x040000 to 0x03af56 (0x50a9 bytes) 000774 1773233279.381226 read 0x003dce thru 0x004cb9 (0xeec bytes) 000775 1773233279.382939 punch from 0x02b82b to 0x0323aa (0x6b80 bytes) 000776 1773233279.384509 mapwrite 0x01b25e thru 0x02ab45 (0xf8e8 bytes) 000778 1773233279.396965 mapread 0x01790a thru 0x01b263 (0x395a bytes) 000779 1773233279.397771 write 0x001944 thru 0x00f2bc (0xd979 bytes) 000780 1773233279.400948 trunc from 0x002842 to 0x00ad47 (0x8506 bytes) 000781 1773233279.404863 mapread 0x0253a7 thru 0x029c4a (0x48a4 bytes) 000782 1773233279.405551 trunc from 0x03af57 to 0x00634a (0x34c0c bytes) 000783 1773233279.411159 punch from 0x000a42 to 0x0039a9 (0x2f68 bytes) 000784 1773233279.412798 read 0x0024c4 thru 0x00634a (0x3e87 bytes) 000785 1773233279.415197 punch from 0x004285 to 0x00634a (0x20c6 bytes) 000786 1773233279.416668 mapread 0x000336 thru 0x00634a (0x6015 bytes) 000787 1773233279.420691 read 0x002ad6 thru 0x0034cb (0x9f6 bytes) 000788 1773233279.421251 read 0x00607a thru 0x00634a (0x2d1 bytes) 000789 1773233279.421731 mapread 0x00201b thru 0x00634a (0x4330 bytes) 000790 1773233279.423729 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x634b): Mirror 3: 0xb30a8404 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x634b): Mirror 3: 0xb30a8404 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x634b: skipped 000791 1773233279.485719 mapread 0x003f48 thru 0x00634a (0x2403 bytes) 000792 1773233279.490252 mapread 0x004ecb thru 0x00634a (0x1480 bytes) 000795 1773233279.492285 trunc from 0x00634b to 0x0026b9 (0x3c91 bytes) 000796 1773233279.497494 punch from 0x0019ae to 0x0026b9 (0xd0c bytes) 000797 1773233279.499136 mapwrite 0x000664 thru 0x00f099 (0xea36 bytes) 000798 1773233279.512145 mapread 0x00cfed thru 0x00f099 (0x20ad bytes) 000800 1773233279.513707 mapwrite 0x019b2e thru 0x019d77 (0x24a bytes) 000802 1773233279.523935 trunc from 0x00e819 to 0x019d77 (0xb55f bytes) 000803 1773233279.525707 read 0x00dd07 thru 0x010d59 (0x3053 bytes) 000804 1773233279.527583 fallocate 0x000b9d thru 0x009406 (0x886a bytes) 000805 1773233279.529147 fallocate 0x023393 thru 0x026b29 (0x3797 bytes) 000806 1773233279.530639 fallocate 0x02eac2 thru 0x03b653 (0xcb92 bytes) 000807 1773233279.532177 mapwrite 0x0342d6 thru 0x03ffff (0xbd2a bytes) 000808 1773233279.545379 trunc from 0x040000 to 0x0396fd (0x6902 bytes) 000809 1773233279.549505 trunc from 0x0396fe to 0x0162fb (0x23402 bytes) 000810 1773233279.554792 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000811 1773233279.585615 punch from 0x006314 to 0x00ac16 (0x4903 bytes) 000812 1773233279.587498 punch from 0x0100f9 to 0x0162fb (0x6203 bytes) 000813 1773233279.589044 fallocate 0x0337af thru 0x03d9c0 (0xa212 bytes) 000814 1773233279.590689 trunc from 0x03d9c1 to 0x02fb72 (0xde4e bytes) 000815 1773233279.595806 mapwrite 0x01235e thru 0x01c7fe (0xa4a1 bytes) 000816 1773233279.607207 punch from 0x0233b7 to 0x02dce2 (0xa92c bytes) 000817 1773233279.608880 trunc from 0x02fb73 to 0x033656 (0x3ae4 bytes) 000818 1773233279.613160 write 0x00bc56 thru 0x00eb2d (0x2ed8 bytes) 000820 1773233279.616949 punch from 0x02e1c8 to 0x033656 (0x548f bytes) 000821 1773233279.618320 read 0x013d06 thru 0x020c81 (0xcf7c bytes) 000822 1773233279.622206 MIRROR_EXTEND to 2 mirrors 000823 1773233279.794550 mapread 0x02b4f5 thru 0x033656 (0x8162 bytes) 000824 1773233279.799108 trunc from 0x033657 to 0x02d104 (0x6552 bytes) 000826 1773233279.813993 punch from 0x0137c8 to 0x01d7e9 (0xa022 bytes) 000827 1773233279.815659 trunc from 0x02d105 to 0x0399c5 (0xc8c1 bytes) 000828 1773233279.820209 write 0x03406a thru 0x03ffff (0xbf96 bytes) 000829 1773233279.823233 mapread 0x0220f4 thru 0x02f66d (0xd57a bytes) 000830 1773233279.827092 trunc from 0x040000 to 0x008bde (0x37421 bytes) 000831 1773233279.831338 punch from 0x000d21 to 0x002980 (0x1c60 bytes) 000832 1773233279.833018 read 0x00152a thru 0x006810 (0x52e7 bytes) 000833 1773233279.836306 trunc from 0x008bdf to 0x01bf59 (0x1337b bytes) 000834 1773233279.840668 punch from 0x005647 to 0x00a0ea (0x4aa4 bytes) 000835 1773233279.842390 mapwrite 0x03f215 thru 0x03ffff (0xdeb bytes) 000836 1773233279.851790 mapread 0x02a6f3 thru 0x03766b (0xcf79 bytes) 000837 1773233279.857031 mapwrite 0x020b99 thru 0x0282dc (0x7744 bytes) 000838 1773233279.865041 punch from 0x008ee1 to 0x013f1d (0xb03d bytes) 000839 1773233279.866529 mapwrite 0x028c43 thru 0x031809 (0x8bc7 bytes) 000840 1773233279.872573 mapwrite 0x034c1f thru 0x036105 (0x14e7 bytes) 000841 1773233279.876734 write 0x00c0db thru 0x011747 (0x566d bytes) 000842 1773233279.880988 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x896b34cb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x978c6fd3 Mirror 3: 0x978c6fd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000843 1773233280.125808 read 0x0027eb thru 0x008131 (0x5947 bytes) 000844 1773233280.130169 trunc from 0x01c51c to 0x025dcc (0x98b1 bytes) 000845 1773233280.141014 read 0x025d0c thru 0x027274 (0x1569 bytes) 000846 1773233280.146924 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x978c6fd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x978c6fd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000847 1773233280.223516 mapread 0x03d2f9 thru 0x03ffff (0x2d07 bytes) 000848 1773233280.230244 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x978c6fd3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9174f182 Mirror 4: 0x9174f182 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000849 1773233280.511482 read 0x0201e8 thru 0x021c28 (0x1a41 bytes) 000851 1773233280.517046 fallocate 0x014a2e thru 0x019f75 (0x5548 bytes) 000852 1773233280.527830 mapwrite 0x00edf2 thru 0x012b57 (0x3d66 bytes) 000854 1773233280.538663 write 0x00c84d thru 0x01801b (0xb7cf bytes) 000855 1773233280.541216 mapwrite 0x0389e0 thru 0x03ffff (0x7620 bytes) 000856 1773233280.550365 fallocate 0x00c207 thru 0x00ccd0 (0xaca bytes) 000857 1773233280.552172 punch from 0x029afa to 0x039006 (0xf50d bytes) 000858 1773233280.553770 mapread 0x02a809 thru 0x02abe1 (0x3d9 bytes) 000861 1773233280.557903 trunc from 0x040000 to 0x01a501 (0x25afe bytes) 000862 1773233280.563652 mapwrite 0x034629 thru 0x03ffff (0xb9d7 bytes) 000863 1773233280.577471 mapwrite 0x03d855 thru 0x03ffff (0x27ab bytes) 000864 1773233280.581459 punch from 0x00155d to 0x006662 (0x5106 bytes) 000865 1773233280.583308 trunc from 0x015ca6 to 0x019e4a (0x41a5 bytes) 000866 1773233280.585050 trunc from 0x0044b8 to 0x013d61 (0xf8aa bytes) 000867 1773233280.586749 fallocate 0x009426 thru 0x01725b (0xde36 bytes) 000868 1773233280.588424 mapwrite 0x01d0f5 thru 0x020ffd (0x3f09 bytes) 000869 1773233280.597159 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9174f182 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9174f182 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000870 1773233280.661985 mapread 0x0278c7 thru 0x02c050 (0x478a bytes) 000872 1773233280.667870 read 0x01b606 thru 0x01f161 (0x3b5c bytes) 000873 1773233280.670088 write 0x032bad thru 0x03f033 (0xc487 bytes) 000874 1773233280.675171 mapread 0x00ffa4 thru 0x01af46 (0xafa3 bytes) 000875 1773233280.680550 trunc from 0x02b3ad to 0x032339 (0x6f8d bytes) 000876 1773233280.684362 fallocate 0x0317de thru 0x03680d (0x5030 bytes) 000877 1773233280.685921 trunc from 0x00e8f5 to 0x014ce1 (0x63ed bytes) 000878 1773233280.687504 punch from 0x012482 to 0x0184d0 (0x604f bytes) 000880 1773233280.689725 trunc from 0x03e3be to 0x03f30e (0xf51 bytes) 000881 1773233280.692570 MIRROR_RESYNC 2 mirrors 000882 1773233280.873536 trunc from 0x039b8f to 0x03ffff (0x6471 bytes) 000883 1773233280.883470 mapread 0x02e25a thru 0x030742 (0x24e9 bytes) 000884 1773233280.889903 mapwrite 0x02f0ec thru 0x03cb8c (0xdaa1 bytes) 000885 1773233280.900242 mapwrite 0x005b9d thru 0x00e50b (0x896f bytes) 000886 1773233280.910001 mapread 0x006cee thru 0x006fbe (0x2d1 bytes) 000887 1773233280.910743 trunc from 0x040000 to 0x02afa5 (0x1505a bytes) 000889 1773233280.921448 mapread 0x01d918 thru 0x024b1f (0x7208 bytes) 000890 1773233280.924580 trunc from 0x02afa6 to 0x00b086 (0x1ff1f bytes) 000891 1773233280.929248 write 0x029150 thru 0x030044 (0x6ef5 bytes) 000893 1773233280.930681 trunc from 0x023598 to 0x027b21 (0x458a bytes) 000894 1773233280.932474 trunc from 0x030045 to 0x0325c4 (0x2580 bytes) 000895 1773233280.936690 write 0x0227d6 thru 0x02b8da (0x9105 bytes) 000896 1773233280.939896 punch from 0x01f2bc to 0x029d86 (0xaacb bytes) 000897 1773233280.943925 trunc from 0x019ae3 to 0x024161 (0xa67f bytes) 000898 1773233280.945361 trunc from 0x002ac8 to 0x004df9 (0x2332 bytes) 000899 1773233280.946731 mapwrite 0x014417 thru 0x014ff2 (0xbdc bytes) 000900 1773233280.952996 trunc from 0x02a4fd to 0x0325c4 (0x80c8 bytes) 000901 1773233280.954542 write 0x035a06 thru 0x0399d3 (0x3fce bytes) 000902 1773233280.955248 trunc from 0x0399d4 to 0x01b5d9 (0x1e3fa bytes) 000903 1773233280.959455 read 0x00987c thru 0x0160bd (0xc842 bytes) 000905 1773233280.962079 punch from 0x0013c7 to 0x0016f8 (0x332 bytes) 000906 1773233280.963745 fallocate 0x020c8b thru 0x02ca5b (0xbdd1 bytes) 000907 1773233280.965561 mapread 0x015033 thru 0x01b5d9 (0x65a7 bytes) 000908 1773233280.966442 read 0x017292 thru 0x01a05b (0x2dca bytes) 000910 1773233280.967408 read 0x017984 thru 0x01b5d9 (0x3c56 bytes) 000911 1773233280.967907 fallocate 0x024d73 thru 0x0294de (0x476c bytes) 000912 1773233280.969688 punch from 0x00b491 to 0x00e58d (0x30fd bytes) 000913 1773233280.971384 mapread 0x00945e thru 0x00bf94 (0x2b37 bytes) 000914 1773233280.974379 punch from 0x007a5c to 0x011e14 (0xa3b9 bytes) 000915 1773233280.976032 read 0x01a6a5 thru 0x01a8dd (0x239 bytes) 000917 1773233280.976892 punch from 0x0071e3 to 0x00b5d4 (0x43f2 bytes) 000918 1773233280.978317 trunc from 0x002e78 to 0x00de63 (0xafec bytes) 000920 1773233280.980288 mapread 0x015452 thru 0x01aa4d (0x55fc bytes) 000921 1773233280.981011 mapwrite 0x01398a thru 0x019bf3 (0x626a bytes) 000923 1773233280.986993 trunc from 0x01b5da to 0x009ed9 (0x11700 bytes) 000924 1773233280.991358 mapwrite 0x036552 thru 0x03ffff (0x9aae bytes) 000925 1773233281.004216 punch from 0x002ee0 to 0x009ca9 (0x6dca bytes) 000926 1773233281.005955 write 0x005e2c thru 0x0068c7 (0xa9c bytes) 000927 1773233281.008734 trunc from 0x040000 to 0x01d049 (0x22fb6 bytes) 000928 1773233281.014838 mapwrite 0x01ee90 thru 0x023636 (0x47a7 bytes) 000929 1773233281.030673 trunc from 0x0126fc to 0x01f4e1 (0xcde6 bytes) 000930 1773233281.032523 fallocate 0x022e48 thru 0x0245b2 (0x176b bytes) 000931 1773233281.035435 MIRROR_RESYNC 2 mirrors 000932 1773233281.202457 trunc from 0x01efb5 to 0x01f9bf (0xa0b bytes) 000934 1773233281.212520 write 0x01383f thru 0x0160bd (0x287f bytes) 000935 1773233281.215998 mapwrite 0x030e40 thru 0x03a5ab (0x976c bytes) 000937 1773233281.235789 fallocate 0x0089ac thru 0x011c71 (0x92c6 bytes) 000938 1773233281.238768 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a5ac): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5ac): Mirror 4: 0xa50e9ce4 (stale) CRC-32 checksum value for chunk [0, 0x3a5ac): Mirror 1: 0x4a4cd6dc Mirror 5: 0x4a4cd6dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5ac: skipped 000939 1773233281.477813 punch from 0x033767 to 0x039aa6 (0x6340 bytes) 000940 1773233281.487135 mapread 0x020014 thru 0x024bf8 (0x4be5 bytes) 000941 1773233281.492385 trunc from 0x03a5ac to 0x01a996 (0x1fc15 bytes) 000943 1773233281.504461 fallocate 0x00676d thru 0x009db5 (0x3649 bytes) 000944 1773233281.506183 mapread 0x012191 thru 0x0162e7 (0x4157 bytes) 000945 1773233281.509377 mapread 0x00f485 thru 0x010546 (0x10c2 bytes) 000947 1773233281.510410 trunc from 0x01a997 to 0x008661 (0x12335 bytes) 000948 1773233281.514765 fallocate 0x0289d4 thru 0x02c823 (0x3e50 bytes) 000949 1773233281.516690 read 0x01eeb7 thru 0x02c823 (0xd96d bytes) 000950 1773233281.519632 write 0x0252fa thru 0x032b78 (0xd87f bytes) 000951 1773233281.520503 fallocate 0x0318c4 thru 0x03c83f (0xaf7c bytes) 000952 1773233281.522179 mapwrite 0x00e5ee thru 0x018c97 (0xa6aa bytes) 000953 1773233281.533658 trunc from 0x032b79 to 0x00bff6 (0x26b82 bytes) 000954 1773233281.539392 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xbff7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbff7): Mirror 4: 0x3dbfe81f (stale) Mirror 5: 0x3dbfe81f (stale) CRC-32 checksum value for chunk [0, 0xbff7): Mirror 1: 0x3dbfe81f Mirror 6: 0x3dbfe81f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbff7: skipped 000955 1773233281.756789 write 0x02983a thru 0x02c2fe (0x2ac5 bytes) 000956 1773233281.767363 mapread 0x0185bf thru 0x024c7e (0xc6c0 bytes) 000959 1773233281.776307 trunc from 0x02c2ff to 0x03e409 (0x1210b bytes) 000960 1773233281.786977 punch from 0x024557 to 0x028e0e (0x48b8 bytes) 000961 1773233281.790049 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 4: 0x3e66fe4e (stale) Mirror 5: 0x2fad5ab (stale) Mirror 6: 0xe2563073 (stale) CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 1: 0x8d20437 Mirror 7: 0x8d20437 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e40a: skipped 000962 1773233282.054831 punch from 0x012165 to 0x01650b (0x43a7 bytes) 000964 1773233282.066568 fallocate 0x0398f6 thru 0x03ffff (0x670a bytes) 000965 1773233282.070480 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 4: 0x3e66fe4e (stale) Mirror 5: 0x2fad5ab (stale) Mirror 6: 0xe2563073 (stale) Mirror 7: 0x8d20437 (stale) CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 1: 0x8d20437 Mirror 8: 0x8d20437 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e40a: skipped 000966 1773233282.359099 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 4: 0x3e66fe4e (stale) Mirror 5: 0x2fad5ab (stale) Mirror 6: 0xe2563073 (stale) Mirror 7: 0x8d20437 (stale) CRC-32 checksum value for chunk [0, 0x3e40a): Mirror 1: 0x8d20437 Mirror 8: 0x8d20437 Mirror 9: 0x8d20437 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e40a: skipped 000967 1773233282.649053 mapwrite 0x01d4ef thru 0x027ccd (0xa7df bytes) 000968 1773233282.688621 trunc from 0x03e40a to 0x019dcd (0x2463c bytes) 000969 1773233282.695683 mapwrite 0x01a429 thru 0x02705b (0xcc33 bytes) 000970 1773233282.711352 read 0x010dd7 thru 0x017136 (0x6360 bytes) 000971 1773233282.713306 fallocate 0x026bf7 thru 0x031986 (0xad90 bytes) 000972 1773233282.715171 trunc from 0x0140dc to 0x01e9b5 (0xa8da bytes) 000973 1773233282.716819 read 0x024e4f thru 0x031986 (0xcb38 bytes) 000974 1773233282.720017 write 0x0016a9 thru 0x002a43 (0x139b bytes) 000975 1773233282.720667 punch from 0x00a443 to 0x0189f9 (0xe5b7 bytes) 000976 1773233282.722503 trunc from 0x031987 to 0x023a08 (0xdf7e bytes) 000977 1773233282.727141 write 0x020afe thru 0x026449 (0x594c bytes) 000978 1773233282.728379 mapread 0x011996 thru 0x01af1b (0x9586 bytes) 000979 1773233282.741551 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2644a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2644a): Mirror 4: 0xc321c425 (stale) Mirror 5: 0x7f7ea076 (stale) Mirror 6: 0xa86573b5 (stale) Mirror 7: 0xa86573b5 (stale) Mirror 8: 0xa86573b5 (stale) Mirror 9: 0xa86573b5 (stale) CRC-32 checksum value for chunk [0, 0x2644a): Mirror 1: 0x6b27710b Mirror 10: 0x6b27710b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2644a: skipped 000980 1773233283.001749 trunc from 0x02644a to 0x02e255 (0x7e0c bytes) 000981 1773233283.018907 mapread 0x019982 thru 0x01efe5 (0x5664 bytes) 000982 1773233283.024233 fallocate 0x02b8a5 thru 0x03b62e (0xfd8a bytes) 000983 1773233283.026120 mapread 0x00eca2 thru 0x017d95 (0x90f4 bytes) 000984 1773233283.029260 trunc from 0x02e256 to 0x02562f (0x8c26 bytes) 000985 1773233283.034547 write 0x009e96 thru 0x00b418 (0x1583 bytes) 000986 1773233283.035278 fallocate 0x0010de thru 0x00588d (0x47b0 bytes) 000987 1773233283.037080 read 0x009fd7 thru 0x00cf2a (0x2f54 bytes) 000988 1773233283.039062 MIRROR_RESYNC 8 mirrors 000989 1773233283.461364 fallocate 0x00cad8 thru 0x00f6bb (0x2be4 bytes) 000990 1773233283.471284 punch from 0x000637 to 0x004f8c (0x4956 bytes) 000991 1773233283.472844 fallocate 0x009c94 thru 0x018fb7 (0xf324 bytes) 000992 1773233283.474838 write 0x03dc0a thru 0x03ffff (0x23f6 bytes) 000993 1773233283.475902 mapread 0x02ee97 thru 0x038175 (0x92df bytes) 000994 1773233283.480733 mapwrite 0x0237c2 thru 0x030c50 (0xd48f bytes) 000995 1773233283.489660 mapread 0x0362e1 thru 0x03c49b (0x61bb bytes) 000996 1773233283.491883 mapread 0x026256 thru 0x035a3f (0xf7ea bytes) 000997 1773233283.492828 mapwrite 0x01de58 thru 0x02a969 (0xcb12 bytes) 000998 1773233283.499522 mapread 0x019a2c thru 0x01e72b (0x4d00 bytes) 000999 1773233283.502336 mapwrite 0x004074 thru 0x011c13 (0xdba0 bytes) 001000 1773233283.513216 mapread 0x0023e4 thru 0x00b86f (0x948c bytes) All operations completed A-OK! PASS 70a (21s) == sanity-flr test 71: check mirror extend parent fid ==== 08:48:05 (1773233285) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x7c6:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 08:48:08 (1773233288) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0665243 s, 31.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354295 s, 29.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbb:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (2s) == sanity-flr test 200a: stress test ===================== 08:48:10 (1773233290) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2253184 .. Extending file size to 5003776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7079520 .. done Extending file size to 6736352 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7558720 .. Extending file size to 9072480 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7686304 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7970880 .. Extending file size to 9238112 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9280128 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64294 64295 64297 64298 64301 done 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) 192.168.203.154@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:49:15 (1773233355) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7559008 .. Extending file size to 6553088 .. Extending file size to 8149184 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8740544 .. Extending file size to 7865312 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8734560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9202976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8998112 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9053664 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 67687 67688 67690 67692 67694 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) 192.168.203.154@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:50:20 (1773233420) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2668 1268636 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2920 1268384 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24864 3582156 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18876 3588144 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43740 7170300 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2668 1268636 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2920 1268384 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24864 3582156 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18876 3588144 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43740 7170300 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.040697 s, 51.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0432117 s, 48.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5db89120 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:50:26 (1773233426) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.02765 s, 75.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 08:50:28 (1773233428) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:50:30 (1773233430) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0224391 s, 93.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb80:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb80:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb81:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:50:32 (1773233432) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0192217 s, 109 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:50:35 (1773233435) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0215778 s, 97.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:50:59 (1773233459) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg354-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158955 s, 66.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb81:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb82:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb81:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb82:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb82:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:51:32 (1773233492) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg354-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb85:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0271221 s, 77.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb85:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb83:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb85:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb83:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:52:05 (1773233525) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg354-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg354-server: Pool lustre.test_204f created Adding targets to pool oleg354-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg354-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0276788 s, 75.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg354-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg354-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg354-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:52:52 (1773233572) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:52:54 (1773233574) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:52:56 (1773233576) oleg354-server: Pool lustre.test_206 created Adding targets to pool oleg354-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg354-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg354-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg354-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg354-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:53:11 (1773233591) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273103 s, 38.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149889 s, 70.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1935d761 Mirror 3: 0x1935d761 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:53:14 (1773233594) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:53:15 (1773233595) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:53:16 (1773233596) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.00871495 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xfd0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x820000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x820000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x820000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe20000 Read this (4 bytes): BBBB 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:53:32 (1773233612) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00290665 s, 2.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:53:35 (1773233615) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194738 s, 53.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc1 lmm_fid: [0x200000405:0xc1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc1 lmm_fid: [0x200000405:0xc1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdf:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611329 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-11 08:53:35.000000000 -0400 Modify: 2026-03-11 08:53:35.000000000 -0400 Change: 2026-03-11 08:53:36.000000000 -0400 Birth: 2026-03-11 08:53:35.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:53:37 (1773233617) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178533 s, 58.7 MB/s oleg354-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg354-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg354-server: O/280000401/d0/e0: File not found by ext2_lookup PASS 210b (15s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:53:52 (1773233632) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0522227 s, 784 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00620466 s, 660 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 11 08:53 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 11 08:53 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:53:55 (1773233635) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:53:57 (1773233637) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 813 sec ============ 08:53:59 (1773233639) === sanity-flr: start cleanup 08:54:00 (1773233640) === === sanity-flr: finish cleanup 08:54:02 (1773233642) ===