-----============= acceptance-small: sanity-flr ============----- Sat May 9 15:28:22 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg251-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:28:28 (1778354908) === oleg251-client.virtnet: executing check_config_client /mnt/lustre oleg251-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg251-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa00f18e6c000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa00f18e6c000.idle_timeout=debug disable quota as required oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:28:40 (1778354920) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8174] == sanity-flr test 0a: lfs mirror create with -N option == 15:28:41 (1778354921) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:28:45 (1778354925) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:28:46 (1778354926) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:28:47 (1778354927) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:28:53 (1778354933) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:28:54 (1778354934) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 15:28:55 (1778354935) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa00f18e6c000.stats=clear osc.lustre-OST0001-osc-ffffa00f18e6c000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:28:58 (1778354938) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:29:01 (1778354941) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:29:03 (1778354943) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:29:04 (1778354944) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:29:06 (1778354946) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 15:29:09 (1778354949) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:29:11 (1778354951) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:29:14 (1778354954) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:29:16 (1778354956) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:29:19 (1778354959) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:29:21 (1778354961) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.269791 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312755 s, 33.5 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:29:24 (1778354964) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.180883 s, 110 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:29:26 (1778354966) Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:29:54 (1778354994) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.177561 s, 47.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:30:07 (1778355007) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:30:31 (1778355031) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.834619 s, 62.8 MB/s -rw-r--r-- 1 root root 52428800 May 9 15:30 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.252s user 0m0.001s sys 0m0.149s Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00f18e6c000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.278s user 0m0.001s sys 0m0.164s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.300s user 0m0.001s sys 0m0.149s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00f18e6c000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:30:59 (1778355059) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:31:00 (1778355060) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:31:01 (1778355061) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 15:31:02 (1778355062) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 15:31:04 (1778355064) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.145175 s, 86.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:31:08 (1778355068) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138458 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144304 s, 72.7 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:31:10 (1778355070) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:31:14 (1778355074) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000649279 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101725 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000696099 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000586777 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 15:31:17 (1778355077) create test files with size 15750654 .. 8d85b2586bcdc7073aee67d916656b2c - 2596bab7cca87df2d93c45c01cb14a53 - 4c59c513a0cca42c5245636836455429 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 15:31:22 (1778355082) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:31:27 (1778355087) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:31:29 (1778355089) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293231 s, 35.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103632 s, 101 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 15:31:31 (1778355091) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0274458 s, 76.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.095196 s, 44.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.03879 s, 54.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 15:31:36 (1778355096) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 15:31:37 (1778355097) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:31:38 (1778355098) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111252 s, 94.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 15:31:40 (1778355100) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0739987 s, 42.5 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:31:43 (1778355103) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:31:46 (1778355106) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0982448 s, 107 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:32:05 (1778355125) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa00f18e6e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0968556 s, 108 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa00f18f41800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:32:09 (1778355129) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:32:22 (1778355142) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:32:33 (1778355153) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 15:32:35 (1778355155) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0688215 s, 45.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0537291 s, 58.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf5bae63e (stale) Mirror 2: 0xf5bae63e (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: bdac85ea2cf0a7f4b3946247876ecd44 - mirror 1 checksum: f73446d49ae8e98e4f833704ad48fbd7 - mirror 2 checksum: bdac85ea2cf0a7f4b3946247876ecd44 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat May 9 15:32:35 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat May 9 15:32:35 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: bdac85ea2cf0a7f4b3946247876ecd44 - mirror 1 checksum: f73446d49ae8e98e4f833704ad48fbd7 - mirror 2 checksum: f73446d49ae8e98e4f833704ad48fbd7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat May 9 15:32:35 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat May 9 15:32:35 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:32:38 (1778355158) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162592 s, 64.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:32:40 (1778355160) Failing mds1 on oleg251-server Stopping /mnt/lustre-mds1 (opts:) on oleg251-server 15:32:48 (1778355168) shut down facet: mds1 facet_host: oleg251-server facet_failover_host: oleg251-server Failover mds1 to oleg251-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 15:33:02 (1778355182) targets are mounted 15:33:02 (1778355182) facet_failover done oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109881 s, 95.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:33:10 (1778355190) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00922014 s, 2.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00510086 s, 4.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:33:13 (1778355193) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.010015 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00354403 s, 5.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:33:16 (1778355196) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00241573 s, 6.8 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:33:18 (1778355198) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0103441 s, 2.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:33:21 (1778355201) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00164258 s, 2.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00927168 s, 442 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:33:23 (1778355203) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:33:29 (1778355209) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:33:48 (1778355228) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:34:05 (1778355245) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:34:23 (1778355263) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 15:34:25 (1778355265) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 56421 mirror split pid 56422 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 56421 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 56422 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 15:35:28 (1778355328) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5328 000001 1778355328.803148 mapwrite 0x01d4fc thru 0x02693d (0x9442 bytes) truncating to largest ever: 0x15d6f 000002 1778355328.822667 trunc from 0x02693e to 0x015d6e (0x10bcf bytes) 000003 1778355328.827298 punch from 0x00f825 to 0x012f64 (0x3740 bytes) 000004 1778355328.829173 punch from 0x007069 to 0x015d6e (0xed06 bytes) 000005 1778355328.831208 mapread 0x003d8c thru 0x00ea68 (0xacdd bytes) 000006 1778355328.835861 read 0x00d2e3 thru 0x012c6c (0x598a bytes) 000007 1778355328.838568 read 0x00bb92 thru 0x015d6e (0xa1dd bytes) truncating to largest ever: 0x15fe7 000008 1778355328.841845 trunc from 0x015d6f to 0x015fe6 (0x278 bytes) truncating to largest ever: 0x37d71 000009 1778355328.846661 trunc from 0x015fe7 to 0x037d70 (0x21d8a bytes) 000010 1778355328.851305 mapread 0x000646 thru 0x00bbf7 (0xb5b2 bytes) 000011 1778355328.852568 write 0x03805d thru 0x03fcb7 (0x7c5b bytes) 000012 1778355328.853868 trunc from 0x011d42 to 0x013d27 (0x1fe6 bytes) 000013 1778355328.855595 fallocate 0x005a78 thru 0x00c4f4 (0x6a7d bytes) 000014 1778355328.857384 fallocate 0x00fdce thru 0x0123d7 (0x260a bytes) 000017 1778355328.860572 mapwrite 0x0346ec thru 0x03e1d5 (0x9aea bytes) 000018 1778355328.870725 punch from 0x019b60 to 0x023d5f (0xa200 bytes) 000020 1778355328.872972 read 0x01a3c4 thru 0x01ebc7 (0x4804 bytes) 000021 1778355328.875818 write 0x02aa0d thru 0x02e42f (0x3a23 bytes) 000022 1778355328.876703 mapread 0x00254b thru 0x009bef (0x76a5 bytes) fallocating to largest ever: 0x40000 000023 1778355328.877469 fallocate 0x03a4d8 thru 0x03ffff (0x5b28 bytes) 000025 1778355328.879980 mapwrite 0x008e9d thru 0x014b54 (0xbcb8 bytes) 000028 1778355328.888119 mapread 0x01c8c9 thru 0x025856 (0x8f8e bytes) 000029 1778355328.888979 fallocate 0x00ec6a thru 0x018aac (0x9e43 bytes) 000030 1778355328.891124 trunc from 0x03fcb8 to 0x029e94 (0x15e23 bytes) 000031 1778355328.896160 punch from 0x01f761 to 0x029e94 (0xa734 bytes) 000032 1778355328.897909 trunc from 0x00fa52 to 0x013259 (0x3808 bytes) 000033 1778355328.899597 trunc from 0x004540 to 0x013738 (0xf1f9 bytes) 000034 1778355328.901319 write 0x004d1d thru 0x0121dc (0xd4c0 bytes) 000035 1778355328.905108 mapread 0x024e43 thru 0x029e94 (0x5052 bytes) 000036 1778355328.913789 write 0x003f01 thru 0x00e19d (0xa29d bytes) 000037 1778355328.914722 write 0x0188f1 thru 0x0218a5 (0x8fb5 bytes) 000038 1778355328.917020 trunc from 0x0269d9 to 0x0280e6 (0x170e bytes) 000039 1778355328.919105 fallocate 0x00aa66 thru 0x00ba52 (0xfed bytes) 000041 1778355328.921672 trunc from 0x016d2e to 0x026013 (0xf2e6 bytes) 000043 1778355328.926919 mapwrite 0x02a346 thru 0x03962e (0xf2e9 bytes) 000044 1778355328.961125 fallocate 0x012d5e thru 0x014b1f (0x1dc2 bytes) 000045 1778355328.963061 write 0x00977a thru 0x00cc3f (0x34c6 bytes) 000047 1778355328.964643 write 0x02703c thru 0x02ccc2 (0x5c87 bytes) 000048 1778355328.966814 punch from 0x01ca0c to 0x0235e9 (0x6bde bytes) 000050 1778355328.968810 write 0x005eb5 thru 0x00c44a (0x6596 bytes) 000051 1778355328.969995 punch from 0x0131ec to 0x022dee (0xfc03 bytes) 000053 1778355328.972428 fallocate 0x0380b1 thru 0x03ffff (0x7f4f bytes) 000054 1778355328.974271 trunc from 0x012cbe to 0x021c43 (0xef86 bytes) 000055 1778355328.976273 trunc from 0x0227b3 to 0x031434 (0xec82 bytes) 000056 1778355328.981432 write 0x032080 thru 0x03b591 (0x9512 bytes) 000057 1778355328.983703 punch from 0x017c27 to 0x018469 (0x843 bytes) 000058 1778355328.985151 fallocate 0x03ee62 thru 0x03ffff (0x119e bytes) 000059 1778355328.987016 fallocate 0x034d40 thru 0x03ffff (0xb2c0 bytes) 000060 1778355328.988891 trunc from 0x024e09 to 0x0298fa (0x4af2 bytes) 000061 1778355328.990544 write 0x015c32 thru 0x02288f (0xcc5e bytes) 000062 1778355328.993841 trunc from 0x001729 to 0x010371 (0xec49 bytes) 000063 1778355328.995541 punch from 0x006af3 to 0x00f124 (0x8632 bytes) 000064 1778355328.997181 punch from 0x039148 to 0x03b2b6 (0x216f bytes) 000065 1778355329.001799 read 0x023f24 thru 0x027cb9 (0x3d96 bytes) 000066 1778355329.005679 trunc from 0x0346d2 to 0x034fae (0x8dd bytes) 000067 1778355329.007465 fallocate 0x030b77 thru 0x034891 (0x3d1b bytes) 000068 1778355329.009558 read 0x013276 thru 0x015da4 (0x2b2f bytes) 000070 1778355329.011244 mapread 0x0223b2 thru 0x02ed24 (0xc973 bytes) 000071 1778355329.012674 fallocate 0x03ea66 thru 0x03ffff (0x159a bytes) 000072 1778355329.014633 fallocate 0x00eea5 thru 0x01d6f9 (0xe855 bytes) 000073 1778355329.016646 read 0x024638 thru 0x02fa33 (0xb3fc bytes) 000074 1778355329.018496 MIRROR_EXTEND to 2 mirrors 000075 1778355329.211095 trunc from 0x0312b5 to 0x031447 (0x193 bytes) 000076 1778355329.221180 write 0x00759a thru 0x012faf (0xba16 bytes) 000077 1778355329.225616 read 0x02fa39 thru 0x039851 (0x9e19 bytes) 000078 1778355329.228173 read 0x03223f thru 0x03a2b0 (0x8072 bytes) 000079 1778355329.231808 mapread 0x027391 thru 0x02eda1 (0x7a11 bytes) 000080 1778355329.232975 mapwrite 0x02f709 thru 0x03c3e0 (0xccd8 bytes) 000081 1778355329.242075 trunc from 0x040000 to 0x0008c2 (0x3f73d bytes) 000082 1778355329.249811 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) 000083 1778355329.285562 mapread 0x000216 thru 0x0008c2 (0x6ad bytes) 000084 1778355329.287933 fallocate 0x00283f thru 0x00548a (0x2c4c bytes) 000085 1778355329.290228 punch from 0x003e20 to 0x00548a (0x166b bytes) 000088 1778355329.293036 punch from 0x0005ad to 0x001b0b (0x155f bytes) 000089 1778355329.294712 trunc from 0x001662 to 0x00548a (0x3e29 bytes) 000091 1778355329.296824 mapwrite 0x00acff thru 0x019fdc (0xf2de bytes) 000092 1778355329.317081 read 0x000cdb thru 0x001dd1 (0x10f7 bytes) 000093 1778355329.319017 trunc from 0x019fdd to 0x01e98e (0x49b2 bytes) 000094 1778355329.322904 read 0x019d71 thru 0x01e98e (0x4c1e bytes) 000095 1778355329.325609 write 0x0144d5 thru 0x02230f (0xde3b bytes) 000096 1778355329.326737 punch from 0x004515 to 0x00d50f (0x8ffb bytes) 000097 1778355329.328729 fallocate 0x0079bd thru 0x00dee9 (0x652d bytes) 000098 1778355329.330452 read 0x0201a8 thru 0x02230f (0x2168 bytes) 000100 1778355329.331553 punch from 0x00d061 to 0x0108a2 (0x3842 bytes) 000101 1778355329.333170 mapread 0x00714d thru 0x0089c2 (0x1876 bytes) 000102 1778355329.336218 trunc from 0x000fa0 to 0x003862 (0x28c3 bytes) 000103 1778355329.338345 mapwrite 0x00728d thru 0x00aa78 (0x37ec bytes) 000104 1778355329.344934 trunc from 0x00f060 to 0x01d2f0 (0xe291 bytes) 000105 1778355329.346692 write 0x02bd3d thru 0x02c9ee (0xcb2 bytes) 000106 1778355329.347422 fallocate 0x022f20 thru 0x02d0f5 (0xa1d6 bytes) 000107 1778355329.349495 mapwrite 0x02f00f thru 0x036932 (0x7924 bytes) 000108 1778355329.362348 trunc from 0x025b78 to 0x0289c5 (0x2e4e bytes) 000110 1778355329.365066 fallocate 0x025689 thru 0x033919 (0xe291 bytes) 000112 1778355329.367779 read 0x00231b thru 0x0059ed (0x36d3 bytes) 000113 1778355329.370784 mapwrite 0x033378 thru 0x035306 (0x1f8f bytes) 000116 1778355329.376394 mapread 0x00a3c4 thru 0x014161 (0x9d9e bytes) 000117 1778355329.377218 punch from 0x01a501 to 0x027697 (0xd197 bytes) 000118 1778355329.379049 mapwrite 0x012cd3 thru 0x0191ab (0x64d9 bytes) 000120 1778355329.385227 fallocate 0x01b206 thru 0x023c21 (0x8a1c bytes) 000122 1778355329.387896 trunc from 0x036933 to 0x0315b6 (0x537c bytes) 000123 1778355329.392706 trunc from 0x028b81 to 0x030036 (0x74b6 bytes) 000124 1778355329.394686 fallocate 0x01dd49 thru 0x026d69 (0x9021 bytes) 000125 1778355329.396803 mapread 0x01c9d7 thru 0x028015 (0xb63f bytes) 000126 1778355329.413617 trunc from 0x0315b7 to 0x014204 (0x1d3b2 bytes) 000128 1778355329.419217 punch from 0x00aac1 to 0x00e621 (0x3b61 bytes) 000130 1778355329.421980 punch from 0x00facf to 0x013d5e (0x4290 bytes) 000131 1778355329.425010 MIRROR_EXTEND to 2 mirrors 000132 1778355329.590374 punch from 0x00838a to 0x012f5a (0xabd1 bytes) 000134 1778355329.599941 read 0x00a081 thru 0x0131f7 (0x9177 bytes) 000135 1778355329.602749 trunc from 0x011769 to 0x014204 (0x2a9c bytes) 000136 1778355329.606782 trunc from 0x003c0d to 0x0046be (0xab2 bytes) 000137 1778355329.608276 trunc from 0x014205 to 0x024e56 (0x10c52 bytes) 000139 1778355329.619569 read 0x020a1e thru 0x024e56 (0x4439 bytes) 000140 1778355329.622736 mapread 0x01bfcd thru 0x024e56 (0x8e8a bytes) 000141 1778355329.623597 mapwrite 0x00ae70 thru 0x00fb74 (0x4d05 bytes) 000142 1778355329.629395 mapwrite 0x024780 thru 0x032a95 (0xe316 bytes) 000143 1778355329.643156 mapread 0x031616 thru 0x032a95 (0x1480 bytes) 000144 1778355329.643876 punch from 0x018527 to 0x01ff03 (0x79dd bytes) 000145 1778355329.645534 read 0x020fcb thru 0x02faaa (0xeae0 bytes) 000146 1778355329.645977 trunc from 0x032a96 to 0x020d7e (0x11d17 bytes) 000147 1778355329.650081 mapwrite 0x022791 thru 0x0298ed (0x715d bytes) 000148 1778355329.662472 trunc from 0x0298ee to 0x00a588 (0x1f365 bytes) 000149 1778355329.666680 mapwrite 0x0204a9 thru 0x027d4c (0x78a4 bytes) 000150 1778355329.687808 read 0x01fe01 thru 0x027d4c (0x7f4c bytes) 000151 1778355329.691111 write 0x0139f9 thru 0x018cff (0x5307 bytes) 000152 1778355329.692388 punch from 0x00369a to 0x011583 (0xdeea bytes) 000153 1778355329.694272 mapwrite 0x000e27 thru 0x009a1f (0x8bf9 bytes) 000155 1778355329.710589 trunc from 0x027d4d to 0x03d464 (0x15718 bytes) 000156 1778355329.714841 trunc from 0x0177b8 to 0x01f181 (0x79ca bytes) 000157 1778355329.716664 mapwrite 0x030a9f thru 0x03a2cc (0x982e bytes) 000158 1778355329.725782 trunc from 0x001e68 to 0x008dc7 (0x6f60 bytes) 000159 1778355329.729263 MIRROR_RESYNC 2 mirrors 000160 1778355329.935127 trunc from 0x008a43 to 0x00f6d8 (0x6c96 bytes) 000161 1778355329.943585 fallocate 0x007f06 thru 0x00851d (0x618 bytes) 000162 1778355329.945459 fallocate 0x0360ff thru 0x03ffff (0x9f01 bytes) 000163 1778355329.948454 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d465): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d465): Mirror 2: 0x8c335375 (stale) CRC-32 checksum value for chunk [0, 0x3d465): Mirror 1: 0x8982fc53 Mirror 3: 0x8982fc53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d465: skipped 000164 1778355330.226274 mapwrite 0x018fd6 thru 0x02215b (0x9186 bytes) 000165 1778355330.266854 MIRROR_RESYNC 3 mirrors 000166 1778355330.526079 trunc from 0x03d465 to 0x03d51f (0xbb bytes) 000167 1778355330.535759 mapwrite 0x015930 thru 0x01c34d (0x6a1e bytes) 000168 1778355330.546487 punch from 0x02bebf to 0x037502 (0xb644 bytes) 000169 1778355330.548236 trunc from 0x03d520 to 0x01e038 (0x1f4e7 bytes) 000170 1778355330.555990 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e039): Mirror 3: 0xaacf93a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e039): Mirror 3: 0xaacf93a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e039: skipped 000171 1778355330.631204 mapread 0x00900f thru 0x0165f3 (0xd5e5 bytes) 000172 1778355330.638966 read 0x0021ca thru 0x00c944 (0xa77b bytes) 000173 1778355330.641130 mapread 0x003ddd thru 0x006594 (0x27b8 bytes) 000175 1778355330.642369 read 0x002725 thru 0x00a081 (0x795d bytes) 000176 1778355330.642876 punch from 0x013446 to 0x016285 (0x2e40 bytes) 000177 1778355330.646087 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e039): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e039): Mirror 3: 0xaacf93a0 (stale) CRC-32 checksum value for chunk [0, 0x1e039): Mirror 1: 0xacffebab Mirror 4: 0xacffebab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e039: skipped 000179 1778355330.888556 read 0x00ba30 thru 0x01b180 (0xf751 bytes) 000180 1778355330.891491 mapwrite 0x010ef2 thru 0x017cdf (0x6dee bytes) 000181 1778355330.925958 mapwrite 0x0270ca thru 0x034e01 (0xdd38 bytes) 000183 1778355330.951109 write 0x02218c thru 0x026ea7 (0x4d1c bytes) 000184 1778355330.952072 punch from 0x001ac7 to 0x011147 (0xf681 bytes) 000185 1778355330.954053 punch from 0x0194b8 to 0x02710b (0xdc54 bytes) 000186 1778355330.957990 mapwrite 0x0209fb thru 0x028f44 (0x854a bytes) 000187 1778355330.967154 trunc from 0x01f5a0 to 0x02d1ac (0xdc0d bytes) 000188 1778355330.969125 trunc from 0x01aafd to 0x01c56a (0x1a6e bytes) 000189 1778355330.971013 read 0x01b325 thru 0x028c43 (0xd91f bytes) 000190 1778355330.974220 mapread 0x013284 thru 0x01d49a (0xa217 bytes) 000191 1778355330.975202 fallocate 0x00f6eb thru 0x01e81d (0xf133 bytes) 000192 1778355330.977286 trunc from 0x03327c to 0x034e01 (0x1b86 bytes) 000193 1778355330.979268 trunc from 0x034e02 to 0x02a1a2 (0xac5f bytes) 000194 1778355330.985026 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a1a3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1a3): Mirror 3: 0xfdb08fea (stale) Mirror 4: 0x8dc53951 (stale) CRC-32 checksum value for chunk [0, 0x2a1a3): Mirror 1: 0xd15bc0f6 Mirror 5: 0xd15bc0f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1a3: skipped 000195 1778355331.249969 write 0x02f203 thru 0x037117 (0x7f15 bytes) 000196 1778355331.261970 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37118): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37118): Mirror 3: 0x19fc78cc (stale) Mirror 4: 0x9cde96de (stale) Mirror 5: 0x31ca921b (stale) CRC-32 checksum value for chunk [0, 0x37118): Mirror 1: 0xf4f08ef4 Mirror 6: 0xf4f08ef4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37118: skipped 000197 1778355331.561604 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37118): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37118): Mirror 3: 0x19fc78cc (stale) Mirror 4: 0x9cde96de (stale) Mirror 5: 0x31ca921b (stale) CRC-32 checksum value for chunk [0, 0x37118): Mirror 1: 0xf4f08ef4 Mirror 6: 0xf4f08ef4 Mirror 7: 0xf4f08ef4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37118: skipped 000200 1778355331.836641 punch from 0x01407d to 0x016104 (0x2088 bytes) 000201 1778355331.847390 punch from 0x01c4e1 to 0x023ed9 (0x79f9 bytes) 000202 1778355331.849023 trunc from 0x02f141 to 0x035499 (0x6359 bytes) 000203 1778355331.851028 mapwrite 0x00c033 thru 0x015450 (0x941e bytes) 000204 1778355331.863866 trunc from 0x027b69 to 0x03571a (0xdbb2 bytes) 000205 1778355331.865721 trunc from 0x037118 to 0x01d4fe (0x19c19 bytes) 000206 1778355331.872623 mapwrite 0x00b908 thru 0x00c3a0 (0xa99 bytes) 000207 1778355331.877126 trunc from 0x01d4ff to 0x0207dd (0x32df bytes) 000208 1778355331.881526 punch from 0x00d064 to 0x01c79a (0xf737 bytes) 000210 1778355331.883759 fallocate 0x010230 thru 0x01893e (0x870f bytes) 000211 1778355331.885669 trunc from 0x0207de to 0x012d3b (0xdaa2 bytes) 000212 1778355331.889767 punch from 0x00d62e to 0x00ff8e (0x2961 bytes) 000213 1778355331.892910 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12d3c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12d3c): Mirror 3: 0xc248e71c (stale) Mirror 4: 0xc248e71c (stale) Mirror 5: 0xaeda2a05 (stale) Mirror 6: 0xaeda2a05 (stale) Mirror 7: 0xaeda2a05 (stale) CRC-32 checksum value for chunk [0, 0x12d3c): Mirror 1: 0xd9c21079 Mirror 8: 0xd9c21079 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12d3c: skipped 000214 1778355332.172145 read 0x00a66a thru 0x012d3b (0x86d2 bytes) 000215 1778355332.176710 fallocate 0x00a111 thru 0x019aa7 (0xf997 bytes) 000218 1778355332.188498 mapread 0x00e67a thru 0x011462 (0x2de9 bytes) 000219 1778355332.193691 fallocate 0x035257 thru 0x03b839 (0x65e3 bytes) 000220 1778355332.195585 trunc from 0x03aaf2 to 0x03b839 (0xd48 bytes) 000221 1778355332.196988 write 0x0372fa thru 0x03ac89 (0x3990 bytes) 000222 1778355332.202493 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b83a): Mirror 3: 0xd7280faa (stale) Mirror 4: 0x1b5faa20 (stale) Mirror 5: 0x3d98fe44 (stale) Mirror 7: 0xf829ed5a (stale) Mirror 8: 0x9f4ac73e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b83a): Mirror 3: 0xd7280faa (stale) Mirror 4: 0x1b5faa20 (stale) Mirror 5: 0x3d98fe44 (stale) Mirror 7: 0xf829ed5a (stale) Mirror 8: 0x9f4ac73e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b83a: skipped 000223 1778355332.309358 trunc from 0x03b83a to 0x001c4c (0x39bed bytes) 000224 1778355332.316352 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c4d): Mirror 3: 0x2d668349 (stale) Mirror 4: 0x2d668349 (stale) Mirror 5: 0xa5ae1049 (stale) Mirror 8: 0xa5ae1049 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c4d): Mirror 3: 0x2d668349 (stale) Mirror 4: 0x2d668349 (stale) Mirror 5: 0xa5ae1049 (stale) Mirror 8: 0xa5ae1049 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c4d: skipped 000226 1778355332.391312 read 0x0005bf thru 0x001c4c (0x168e bytes) 000227 1778355332.394355 MIRROR_RESYNC 5 mirrors 000228 1778355332.578596 mapwrite 0x00f3eb thru 0x011d52 (0x2968 bytes) 000229 1778355332.602300 trunc from 0x011d53 to 0x008f66 (0x8dec bytes) 000230 1778355332.609002 MIRROR_RESYNC 5 mirrors 000231 1778355332.819876 write 0x020dc6 thru 0x0248d5 (0x3b10 bytes) 000232 1778355332.829272 trunc from 0x0131e5 to 0x013db1 (0xbcd bytes) 000233 1778355332.832546 MIRROR_RESYNC 5 mirrors 000234 1778355333.060776 read 0x00446a thru 0x011654 (0xd1eb bytes) 000235 1778355333.065010 MIRROR_RESYNC 5 mirrors 000236 1778355333.207252 fallocate 0x01abe5 thru 0x023cb4 (0x90d0 bytes) 000237 1778355333.217546 mapwrite 0x00e86c thru 0x01be2a (0xd5bf bytes) 000238 1778355333.231993 mapwrite 0x0001dd thru 0x002445 (0x2269 bytes) 000239 1778355333.238875 mapread 0x021cf1 thru 0x0248d5 (0x2be5 bytes) 000240 1778355333.242881 MIRROR_RESYNC 5 mirrors 000241 1778355333.471190 trunc from 0x020aa7 to 0x0248d5 (0x3e2f bytes) 000242 1778355333.479517 mapwrite 0x034601 thru 0x03be68 (0x7868 bytes) 000243 1778355333.499627 trunc from 0x03be69 to 0x02d5cb (0xe89d bytes) 000244 1778355333.505827 MIRROR_RESYNC 5 mirrors 000246 1778355333.800250 fallocate 0x00a85f thru 0x00cf5b (0x26fd bytes) 000247 1778355333.810103 mapwrite 0x0354fd thru 0x035d8e (0x892 bytes) 000248 1778355333.825361 punch from 0x008467 to 0x00e235 (0x5dcf bytes) 000249 1778355333.827502 mapread 0x023c3e thru 0x0286c6 (0x4a89 bytes) 000250 1778355333.832685 trunc from 0x035d8f to 0x0172e4 (0x1eaaa bytes) 000253 1778355333.838723 mapwrite 0x026bfe thru 0x0348ac (0xdcaf bytes) 000254 1778355333.855783 mapread 0x01fe30 thru 0x029ea0 (0xa071 bytes) 000255 1778355333.857152 trunc from 0x01c39e to 0x01dd91 (0x19f4 bytes) 000256 1778355333.859107 mapread 0x026b8b thru 0x02eeef (0x8365 bytes) 000257 1778355333.860147 read 0x02477c thru 0x0282ab (0x3b30 bytes) 000258 1778355333.860658 trunc from 0x0348ad to 0x03da5a (0x91ae bytes) 000259 1778355333.865030 mapwrite 0x0332a2 thru 0x03d40f (0xa16e bytes) 000260 1778355333.874815 trunc from 0x03da5b to 0x03772b (0x632f bytes) 000261 1778355333.879753 punch from 0x0348f4 to 0x03772b (0x2e38 bytes) 000262 1778355333.883243 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3772c): Mirror 3: 0xa4c658a9 (stale) Mirror 4: 0xa4c658a9 (stale) Mirror 8: 0xa4c658a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3772c): Mirror 3: 0xa4c658a9 (stale) Mirror 4: 0xa4c658a9 (stale) Mirror 8: 0xa4c658a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3772c: skipped 000263 1778355333.977285 MIRROR_RESYNC 4 mirrors 000264 1778355334.250545 punch from 0x018457 to 0x01b2f7 (0x2ea1 bytes) 000265 1778355334.261322 read 0x02decc thru 0x0306a3 (0x27d8 bytes) 000266 1778355334.263620 trunc from 0x033ee1 to 0x03772b (0x384b bytes) 000267 1778355334.265645 read 0x036ddc thru 0x03772b (0x950 bytes) 000269 1778355334.269766 punch from 0x0155d9 to 0x01aa18 (0x5440 bytes) 000270 1778355334.271546 mapread 0x021a90 thru 0x02fcaa (0xe21b bytes) 000272 1778355334.273012 trunc from 0x01f4fa to 0x0257de (0x62e5 bytes) 000273 1778355334.274950 fallocate 0x017d12 thru 0x01895d (0xc4c bytes) 000274 1778355334.277027 mapwrite 0x032a2f thru 0x03f631 (0xcc03 bytes) 000275 1778355334.293961 mapread 0x0222ac thru 0x02ecf5 (0xca4a bytes) 000276 1778355334.300712 mapwrite 0x01de37 thru 0x023d86 (0x5f50 bytes) 000277 1778355334.311917 write 0x01a438 thru 0x01e3da (0x3fa3 bytes) 000278 1778355334.314001 write 0x01f13c thru 0x026b34 (0x79f9 bytes) 000279 1778355334.314912 punch from 0x03b2ed to 0x03f631 (0x4345 bytes) 000280 1778355334.316810 trunc from 0x03f632 to 0x02e149 (0x114e8 bytes) 000281 1778355334.322351 trunc from 0x009701 to 0x0168b9 (0xd1b9 bytes) 000282 1778355334.324323 read 0x016b88 thru 0x02632e (0xf7a7 bytes) 000283 1778355334.329071 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e14a): Mirror 3: 0x4153999a (stale) Mirror 4: 0x4153999a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e14a): Mirror 3: 0x4153999a (stale) Mirror 4: 0x4153999a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e14a: skipped 000285 1778355334.414566 mapread 0x0081e5 thru 0x01061a (0x8436 bytes) 000286 1778355334.419759 read 0x02b8b6 thru 0x02e149 (0x2894 bytes) 000287 1778355334.421734 fallocate 0x01e281 thru 0x029af0 (0xb870 bytes) 000288 1778355334.423655 trunc from 0x01c9f5 to 0x02afa9 (0xe5b5 bytes) 000289 1778355334.425517 mapwrite 0x03b5fb thru 0x03c2e9 (0xcef bytes) 000290 1778355334.436829 mapread 0x003521 thru 0x00e67e (0xb15e bytes) 000292 1778355334.438270 fallocate 0x032f69 thru 0x03b147 (0x81df bytes) 000293 1778355334.440258 mapread 0x00c49f thru 0x018faf (0xcb11 bytes) 000294 1778355334.443228 mapread 0x00f29e thru 0x01c915 (0xd678 bytes) 000295 1778355334.446699 trunc from 0x03c2ea to 0x03034c (0xbf9d bytes) 000297 1778355334.451539 read 0x02ecaf thru 0x03034c (0x169e bytes) 000298 1778355334.454359 mapread 0x02e511 thru 0x03034c (0x1e3c bytes) 000299 1778355334.455171 mapread 0x0189d5 thru 0x018cd9 (0x305 bytes) 000300 1778355334.456054 fallocate 0x00f2d2 thru 0x01eb11 (0xf840 bytes) 000301 1778355334.458192 punch from 0x0059e4 to 0x006ed3 (0x14f0 bytes) 000302 1778355334.459875 trunc from 0x01417b to 0x02050c (0xc392 bytes) 000303 1778355334.461646 fallocate 0x0392d1 thru 0x039c9f (0x9cf bytes) 000304 1778355334.463345 write 0x03fa25 thru 0x03ffff (0x5db bytes) 000306 1778355334.464504 mapwrite 0x03333e thru 0x036cf4 (0x39b7 bytes) 000307 1778355334.472704 mapwrite 0x00ab2a thru 0x0131eb (0x86c2 bytes) 000308 1778355334.479037 punch from 0x00e447 to 0x017670 (0x922a bytes) 000309 1778355334.480721 write 0x03d2d7 thru 0x03ffff (0x2d29 bytes) 000310 1778355334.481535 trunc from 0x008072 to 0x015dd5 (0xdd64 bytes) 000311 1778355334.483662 punch from 0x02f1c7 to 0x032131 (0x2f6b bytes) 000312 1778355334.485286 fallocate 0x000ba5 thru 0x010a03 (0xfe5f bytes) 000313 1778355334.487341 trunc from 0x038193 to 0x03ffff (0x7e6d bytes) 000314 1778355334.490234 trunc from 0x040000 to 0x00cef5 (0x3310a bytes) 000315 1778355334.494766 fallocate 0x0216ad thru 0x02758f (0x5ee3 bytes) 000316 1778355334.496766 read 0x00f4ec thru 0x01ea54 (0xf569 bytes) 000317 1778355334.500569 punch from 0x01cd96 to 0x021e1e (0x5089 bytes) 000318 1778355334.502476 mapwrite 0x03854e thru 0x039d16 (0x17c9 bytes) 000319 1778355334.513001 punch from 0x02eb76 to 0x039b3d (0xafc8 bytes) 000320 1778355334.514919 trunc from 0x039d17 to 0x0060d0 (0x33c46 bytes) 000321 1778355334.519199 punch from 0x005d96 to 0x0060d0 (0x33b bytes) 000322 1778355334.520814 fallocate 0x011f95 thru 0x013d6f (0x1ddb bytes) 000323 1778355334.522732 punch from 0x00ef92 to 0x013d6f (0x4dde bytes) 000324 1778355334.524357 mapwrite 0x0099f9 thru 0x00dece (0x44d6 bytes) 000325 1778355334.537121 mapwrite 0x02357d thru 0x031a6a (0xe4ee bytes) 000326 1778355334.569320 punch from 0x03051d to 0x031a6a (0x154e bytes) 000327 1778355334.571356 mapread 0x01128f thru 0x01cedd (0xbc4f bytes) 000328 1778355334.587231 mapread 0x028589 thru 0x031a6a (0x94e2 bytes) 000329 1778355334.591722 trunc from 0x031a6b to 0x004416 (0x2d654 bytes) 000330 1778355334.597056 punch from 0x00318f to 0x004416 (0x1288 bytes) 000331 1778355334.598583 fallocate 0x024d8d thru 0x0259fe (0xc72 bytes) 000332 1778355334.600516 read 0x0041fd thru 0x004416 (0x21a bytes) 000333 1778355334.602622 trunc from 0x003c72 to 0x004416 (0x7a5 bytes) 000334 1778355334.605493 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x4417): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4417): Mirror 3: 0x9db0e9d5 (stale) Mirror 4: 0x9db0e9d5 (stale) CRC-32 checksum value for chunk [0, 0x4417): Mirror 1: 0x9db0e9d5 Mirror 5: 0x9db0e9d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4417: skipped 000335 1778355334.827558 write 0x00e677 thru 0x019635 (0xafbf bytes) 000336 1778355334.837919 write 0x01ff14 thru 0x025b8a (0x5c77 bytes) 000337 1778355334.838799 mapread 0x0047f1 thru 0x011c8c (0xd49c bytes) 000339 1778355334.844404 write 0x012fbf thru 0x01cd02 (0x9d44 bytes) 000340 1778355334.846351 fallocate 0x000e5a thru 0x00554e (0x46f5 bytes) 000341 1778355334.848123 fallocate 0x01cd35 thru 0x0260f3 (0x93bf bytes) 000342 1778355334.849811 mapwrite 0x00f533 thru 0x0184b1 (0x8f7f bytes) 000343 1778355334.856076 trunc from 0x005881 to 0x00e01d (0x879d bytes) 000344 1778355334.857823 read 0x00cca3 thru 0x0148c7 (0x7c25 bytes) 000345 1778355334.859626 trunc from 0x013716 to 0x018e13 (0x56fe bytes) 000346 1778355334.861246 write 0x039ccb thru 0x03ffff (0x6335 bytes) 000347 1778355334.862322 read 0x02a9ac thru 0x03a076 (0xf6cb bytes) 000348 1778355334.866721 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x86bb8bc9 (stale) Mirror 4: 0x86bb8bc9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x86bb8bc9 (stale) Mirror 4: 0x86bb8bc9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000350 1778355334.953459 MIRROR_RESYNC 3 mirrors 000351 1778355335.216130 punch from 0x00630b to 0x00bffb (0x5cf1 bytes) 000352 1778355335.225776 write 0x03f226 thru 0x03ffff (0xdda bytes) 000353 1778355335.228210 read 0x036125 thru 0x03ffff (0x9edb bytes) 000355 1778355335.232241 mapwrite 0x02d376 thru 0x037875 (0xa500 bytes) 000356 1778355335.243375 fallocate 0x012042 thru 0x01bb52 (0x9b11 bytes) 000357 1778355335.245210 read 0x00fa95 thru 0x01b9e8 (0xbf54 bytes) 000358 1778355335.248491 mapwrite 0x00c22f thru 0x01419f (0x7f71 bytes) 000359 1778355335.255168 trunc from 0x040000 to 0x02dda2 (0x1225d bytes) 000360 1778355335.261063 trunc from 0x02bbec to 0x02dda2 (0x21b7 bytes) 000362 1778355335.263064 fallocate 0x028392 thru 0x02d6e1 (0x5350 bytes) 000363 1778355335.264793 mapwrite 0x01bc12 thru 0x01d681 (0x1a70 bytes) 000364 1778355335.271555 MIRROR_RESYNC 3 mirrors 000365 1778355335.520335 mapwrite 0x0249ad thru 0x0334b9 (0xeb0d bytes) 000366 1778355335.550526 mapwrite 0x025376 thru 0x02b4c2 (0x614d bytes) 000367 1778355335.556875 mapwrite 0x013d34 thru 0x01d6ad (0x997a bytes) 000369 1778355335.568293 fallocate 0x0107a4 thru 0x019a17 (0x9274 bytes) 000370 1778355335.571764 MIRROR_RESYNC 3 mirrors 000371 1778355335.773606 mapread 0x00acd7 thru 0x0175bc (0xc8e6 bytes) 000372 1778355335.781267 read 0x016217 thru 0x0206a7 (0xa491 bytes) 000373 1778355335.783059 mapread 0x006f17 thru 0x00c41f (0x5509 bytes) 000374 1778355335.784788 MIRROR_EXTEND to 4 mirrors 000375 1778355335.970431 punch from 0x018461 to 0x0271f4 (0xed94 bytes) 000378 1778355335.981590 punch from 0x0203d4 to 0x02c707 (0xc334 bytes) 000380 1778355335.985524 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x334ba): Mirror 4: 0xbc30efc7 (stale) Mirror 5: 0xbc30efc7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334ba): Mirror 4: 0xbc30efc7 (stale) Mirror 5: 0xbc30efc7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334ba: skipped 000381 1778355336.061542 mapread 0x010505 thru 0x018909 (0x8405 bytes) 000382 1778355336.068339 MIRROR_RESYNC 3 mirrors 000383 1778355336.285386 mapread 0x013122 thru 0x021ed4 (0xedb3 bytes) 000384 1778355336.293228 mapwrite 0x007f51 thru 0x00e1e9 (0x6299 bytes) 000385 1778355336.324733 punch from 0x0058f6 to 0x00974e (0x3e59 bytes) 000386 1778355336.326608 read 0x01b77e thru 0x01d943 (0x21c6 bytes) 000387 1778355336.330291 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x334ba): Mirror 5: 0x88f6990f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334ba): Mirror 5: 0x88f6990f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334ba: skipped 000388 1778355336.405201 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) 000389 1778355336.442187 trunc from 0x00b65d to 0x00bcb9 (0x65d bytes) 000390 1778355336.443933 mapread 0x02242b thru 0x02889a (0x6470 bytes) 000391 1778355336.450138 fallocate 0x0219c5 thru 0x02bfac (0xa5e8 bytes) 000392 1778355336.451908 write 0x021866 thru 0x024f6d (0x3708 bytes) 000393 1778355336.453769 read 0x00c6ef thru 0x0117b3 (0x50c5 bytes) 000395 1778355336.456927 trunc from 0x020480 to 0x02c96c (0xc4ed bytes) 000396 1778355336.460166 trunc from 0x02cab0 to 0x0334b9 (0x6a0a bytes) 000397 1778355336.461934 trunc from 0x0334ba to 0x015f7d (0x1d53c bytes) 000398 1778355336.466291 punch from 0x0147f4 to 0x015f7d (0x178a bytes) 000400 1778355336.468272 write 0x011a13 thru 0x016873 (0x4e61 bytes) 000401 1778355336.469119 write 0x01b9b0 thru 0x021f2c (0x657d bytes) 000402 1778355336.470222 fallocate 0x0071da thru 0x00f765 (0x858c bytes) 000403 1778355336.472294 fallocate 0x02938c thru 0x02a0b6 (0xd2b bytes) 000404 1778355336.474271 fallocate 0x027d26 thru 0x02fdda (0x80b5 bytes) 000405 1778355336.476390 write 0x003e11 thru 0x00fdd1 (0xbfc1 bytes) 000406 1778355336.479797 fallocate 0x001dc5 thru 0x009e1a (0x8056 bytes) 000407 1778355336.481677 trunc from 0x02fddb to 0x03d3e2 (0xd608 bytes) 000409 1778355336.486698 trunc from 0x00ae38 to 0x00ddcd (0x2f96 bytes) 000410 1778355336.491126 read 0x026058 thru 0x032f03 (0xceac bytes) 000412 1778355336.495279 fallocate 0x0249fa thru 0x02a8fa (0x5f01 bytes) 000413 1778355336.498562 MIRROR_EXTEND to 2 mirrors 000414 1778355336.687224 punch from 0x02c098 to 0x03843a (0xc3a3 bytes) 000415 1778355336.697312 read 0x00696c thru 0x01613b (0xf7d0 bytes) 000416 1778355336.701909 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) 000417 1778355336.736646 mapread 0x005bbf thru 0x011368 (0xb7aa bytes) 000418 1778355336.744635 read 0x025d6a thru 0x034a5d (0xecf4 bytes) 000419 1778355336.747812 read 0x02d23c thru 0x031311 (0x40d6 bytes) 000420 1778355336.748542 trunc from 0x03d3e3 to 0x020d04 (0x1c6de bytes) 000421 1778355336.759140 read 0x02022b thru 0x020d04 (0xada bytes) 000422 1778355336.761852 read 0x017f7c thru 0x020d04 (0x8d89 bytes) 000423 1778355336.762280 trunc from 0x011e36 to 0x01ef98 (0xd163 bytes) 000424 1778355336.765309 MIRROR_EXTEND to 2 mirrors 000425 1778355336.941491 mapwrite 0x024e03 thru 0x02d4c8 (0x86c6 bytes) 000426 1778355336.968694 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d4c9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d4c9): Mirror 2: 0x440c25e6 (stale) CRC-32 checksum value for chunk [0, 0x2d4c9): Mirror 1: 0x9cdba55 Mirror 3: 0x9cdba55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d4c9: skipped 000427 1778355337.226019 mapread 0x026f2b thru 0x02d4c8 (0x659e bytes) 000428 1778355337.231703 fallocate 0x01e267 thru 0x02d080 (0xee1a bytes) 000429 1778355337.242062 mapwrite 0x011781 thru 0x01e4b8 (0xcd38 bytes) 000431 1778355337.256531 MIRROR_RESYNC 3 mirrors 000432 1778355337.471429 read 0x021be8 thru 0x0246e8 (0x2b01 bytes) 000434 1778355337.474798 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) 000435 1778355337.571467 write 0x01ccac thru 0x01e3f2 (0x1747 bytes) 000436 1778355337.584050 fallocate 0x0318df thru 0x037ef2 (0x6614 bytes) 000437 1778355337.585888 trunc from 0x037ef3 to 0x0328c2 (0x5630 bytes) 000438 1778355337.594716 write 0x023f2b thru 0x031ee3 (0xdfb9 bytes) 000439 1778355337.598746 trunc from 0x0328c3 to 0x025e4b (0xca77 bytes) 000440 1778355337.603290 trunc from 0x025e4c to 0x01ae03 (0xb048 bytes) 000441 1778355337.607666 read 0x018c8d thru 0x01ae03 (0x2177 bytes) 000442 1778355337.609328 punch from 0x013cbc to 0x01ae03 (0x7148 bytes) 000443 1778355337.611128 trunc from 0x0090b2 to 0x0180d7 (0xf026 bytes) 000444 1778355337.612822 trunc from 0x01ae04 to 0x009158 (0x11cab bytes) 000445 1778355337.616909 write 0x0204d6 thru 0x02ba97 (0xb5c2 bytes) 000446 1778355337.617977 read 0x026c06 thru 0x02ba97 (0x4e92 bytes) 000447 1778355337.618546 trunc from 0x0015ae to 0x001ba5 (0x5f8 bytes) 000448 1778355337.620164 write 0x0009ef thru 0x00d7ac (0xcdbe bytes) 000449 1778355337.623781 write 0x008038 thru 0x011ace (0x9a97 bytes) 000451 1778355337.625012 fallocate 0x035276 thru 0x03e87d (0x9608 bytes) 000452 1778355337.626890 trunc from 0x020f94 to 0x024c1b (0x3c88 bytes) 000453 1778355337.630998 mapread 0x003256 thru 0x004b2a (0x18d5 bytes) 000454 1778355337.633082 MIRROR_RESYNC 2 mirrors 000455 1778355337.811063 write 0x0337b6 thru 0x03ffff (0xc84a bytes) 000456 1778355337.819485 read 0x029467 thru 0x035f38 (0xcad2 bytes) 000457 1778355337.822133 fallocate 0x0092b5 thru 0x016171 (0xcebd bytes) 000458 1778355337.825589 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x248d69ef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf91488e1 Mirror 4: 0xf91488e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000459 1778355338.074289 mapread 0x022352 thru 0x02f8f3 (0xd5a2 bytes) 000460 1778355338.083084 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x248d69ef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf91488e1 Mirror 4: 0xf91488e1 Mirror 5: 0xf91488e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000461 1778355338.344556 fallocate 0x015a79 thru 0x01ab8a (0x5112 bytes) 000462 1778355338.355584 punch from 0x0239d0 to 0x025fd0 (0x2601 bytes) 000463 1778355338.357767 punch from 0x009341 to 0x012735 (0x93f5 bytes) 000464 1778355338.359798 trunc from 0x040000 to 0x0148d6 (0x2b729 bytes) 000465 1778355338.367737 punch from 0x008791 to 0x011187 (0x89f7 bytes) 000466 1778355338.369589 mapread 0x010fc7 thru 0x0148d6 (0x3910 bytes) 000467 1778355338.377580 MIRROR_RESYNC 4 mirrors 000468 1778355338.571396 mapread 0x013ade thru 0x0148d6 (0xdf9 bytes) 000469 1778355338.574877 mapwrite 0x010bbb thru 0x01e5f4 (0xda3a bytes) 000470 1778355338.604901 read 0x00ab34 thru 0x016c0e (0xc0db bytes) 000471 1778355338.605607 read 0x01e0b5 thru 0x01e5f4 (0x540 bytes) 000472 1778355338.606281 mapwrite 0x01e9ad thru 0x022140 (0x3794 bytes) 000473 1778355338.620232 punch from 0x0181bb to 0x022140 (0x9f86 bytes) 000474 1778355338.621932 mapwrite 0x032b42 thru 0x034a83 (0x1f42 bytes) 000475 1778355338.633433 mapwrite 0x00eb52 thru 0x015030 (0x64df bytes) 000476 1778355338.641178 fallocate 0x03715d thru 0x03939a (0x223e bytes) 000477 1778355338.643143 mapwrite 0x027b11 thru 0x0373e4 (0xf8d4 bytes) 000478 1778355338.657142 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3939b): Mirror 3: 0x815fead3 (stale) Mirror 4: 0x815fead3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3939b): Mirror 3: 0x815fead3 (stale) Mirror 4: 0x815fead3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3939b: skipped 000479 1778355338.746288 punch from 0x0205f7 to 0x02f14b (0xeb55 bytes) 000480 1778355338.748557 mapwrite 0x0292a7 thru 0x032785 (0x94df bytes) 000481 1778355338.762547 MIRROR_RESYNC 3 mirrors 000482 1778355338.971546 trunc from 0x03939b to 0x03f8d4 (0x653a bytes) 000483 1778355338.987564 punch from 0x016049 to 0x0243f2 (0xe3aa bytes) 000484 1778355338.989386 read 0x0181be thru 0x01be9b (0x3cde bytes) 000485 1778355338.991989 punch from 0x010f2b to 0x01cf55 (0xc02b bytes) 000486 1778355338.993707 write 0x02fa7f thru 0x031913 (0x1e95 bytes) 000487 1778355338.996680 mapwrite 0x023dd7 thru 0x0308c4 (0xcaee bytes) 000488 1778355339.009307 trunc from 0x03f8d5 to 0x02da13 (0x11ec1 bytes) 000489 1778355339.015295 fallocate 0x015bdb thru 0x025aef (0xff15 bytes) 000490 1778355339.018580 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2da14): Mirror 4: 0x5fac6dbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da14): Mirror 4: 0x5fac6dbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da14: skipped 000491 1778355339.095790 read 0x0295c9 thru 0x02ba24 (0x245c bytes) 000492 1778355339.097772 trunc from 0x003e9d to 0x00bd35 (0x7e99 bytes) 000493 1778355339.099475 mapwrite 0x02a43f thru 0x02bbc0 (0x1782 bytes) 000494 1778355339.104079 read 0x01456a thru 0x01af7a (0x6a11 bytes) 000495 1778355339.107159 punch from 0x009285 to 0x012da1 (0x9b1d bytes) 000496 1778355339.108970 read 0x0096ba thru 0x00ca12 (0x3359 bytes) 000497 1778355339.111842 punch from 0x011477 to 0x011a8d (0x617 bytes) 000498 1778355339.113470 mapwrite 0x02c497 thru 0x037f7c (0xbae6 bytes) 000500 1778355339.128806 write 0x001a26 thru 0x006980 (0x4f5b bytes) 000501 1778355339.129836 read 0x0070ab thru 0x008a30 (0x1986 bytes) 000502 1778355339.130289 read 0x00ff94 thru 0x011098 (0x1105 bytes) 000503 1778355339.132305 trunc from 0x037f7d to 0x03a009 (0x208d bytes) 000504 1778355339.137668 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a00a): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a00a): Mirror 4: 0x1314a22a (stale) CRC-32 checksum value for chunk [0, 0x3a00a): Mirror 1: 0xf14b5b08 Mirror 5: 0xf14b5b08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a00a: skipped 000505 1778355339.383030 trunc from 0x03a00a to 0x0295a8 (0x10a61 bytes) 000506 1778355339.400367 punch from 0x01e0a2 to 0x0295a8 (0xb507 bytes) 000507 1778355339.402238 fallocate 0x03df1d thru 0x03ffff (0x20e3 bytes) 000508 1778355339.404169 trunc from 0x01e35c to 0x0205d7 (0x227c bytes) 000509 1778355339.405772 mapwrite 0x0367cf thru 0x03ffff (0x9831 bytes) 000510 1778355339.423808 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9242954c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9242954c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1778355339.500737 read 0x038cfd thru 0x03acf4 (0x1ff8 bytes) 000512 1778355339.503130 trunc from 0x040000 to 0x02c690 (0x1396f bytes) 000513 1778355339.508820 punch from 0x02ac1c to 0x02c690 (0x1a75 bytes) 000514 1778355339.510506 mapread 0x023347 thru 0x02c690 (0x934a bytes) 000516 1778355339.516489 mapwrite 0x02072d thru 0x0275ac (0x6e80 bytes) 000517 1778355339.523947 MIRROR_RESYNC 2 mirrors 000518 1778355339.711771 write 0x003da9 thru 0x00e1d0 (0xa428 bytes) 000519 1778355339.723631 trunc from 0x02556f to 0x02c690 (0x7122 bytes) 000520 1778355339.725581 read 0x018648 thru 0x023fbd (0xb976 bytes) 000521 1778355339.728467 trunc from 0x02c691 to 0x02a525 (0x216b bytes) 000522 1778355339.738825 write 0x0272d9 thru 0x02c7a7 (0x54cf bytes) 000523 1778355339.742035 mapread 0x00e59e thru 0x01967c (0xb0df bytes) 000524 1778355339.745010 mapwrite 0x030d4f thru 0x037629 (0x68db bytes) 000525 1778355339.758172 punch from 0x0020f1 to 0x0051dc (0x30ec bytes) 000526 1778355339.760055 write 0x015514 thru 0x01a0af (0x4b9c bytes) 000528 1778355339.761682 fallocate 0x031902 thru 0x03ffff (0xe6fe bytes) 000529 1778355339.763666 trunc from 0x03762a to 0x03ff37 (0x890e bytes) 000530 1778355339.768108 fallocate 0x031f2a thru 0x037f11 (0x5fe8 bytes) 000531 1778355339.769937 punch from 0x030835 to 0x03322b (0x29f7 bytes) 000532 1778355339.772177 mapread 0x03665a thru 0x03b6a5 (0x504c bytes) 000533 1778355339.775703 mapwrite 0x0243a4 thru 0x02457f (0x1dc bytes) 000535 1778355339.781483 write 0x03967e thru 0x03ffff (0x6982 bytes) 000536 1778355339.782277 mapread 0x0082c2 thru 0x008a54 (0x793 bytes) 000537 1778355339.782881 fallocate 0x02841e thru 0x0330cb (0xacae bytes) 000538 1778355339.784730 trunc from 0x02994a to 0x038fde (0xf695 bytes) 000539 1778355339.786938 trunc from 0x032b4d to 0x03ffff (0xd4b3 bytes) 000540 1778355339.792229 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa7080257 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x249251be Mirror 6: 0x249251be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000541 1778355340.061476 trunc from 0x040000 to 0x030e96 (0xf169 bytes) 000542 1778355340.079140 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30e97): Mirror 6: 0xd692b95b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e97): Mirror 6: 0xd692b95b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e97: skipped 000543 1778355340.160746 read 0x02a385 thru 0x030e96 (0x6b12 bytes) 000544 1778355340.163834 read 0x022988 thru 0x030e96 (0xe50f bytes) 000545 1778355340.168401 MIRROR_RESYNC 2 mirrors 000546 1778355340.392166 mapwrite 0x03d6ca thru 0x03ffff (0x2936 bytes) 000547 1778355340.416357 trunc from 0x040000 to 0x027533 (0x18acc bytes) 000548 1778355340.421839 punch from 0x0063a9 to 0x00b995 (0x55ed bytes) 000549 1778355340.423809 read 0x01f31f thru 0x022cad (0x398f bytes) 000551 1778355340.426705 read 0x0171ba thru 0x022928 (0xb76f bytes) 000552 1778355340.431320 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) 000553 1778355340.467566 trunc from 0x027534 to 0x012db8 (0x1477b bytes) 000554 1778355340.473234 read 0x002ccf thru 0x00e651 (0xb983 bytes) 000556 1778355340.476651 write 0x0002bc thru 0x000e5e (0xba3 bytes) 000557 1778355340.478532 punch from 0x011922 to 0x012db8 (0x1497 bytes) 000558 1778355340.480110 mapread 0x002550 thru 0x00fecb (0xd97c bytes) 000559 1778355340.482233 mapread 0x011fc8 thru 0x012db8 (0xdf1 bytes) 000560 1778355340.486262 trunc from 0x012db9 to 0x02b80f (0x18a57 bytes) 000561 1778355340.490735 mapread 0x022299 thru 0x024116 (0x1e7e bytes) 000562 1778355340.493780 write 0x01e112 thru 0x026f6a (0x8e59 bytes) 000563 1778355340.494675 fallocate 0x0180fd thru 0x01d976 (0x587a bytes) 000564 1778355340.496451 trunc from 0x02b810 to 0x003526 (0x282e9 bytes) 000565 1778355340.501018 trunc from 0x002c3f to 0x002d95 (0x157 bytes) 000566 1778355340.502941 trunc from 0x002852 to 0x003526 (0xcd5 bytes) 000567 1778355340.504559 punch from 0x0033dd to 0x003526 (0x14a bytes) 000568 1778355340.506252 read 0x0024dd thru 0x003526 (0x104a bytes) 000569 1778355340.508602 trunc from 0x003527 to 0x0295cd (0x260a7 bytes) 000570 1778355340.513250 fallocate 0x0001c3 thru 0x00838c (0x81ca bytes) 000571 1778355340.515730 trunc from 0x002fa3 to 0x003bec (0xc4a bytes) 000572 1778355340.517441 trunc from 0x017935 to 0x022226 (0xa8f2 bytes) 000573 1778355340.519163 mapread 0x020f0f thru 0x0273e5 (0x64d7 bytes) 000575 1778355340.523931 trunc from 0x01fff2 to 0x0295cd (0x95dc bytes) 000576 1778355340.525841 mapwrite 0x02a6ca thru 0x02c66a (0x1fa1 bytes) 000579 1778355340.538712 mapwrite 0x01f276 thru 0x0209af (0x173a bytes) 000580 1778355340.546712 write 0x03dac9 thru 0x03ffff (0x2537 bytes) 000582 1778355340.548295 read 0x03b541 thru 0x03ffff (0x4abf bytes) 000583 1778355340.551637 fallocate 0x01e5ba thru 0x01fb01 (0x1548 bytes) 000584 1778355340.553567 mapread 0x02374a thru 0x02a7c5 (0x707c bytes) 000585 1778355340.554515 write 0x009687 thru 0x00e403 (0x4d7d bytes) 000587 1778355340.555924 write 0x015025 thru 0x01f1fc (0xa1d8 bytes) 000588 1778355340.556829 trunc from 0x01021a to 0x014c24 (0x4a0b bytes) 000589 1778355340.559003 mapwrite 0x03f998 thru 0x03ffff (0x668 bytes) 000591 1778355340.564744 mapwrite 0x02ede4 thru 0x0326ec (0x3909 bytes) 000592 1778355340.569786 write 0x025411 thru 0x035166 (0xfd56 bytes) 000594 1778355340.571206 punch from 0x0000aa to 0x008958 (0x88af bytes) 000595 1778355340.573071 fallocate 0x00fcc2 thru 0x0132b6 (0x35f5 bytes) 000596 1778355340.575001 read 0x015e34 thru 0x018481 (0x264e bytes) 000597 1778355340.575463 read 0x03c817 thru 0x03ffff (0x37e9 bytes) 000598 1778355340.577397 MIRROR_EXTEND to 2 mirrors 000599 1778355340.783827 MIRROR_RESYNC 2 mirrors 000600 1778355340.906357 trunc from 0x040000 to 0x03f4a2 (0xb5d bytes) 000601 1778355340.922499 trunc from 0x03f4a3 to 0x0188eb (0x26bb7 bytes) 000603 1778355340.928535 punch from 0x002427 to 0x00f174 (0xcd4e bytes) 000604 1778355340.930128 fallocate 0x02f0df thru 0x03e56e (0xf490 bytes) 000605 1778355340.931984 trunc from 0x03e56f to 0x01a2cf (0x2429f bytes) 000606 1778355340.936558 punch from 0x00e666 to 0x011649 (0x2fe4 bytes) 000607 1778355340.938124 punch from 0x014b9d to 0x01a2cf (0x5733 bytes) 000608 1778355340.940033 write 0x033042 thru 0x03d89c (0xa85b bytes) 000609 1778355340.941305 read 0x01c06d thru 0x025251 (0x91e5 bytes) 000610 1778355340.944014 punch from 0x02463f to 0x028d94 (0x4756 bytes) 000612 1778355340.945986 trunc from 0x03d89d to 0x03e45b (0xbbf bytes) 000613 1778355340.950315 fallocate 0x032106 thru 0x0324d5 (0x3d0 bytes) 000614 1778355340.952181 mapread 0x00ad72 thru 0x016f77 (0xc206 bytes) 000615 1778355340.960165 trunc from 0x026e97 to 0x02eb64 (0x7cce bytes) 000616 1778355340.963345 MIRROR_RESYNC 2 mirrors 000617 1778355341.145470 trunc from 0x01a5a5 to 0x01be64 (0x18c0 bytes) 000619 1778355341.155162 trunc from 0x03822b to 0x03ad31 (0x2b07 bytes) 000620 1778355341.158491 MIRROR_RESYNC 2 mirrors 000622 1778355341.352266 write 0x029bdc thru 0x02f19d (0x55c2 bytes) 000623 1778355341.362641 write 0x01a98b thru 0x026e83 (0xc4f9 bytes) 000624 1778355341.368844 MIRROR_RESYNC 2 mirrors 000625 1778355341.602554 MIRROR_EXTEND to 3 mirrors 000626 1778355341.829096 fallocate 0x02a519 thru 0x0307f9 (0x62e1 bytes) 000627 1778355341.839618 trunc from 0x00feb1 to 0x0112e8 (0x1438 bytes) 000629 1778355341.841793 punch from 0x0033d4 to 0x010a18 (0xd645 bytes) 000630 1778355341.843629 trunc from 0x03e45c to 0x0285b5 (0x15ea6 bytes) 000631 1778355341.850622 mapwrite 0x0264d3 thru 0x0300aa (0x9bd8 bytes) 000632 1778355341.867372 trunc from 0x0300ab to 0x02bce2 (0x43c8 bytes) 000633 1778355341.872039 mapread 0x022738 thru 0x0238e3 (0x11ac bytes) 000634 1778355341.873162 mapwrite 0x002871 thru 0x00f257 (0xc9e7 bytes) 000635 1778355341.883865 mapread 0x00eb57 thru 0x016b35 (0x7fdf bytes) 000636 1778355341.886784 trunc from 0x02b7a7 to 0x02bce2 (0x53c bytes) 000637 1778355341.888447 trunc from 0x02bce3 to 0x0009f0 (0x2b2f2 bytes) 000639 1778355341.893872 trunc from 0x000528 to 0x0009f0 (0x4c9 bytes) 000640 1778355341.895583 mapread 0x0001d6 thru 0x0009f0 (0x81b bytes) 000641 1778355341.897953 punch from 0x00083c to 0x0009f0 (0x1b5 bytes) 000642 1778355341.900909 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9f1): Mirror 3: 0x4f9c5008 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f1): Mirror 3: 0x4f9c5008 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f1: skipped 000643 1778355341.972914 trunc from 0x0009f1 to 0x005d7d (0x538d bytes) 000644 1778355341.978091 write 0x02a0b7 thru 0x03933a (0xf284 bytes) 000646 1778355341.979864 mapread 0x034562 thru 0x03933a (0x4dd9 bytes) 000647 1778355341.980821 punch from 0x020845 to 0x02c37b (0xbb37 bytes) 000648 1778355341.986822 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) 000649 1778355342.024150 write 0x03635f thru 0x037211 (0xeb3 bytes) 000650 1778355342.028660 fallocate 0x00f9d8 thru 0x014d1d (0x5346 bytes) 000651 1778355342.030904 punch from 0x015235 to 0x019c9e (0x4a6a bytes) 000652 1778355342.032538 trunc from 0x01f04b to 0x024af6 (0x5aac bytes) 000653 1778355342.034264 fallocate 0x01ab0a thru 0x022855 (0x7d4c bytes) 000654 1778355342.036207 write 0x03f034 thru 0x03ffff (0xfcc bytes) 000655 1778355342.037103 trunc from 0x040000 to 0x023ae7 (0x1c518 bytes) 000656 1778355342.041663 trunc from 0x023ae8 to 0x015bfe (0xdee9 bytes) 000657 1778355342.046936 MIRROR_EXTEND to 2 mirrors 000658 1778355342.233746 read 0x00b361 thru 0x00b667 (0x307 bytes) 000659 1778355342.235535 mapread 0x01241b thru 0x015bfe (0x37e4 bytes) 000660 1778355342.240552 read 0x00dcd6 thru 0x015bfe (0x7f29 bytes) 000661 1778355342.241913 MIRROR_EXTEND to 3 mirrors 000662 1778355342.428869 mapread 0x00c110 thru 0x00ff6c (0x3e5d bytes) 000663 1778355342.434961 trunc from 0x015bff to 0x01f05e (0x9460 bytes) 000664 1778355342.447920 write 0x038261 thru 0x03c4cd (0x426d bytes) 000665 1778355342.448799 mapwrite 0x00be1f thru 0x01456b (0x874d bytes) 000666 1778355342.460914 trunc from 0x03b5dd to 0x03c4cd (0xef1 bytes) 000667 1778355342.463032 write 0x029026 thru 0x032b2c (0x9b07 bytes) 000668 1778355342.467719 fallocate 0x02e835 thru 0x030e21 (0x25ed bytes) 000671 1778355342.470731 punch from 0x02b11c to 0x03ab6b (0xfa50 bytes) 000672 1778355342.475250 punch from 0x00c82a to 0x01bb66 (0xf33d bytes) 000673 1778355342.477209 punch from 0x025275 to 0x026cca (0x1a56 bytes) 000674 1778355342.478847 trunc from 0x024691 to 0x0265cd (0x1f3d bytes) 000675 1778355342.480458 write 0x02c4c1 thru 0x036851 (0xa391 bytes) 000676 1778355342.484948 mapwrite 0x01efb1 thru 0x02172b (0x277b bytes) 000677 1778355342.493797 trunc from 0x03c4ce to 0x03906d (0x3460 bytes) 000679 1778355342.500059 fallocate 0x027fcb thru 0x029414 (0x144a bytes) 000680 1778355342.502053 read 0x012284 thru 0x019c10 (0x798d bytes) 000681 1778355342.504762 trunc from 0x03906e to 0x013790 (0x258dd bytes) 000682 1778355342.509136 trunc from 0x007f0d to 0x007fc0 (0xb4 bytes) 000683 1778355342.511057 fallocate 0x02f014 thru 0x036a89 (0x7a76 bytes) 000684 1778355342.513434 mapread 0x00029f thru 0x003ed0 (0x3c32 bytes) 000686 1778355342.516805 read 0x01b2ce thru 0x02742b (0xc15e bytes) 000687 1778355342.520052 read 0x02252c thru 0x02e947 (0xc41c bytes) 000688 1778355342.520658 mapread 0x021a3f thru 0x030ed2 (0xf494 bytes) 000689 1778355342.521628 read 0x0137d7 thru 0x01bbfa (0x8424 bytes) 000690 1778355342.522119 punch from 0x01b098 to 0x01deec (0x2e55 bytes) 000691 1778355342.523848 mapwrite 0x00ff9f thru 0x01f299 (0xf2fb bytes) 000692 1778355342.534045 read 0x002dcb thru 0x010902 (0xdb38 bytes) 000693 1778355342.536304 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36a8a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a8a): Mirror 2: 0x67f5570f (stale) Mirror 3: 0x67f5570f (stale) CRC-32 checksum value for chunk [0, 0x36a8a): Mirror 1: 0x774f3b4 Mirror 4: 0x774f3b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a8a: skipped 000694 1778355342.782482 punch from 0x01ae4b to 0x024db8 (0x9f6e bytes) 000695 1778355342.791899 trunc from 0x036a8a to 0x033b26 (0x2f63 bytes) 000696 1778355342.803544 write 0x004b74 thru 0x00f4ae (0xa93b bytes) 000697 1778355342.807015 mapwrite 0x013693 thru 0x01f998 (0xc306 bytes) 000698 1778355342.822442 MIRROR_RESYNC 4 mirrors 000699 1778355343.064223 punch from 0x029110 to 0x033b26 (0xaa17 bytes) 000700 1778355343.073593 write 0x0095e7 thru 0x014486 (0xaea0 bytes) 000701 1778355343.076958 trunc from 0x022f6d to 0x02cb3b (0x9bcf bytes) 000702 1778355343.078609 fallocate 0x005c36 thru 0x00870c (0x2ad7 bytes) 000703 1778355343.080255 write 0x0123c1 thru 0x019264 (0x6ea4 bytes) 000704 1778355343.083461 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33b27): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b27): Mirror 2: 0x2db6f48 (stale) Mirror 3: 0x2db6f48 (stale) Mirror 4: 0x2db6f48 (stale) CRC-32 checksum value for chunk [0, 0x33b27): Mirror 1: 0x68045ca1 Mirror 5: 0x68045ca1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b27: skipped 000705 1778355343.353059 trunc from 0x0119c6 to 0x019cb0 (0x82eb bytes) 000706 1778355343.362995 write 0x02679f thru 0x033ce4 (0xd546 bytes) 000707 1778355343.368756 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33ce5): Mirror 3: 0x334554bb (stale) Mirror 4: 0x334554bb (stale) Mirror 5: 0x3529a502 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ce5): Mirror 3: 0x334554bb (stale) Mirror 4: 0x334554bb (stale) Mirror 5: 0x3529a502 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ce5: skipped 000708 1778355343.463930 mapwrite 0x02b864 thru 0x036676 (0xae13 bytes) 000710 1778355343.483756 write 0x03710a thru 0x03b5bf (0x44b6 bytes) 000711 1778355343.484838 mapwrite 0x001b3c thru 0x0083ff (0x68c4 bytes) 000712 1778355343.494083 trunc from 0x03b5c0 to 0x037267 (0x4358 bytes) 000713 1778355343.499066 fallocate 0x0124f7 thru 0x020bda (0xe6e4 bytes) 000714 1778355343.501364 fallocate 0x02abed thru 0x0324aa (0x78be bytes) 000715 1778355343.503290 punch from 0x019858 to 0x025f5a (0xc703 bytes) 000716 1778355343.504990 trunc from 0x037268 to 0x0036ec (0x33b7b bytes) 000717 1778355343.511160 MIRROR_RESYNC 4 mirrors 000718 1778355343.694937 mapread 0x002536 thru 0x0036ec (0x11b7 bytes) 000719 1778355343.698767 fallocate 0x000eb6 thru 0x0106e1 (0xf82c bytes) 000720 1778355343.708486 mapwrite 0x008950 thru 0x00d22f (0x48e0 bytes) 000721 1778355343.721647 punch from 0x002356 to 0x0092d6 (0x6f81 bytes) 000722 1778355343.723663 read 0x00c686 thru 0x00d22f (0xbaa bytes) 000723 1778355343.724087 trunc from 0x00d230 to 0x026756 (0x19527 bytes) 000724 1778355343.728556 punch from 0x022d1a to 0x026756 (0x3a3d bytes) 000725 1778355343.730215 mapwrite 0x00dada thru 0x011a4c (0x3f73 bytes) 000726 1778355343.738124 trunc from 0x026757 to 0x00dafa (0x18c5c bytes) 000728 1778355343.743692 trunc from 0x00dafb to 0x02750a (0x19a10 bytes) 000729 1778355343.750537 MIRROR_RESYNC 4 mirrors 000731 1778355343.984894 mapwrite 0x00f7c6 thru 0x01df7e (0xe7b9 bytes) 000732 1778355344.021526 read 0x00ed13 thru 0x012918 (0x3c06 bytes) 000733 1778355344.022044 fallocate 0x0302d2 thru 0x0307ba (0x4e9 bytes) 000734 1778355344.024275 punch from 0x016e94 to 0x01e45e (0x75cb bytes) 000736 1778355344.026446 trunc from 0x0307bb to 0x020e15 (0xf9a5 bytes) 000737 1778355344.033585 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20e16): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e16): Mirror 3: 0xc0a921cc (stale) Mirror 4: 0xc0a921cc (stale) Mirror 5: 0xc0a921cc (stale) CRC-32 checksum value for chunk [0, 0x20e16): Mirror 1: 0x1dc0faaa Mirror 6: 0x1dc0faaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e16: skipped 000738 1778355344.296137 punch from 0x00a721 to 0x012186 (0x7a66 bytes) 000740 1778355344.307724 mapread 0x013154 thru 0x01a811 (0x76be bytes) 000741 1778355344.313902 mapread 0x000325 thru 0x008859 (0x8535 bytes) 000742 1778355344.316913 punch from 0x005a13 to 0x00f091 (0x967f bytes) 000743 1778355344.318666 write 0x027c67 thru 0x02c6c4 (0x4a5e bytes) 000744 1778355344.320762 trunc from 0x02c6c5 to 0x00d985 (0x1ed3f bytes) 000745 1778355344.326173 punch from 0x00c6a3 to 0x00d985 (0x12e3 bytes) 000746 1778355344.327914 punch from 0x0033d0 to 0x00d985 (0xa5b6 bytes) 000747 1778355344.329522 mapread 0x00430d thru 0x007f85 (0x3c79 bytes) 000748 1778355344.332354 mapread 0x003d7b thru 0x007daa (0x4030 bytes) 000750 1778355344.333442 write 0x00e187 thru 0x018e7c (0xacf6 bytes) 000751 1778355344.334421 mapread 0x010787 thru 0x018e7c (0x86f6 bytes) 000752 1778355344.336495 MIRROR_RESYNC 5 mirrors 000753 1778355344.579757 MIRROR_EXTEND to 6 mirrors 000754 1778355344.756962 read 0x00dc3e thru 0x014fb1 (0x7374 bytes) 000755 1778355344.759806 trunc from 0x0122fa to 0x0140eb (0x1df2 bytes) 000756 1778355344.770916 trunc from 0x018e7d to 0x03fce7 (0x26e6b bytes) 000757 1778355344.781021 fallocate 0x01a6b2 thru 0x0216b0 (0x6fff bytes) 000758 1778355344.783044 mapread 0x01181c thru 0x017358 (0x5b3d bytes) 000759 1778355344.789090 write 0x0129d3 thru 0x020dd2 (0xe400 bytes) 000760 1778355344.791705 punch from 0x038415 to 0x03fce7 (0x78d3 bytes) 000761 1778355344.793752 fallocate 0x001e99 thru 0x00ecac (0xce14 bytes) 000762 1778355344.795612 mapwrite 0x0198e0 thru 0x020dec (0x750d bytes) 000763 1778355344.802025 write 0x035db3 thru 0x03ffff (0xa24d bytes) 000764 1778355344.805430 trunc from 0x040000 to 0x01a3f4 (0x25c0b bytes) 000765 1778355344.811586 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a3f5): Mirror 4: 0xe1e93b1 (stale) Mirror 5: 0xe1e93b1 (stale) Mirror 6: 0xe1e93b1 (stale) Mirror 7: 0xe1e93b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a3f5): Mirror 4: 0xe1e93b1 (stale) Mirror 5: 0xe1e93b1 (stale) Mirror 6: 0xe1e93b1 (stale) Mirror 7: 0xe1e93b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a3f5: skipped 000766 1778355344.910153 mapwrite 0x01bb09 thru 0x02b904 (0xfdfc bytes) 000767 1778355344.930739 trunc from 0x02b905 to 0x01d703 (0xe201 bytes) 000768 1778355344.937553 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d704): Mirror 4: 0x8da23047 (stale) Mirror 6: 0x8da23047 (stale) Mirror 7: 0x8da23047 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d704): Mirror 4: 0x8da23047 (stale) Mirror 6: 0x8da23047 (stale) Mirror 7: 0x8da23047 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d704: skipped 000769 1778355345.023770 write 0x014a38 thru 0x02128a (0xc853 bytes) 000771 1778355345.028174 read 0x0012ac thru 0x0101e6 (0xef3b bytes) 000772 1778355345.031240 trunc from 0x00b38a to 0x0185cd (0xd244 bytes) 000773 1778355345.035800 mapwrite 0x02b679 thru 0x03122e (0x5bb6 bytes) 000774 1778355345.050509 write 0x020e92 thru 0x0300ce (0xf23d bytes) 000775 1778355345.051877 fallocate 0x027d91 thru 0x02ddbe (0x602e bytes) 000776 1778355345.053731 fallocate 0x014279 thru 0x018f02 (0x4c8a bytes) 000777 1778355345.055806 punch from 0x01d5d3 to 0x02c60e (0xf03c bytes) 000778 1778355345.060173 write 0x017618 thru 0x0183ad (0xd96 bytes) 000779 1778355345.063059 mapread 0x0157f8 thru 0x01b6aa (0x5eb3 bytes) 000780 1778355345.063848 mapwrite 0x02fef1 thru 0x03a66f (0xa77f bytes) 000781 1778355345.078472 read 0x0360af thru 0x03a66f (0x45c1 bytes) 000782 1778355345.078962 fallocate 0x0296b6 thru 0x034183 (0xaace bytes) 000783 1778355345.082357 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a670): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a670): Mirror 4: 0xbad71ccb (stale) Mirror 6: 0xbad71ccb (stale) Mirror 7: 0xbad71ccb (stale) CRC-32 checksum value for chunk [0, 0x3a670): Mirror 1: 0x86477504 Mirror 8: 0x86477504 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a670: skipped 000784 1778355345.338374 mapwrite 0x03c3bd thru 0x03e635 (0x2279 bytes) 000785 1778355345.363746 trunc from 0x03e636 to 0x029d2d (0x14908 bytes) 000787 1778355345.369775 write 0x0099a7 thru 0x015c12 (0xc26c bytes) 000788 1778355345.376386 MIRROR_RESYNC 5 mirrors 000789 1778355345.691831 trunc from 0x01f40e to 0x01fe84 (0xa77 bytes) 000790 1778355345.701581 read 0x0070f7 thru 0x015c17 (0xeb21 bytes) 000791 1778355345.704418 fallocate 0x015985 thru 0x015f4b (0x5c7 bytes) 000792 1778355345.706213 read 0x0259a6 thru 0x029d2d (0x4388 bytes) 000793 1778355345.709331 mapread 0x020e21 thru 0x029d2d (0x8f0d bytes) 000794 1778355345.710170 read 0x016155 thru 0x0225b0 (0xc45c bytes) 000795 1778355345.710644 trunc from 0x029d2e to 0x02ac2e (0xf01 bytes) 000796 1778355345.720850 fallocate 0x01ca5c thru 0x02c727 (0xfccc bytes) 000797 1778355345.722965 write 0x003df3 thru 0x007738 (0x3946 bytes) 000799 1778355345.724140 write 0x009beb thru 0x00a186 (0x59c bytes) 000800 1778355345.724826 trunc from 0x02ac2f to 0x028c0e (0x2020 bytes) 000801 1778355345.729557 read 0x00fe8a thru 0x019475 (0x95ec bytes) 000802 1778355345.730029 mapread 0x017327 thru 0x023e94 (0xcb6e bytes) 000803 1778355345.730946 mapwrite 0x00439f thru 0x00b6a5 (0x7307 bytes) 000804 1778355345.738555 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x28c0f): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c0f): Mirror 4: 0x95f5d919 (stale) Mirror 6: 0x95f5d919 (stale) Mirror 7: 0x95f5d919 (stale) Mirror 8: 0x95f5d919 (stale) CRC-32 checksum value for chunk [0, 0x28c0f): Mirror 1: 0xe24df35d Mirror 9: 0xe24df35d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c0f: skipped 000805 1778355345.989733 write 0x03e17b thru 0x03ffff (0x1e85 bytes) 000806 1778355345.999953 mapread 0x0003c5 thru 0x0041e0 (0x3e1c bytes) 000807 1778355346.006511 write 0x03bd24 thru 0x03ffff (0x42dc bytes) 000808 1778355346.009217 fallocate 0x009df9 thru 0x016468 (0xc670 bytes) 000810 1778355346.011619 read 0x010483 thru 0x013e2b (0x39a9 bytes) 000811 1778355346.013561 mapread 0x021f9f thru 0x024e47 (0x2ea9 bytes) 000812 1778355346.016554 write 0x015da5 thru 0x019536 (0x3792 bytes) 000813 1778355346.019613 fallocate 0x01a987 thru 0x0290e5 (0xe75f bytes) 000814 1778355346.021563 mapwrite 0x020d14 thru 0x02abdf (0x9ecc bytes) 000815 1778355346.030899 trunc from 0x000570 to 0x00cd9a (0xc82b bytes) 000816 1778355346.032648 mapwrite 0x00be5c thru 0x015d7b (0x9f20 bytes) 000818 1778355346.042351 mapwrite 0x0180a0 thru 0x0221f9 (0xa15a bytes) 000819 1778355346.049831 read 0x00ffea thru 0x01da9a (0xdab1 bytes) 000820 1778355346.050289 write 0x02372b thru 0x02483d (0x1113 bytes) 000821 1778355346.051122 trunc from 0x040000 to 0x0170f7 (0x28f08 bytes) 000823 1778355346.061888 mapwrite 0x003252 thru 0x006057 (0x2e06 bytes) 000824 1778355346.068905 read 0x007dbd thru 0x014b53 (0xcd97 bytes) 000825 1778355346.069590 mapread 0x00f625 thru 0x0170f7 (0x7ad3 bytes) 000826 1778355346.070401 punch from 0x0037a3 to 0x010a99 (0xd2f7 bytes) 000827 1778355346.072400 write 0x039d4b thru 0x03ac5e (0xf14 bytes) 000828 1778355346.073314 fallocate 0x02e986 thru 0x02fae7 (0x1162 bytes) 000829 1778355346.076800 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac5f): Mirror 6: 0x6c5f4a59 (stale) Mirror 7: 0x6c5f4a59 (stale) Mirror 8: 0x6c5f4a59 (stale) Mirror 9: 0x19e46faf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac5f): Mirror 6: 0x6c5f4a59 (stale) Mirror 7: 0x6c5f4a59 (stale) Mirror 8: 0x6c5f4a59 (stale) Mirror 9: 0x19e46faf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac5f: skipped 000830 1778355346.177537 mapread 0x01862a thru 0x0280bb (0xfa92 bytes) 000831 1778355346.185658 write 0x03848c thru 0x03ffff (0x7b74 bytes) 000832 1778355346.190014 trunc from 0x040000 to 0x03d93e (0x26c1 bytes) 000833 1778355346.194522 punch from 0x020862 to 0x02acf4 (0xa493 bytes) 000834 1778355346.196550 write 0x00418b thru 0x011ab8 (0xd92e bytes) 000835 1778355346.200885 trunc from 0x000ef5 to 0x00be6b (0xaf77 bytes) 000836 1778355346.205217 mapwrite 0x00a1cf thru 0x00e4dd (0x430f bytes) 000837 1778355346.213986 read 0x0172e5 thru 0x019fa6 (0x2cc2 bytes) 000838 1778355346.214701 trunc from 0x03d93f to 0x02c14e (0x117f0 bytes) 000839 1778355346.219536 mapread 0x020bbb thru 0x02bcf2 (0xb138 bytes) 000840 1778355346.224677 trunc from 0x00efa9 to 0x01c684 (0xd6dc bytes) 000841 1778355346.226744 trunc from 0x02c14f to 0x02c179 (0x2b bytes) 000842 1778355346.230996 mapwrite 0x02e089 thru 0x03506e (0x6fe6 bytes) 000843 1778355346.244473 mapwrite 0x024c17 thru 0x030a45 (0xbe2f bytes) 000844 1778355346.252933 write 0x01de00 thru 0x0287df (0xa9e0 bytes) 000845 1778355346.254077 fallocate 0x00360c thru 0x00956b (0x5f60 bytes) 000847 1778355346.257033 write 0x035c1f thru 0x037bcd (0x1faf bytes) 000848 1778355346.257874 trunc from 0x037bce to 0x013614 (0x245b9 bytes) 000849 1778355346.262878 trunc from 0x006d2d to 0x013614 (0xc8e8 bytes) 000850 1778355346.264905 trunc from 0x013615 to 0x0282a4 (0x14c90 bytes) 000851 1778355346.269265 trunc from 0x00d1dd to 0x015dcf (0x8bf3 bytes) 000852 1778355346.270808 write 0x03effa thru 0x03ffff (0x1006 bytes) 000853 1778355346.271595 write 0x021f7c thru 0x02e30c (0xc391 bytes) 000854 1778355346.275806 trunc from 0x03ad44 to 0x03ffff (0x52bc bytes) 000855 1778355346.279805 mapread 0x032673 thru 0x03ceb8 (0xa846 bytes) 000856 1778355346.285264 trunc from 0x00c0dd to 0x0141d1 (0x80f5 bytes) 000857 1778355346.287466 trunc from 0x00db9e to 0x01485f (0x6cc2 bytes) 000858 1778355346.289368 write 0x03a65b thru 0x03ffff (0x59a5 bytes) 000859 1778355346.292220 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 7: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a975a7 Mirror 10: 0x8a975a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000860 1778355346.554960 punch from 0x00629b to 0x0071f6 (0xf5c bytes) 000861 1778355346.565554 punch from 0x02b1fb to 0x0326fa (0x7500 bytes) 000862 1778355346.568883 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 7: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) Mirror 10: 0x8a975a7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb485d146 Mirror 11: 0xb485d146 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000863 1778355346.830109 read 0x00fbe4 thru 0x01f746 (0xfb63 bytes) 000864 1778355346.834671 read 0x010371 thru 0x015fa9 (0x5c39 bytes) 000865 1778355346.836759 trunc from 0x040000 to 0x00decc (0x32133 bytes) 000866 1778355346.853580 punch from 0x001fff to 0x00decc (0xbece bytes) 000867 1778355346.855376 punch from 0x00388d to 0x00d5e7 (0x9d5b bytes) 000869 1778355346.857762 read 0x00318b thru 0x007b41 (0x49b7 bytes) 000870 1778355346.860631 mapread 0x009a42 thru 0x00decc (0x448b bytes) 000871 1778355346.861378 write 0x0141c7 thru 0x014988 (0x7c2 bytes) 000872 1778355346.862227 punch from 0x003b87 to 0x0078ad (0x3d27 bytes) 000873 1778355346.863818 read 0x00d39f thru 0x014988 (0x75ea bytes) 000874 1778355346.866223 fallocate 0x01804a thru 0x01d856 (0x580d bytes) 000875 1778355346.868038 write 0x03c51a thru 0x03ffff (0x3ae6 bytes) 000876 1778355346.868818 read 0x01c563 thru 0x0250f6 (0x8b94 bytes) 000877 1778355346.871703 read 0x022f0b thru 0x029b70 (0x6c66 bytes) 000878 1778355346.872337 write 0x019431 thru 0x0212ef (0x7ebf bytes) 000879 1778355346.873244 read 0x02812a thru 0x036285 (0xe15c bytes) 000880 1778355346.873919 fallocate 0x03ea5e thru 0x03ffff (0x15a2 bytes) 000881 1778355346.875929 read 0x01fd20 thru 0x027a73 (0x7d54 bytes) 000882 1778355346.876394 mapwrite 0x01d722 thru 0x028685 (0xaf64 bytes) 000884 1778355346.884064 mapread 0x01ee23 thru 0x02645c (0x763a bytes) 000885 1778355346.884955 punch from 0x00abd2 to 0x012b7e (0x7fad bytes) 000886 1778355346.886706 mapread 0x00e661 thru 0x01ae3c (0xc7dc bytes) 000887 1778355346.889809 mapwrite 0x03266f thru 0x038826 (0x61b8 bytes) 000888 1778355346.896971 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 7: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) Mirror 10: 0x8a975a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 7: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) Mirror 10: 0x8a975a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000891 1778355347.020098 write 0x02eb12 thru 0x03d056 (0xe545 bytes) 000892 1778355347.024766 mapread 0x024094 thru 0x032be3 (0xeb50 bytes) 000893 1778355347.031043 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 7: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 7: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000894 1778355347.127039 mapwrite 0x00dade thru 0x011291 (0x37b4 bytes) 000895 1778355347.137397 read 0x0270ac thru 0x0332cc (0xc221 bytes) 000896 1778355347.140157 punch from 0x022362 to 0x026f5a (0x4bf9 bytes) 000897 1778355347.142106 read 0x016efd thru 0x025860 (0xe964 bytes) 000898 1778355347.145377 mapwrite 0x004b87 thru 0x00ee62 (0xa2dc bytes) 000899 1778355347.153362 write 0x005a26 thru 0x0122b2 (0xc88d bytes) 000900 1778355347.155572 trunc from 0x0101aa to 0x01de7f (0xdcd6 bytes) 000901 1778355347.159983 write 0x021a3b thru 0x02b450 (0x9a16 bytes) 000902 1778355347.161319 mapwrite 0x02ad2d thru 0x033d79 (0x904d bytes) 000903 1778355347.168614 mapread 0x00be76 thru 0x01682c (0xa9b7 bytes) 000904 1778355347.178062 mapwrite 0x005171 thru 0x013e87 (0xed17 bytes) 000905 1778355347.186403 punch from 0x032883 to 0x03f09f (0xc81d bytes) 000906 1778355347.188326 punch from 0x02ebda to 0x03c8e1 (0xdd08 bytes) 000907 1778355347.190147 mapread 0x02d5ec thru 0x032eda (0x58ef bytes) 000908 1778355347.197379 fallocate 0x015c69 thru 0x022f4a (0xd2e2 bytes) 000909 1778355347.199130 read 0x007ca5 thru 0x015998 (0xdcf4 bytes) 000910 1778355347.199616 punch from 0x036842 to 0x03ffff (0x97be bytes) 000912 1778355347.201499 trunc from 0x019abd to 0x01aca2 (0x11e6 bytes) 000913 1778355347.202955 trunc from 0x0038db to 0x00d3b3 (0x9ad9 bytes) 000914 1778355347.206077 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77e33f01 (stale) Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000916 1778355347.293662 read 0x01bdcd thru 0x02066c (0x48a0 bytes) 000917 1778355347.296649 write 0x0399ec thru 0x03b1cb (0x17e0 bytes) 000918 1778355347.300680 write 0x039327 thru 0x03ffff (0x6cd9 bytes) 000919 1778355347.301583 trunc from 0x03497c to 0x03c351 (0x79d6 bytes) 000920 1778355347.306705 punch from 0x00d529 to 0x0145ca (0x70a2 bytes) 000921 1778355347.308481 fallocate 0x022eb2 thru 0x029f82 (0x70d1 bytes) 000922 1778355347.310221 read 0x0108c5 thru 0x014880 (0x3fbc bytes) 000924 1778355347.314759 mapwrite 0x0287a1 thru 0x02a14e (0x19ae bytes) 000925 1778355347.319930 trunc from 0x006e69 to 0x013fd0 (0xd168 bytes) 000926 1778355347.321999 fallocate 0x01e468 thru 0x02012d (0x1cc6 bytes) 000927 1778355347.323779 mapwrite 0x00c0f5 thru 0x01a047 (0xdf53 bytes) 000928 1778355347.341944 read 0x022d56 thru 0x026f94 (0x423f bytes) 000929 1778355347.342590 mapread 0x02a122 thru 0x034105 (0x9fe4 bytes) 000930 1778355347.343518 mapwrite 0x039756 thru 0x039f10 (0x7bb bytes) 000931 1778355347.348834 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000932 1778355347.427328 mapwrite 0x03db69 thru 0x03ffff (0x2497 bytes) 000933 1778355347.436741 read 0x03f8f1 thru 0x03ffff (0x70f bytes) 000934 1778355347.437261 mapread 0x01a7aa thru 0x024f6f (0xa7c6 bytes) 000935 1778355347.443253 read 0x02f590 thru 0x034258 (0x4cc9 bytes) 000936 1778355347.446778 fallocate 0x029edd thru 0x03294c (0x8a70 bytes) 000937 1778355347.448903 mapwrite 0x03a951 thru 0x03ffff (0x56af bytes) 000938 1778355347.455260 punch from 0x02c945 to 0x034a50 (0x810c bytes) 000939 1778355347.457193 read 0x002a4f thru 0x01172f (0xece1 bytes) 000940 1778355347.457778 read 0x02a10c thru 0x03995b (0xf850 bytes) 000941 1778355347.460639 trunc from 0x0064e2 to 0x015974 (0xf493 bytes) 000942 1778355347.462535 mapread 0x0027e7 thru 0x007c78 (0x5492 bytes) 000943 1778355347.465724 trunc from 0x040000 to 0x0244e3 (0x1bb1c bytes) 000944 1778355347.473080 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x244e4): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x244e4): Mirror 8: 0x747cc770 (stale) Mirror 9: 0x79c49f70 (stale) CRC-32 checksum value for chunk [0, 0x244e4): Mirror 1: 0x47654258 Mirror 10: 0x47654258 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x244e4: skipped 000946 1778355347.720595 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x244e4): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x244e4): Mirror 8: 0x747cc770 (stale) Mirror 9: 0x79c49f70 (stale) CRC-32 checksum value for chunk [0, 0x244e4): Mirror 1: 0x47654258 Mirror 10: 0x47654258 Mirror 11: 0x47654258 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x244e4: skipped 000947 1778355347.959232 write 0x01db8d thru 0x02106f (0x34e3 bytes) 000948 1778355347.972005 fallocate 0x021514 thru 0x03087c (0xf369 bytes) 000949 1778355347.974185 mapread 0x001917 thru 0x010f97 (0xf681 bytes) 000950 1778355347.980010 trunc from 0x003a13 to 0x00d51f (0x9b0d bytes) 000951 1778355347.981676 write 0x030152 thru 0x03fe8d (0xfd3c bytes) 000952 1778355347.984866 fallocate 0x032235 thru 0x039354 (0x7120 bytes) 000953 1778355347.986650 mapread 0x0291de thru 0x02b6d5 (0x24f8 bytes) 000954 1778355347.987481 mapread 0x03ecf5 thru 0x03fe8d (0x1199 bytes) 000955 1778355347.988232 mapwrite 0x033748 thru 0x03ffff (0xc8b8 bytes) 000956 1778355348.004921 trunc from 0x040000 to 0x00a744 (0x358bb bytes) 000957 1778355348.010111 write 0x0096a3 thru 0x013af0 (0xa44e bytes) 000958 1778355348.014111 write 0x022421 thru 0x02e3c5 (0xbfa5 bytes) 000959 1778355348.015704 trunc from 0x02e3c6 to 0x010b6a (0x1d85b bytes) 000960 1778355348.020549 mapwrite 0x015245 thru 0x019594 (0x4350 bytes) 000962 1778355348.034480 fallocate 0x018956 thru 0x0281e5 (0xf890 bytes) 000963 1778355348.036747 write 0x03a84e thru 0x03ffff (0x57b2 bytes) 000964 1778355348.037759 read 0x026348 thru 0x028d29 (0x29e2 bytes) 000965 1778355348.039837 mapread 0x01ef24 thru 0x02a38b (0xb468 bytes) 000966 1778355348.045461 read 0x022945 thru 0x028fb7 (0x6673 bytes) 000967 1778355348.046098 mapread 0x03e23e thru 0x03ffff (0x1dc2 bytes) 000969 1778355348.047639 punch from 0x027486 to 0x02a7e1 (0x335c bytes) 000970 1778355348.049230 trunc from 0x040000 to 0x038e47 (0x71b8 bytes) 000971 1778355348.054097 trunc from 0x038e48 to 0x02fc4b (0x91fc bytes) 000972 1778355348.058512 write 0x02ea9d thru 0x030796 (0x1cfa bytes) 000973 1778355348.059597 read 0x003091 thru 0x00d38c (0xa2fc bytes) 000974 1778355348.060248 fallocate 0x01b702 thru 0x023aec (0x83eb bytes) 000975 1778355348.062542 mapwrite 0x034cd8 thru 0x03ffff (0xb328 bytes) 000976 1778355348.078161 punch from 0x031399 to 0x039af5 (0x875d bytes) 000978 1778355348.080425 punch from 0x023670 to 0x0239a8 (0x339 bytes) 000979 1778355348.082220 mapread 0x021882 thru 0x02fb66 (0xe2e5 bytes) 000980 1778355348.084880 fallocate 0x03bc10 thru 0x03ffff (0x43f0 bytes) 000982 1778355348.089358 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x724:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) Mirror 10: 0xa1343574 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x77e33f01 (stale) Mirror 9: 0xa7754a1c (stale) Mirror 10: 0xa1343574 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000983 1778355348.187803 write 0x0324e3 thru 0x039729 (0x7247 bytes) 000984 1778355348.192299 write 0x0367e4 thru 0x03d143 (0x6960 bytes) 000985 1778355348.194812 fallocate 0x01e6a4 thru 0x01f9e0 (0x133d bytes) 000986 1778355348.196612 read 0x015cb0 thru 0x0176b4 (0x1a05 bytes) 000987 1778355348.198487 punch from 0x03b6fb to 0x03ffff (0x4905 bytes) 000988 1778355348.202739 trunc from 0x040000 to 0x03dfd4 (0x202b bytes) 000989 1778355348.207367 write 0x03fb13 thru 0x03ffff (0x4ed bytes) 000990 1778355348.208075 mapread 0x03e5a1 thru 0x03ffff (0x1a5f bytes) 000993 1778355348.211233 trunc from 0x00b718 to 0x010be5 (0x54ce bytes) 000994 1778355348.213058 punch from 0x0392ea to 0x03ffff (0x6d16 bytes) 000995 1778355348.216387 read 0x01a378 thru 0x0282d7 (0xdf60 bytes) 000996 1778355348.220393 mapwrite 0x03d5d8 thru 0x03ffff (0x2a28 bytes) 000997 1778355348.225966 fallocate 0x0353a3 thru 0x03b4f4 (0x6152 bytes) 000998 1778355348.228809 mapread 0x00d82c thru 0x01cd8c (0xf561 bytes) 000999 1778355348.230338 write 0x03847d thru 0x03ffff (0x7b83 bytes) 001000 1778355348.231507 mapwrite 0x0034ea thru 0x0114e0 (0xdff7 bytes) All operations completed A-OK! PASS 70a (21s) == sanity-flr test 71: check mirror extend parent fid ==== 15:35:49 (1778355349) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x764:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 72: check append on FLR file ========== 15:35:53 (1778355353) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.06683 s, 31.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387829 s, 27.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (2s) == sanity-flr test 200a: stress test ===================== 15:35:55 (1778355355) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5214656 .. Extending file size to 7402432 .. done Extending file size to 5967392 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8157984 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8168416 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8348672 .. Extending file size to 8705792 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8586624 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9077440 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9054080 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9126784 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9161408 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9183392 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 64472 64473 64475 64477 64479 failed 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) 192.168.202.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:37:00 (1778355420) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2892640 .. Extending file size to 2066848 .. done Extending file size to 6926400 .. Extending file size to 6250016 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8227072 .. Extending file size to 8942592 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7525472 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7773792 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9173952 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9218752 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 67926 67927 67929 67931 67933 done 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) 192.168.202.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:38:06 (1778355486) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2576 1268728 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2960 1268344 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25892 3581128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17892 3589128 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43784 7170256 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2576 1268728 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2960 1268344 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25892 3581128 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17892 3589128 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43784 7170256 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0424007 s, 49.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0520917 s, 40.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa0d68cbb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:38:13 (1778355493) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0319747 s, 65.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 15:38:15 (1778355495) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:38:17 (1778355497) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0263146 s, 79.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:38:20 (1778355500) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.025251 s, 83.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:38:22 (1778355502) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0236457 s, 88.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:38:46 (1778355526) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190705 s, 55.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xade:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xadd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xadc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xade:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xadd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xadc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xadf:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:39:22 (1778355562) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0239952 s, 87.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:39:57 (1778355597) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg251-server: Pool lustre.test_204f created Adding targets to pool oleg251-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg251-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0222786 s, 94.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg251-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg251-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg251-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:40:46 (1778355646) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:40:48 (1778355648) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:40:50 (1778355650) oleg251-server: Pool lustre.test_206 created Adding targets to pool oleg251-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg251-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg251-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg251-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg251-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:41:03 (1778355663) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233274 s, 45.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117272 s, 89.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3171fdf9 Mirror 3: 0x3171fdf9 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:41:06 (1778355666) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:41:07 (1778355667) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:41:08 (1778355668) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.00928575 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x8f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2460000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2460000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2460000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24f0000 Read this (4 bytes): BBBB 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:41:24 (1778355684) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00330531 s, 2.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:41:26 (1778355686) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013038 s, 80.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 15:41:26.000000000 -0400 Modify: 2026-05-09 15:41:26.000000000 -0400 Change: 2026-05-09 15:41:27.000000000 -0400 Birth: 2026-05-09 15:41:26.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:41:28 (1778355688) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131719 s, 79.6 MB/s oleg251-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg251-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg251-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:41:41 (1778355701) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0396929 s, 1.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00494164 s, 829 kB/s size after second write -rw-r--r-- 1 root root 4096 May 9 15:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 9 15:41 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:41:44 (1778355704) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:41:46 (1778355706) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 805 sec ============ 15:41:48 (1778355708) === sanity-flr: start cleanup 15:41:49 (1778355709) === === sanity-flr: finish cleanup 15:41:51 (1778355711) ===