-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 22:34:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg657-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:34:31 (1787884471) === oleg657-client.virtnet: executing check_config_client /mnt/lustre oleg657-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg657-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b4883558800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b4883558800.idle_timeout=debug disable quota as required oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:35:14 (1787884514) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8218] == sanity-flr test 0a: lfs mirror create with -N option == 22:35:16 (1787884516) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:35:35 (1787884535) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:35:38 (1787884538) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:35:42 (1787884542) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:36:04 (1787884564) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:36:08 (1787884568) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:36:12 (1787884572) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b4883558800.stats=clear osc.lustre-OST0001-osc-ffff9b4883558800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:36:21 (1787884581) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:36:32 (1787884592) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:36:41 (1787884601) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:36:45 (1787884605) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:36:53 (1787884613) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 22:37:04 (1787884624) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:37:12 (1787884632) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:37:21 (1787884641) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:37:29 (1787884649) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:37:41 (1787884661) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:37:49 (1787884669) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999526 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105008 s, 10.0 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:37:58 (1787884678) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.666007 s, 20.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:38:08 (1787884688) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:38:19 (1787884699) Stopping /mnt/lustre-ost1 (opts:) on oleg657-server oleg657-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg657-server: oleg657-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg657-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg657-server oleg657-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg657-server: oleg657-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg657-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:39:34 (1787884774) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.760411 s, 15.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg657-server oleg657-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg657-server: oleg657-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg657-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:40:18 (1787884818) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg657-server oleg657-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg657-server: oleg657-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg657-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg657-server oleg657-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg657-server: oleg657-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg657-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:41:39 (1787884899) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.67761 s, 19.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 22:41 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.003s user 0m0.001s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg657-server oleg657-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b4883558800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.097s user 0m0.001s sys 0m0.180s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg657-server: oleg657-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg657-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg657-server oleg657-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.002s user 0m0.000s sys 0m0.188s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg657-server: oleg657-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg657-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg657-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b4883558800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:43:14 (1787884994) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:43:18 (1787884998) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:43:22 (1787885002) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:43:27 (1787885007) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 22:43:36 (1787885016) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.632648 s, 19.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:43:50 (1787885030) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435423 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240323 s, 43.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:43:58 (1787885038) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:44:07 (1787885047) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000703993 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0008023 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000797101 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00452964 s, 226 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 22:44:20 (1787885060) create test files with size 15738674 .. 7d514796f1565ce9d57b6e382fa13bc6 - 6697728fa1f64398b767fbc6d6e4fb0e - 720ffac49611099d46a215edc29ba1f2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg657-client.virtnet /mnt/lustre (opts:) Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 22:44:47 (1787885087) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:45:00 (1787885100) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:45:08 (1787885108) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914294 s, 11.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411964 s, 25.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 22:45:18 (1787885118) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105923 s, 19.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.388471 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186558 s, 11.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 22:45:33 (1787885133) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 22:45:37 (1787885137) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:45:40 (1787885140) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419939 s, 25.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 22:45:48 (1787885148) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.381995 s, 8.2 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:45:58 (1787885158) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:46:08 (1787885168) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.377865 s, 27.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg657-client.virtnet /mnt/lustre (opts:) Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:46:18 (1787885178) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b48a04c8800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.301008 s, 34.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg657-client.virtnet /mnt/lustre (opts:) Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b4888d9a800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:46:35 (1787885195) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:47:13 (1787885233) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:47:46 (1787885266) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:47:49 (1787885269) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.31894 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.161705 s, 19.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfe8a0068 (nosync,stale) Mirror 2: 0xfe8a0068 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 95fba8bb44ea7a639bee45ab00c1c69f - mirror 1 checksum: e93e8da9571ae82ded7b90637a5c0ec8 - mirror 2 checksum: 95fba8bb44ea7a639bee45ab00c1c69f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T22:47:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T22:47:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 95fba8bb44ea7a639bee45ab00c1c69f - mirror 1 checksum: e93e8da9571ae82ded7b90637a5c0ec8 - mirror 2 checksum: e93e8da9571ae82ded7b90637a5c0ec8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T22:47:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T22:47:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:47:59 (1787885279) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624616 s, 16.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:48:08 (1787885288) Failing mds1 on oleg657-server Stopping /mnt/lustre-mds1 (opts:) on oleg657-server 22:48:20 (1787885300) shut down facet: mds1 facet_host: oleg657-server facet_failover_host: oleg657-server Failover mds1 to oleg657-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg657-server: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Started lustre-MDT0000 22:48:53 (1787885333) targets are mounted 22:48:53 (1787885333) facet_failover done oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587875 s, 17.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:49:09 (1787885349) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0402106 s, 509 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0324694 s, 631 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:49:18 (1787885358) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0563974 s, 363 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00949934 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:49:26 (1787885366) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0142115 s, 1.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:49:35 (1787885375) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0742475 s, 276 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:49:43 (1787885383) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00598576 s, 684 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0216861 s, 189 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:49:51 (1787885391) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:50:14 (1787885414) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:50:42 (1787885442) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:51:06 (1787885466) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:51:29 (1787885489) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:51:36 (1787885496) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57560 mirror split pid 57561 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57560 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57561 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 22:52:44 (1787885564) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5566 skipping zero size read 000004 1787885566.399529 mapwrite 0x02c079 thru 0x032fba (0x6f42 bytes) 000005 1787885566.464521 write 0x013c9e thru 0x01e4cc (0xa82f bytes) fallocating to largest ever: 0x298ee 000006 1787885566.477726 fallocate 0x01be8c thru 0x0298ed (0xda62 bytes) 000007 1787885566.484815 mapwrite 0x02580b thru 0x033daf (0xe5a5 bytes) 000008 1787885566.573816 read 0x02bee6 thru 0x0312d9 (0x53f4 bytes) truncating to largest ever: 0x3875e 000009 1787885566.575741 trunc from 0x033db0 to 0x03875d (0x49ae bytes) 000010 1787885566.593119 mapwrite 0x02ef4c thru 0x0307e7 (0x189c bytes) 000011 1787885566.613188 trunc from 0x03875e to 0x02ebde (0x9b7f bytes) 000012 1787885566.633033 trunc from 0x0295eb to 0x02ebde (0x55f4 bytes) 000013 1787885566.640973 mapread 0x00c54b thru 0x00f5c9 (0x307f bytes) fallocating to largest ever: 0x40000 000014 1787885566.655992 fallocate 0x03b9a6 thru 0x03ffff (0x465a bytes) 000015 1787885566.661779 mapread 0x000f6e thru 0x004721 (0x37b4 bytes) 000016 1787885566.673029 write 0x010a04 thru 0x020482 (0xfa7f bytes) 000018 1787885566.676571 write 0x000a42 thru 0x00ba20 (0xafdf bytes) 000019 1787885566.680278 punch from 0x01123e to 0x015615 (0x43d8 bytes) 000020 1787885566.700240 mapread 0x00e7c8 thru 0x01d1d3 (0xea0c bytes) 000021 1787885566.713128 read 0x023f99 thru 0x0257bc (0x1824 bytes) 000022 1787885566.713801 fallocate 0x028907 thru 0x033994 (0xb08e bytes) 000023 1787885566.721279 fallocate 0x01fd18 thru 0x020427 (0x710 bytes) 000024 1787885566.728939 mapwrite 0x02c484 thru 0x030a5c (0x45d9 bytes) 000025 1787885566.788248 MIRROR_EXTEND to 2 mirrors 000026 1787885567.905418 write 0x037466 thru 0x03d437 (0x5fd2 bytes) 000027 1787885567.977246 punch from 0x0117e5 to 0x02123d (0xfa59 bytes) 000028 1787885567.985432 fallocate 0x000cfc thru 0x0049fe (0x3d03 bytes) 000029 1787885567.994405 mapread 0x036dca thru 0x03ffff (0x9236 bytes) 000030 1787885568.042741 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000031 1787885568.350063 punch from 0x009d34 to 0x018029 (0xe2f6 bytes) 000032 1787885568.361990 trunc from 0x004554 to 0x00a152 (0x5bff bytes) 000033 1787885568.369982 write 0x013bd1 thru 0x01ca1a (0x8e4a bytes) 000034 1787885568.390235 punch from 0x019958 to 0x0239d3 (0xa07c bytes) 000035 1787885568.406995 read 0x013dca thru 0x0194b3 (0x56ea bytes) 000036 1787885568.412024 trunc from 0x040000 to 0x00aa88 (0x35577 bytes) 000037 1787885568.430985 read 0x0010b4 thru 0x007478 (0x63c5 bytes) 000038 1787885568.442253 write 0x03be55 thru 0x03ffff (0x41ab bytes) 000039 1787885568.444146 fallocate 0x022ec7 thru 0x03265f (0xf799 bytes) 000040 1787885568.450026 read 0x0139ba thru 0x020070 (0xc6b7 bytes) 000041 1787885568.466213 trunc from 0x040000 to 0x022764 (0x1d89b bytes) 000042 1787885568.489919 trunc from 0x022765 to 0x0234d6 (0xd72 bytes) 000044 1787885568.509416 fallocate 0x031966 thru 0x03726b (0x5906 bytes) 000045 1787885568.514873 write 0x03f040 thru 0x03ffff (0xfc0 bytes) 000046 1787885568.516363 punch from 0x031526 to 0x03a864 (0x933f bytes) 000048 1787885568.522260 punch from 0x039fdb to 0x03ad2d (0xd53 bytes) 000049 1787885568.527063 trunc from 0x00c3b6 to 0x0103f8 (0x4043 bytes) 000050 1787885568.532364 read 0x001011 thru 0x00a6ff (0x96ef bytes) 000051 1787885568.533175 fallocate 0x00ce9a thru 0x01b8d1 (0xea38 bytes) 000052 1787885568.538844 fallocate 0x019c9f thru 0x01dc9b (0x3ffd bytes) 000053 1787885568.543842 punch from 0x00f0ce to 0x01b700 (0xc633 bytes) 000054 1787885568.548440 mapread 0x0112ed thru 0x01d16d (0xbe81 bytes) 000055 1787885568.563588 fallocate 0x004168 thru 0x012ca7 (0xeb40 bytes) 000056 1787885568.569203 trunc from 0x01dfa6 to 0x025677 (0x76d2 bytes) 000057 1787885568.574427 punch from 0x00116b to 0x005dfa (0x4c90 bytes) 000058 1787885568.580153 fallocate 0x0204d1 thru 0x021648 (0x1178 bytes) 000059 1787885568.585590 mapwrite 0x00415a thru 0x00aad0 (0x6977 bytes) 000060 1787885568.621025 mapwrite 0x020f7c thru 0x030245 (0xf2ca bytes) 000061 1787885568.691228 read 0x013db3 thru 0x01bc84 (0x7ed2 bytes) 000062 1787885568.692126 trunc from 0x040000 to 0x01e958 (0x216a7 bytes) 000063 1787885568.717311 trunc from 0x01e959 to 0x009b29 (0x14e2f bytes) 000064 1787885568.745065 mapread 0x00439e thru 0x009b29 (0x578c bytes) 000066 1787885568.756624 fallocate 0x037efa thru 0x03ffff (0x8106 bytes) 000068 1787885568.766211 trunc from 0x0050e1 to 0x00568b (0x5ab bytes) 000069 1787885568.774876 mapwrite 0x005ab2 thru 0x0077fe (0x1d4d bytes) 000070 1787885568.800944 mapread 0x003a72 thru 0x009b29 (0x60b8 bytes) 000071 1787885568.805236 write 0x000725 thru 0x00ed71 (0xe64d bytes) 000072 1787885568.809520 trunc from 0x003fde to 0x00a789 (0x67ac bytes) 000073 1787885568.827728 mapwrite 0x0276ea thru 0x02f3c5 (0x7cdc bytes) 000075 1787885568.866345 mapwrite 0x0014d9 thru 0x00c264 (0xad8c bytes) 000077 1787885568.959211 write 0x035017 thru 0x03ffff (0xafe9 bytes) 000078 1787885568.962146 write 0x02113f thru 0x030761 (0xf623 bytes) 000080 1787885568.990866 MIRROR_EXTEND to 2 mirrors 000081 1787885569.903915 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000082 1787885570.110011 punch from 0x016ecd to 0x0246e9 (0xd81d bytes) 000083 1787885570.123734 punch from 0x03e3ce to 0x03ffff (0x1c32 bytes) 000084 1787885570.128667 mapread 0x02de31 thru 0x02ef91 (0x1161 bytes) 000085 1787885570.142137 fallocate 0x02c6b0 thru 0x03451a (0x7e6b bytes) 000086 1787885570.148088 fallocate 0x031ae2 thru 0x0375ae (0x5acd bytes) 000088 1787885570.153953 mapread 0x0366b0 thru 0x03ffff (0x9950 bytes) 000089 1787885570.172718 punch from 0x01a13f to 0x01cd31 (0x2bf3 bytes) 000090 1787885570.182187 write 0x00d810 thru 0x01b93b (0xe12c bytes) 000091 1787885570.199870 write 0x01d6e0 thru 0x027bd6 (0xa4f7 bytes) 000092 1787885570.209983 write 0x01e628 thru 0x02894c (0xa325 bytes) 000093 1787885570.212684 mapwrite 0x013f48 thru 0x0238c8 (0xf981 bytes) 000095 1787885570.233378 punch from 0x01e0c3 to 0x022e42 (0x4d80 bytes) 000096 1787885570.238684 mapwrite 0x02bf6e thru 0x037c15 (0xbca8 bytes) 000097 1787885570.257007 punch from 0x0225b1 to 0x0257e4 (0x3234 bytes) 000098 1787885570.261545 punch from 0x0282ad to 0x034244 (0xbf98 bytes) 000099 1787885570.267764 trunc from 0x040000 to 0x014979 (0x2b686 bytes) 000100 1787885570.287117 read 0x0143c1 thru 0x014979 (0x5b9 bytes) 000101 1787885570.288661 trunc from 0x01497a to 0x011b8f (0x2dea bytes) 000102 1787885570.313643 trunc from 0x010da1 to 0x011b8f (0xdef bytes) 000103 1787885570.319207 write 0x014b9d thru 0x023d13 (0xf177 bytes) 000104 1787885570.322077 mapread 0x0148b7 thru 0x01c21b (0x7965 bytes) 000105 1787885570.323458 trunc from 0x019c5f to 0x01fb07 (0x5ea9 bytes) 000106 1787885570.343437 mapread 0x0195ce thru 0x023d13 (0xa746 bytes) 000107 1787885570.358744 mapwrite 0x01e04f thru 0x0235bf (0x5571 bytes) 000108 1787885570.381861 mapwrite 0x02ac9c thru 0x02e885 (0x3bea bytes) 000109 1787885570.430839 fallocate 0x01da23 thru 0x020a52 (0x3030 bytes) 000110 1787885570.437864 write 0x007e5a thru 0x0086cd (0x874 bytes) 000111 1787885570.449852 punch from 0x01e6c9 to 0x028d6c (0xa6a4 bytes) 000112 1787885570.462248 fallocate 0x01b408 thru 0x026799 (0xb392 bytes) 000113 1787885570.466209 mapread 0x00c264 thru 0x00d024 (0xdc1 bytes) 000114 1787885570.480034 read 0x0298cf thru 0x02e885 (0x4fb7 bytes) 000115 1787885570.481050 read 0x0221a2 thru 0x02cf3b (0xad9a bytes) 000116 1787885570.499137 write 0x01d332 thru 0x02be62 (0xeb31 bytes) 000117 1787885570.500578 mapwrite 0x02e1e0 thru 0x032f90 (0x4db1 bytes) 000118 1787885570.560983 write 0x00cf55 thru 0x018468 (0xb514 bytes) 000119 1787885570.562550 write 0x02ad82 thru 0x02c35d (0x15dc bytes) 000120 1787885570.564015 write 0x025632 thru 0x02cd5e (0x772d bytes) 000121 1787885570.565722 fallocate 0x02805c thru 0x0371d8 (0xf17d bytes) 000122 1787885570.570552 mapread 0x02db4b thru 0x02e626 (0xadc bytes) 000123 1787885570.571759 read 0x002ecf thru 0x008a0b (0x5b3d bytes) 000125 1787885570.587184 trunc from 0x029167 to 0x0330f7 (0x9f91 bytes) 000126 1787885570.602741 trunc from 0x0371d9 to 0x01ee55 (0x18383 bytes) 000127 1787885570.616896 write 0x007093 thru 0x015e6f (0xeddd bytes) 000128 1787885570.620137 trunc from 0x01ee56 to 0x0328f1 (0x13a9c bytes) 000129 1787885570.637018 fallocate 0x023839 thru 0x024e37 (0x15ff bytes) 000130 1787885570.642580 write 0x01f3b8 thru 0x029326 (0x9f6f bytes) 000131 1787885570.652099 mapwrite 0x0162ef thru 0x0219e2 (0xb6f4 bytes) 000132 1787885570.686329 fallocate 0x00704d thru 0x010c6a (0x9c1e bytes) 000133 1787885570.698400 punch from 0x0038d9 to 0x004dc9 (0x14f1 bytes) 000134 1787885570.708068 write 0x016fbe thru 0x02402d (0xd070 bytes) 000136 1787885570.711353 mapread 0x026b73 thru 0x02ede2 (0x8270 bytes) 000137 1787885570.712602 read 0x02c833 thru 0x0328f1 (0x60bf bytes) 000138 1787885570.723098 read 0x000274 thru 0x0064b8 (0x6245 bytes) 000140 1787885570.741872 punch from 0x013f14 to 0x0214cf (0xd5bc bytes) 000142 1787885570.768856 trunc from 0x0328f2 to 0x03de35 (0xb544 bytes) 000143 1787885570.818317 punch from 0x030a02 to 0x03de35 (0xd434 bytes) 000144 1787885570.827002 fallocate 0x01da90 thru 0x020b2f (0x30a0 bytes) 000145 1787885570.845596 trunc from 0x02c922 to 0x030bdf (0x42be bytes) 000147 1787885570.864859 mapread 0x00c106 thru 0x00c6f0 (0x5eb bytes) 000148 1787885570.869732 read 0x03ae6f thru 0x03de35 (0x2fc7 bytes) 000149 1787885570.921591 punch from 0x000c8e to 0x001d84 (0x10f7 bytes) 000150 1787885570.935879 fallocate 0x0331c1 thru 0x03d9de (0xa81e bytes) 000152 1787885570.948795 write 0x012200 thru 0x0127cf (0x5d0 bytes) 000153 1787885570.949714 mapread 0x019bea thru 0x01ab88 (0xf9f bytes) 000154 1787885570.967812 MIRROR_EXTEND to 2 mirrors 000155 1787885571.862159 MIRROR_RESYNC 2 mirrors 000156 1787885572.364246 punch from 0x02b692 to 0x035fe7 (0xa956 bytes) 000158 1787885572.416653 mapwrite 0x001a9b thru 0x00de83 (0xc3e9 bytes) 000159 1787885572.502350 mapread 0x013f68 thru 0x01be8d (0x7f26 bytes) 000160 1787885572.515177 trunc from 0x03de36 to 0x0206b5 (0x1d780 bytes) 000161 1787885572.550319 trunc from 0x0206b6 to 0x0146de (0xbfd7 bytes) 000162 1787885572.580997 fallocate 0x009306 thru 0x00dc19 (0x4914 bytes) 000163 1787885572.588375 read 0x012c0f thru 0x0146de (0x1ad0 bytes) 000164 1787885572.589734 write 0x02ad6e thru 0x03a818 (0xfaab bytes) 000166 1787885572.602879 fallocate 0x002a5d thru 0x008050 (0x55f4 bytes) 000167 1787885572.616600 punch from 0x00347a to 0x005016 (0x1b9d bytes) 000168 1787885572.622424 trunc from 0x01e77c to 0x026cdf (0x8564 bytes) 000170 1787885572.649017 MIRROR_RESYNC 2 mirrors 000171 1787885573.564614 trunc from 0x03a819 to 0x023306 (0x17512 bytes) 000172 1787885573.625914 read 0x006aa2 thru 0x00fcd5 (0x9234 bytes) 000174 1787885573.667350 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000175 1787885573.982069 MIRROR_EXTEND to 2 mirrors 000176 1787885575.121174 mapread 0x01f661 thru 0x023306 (0x3ca6 bytes) 000177 1787885575.185228 trunc from 0x023307 to 0x030b52 (0xd84c bytes) 000178 1787885575.281606 MIRROR_RESYNC 2 mirrors 000179 1787885576.273449 trunc from 0x030b53 to 0x009f52 (0x26c00 bytes) 000180 1787885576.316127 mapread 0x0047ca thru 0x008a7c (0x42b3 bytes) 000181 1787885576.352383 punch from 0x008b9d to 0x009f52 (0x13b6 bytes) 000182 1787885576.366136 trunc from 0x009f53 to 0x01518a (0xb238 bytes) 000185 1787885576.401204 trunc from 0x001e01 to 0x006f9d (0x519d bytes) 000186 1787885576.407604 trunc from 0x01518b to 0x01e098 (0x8f0e bytes) 000188 1787885576.428135 read 0x00ce69 thru 0x012ca9 (0x5e41 bytes) 000189 1787885576.437735 read 0x019cd7 thru 0x01e098 (0x43c2 bytes) 000190 1787885576.459047 mapwrite 0x03ac94 thru 0x03ffff (0x536c bytes) 000191 1787885576.531952 MIRROR_RESYNC 2 mirrors 000192 1787885577.397582 write 0x008f2b thru 0x014c57 (0xbd2d bytes) 000193 1787885577.484259 mapread 0x009a81 thru 0x00d725 (0x3ca5 bytes) 000194 1787885577.486943 read 0x02ceae thru 0x030b83 (0x3cd6 bytes) 000195 1787885577.500879 read 0x0381df thru 0x03ffff (0x7e21 bytes) 000196 1787885577.522063 trunc from 0x040000 to 0x0256de (0x1a921 bytes) 000197 1787885577.547882 fallocate 0x01b52f thru 0x022f0c (0x79de bytes) 000198 1787885577.559379 MIRROR_RESYNC 2 mirrors 000199 1787885578.205194 write 0x03d134 thru 0x03ffff (0x2ecc bytes) 000200 1787885578.238104 mapwrite 0x0221dd thru 0x02eec7 (0xcceb bytes) 000201 1787885578.289826 write 0x0312c1 thru 0x032cc4 (0x1a04 bytes) 000202 1787885578.291634 mapwrite 0x000897 thru 0x00a05a (0x97c4 bytes) 000203 1787885578.332981 trunc from 0x040000 to 0x02e6fe (0x11901 bytes) 000204 1787885578.387003 fallocate 0x01ab75 thru 0x01de18 (0x32a4 bytes) 000205 1787885578.392951 mapread 0x01127d thru 0x01d273 (0xbff7 bytes) 000206 1787885578.411960 trunc from 0x02e6ff to 0x034894 (0x6196 bytes) 000207 1787885578.427308 trunc from 0x034895 to 0x02495f (0xff35 bytes) 000208 1787885578.441942 mapwrite 0x00d1ab thru 0x00dded (0xc43 bytes) 000209 1787885578.455408 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000210 1787885578.581461 read 0x0154c6 thru 0x01e8c9 (0x9404 bytes) 000211 1787885578.590769 mapread 0x01e0ba thru 0x02495f (0x68a6 bytes) 000212 1787885578.607319 read 0x012455 thru 0x01d9cc (0xb578 bytes) 000213 1787885578.616104 trunc from 0x0123c5 to 0x014e84 (0x2ac0 bytes) 000214 1787885578.620958 mapwrite 0x02cfa0 thru 0x02f2a5 (0x2306 bytes) 000215 1787885578.664224 trunc from 0x02f2a6 to 0x01ddcd (0x114d8 bytes) 000217 1787885578.677722 punch from 0x0048cb to 0x008c80 (0x43b6 bytes) 000218 1787885578.683053 trunc from 0x004414 to 0x005037 (0xc24 bytes) 000219 1787885578.689060 mapread 0x00d976 thru 0x0123fe (0x4a89 bytes) 000220 1787885578.695656 fallocate 0x030cbe thru 0x0369ed (0x5d30 bytes) 000221 1787885578.700261 punch from 0x0098a2 to 0x00c621 (0x2d80 bytes) 000224 1787885578.710853 MIRROR_EXTEND to 2 mirrors 000225 1787885579.477356 read 0x00aea8 thru 0x0191a6 (0xe2ff bytes) 000226 1787885579.493845 mapread 0x008ce4 thru 0x00d39d (0x46ba bytes) 000227 1787885579.520170 fallocate 0x03aeba thru 0x03ffff (0x5146 bytes) 000228 1787885579.552787 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1088f5da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1088f5da Mirror 3: 0x1088f5da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000229 1787885580.735898 mapread 0x02463d thru 0x032906 (0xe2ca bytes) 000230 1787885580.799019 mapwrite 0x01c9f8 thru 0x01cca7 (0x2b0 bytes) 000231 1787885580.913212 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1088f5da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1088f5da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000232 1787885581.159765 trunc from 0x01ce00 to 0x023737 (0x6938 bytes) 000233 1787885581.164082 mapread 0x001ffc thru 0x00d4c9 (0xb4ce bytes) 000234 1787885581.183096 mapread 0x033952 thru 0x03ffff (0xc6ae bytes) 000235 1787885581.206326 write 0x01c71f thru 0x022a96 (0x6378 bytes) 000237 1787885581.232324 punch from 0x010be5 to 0x011bb2 (0xfce bytes) 000238 1787885581.248171 MIRROR_RESYNC 2 mirrors 000239 1787885582.328113 mapwrite 0x03607e thru 0x03ffff (0x9f82 bytes) 000240 1787885582.422215 write 0x004548 thru 0x00e2b6 (0x9d6f bytes) 000241 1787885582.447745 trunc from 0x040000 to 0x033147 (0xceb8 bytes) 000242 1787885582.477977 trunc from 0x007886 to 0x00e104 (0x687f bytes) 000243 1787885582.506795 fallocate 0x0084b3 thru 0x00905f (0xbad bytes) 000244 1787885582.525124 read 0x029d47 thru 0x033147 (0x9401 bytes) 000245 1787885582.554456 read 0x00bcd4 thru 0x014743 (0x8a70 bytes) 000246 1787885582.576079 mapwrite 0x01a717 thru 0x01e19c (0x3a86 bytes) 000247 1787885582.605893 fallocate 0x03842a thru 0x03e903 (0x64da bytes) 000248 1787885582.612524 mapwrite 0x0109d4 thru 0x019405 (0x8a32 bytes) 000249 1787885582.642616 trunc from 0x001b4e to 0x005449 (0x38fc bytes) 000250 1787885582.652869 mapwrite 0x0240bb thru 0x02b74a (0x7690 bytes) 000251 1787885582.678646 trunc from 0x033148 to 0x030dee (0x2359 bytes) 000252 1787885582.698365 read 0x0281cc thru 0x02ff48 (0x7d7d bytes) 000253 1787885582.699068 write 0x0063e4 thru 0x00e5d5 (0x81f2 bytes) 000254 1787885582.700646 trunc from 0x030def to 0x027fad (0x8e41 bytes) 000255 1787885582.731902 punch from 0x01336e to 0x01a90b (0x759e bytes) 000256 1787885582.742503 mapwrite 0x00ed20 thru 0x013dbb (0x509c bytes) 000257 1787885582.785194 read 0x00cfda thru 0x0136b6 (0x66dd bytes) 000258 1787885582.791030 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27fae): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27fae): Mirror 2: 0xfa659229 (stale) CRC-32 checksum value for chunk [0, 0x27fae): Mirror 1: 0x3972ccf5 Mirror 3: 0x3972ccf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27fae: skipped 000259 1787885583.673151 trunc from 0x027fae to 0x035e0b (0xde5e bytes) 000260 1787885583.767955 mapread 0x016dc2 thru 0x01ad77 (0x3fb6 bytes) 000261 1787885583.794175 trunc from 0x035e0c to 0x00a35d (0x2baae bytes) 000262 1787885583.819249 punch from 0x008ef2 to 0x00a35d (0x146c bytes) 000263 1787885583.826901 read 0x00260d thru 0x00a35d (0x7d51 bytes) 000264 1787885583.855869 write 0x028272 thru 0x02c3f5 (0x4184 bytes) 000265 1787885583.862556 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c3f6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3f6): Mirror 2: 0xcb3e6f3f (stale) Mirror 3: 0x3ce1357c (stale) CRC-32 checksum value for chunk [0, 0x2c3f6): Mirror 1: 0x5fd18f5a Mirror 4: 0x5fd18f5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3f6: skipped 000266 1787885584.787946 mapread 0x0278be thru 0x02a382 (0x2ac5 bytes) 000267 1787885584.824970 mapwrite 0x0279ec thru 0x02e9f7 (0x700c bytes) 000268 1787885584.908896 mapwrite 0x0047f7 thru 0x008412 (0x3c1c bytes) 000269 1787885584.933761 mapread 0x01d164 thru 0x028911 (0xb7ae bytes) 000270 1787885584.946209 trunc from 0x01704e to 0x01cf6f (0x5f22 bytes) 000272 1787885584.951873 fallocate 0x00c2f1 thru 0x00e0aa (0x1dba bytes) 000273 1787885584.957020 punch from 0x0010f4 to 0x00b35a (0xa267 bytes) 000274 1787885584.962685 mapread 0x0105d5 thru 0x017ed7 (0x7903 bytes) 000275 1787885584.973918 mapwrite 0x018678 thru 0x019b41 (0x14ca bytes) 000276 1787885584.988779 read 0x01323e thru 0x0142c1 (0x1084 bytes) 000277 1787885584.996986 MIRROR_RESYNC 4 mirrors 000278 1787885586.131256 read 0x00e55b thru 0x016dad (0x8853 bytes) 000279 1787885586.138719 fallocate 0x009349 thru 0x00f63c (0x62f4 bytes) 000280 1787885586.191571 punch from 0x019be5 to 0x0237fb (0x9c17 bytes) 000281 1787885586.213331 MIRROR_RESYNC 4 mirrors 000282 1787885587.232890 punch from 0x008805 to 0x00ee0a (0x6606 bytes) 000283 1787885587.272352 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e9f8): Mirror 2: 0xeb77a8ab (stale) Mirror 3: 0xeb77a8ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9f8): Mirror 2: 0xeb77a8ab (stale) Mirror 3: 0xeb77a8ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9f8: skipped 000284 1787885587.601074 mapwrite 0x020d45 thru 0x02959a (0x8856 bytes) 000285 1787885587.639063 trunc from 0x02e9f8 to 0x033697 (0x4ca0 bytes) 000286 1787885587.657759 trunc from 0x00c66d to 0x018ccf (0xc663 bytes) 000287 1787885587.665144 trunc from 0x025f79 to 0x02633d (0x3c5 bytes) 000288 1787885587.671331 write 0x018e30 thru 0x01c2a7 (0x3478 bytes) 000289 1787885587.675995 fallocate 0x01b269 thru 0x026fe9 (0xbd81 bytes) 000290 1787885587.680826 read 0x007894 thru 0x00e363 (0x6ad0 bytes) 000291 1787885587.689672 mapwrite 0x03d5b5 thru 0x03ffff (0x2a4b bytes) 000292 1787885587.755316 mapwrite 0x016548 thru 0x019f71 (0x3a2a bytes) 000293 1787885587.791741 punch from 0x03647a to 0x03760b (0x1192 bytes) 000294 1787885587.802868 fallocate 0x01b0da thru 0x027166 (0xc08d bytes) 000295 1787885587.823562 punch from 0x01c047 to 0x02626d (0xa227 bytes) 000296 1787885587.834073 read 0x01dcc7 thru 0x020a5e (0x2d98 bytes) 000297 1787885587.879040 fallocate 0x0319ee thru 0x036a3d (0x5050 bytes) 000298 1787885587.892785 mapread 0x03fb9b thru 0x03ffff (0x465 bytes) 000299 1787885587.901643 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd68d2684 (stale) Mirror 3: 0xd68d2684 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb86ffb85 Mirror 4: 0xb86ffb85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000300 1787885589.197886 read 0x012eb4 thru 0x01e3b8 (0xb505 bytes) 000301 1787885589.228293 mapread 0x03a694 thru 0x03ffff (0x596c bytes) 000302 1787885589.287327 trunc from 0x03f912 to 0x03ffff (0x6ee bytes) 000303 1787885589.338663 mapwrite 0x0153b7 thru 0x016720 (0x136a bytes) 000304 1787885589.363368 mapwrite 0x025d89 thru 0x02846c (0x26e4 bytes) 000305 1787885589.388914 read 0x017efa thru 0x0183e3 (0x4ea bytes) 000306 1787885589.390129 read 0x032b76 thru 0x03ffff (0xd48a bytes) 000307 1787885589.416294 fallocate 0x02f7c0 thru 0x03548f (0x5cd0 bytes) 000308 1787885589.430535 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd68d2684 (stale) Mirror 3: 0xd68d2684 (stale) Mirror 4: 0xb86ffb85 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7b13be1b Mirror 5: 0x7b13be1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000309 1787885590.617150 trunc from 0x03c97b to 0x03ffff (0x3685 bytes) 000310 1787885590.683691 read 0x0159bb thru 0x01a093 (0x46d9 bytes) 000311 1787885590.699102 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd68d2684 (stale) Mirror 3: 0xd68d2684 (stale) Mirror 4: 0xb86ffb85 (stale) Mirror 5: 0x7b13be1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86c3f08c Mirror 6: 0x86c3f08c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000312 1787885591.857837 read 0x030b52 thru 0x034ad5 (0x3f84 bytes) 000313 1787885591.875240 write 0x020e1e thru 0x027bfa (0x6ddd bytes) 000314 1787885591.929964 trunc from 0x00461c to 0x0092bf (0x4ca4 bytes) 000315 1787885591.935262 fallocate 0x01af4c thru 0x024cba (0x9d6f bytes) 000316 1787885591.943842 trunc from 0x022a91 to 0x031860 (0xedd0 bytes) 000317 1787885591.959313 mapread 0x0174f0 thru 0x020cba (0x97cb bytes) 000318 1787885591.978705 write 0x01f758 thru 0x021c48 (0x24f1 bytes) 000320 1787885591.980685 punch from 0x02ab69 to 0x02f4a2 (0x493a bytes) 000322 1787885591.985943 trunc from 0x040000 to 0x00dd54 (0x322ab bytes) 000323 1787885592.016720 read 0x005253 thru 0x00ce86 (0x7c34 bytes) 000324 1787885592.026863 mapwrite 0x01462a thru 0x01b325 (0x6cfc bytes) 000325 1787885592.085238 trunc from 0x01b326 to 0x038b22 (0x1d7fd bytes) 000326 1787885592.102996 fallocate 0x03f499 thru 0x03ffff (0xb67 bytes) 000327 1787885592.109090 write 0x01a97a thru 0x020aa9 (0x6130 bytes) 000328 1787885592.116045 trunc from 0x040000 to 0x01f752 (0x208ad bytes) 000329 1787885592.130285 mapread 0x016bb2 thru 0x01e1f4 (0x7643 bytes) 000330 1787885592.135466 MIRROR_RESYNC 6 mirrors 000331 1787885593.367615 mapwrite 0x028981 thru 0x02d603 (0x4c83 bytes) 000332 1787885593.494908 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d604): Mirror 2: 0xabe33f06 (stale) Mirror 4: 0xabe33f06 (stale) Mirror 5: 0xabe33f06 (stale) Mirror 6: 0xabe33f06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d604): Mirror 2: 0xabe33f06 (stale) Mirror 4: 0xabe33f06 (stale) Mirror 5: 0xabe33f06 (stale) Mirror 6: 0xabe33f06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d604: skipped 000333 1787885593.974198 punch from 0x024fca to 0x025ce7 (0xd1e bytes) 000334 1787885593.978455 read 0x0178e9 thru 0x01f30b (0x7a23 bytes) 000335 1787885593.988543 trunc from 0x02d604 to 0x03fff0 (0x129ed bytes) 000336 1787885594.009710 read 0x020b52 thru 0x0257d7 (0x4c86 bytes) 000337 1787885594.020843 punch from 0x039308 to 0x03fff0 (0x6ce9 bytes) 000338 1787885594.031939 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fff1): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fff1): Mirror 2: 0xe9d1de67 (stale) Mirror 4: 0xe9d1de67 (stale) Mirror 5: 0xe9d1de67 (stale) Mirror 6: 0xe9d1de67 (stale) CRC-32 checksum value for chunk [0, 0x3fff1): Mirror 1: 0x3fa38f75 Mirror 7: 0x3fa38f75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fff1: skipped 000339 1787885595.564919 fallocate 0x025e41 thru 0x0301df (0xa39f bytes) 000340 1787885595.613193 write 0x00c05d thru 0x0153c6 (0x936a bytes) 000341 1787885595.632474 write 0x014b6e thru 0x02128e (0xc721 bytes) 000343 1787885595.642950 trunc from 0x03fff1 to 0x01a310 (0x25ce0 bytes) 000345 1787885595.685591 trunc from 0x010d8e to 0x011163 (0x3d6 bytes) 000346 1787885595.709569 punch from 0x000b86 to 0x00d843 (0xccbe bytes) 000347 1787885595.715357 trunc from 0x0108d9 to 0x019b0b (0x9233 bytes) 000348 1787885595.722192 fallocate 0x021716 thru 0x02eac7 (0xd3b2 bytes) 000349 1787885595.728035 mapwrite 0x008d16 thru 0x00afc9 (0x22b4 bytes) 000350 1787885595.765107 trunc from 0x01a311 to 0x0256f8 (0xb3e8 bytes) 000351 1787885595.798459 read 0x01b0b3 thru 0x0256f8 (0xa646 bytes) 000352 1787885595.818694 punch from 0x014ec9 to 0x022b7c (0xdcb4 bytes) 000353 1787885595.828030 trunc from 0x0078f3 to 0x00f8d0 (0x7fde bytes) 000354 1787885595.836895 trunc from 0x0256f9 to 0x0113a1 (0x14357 bytes) 000355 1787885595.861462 mapread 0x0017cd thru 0x00e511 (0xcd45 bytes) 000356 1787885595.873134 fallocate 0x034489 thru 0x037f38 (0x3ab0 bytes) 000357 1787885595.879230 read 0x028f34 thru 0x02e8f4 (0x59c1 bytes) 000358 1787885595.911117 mapwrite 0x019d9d thru 0x01a394 (0x5f8 bytes) 000359 1787885595.926307 trunc from 0x032c24 to 0x037f38 (0x5315 bytes) 000360 1787885595.932773 trunc from 0x01d413 to 0x02385c (0x644a bytes) 000361 1787885595.937770 trunc from 0x037f39 to 0x03c857 (0x491f bytes) 000362 1787885595.956856 mapwrite 0x0313d0 thru 0x035b6f (0x47a0 bytes) 000363 1787885596.001884 trunc from 0x03c858 to 0x01f7e4 (0x1d073 bytes) 000364 1787885596.025862 read 0x01e843 thru 0x01ec38 (0x3f6 bytes) 000365 1787885596.031915 trunc from 0x01f7e5 to 0x003c54 (0x1bb90 bytes) 000366 1787885596.055333 trunc from 0x003c55 to 0x00aa2d (0x6dd9 bytes) 000367 1787885596.081172 trunc from 0x00aa2e to 0x032b1e (0x280f1 bytes) 000369 1787885596.103608 mapwrite 0x006b8a thru 0x00e9ab (0x7e22 bytes) 000370 1787885596.189013 punch from 0x01e7f7 to 0x029ac0 (0xb2ca bytes) 000371 1787885596.194841 trunc from 0x032b1f to 0x03f380 (0xc862 bytes) 000373 1787885596.219462 trunc from 0x031525 to 0x03b037 (0x9b13 bytes) 000374 1787885596.227547 mapwrite 0x028e7a thru 0x033788 (0xa90f bytes) 000375 1787885596.306435 fallocate 0x03a9c0 thru 0x03ffff (0x5640 bytes) 000376 1787885596.313861 write 0x0377b8 thru 0x037f49 (0x792 bytes) 000377 1787885596.315923 read 0x0058b1 thru 0x00fcd2 (0xa422 bytes) 000378 1787885596.331698 mapread 0x032c86 thru 0x03ffff (0xd37a bytes) 000379 1787885596.333814 write 0x01bf03 thru 0x022237 (0x6335 bytes) 000380 1787885596.336016 mapwrite 0x032dc5 thru 0x03ffff (0xd23b bytes) 000381 1787885596.355905 trunc from 0x0089e7 to 0x012cdf (0xa2f9 bytes) 000383 1787885596.362880 fallocate 0x02b33c thru 0x02f9cc (0x4691 bytes) 000384 1787885596.369699 write 0x00c872 thru 0x00ce64 (0x5f3 bytes) 000386 1787885596.381677 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc045ca7b (stale) Mirror 4: 0xc045ca7b (stale) Mirror 5: 0xc045ca7b (stale) Mirror 6: 0xc045ca7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc045ca7b (stale) Mirror 4: 0xc045ca7b (stale) Mirror 5: 0xc045ca7b (stale) Mirror 6: 0xc045ca7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000387 1787885596.950903 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc045ca7b (stale) Mirror 4: 0xc045ca7b (stale) Mirror 5: 0xc045ca7b (stale) Mirror 6: 0xc045ca7b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7fbb2a86 Mirror 7: 0x7fbb2a86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1787885598.137606 trunc from 0x040000 to 0x0301cf (0xfe30 bytes) 000389 1787885598.200997 trunc from 0x01decf to 0x0282d3 (0xa405 bytes) 000391 1787885598.207967 write 0x00b02e thru 0x010c25 (0x5bf8 bytes) 000392 1787885598.217675 trunc from 0x0301d0 to 0x011af4 (0x1e6db bytes) 000393 1787885598.245269 trunc from 0x011af5 to 0x018d75 (0x7281 bytes) 000394 1787885598.259598 fallocate 0x021eeb thru 0x028f56 (0x706c bytes) 000395 1787885598.266295 trunc from 0x00aa02 to 0x00c127 (0x1726 bytes) 000396 1787885598.282582 trunc from 0x018d76 to 0x01f03f (0x62ca bytes) 000397 1787885598.307390 punch from 0x0032b3 to 0x005507 (0x2255 bytes) 000399 1787885598.314648 punch from 0x01d87a to 0x01f03f (0x17c6 bytes) 000400 1787885598.319420 trunc from 0x00df7e to 0x01a91b (0xc99e bytes) 000401 1787885598.326456 fallocate 0x026be6 thru 0x029c26 (0x3041 bytes) 000402 1787885598.333768 write 0x0214b1 thru 0x02f837 (0xe387 bytes) 000403 1787885598.336878 trunc from 0x017c4d to 0x020a08 (0x8dbc bytes) 000404 1787885598.363741 mapread 0x017923 thru 0x0232f1 (0xb9cf bytes) 000405 1787885598.384981 read 0x0051b8 thru 0x0080d1 (0x2f1a bytes) 000406 1787885598.390797 punch from 0x002dc1 to 0x002ebb (0xfb bytes) 000407 1787885598.396298 mapwrite 0x02dbbe thru 0x038ce5 (0xb128 bytes) 000408 1787885598.446898 read 0x01c73a thru 0x01d272 (0xb39 bytes) 000409 1787885598.448542 write 0x0179e5 thru 0x018385 (0x9a1 bytes) 000410 1787885598.450767 mapwrite 0x016eb4 thru 0x01db01 (0x6c4e bytes) 000411 1787885598.474888 read 0x03760c thru 0x038ce5 (0x16da bytes) 000412 1787885598.475431 mapwrite 0x025e02 thru 0x02f4d2 (0x96d1 bytes) 000413 1787885598.500685 trunc from 0x038ce6 to 0x03168d (0x7658 bytes) 000414 1787885598.526753 MIRROR_RESYNC 6 mirrors 000415 1787885600.286897 punch from 0x0019a5 to 0x0022df (0x93b bytes) 000416 1787885600.311432 read 0x00731e thru 0x00a18e (0x2e71 bytes) 000417 1787885600.319829 punch from 0x01c1bc to 0x025945 (0x978a bytes) 000418 1787885600.325090 mapwrite 0x02b4e3 thru 0x038f85 (0xdaa3 bytes) 000419 1787885600.406148 write 0x0050a6 thru 0x008aeb (0x3a46 bytes) 000420 1787885600.415319 read 0x008412 thru 0x00e321 (0x5f10 bytes) 000421 1787885600.431888 mapwrite 0x037c73 thru 0x03ffff (0x838d bytes) 000422 1787885600.501311 mapread 0x00c083 thru 0x016ed7 (0xae55 bytes) 000423 1787885600.503209 mapread 0x003eb2 thru 0x00f0b3 (0xb202 bytes) 000424 1787885600.510100 read 0x01983f thru 0x026c4e (0xd410 bytes) 000426 1787885600.514379 read 0x004925 thru 0x009a5a (0x5136 bytes) 000428 1787885600.515652 write 0x016adf thru 0x01d46d (0x698f bytes) 000429 1787885600.520093 write 0x007179 thru 0x0104a1 (0x9329 bytes) 000430 1787885600.523278 trunc from 0x010526 to 0x0178dd (0x73b8 bytes) 000431 1787885600.548945 mapwrite 0x006855 thru 0x011adb (0xb287 bytes) 000432 1787885600.614057 punch from 0x016d71 to 0x018be8 (0x1e78 bytes) 000433 1787885600.637791 write 0x009d2d thru 0x01322f (0x9503 bytes) 000434 1787885600.650945 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3dd80d3f (stale) Mirror 5: 0x3dd80d3f (stale) Mirror 6: 0x3dd80d3f (stale) Mirror 7: 0x3dd80d3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3dd80d3f (stale) Mirror 5: 0x3dd80d3f (stale) Mirror 6: 0x3dd80d3f (stale) Mirror 7: 0x3dd80d3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000435 1787885601.112176 write 0x022816 thru 0x023c90 (0x147b bytes) 000436 1787885601.132466 mapwrite 0x00a1a7 thru 0x01265f (0x84b9 bytes) 000437 1787885601.170836 fallocate 0x02189f thru 0x02e195 (0xc8f7 bytes) 000438 1787885601.176805 fallocate 0x00735f thru 0x00cbb4 (0x5856 bytes) 000439 1787885601.182070 read 0x002578 thru 0x012530 (0xffb9 bytes) 000440 1787885601.188660 write 0x024634 thru 0x029869 (0x5236 bytes) 000441 1787885601.199281 mapread 0x023b24 thru 0x02d054 (0x9531 bytes) 000442 1787885601.212974 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3dd80d3f (stale) Mirror 6: 0x3dd80d3f (stale) Mirror 7: 0x3dd80d3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3dd80d3f (stale) Mirror 6: 0x3dd80d3f (stale) Mirror 7: 0x3dd80d3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000443 1787885601.579083 punch from 0x028489 to 0x02b130 (0x2ca8 bytes) 000444 1787885601.585911 trunc from 0x040000 to 0x03799b (0x8664 bytes) 000445 1787885601.619871 mapread 0x012eb3 thru 0x021b74 (0xecc2 bytes) 000446 1787885601.660089 trunc from 0x03799c to 0x012d04 (0x24c97 bytes) 000447 1787885601.679907 trunc from 0x00e5df to 0x00f4db (0xefd bytes) 000448 1787885601.685933 fallocate 0x00f4a0 thru 0x00f683 (0x1e4 bytes) 000449 1787885601.691924 read 0x00b8bc thru 0x012d04 (0x7449 bytes) 000450 1787885601.708802 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12d05): Mirror 5: 0x92b7ae94 (stale) Mirror 6: 0x92b7ae94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12d05): Mirror 5: 0x92b7ae94 (stale) Mirror 6: 0x92b7ae94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12d05: skipped 000451 1787885602.073381 mapread 0x012068 thru 0x012d04 (0xc9d bytes) 000452 1787885602.092455 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12d05): Mirror 6: 0x92b7ae94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12d05): Mirror 6: 0x92b7ae94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12d05: skipped 000454 1787885602.387921 mapread 0x0081ab thru 0x012d04 (0xab5a bytes) 000455 1787885602.412328 fallocate 0x02e340 thru 0x03328b (0x4f4c bytes) 000456 1787885602.418571 mapread 0x00cd72 thru 0x012d04 (0x5f93 bytes) 000457 1787885602.420248 mapread 0x00ccdd thru 0x011447 (0x476b bytes) 000458 1787885602.423909 mapwrite 0x01d919 thru 0x02d75e (0xfe46 bytes) 000460 1787885602.512445 trunc from 0x02d75f to 0x032e90 (0x5732 bytes) 000461 1787885602.535567 mapread 0x025fea thru 0x028b09 (0x2b20 bytes) 000462 1787885602.537011 punch from 0x021b84 to 0x023923 (0x1da0 bytes) 000463 1787885602.550398 trunc from 0x032e91 to 0x02f001 (0x3e8f bytes) 000464 1787885602.576449 trunc from 0x02f002 to 0x010449 (0x1ebb8 bytes) 000465 1787885602.599709 mapread 0x002924 thru 0x008527 (0x5c04 bytes) 000467 1787885602.608767 punch from 0x000409 to 0x005e4d (0x5a45 bytes) 000468 1787885602.614970 write 0x03f4af thru 0x03ffff (0xb51 bytes) 000469 1787885602.620796 mapread 0x036b18 thru 0x03b5e8 (0x4ad1 bytes) 000470 1787885602.648525 fallocate 0x02d380 thru 0x032303 (0x4f84 bytes) 000471 1787885602.657705 mapwrite 0x025a24 thru 0x03441a (0xe9f7 bytes) 000472 1787885602.757026 trunc from 0x03c5d7 to 0x03ffff (0x3a29 bytes) 000473 1787885602.764591 punch from 0x014f0b to 0x01a14d (0x5243 bytes) 000475 1787885602.770652 punch from 0x030715 to 0x03c455 (0xbd41 bytes) 000476 1787885602.776034 trunc from 0x040000 to 0x022e56 (0x1d1a9 bytes) 000477 1787885602.800032 punch from 0x008da2 to 0x00b5f7 (0x2856 bytes) 000478 1787885602.811971 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22e57): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e57): Mirror 6: 0x2e753b00 (stale) CRC-32 checksum value for chunk [0, 0x22e57): Mirror 1: 0x8c5e211d Mirror 7: 0x8c5e211d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e57: skipped 000479 1787885604.041924 trunc from 0x00ab46 to 0x00fdf5 (0x52b0 bytes) 000480 1787885604.097379 mapread 0x011860 thru 0x019f88 (0x8729 bytes) 000481 1787885604.132542 mapread 0x020077 thru 0x022e56 (0x2de0 bytes) 000482 1787885604.169347 MIRROR_RESYNC 3 mirrors 000483 1787885605.327065 mapread 0x00c362 thru 0x014959 (0x85f8 bytes) 000484 1787885605.360139 fallocate 0x0061fb thru 0x011575 (0xb37b bytes) 000485 1787885605.426404 trunc from 0x006228 to 0x009ac6 (0x389f bytes) 000486 1787885605.445924 punch from 0x007952 to 0x011848 (0x9ef7 bytes) 000487 1787885605.455858 fallocate 0x018ced thru 0x021e89 (0x919d bytes) 000488 1787885605.468439 read 0x020e22 thru 0x022e56 (0x2035 bytes) 000489 1787885605.492331 punch from 0x00f713 to 0x01f211 (0xfaff bytes) 000490 1787885605.504904 fallocate 0x0209ee thru 0x02a0a6 (0x96b9 bytes) 000491 1787885605.515313 read 0x021271 thru 0x022e56 (0x1be6 bytes) 000492 1787885605.519019 trunc from 0x0142c3 to 0x022371 (0xe0af bytes) 000493 1787885605.529190 mapread 0x01072f thru 0x015447 (0x4d19 bytes) 000494 1787885605.566283 trunc from 0x015a7f to 0x01b09d (0x561f bytes) 000495 1787885605.572025 fallocate 0x01c0bd thru 0x02bbfe (0xfb42 bytes) 000496 1787885605.578725 write 0x003892 thru 0x007baf (0x431e bytes) 000497 1787885605.579622 read 0x02a552 thru 0x02bbfe (0x16ad bytes) 000498 1787885605.587789 trunc from 0x00cfbf to 0x01132c (0x436e bytes) 000499 1787885605.596581 fallocate 0x0188c7 thru 0x019a2b (0x1165 bytes) 000500 1787885605.602411 trunc from 0x00b383 to 0x00c5b2 (0x1230 bytes) 000501 1787885605.614311 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bbff): Mirror 6: 0xf4f51b8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bbff): Mirror 6: 0xf4f51b8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bbff: skipped 000502 1787885606.032084 read 0x01b806 thru 0x020e54 (0x564f bytes) 000503 1787885606.046017 mapread 0x024054 thru 0x026384 (0x2331 bytes) 000504 1787885606.068762 mapread 0x018f19 thru 0x01ae77 (0x1f5f bytes) 000505 1787885606.080252 mapwrite 0x01ef41 thru 0x0251f8 (0x62b8 bytes) 000506 1787885606.104398 MIRROR_RESYNC 2 mirrors 000507 1787885607.013288 mapread 0x01ba64 thru 0x02963b (0xdbd8 bytes) 000508 1787885607.044256 punch from 0x0035d4 to 0x005c9d (0x26ca bytes) 000509 1787885607.085721 fallocate 0x011bb0 thru 0x01387b (0x1ccc bytes) 000510 1787885607.092816 punch from 0x01f65a to 0x02bbfe (0xc5a5 bytes) 000511 1787885607.102889 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000512 1787885607.263234 punch from 0x0291b2 to 0x02bbfe (0x2a4d bytes) 000513 1787885607.268651 mapwrite 0x01c131 thru 0x02a629 (0xe4f9 bytes) 000514 1787885607.342135 trunc from 0x02bbff to 0x0048fe (0x27300 bytes) 000515 1787885607.391121 trunc from 0x0048ff to 0x0006ac (0x4252 bytes) 000516 1787885607.407010 write 0x02468e thru 0x024d2a (0x69d bytes) 000517 1787885607.408982 write 0x03dfc0 thru 0x03ffff (0x2040 bytes) 000518 1787885607.410662 punch from 0x01fe93 to 0x0218bf (0x1a2d bytes) 000519 1787885607.416977 fallocate 0x01c96a thru 0x02a6c0 (0xdd57 bytes) 000520 1787885607.422590 fallocate 0x02b72b thru 0x037138 (0xba0e bytes) 000521 1787885607.427475 fallocate 0x02f58f thru 0x02ff77 (0x9e9 bytes) 000522 1787885607.432977 read 0x0083cf thru 0x014b04 (0xc736 bytes) 000523 1787885607.444740 fallocate 0x02d49a thru 0x02f980 (0x24e7 bytes) 000524 1787885607.450627 mapwrite 0x028454 thru 0x028bb7 (0x764 bytes) 000526 1787885607.509551 trunc from 0x040000 to 0x006426 (0x39bd9 bytes) 000527 1787885607.524353 trunc from 0x0019ab to 0x006426 (0x4a7c bytes) 000528 1787885607.537057 write 0x01e796 thru 0x01fcdf (0x154a bytes) 000529 1787885607.540160 trunc from 0x01fce0 to 0x02bb4f (0xbe70 bytes) 000530 1787885607.571726 punch from 0x0163f3 to 0x01d6a3 (0x72b1 bytes) 000531 1787885607.577394 mapwrite 0x03cfd6 thru 0x03ffff (0x302a bytes) 000532 1787885607.673668 read 0x01c155 thru 0x02457f (0x842b bytes) 000533 1787885607.702469 read 0x00148e thru 0x0091eb (0x7d5e bytes) 000534 1787885607.703121 trunc from 0x040000 to 0x014ba9 (0x2b456 bytes) 000535 1787885607.740686 fallocate 0x033b6c thru 0x03ffff (0xc494 bytes) 000536 1787885607.762356 read 0x03da49 thru 0x03ffff (0x25b7 bytes) 000537 1787885607.791907 mapread 0x017407 thru 0x019662 (0x225c bytes) 000538 1787885607.794464 mapwrite 0x001192 thru 0x00d065 (0xbed4 bytes) 000540 1787885607.842577 trunc from 0x017b54 to 0x01d0ff (0x55ac bytes) 000541 1787885607.850826 read 0x026db8 thru 0x028951 (0x1b9a bytes) 000543 1787885607.858176 write 0x02f211 thru 0x031633 (0x2423 bytes) 000544 1787885607.862333 punch from 0x0023bc to 0x00ef80 (0xcbc5 bytes) 000545 1787885607.869355 read 0x00da17 thru 0x00fb0b (0x20f5 bytes) 000546 1787885607.880932 mapwrite 0x027506 thru 0x0327f6 (0xb2f1 bytes) 000547 1787885607.925326 read 0x018977 thru 0x01df60 (0x55ea bytes) 000548 1787885607.927609 mapwrite 0x03b3a7 thru 0x03dde0 (0x2a3a bytes) 000549 1787885607.986588 MIRROR_EXTEND to 2 mirrors 000550 1787885608.879115 mapread 0x0052ef thru 0x00590b (0x61d bytes) 000551 1787885608.906474 mapread 0x03aa00 thru 0x03ffff (0x5600 bytes) 000552 1787885608.975992 trunc from 0x040000 to 0x01d259 (0x22da6 bytes) 000553 1787885609.060752 trunc from 0x01d25a to 0x029550 (0xc2f7 bytes) 000554 1787885609.081753 trunc from 0x01e0ca to 0x0248b1 (0x67e8 bytes) 000555 1787885609.087150 trunc from 0x029551 to 0x03df1e (0x149ce bytes) 000556 1787885609.103420 trunc from 0x03df1f to 0x011688 (0x2c896 bytes) 000557 1787885609.116209 trunc from 0x009e43 to 0x011688 (0x7846 bytes) 000558 1787885609.122442 mapwrite 0x02af43 thru 0x037edf (0xcf9d bytes) 000559 1787885609.175394 read 0x02097f thru 0x02c1fa (0xb87c bytes) 000561 1787885609.185764 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 2: 0x21d1f72e (stale) CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 1: 0x84ca8d91 Mirror 3: 0x84ca8d91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ee0: skipped 000562 1787885609.999723 trunc from 0x013bed to 0x013ea8 (0x2bc bytes) 000563 1787885610.047247 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 3: 0x84ca8d91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 3: 0x84ca8d91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ee0: skipped 000564 1787885610.400417 mapwrite 0x013172 thru 0x018d1c (0x5bab bytes) 000565 1787885610.456647 fallocate 0x0323e8 thru 0x03ae4c (0x8a65 bytes) 000566 1787885610.470384 write 0x0269b8 thru 0x02a5ee (0x3c37 bytes) 000567 1787885610.506832 trunc from 0x037ee0 to 0x002b9b (0x35344 bytes) 000568 1787885610.572844 fallocate 0x0324f3 thru 0x039f93 (0x7aa1 bytes) 000569 1787885610.584253 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b9c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9c): Mirror 3: 0x752a306c (stale) CRC-32 checksum value for chunk [0, 0x2b9c): Mirror 1: 0x752a306c Mirror 4: 0x752a306c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9c: skipped 000570 1787885611.315356 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000571 1787885611.639649 mapwrite 0x03f0da thru 0x03ffff (0xf26 bytes) 000572 1787885611.704114 punch from 0x00747f to 0x00757d (0xff bytes) 000573 1787885611.709536 mapwrite 0x007391 thru 0x0086af (0x131f bytes) 000574 1787885611.740590 trunc from 0x040000 to 0x026a47 (0x195b8 bytes) 000575 1787885611.764368 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26a48): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a48): Mirror 4: 0x93df558f (stale) CRC-32 checksum value for chunk [0, 0x26a48): Mirror 1: 0x54f98d81 Mirror 5: 0x54f98d81 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a48: skipped 000576 1787885612.718433 mapwrite 0x01a3dc thru 0x0218c9 (0x74ee bytes) 000577 1787885612.863778 mapwrite 0x02dfce thru 0x038f3b (0xaf6e bytes) 000578 1787885612.934557 write 0x02ffe0 thru 0x0342f0 (0x4311 bytes) 000579 1787885612.936612 mapread 0x015699 thru 0x01641d (0xd85 bytes) 000580 1787885612.938732 trunc from 0x0296d6 to 0x02ff6e (0x6899 bytes) 000581 1787885612.953270 mapwrite 0x01f053 thru 0x022446 (0x33f4 bytes) 000582 1787885612.970228 mapread 0x022e94 thru 0x028612 (0x577f bytes) 000584 1787885612.986994 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38f3c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f3c): Mirror 4: 0x4cfe5df5 (stale) Mirror 5: 0xc8fe0749 (stale) CRC-32 checksum value for chunk [0, 0x38f3c): Mirror 1: 0xca59ea9e Mirror 6: 0xca59ea9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f3c: skipped 000585 1787885614.206459 write 0x029084 thru 0x02c15c (0x30d9 bytes) 000586 1787885614.269640 punch from 0x01d1f3 to 0x01fd37 (0x2b45 bytes) 000587 1787885614.275434 mapwrite 0x03998c thru 0x039b40 (0x1b5 bytes) 000589 1787885614.320799 trunc from 0x00f4d9 to 0x01a9d2 (0xb4fa bytes) 000590 1787885614.326782 trunc from 0x004f91 to 0x009841 (0x48b1 bytes) 000592 1787885614.332815 read 0x01343e thru 0x022fed (0xfbb0 bytes) 000593 1787885614.348354 mapread 0x00038d thru 0x00a22c (0x9ea0 bytes) 000594 1787885614.360970 read 0x012707 thru 0x017dc9 (0x56c3 bytes) 000595 1787885614.371518 punch from 0x0092e5 to 0x00c489 (0x31a5 bytes) 000596 1787885614.380172 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b41): Mirror 5: 0x3929d5e6 (stale) Mirror 6: 0x3a1bfccc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b41): Mirror 5: 0x3929d5e6 (stale) Mirror 6: 0x3a1bfccc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b41: skipped 000597 1787885614.673467 write 0x00f21c thru 0x0129c7 (0x37ac bytes) 000598 1787885614.697076 MIRROR_RESYNC 3 mirrors 000599 1787885616.081238 trunc from 0x00e397 to 0x01d2ad (0xef17 bytes) 000600 1787885616.137291 write 0x039a9e thru 0x03ffff (0x6562 bytes) 000601 1787885616.147010 read 0x013921 thru 0x021415 (0xdaf5 bytes) 000602 1787885616.169254 trunc from 0x040000 to 0x027da5 (0x1825a bytes) 000603 1787885616.201971 write 0x01aea6 thru 0x01e6eb (0x3846 bytes) 000604 1787885616.204344 read 0x01f8f7 thru 0x0249e3 (0x50ed bytes) 000605 1787885616.222682 punch from 0x01166f to 0x01e29b (0xcc2d bytes) 000606 1787885616.254907 fallocate 0x03ef6a thru 0x03ffff (0x1096 bytes) 000607 1787885616.260995 mapwrite 0x01f8de thru 0x02cf0f (0xd632 bytes) 000608 1787885616.324368 trunc from 0x02cf10 to 0x01f8e6 (0xd629 bytes) 000609 1787885616.354729 punch from 0x01858a to 0x01c1f4 (0x3c6b bytes) 000610 1787885616.368460 write 0x0385a9 thru 0x03ffff (0x7a57 bytes) 000611 1787885616.388786 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5d1f0361 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5d1f0361 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000612 1787885616.742030 mapwrite 0x00d12b thru 0x01b421 (0xe2f7 bytes) 000613 1787885616.853787 punch from 0x00d329 to 0x00e049 (0xd21 bytes) 000614 1787885616.861679 punch from 0x0290dd to 0x031b23 (0x8a47 bytes) 000615 1787885616.871279 write 0x01e75c thru 0x024b83 (0x6428 bytes) 000616 1787885616.888919 write 0x024aa8 thru 0x02a1c7 (0x5720 bytes) 000617 1787885616.905252 mapwrite 0x02e278 thru 0x03b64e (0xd3d7 bytes) 000618 1787885616.972541 fallocate 0x0316ba thru 0x0346ea (0x3031 bytes) 000619 1787885616.984918 punch from 0x029d20 to 0x0348a7 (0xab88 bytes) 000621 1787885616.997295 trunc from 0x040000 to 0x03dcae (0x2351 bytes) 000622 1787885617.016545 write 0x01d0ff thru 0x02cfaa (0xfeac bytes) 000623 1787885617.031207 mapwrite 0x032729 thru 0x034b2f (0x2407 bytes) 000624 1787885617.079692 write 0x02742a thru 0x0285a5 (0x117c bytes) 000625 1787885617.080576 mapwrite 0x022136 thru 0x031591 (0xf45c bytes) 000626 1787885617.111767 trunc from 0x03dcaf to 0x02c8b7 (0x113f7 bytes) 000627 1787885617.142756 trunc from 0x02c8b8 to 0x02b805 (0x10b2 bytes) 000628 1787885617.167411 punch from 0x00b36f to 0x016115 (0xada7 bytes) 000629 1787885617.172772 read 0x021b67 thru 0x02b805 (0x9c9f bytes) 000630 1787885617.173551 trunc from 0x02b806 to 0x009fd9 (0x2182c bytes) 000631 1787885617.189191 write 0x03dfef thru 0x03ffff (0x2011 bytes) 000632 1787885617.190527 read 0x022616 thru 0x02f11a (0xcb05 bytes) 000633 1787885617.206960 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5d1f0361 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x225fc1f Mirror 7: 0x225fc1f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000634 1787885618.180468 trunc from 0x021e61 to 0x02f71b (0xd8bb bytes) 000635 1787885618.226126 trunc from 0x040000 to 0x00160d (0x3e9f2 bytes) 000636 1787885618.294936 write 0x017dfa thru 0x01cf51 (0x5158 bytes) 000637 1787885618.298465 mapwrite 0x00bece thru 0x010cc3 (0x4df6 bytes) 000638 1787885618.332301 write 0x036bc2 thru 0x03ffff (0x943e bytes) 000639 1787885618.335565 trunc from 0x040000 to 0x01efc3 (0x2103c bytes) 000640 1787885618.356450 fallocate 0x032b7c thru 0x03ffff (0xd484 bytes) 000641 1787885618.364105 mapread 0x01594e thru 0x01efc3 (0x9676 bytes) 000642 1787885618.374559 write 0x020ad0 thru 0x02668d (0x5bbe bytes) 000643 1787885618.375910 fallocate 0x0336fa thru 0x0387c8 (0x50cf bytes) 000644 1787885618.384093 write 0x01a92e thru 0x025fe8 (0xb6bb bytes) 000645 1787885618.387439 mapread 0x01491b thru 0x016611 (0x1cf7 bytes) 000646 1787885618.402614 MIRROR_RESYNC 3 mirrors 000647 1787885619.172707 trunc from 0x00325b to 0x011342 (0xe0e8 bytes) 000648 1787885619.245048 punch from 0x01b989 to 0x0245f6 (0x8c6e bytes) 000649 1787885619.262208 write 0x03fa8f thru 0x03ffff (0x571 bytes) 000650 1787885619.264815 mapread 0x0380cb thru 0x03fcea (0x7c20 bytes) 000651 1787885619.297287 mapwrite 0x01fbac thru 0x026e54 (0x72a9 bytes) 000653 1787885619.342783 write 0x010a10 thru 0x01d614 (0xcc05 bytes) 000654 1787885619.351798 punch from 0x02bfe5 to 0x039cd7 (0xdcf3 bytes) 000656 1787885619.358019 fallocate 0x020cc0 thru 0x02ce94 (0xc1d5 bytes) 000657 1787885619.373460 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x633457d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x633457d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000658 1787885619.859542 punch from 0x0327ba to 0x03637e (0x3bc5 bytes) 000659 1787885619.868073 write 0x00efd9 thru 0x01ef15 (0xff3d bytes) 000660 1787885619.884631 trunc from 0x040000 to 0x023cb2 (0x1c34d bytes) 000661 1787885619.924921 trunc from 0x006a5c to 0x00c3d7 (0x597c bytes) 000662 1787885619.930124 read 0x01cc72 thru 0x023cb2 (0x7041 bytes) 000663 1787885619.936645 mapwrite 0x02c09d thru 0x03518b (0x90ef bytes) 000664 1787885620.027869 write 0x034787 thru 0x03b4af (0x6d29 bytes) 000665 1787885620.029270 trunc from 0x03a088 to 0x03b4af (0x1428 bytes) 000666 1787885620.062241 MIRROR_RESYNC 2 mirrors 000667 1787885620.913875 fallocate 0x01b1f2 thru 0x029fce (0xeddd bytes) 000668 1787885620.962318 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b4b0): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4b0): Mirror 6: 0x6f9e5a6c (stale) CRC-32 checksum value for chunk [0, 0x3b4b0): Mirror 1: 0x6f9e5a6c Mirror 7: 0x6f9e5a6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4b0: skipped 000669 1787885622.289720 fallocate 0x02198d thru 0x02f7fc (0xde70 bytes) 000670 1787885622.364074 mapread 0x00403a thru 0x01321e (0xf1e5 bytes) 000671 1787885622.435768 trunc from 0x03b4b0 to 0x02aa82 (0x10a2d bytes) 000672 1787885622.476588 read 0x020734 thru 0x0271e8 (0x6ab5 bytes) 000673 1787885622.484105 mapwrite 0x012642 thru 0x01dd9d (0xb75c bytes) 000674 1787885622.514560 write 0x008b71 thru 0x009f22 (0x13b2 bytes) 000675 1787885622.515935 mapread 0x008f12 thru 0x009338 (0x427 bytes) 000676 1787885622.518588 read 0x014205 thru 0x0213fd (0xd1f9 bytes) 000677 1787885622.519772 mapwrite 0x020a90 thru 0x02c9be (0xbf2f bytes) 000678 1787885622.572304 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c9bf): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c9bf): Mirror 6: 0xc5f033cf (stale) Mirror 7: 0xc5f033cf (stale) CRC-32 checksum value for chunk [0, 0x2c9bf): Mirror 1: 0x9e8c0d1 Mirror 8: 0x9e8c0d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c9bf: skipped 000679 1787885623.812660 fallocate 0x02131e thru 0x02a96b (0x964e bytes) 000680 1787885623.883981 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c9bf): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c9bf): Mirror 6: 0xc5f033cf (stale) Mirror 7: 0xc5f033cf (stale) Mirror 8: 0x9e8c0d1 (stale) CRC-32 checksum value for chunk [0, 0x2c9bf): Mirror 1: 0x9e8c0d1 Mirror 9: 0x9e8c0d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c9bf: skipped 000681 1787885625.086098 trunc from 0x010dc1 to 0x01b4cf (0xa70f bytes) 000683 1787885625.154582 trunc from 0x001ba3 to 0x003358 (0x17b6 bytes) 000687 1787885625.163651 mapwrite 0x026667 thru 0x026b14 (0x4ae bytes) 000688 1787885625.185540 write 0x03a3e3 thru 0x03ffff (0x5c1d bytes) 000689 1787885625.194266 mapread 0x026dc4 thru 0x02c435 (0x5672 bytes) 000690 1787885625.217824 mapread 0x02f3e6 thru 0x039590 (0xa1ab bytes) 000691 1787885625.225246 read 0x002706 thru 0x00adcc (0x86c7 bytes) 000692 1787885625.233658 write 0x0089a6 thru 0x00ca50 (0x40ab bytes) 000693 1787885625.238331 fallocate 0x02e17c thru 0x03dbab (0xfa30 bytes) 000694 1787885625.248044 MIRROR_RESYNC 5 mirrors 000695 1787885626.890358 write 0x021fd7 thru 0x02957c (0x75a6 bytes) 000696 1787885626.932683 read 0x038d5e thru 0x03f1b2 (0x6455 bytes) 000697 1787885626.941877 fallocate 0x002d10 thru 0x00bb14 (0x8e05 bytes) 000698 1787885626.947274 trunc from 0x02744e to 0x034bfe (0xd7b1 bytes) 000699 1787885626.976113 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8327c69f (stale) Mirror 7: 0x8327c69f (stale) Mirror 8: 0x8327c69f (stale) Mirror 9: 0x8327c69f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x17162f5b Mirror 10: 0x17162f5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000700 1787885628.049238 MIRROR_RESYNC 6 mirrors 000702 1787885629.756242 MIRROR_RESYNC 6 mirrors 000703 1787885630.517697 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000705 1787885630.967014 fallocate 0x018ce6 thru 0x022bec (0x9f07 bytes) 000706 1787885631.029887 trunc from 0x040000 to 0x0321ea (0xde15 bytes) 000707 1787885631.071327 mapwrite 0x03abca thru 0x03e95f (0x3d96 bytes) 000708 1787885631.138211 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e960): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e960): Mirror 7: 0xa41889a3 (stale) Mirror 8: 0xa41889a3 (stale) Mirror 9: 0xa41889a3 (stale) Mirror 10: 0xa41889a3 (stale) CRC-32 checksum value for chunk [0, 0x3e960): Mirror 1: 0xc682b7f Mirror 11: 0xc682b7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e960: skipped 000709 1787885632.155626 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e960): Mirror 7: 0xa41889a3 (stale) Mirror 8: 0xa41889a3 (stale) Mirror 9: 0xa41889a3 (stale) Mirror 10: 0xa41889a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e960): Mirror 7: 0xa41889a3 (stale) Mirror 8: 0xa41889a3 (stale) Mirror 9: 0xa41889a3 (stale) Mirror 10: 0xa41889a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e960: skipped 000710 1787885632.540641 fallocate 0x02bb80 thru 0x02c9b2 (0xe33 bytes) 000711 1787885632.563267 mapwrite 0x026874 thru 0x02b13a (0x48c7 bytes) 000713 1787885632.598245 mapread 0x036668 thru 0x03e95f (0x82f8 bytes) 000714 1787885632.609400 fallocate 0x0305a1 thru 0x0312dc (0xd3c bytes) 000715 1787885632.624671 punch from 0x00d76b to 0x00ffd0 (0x2866 bytes) 000716 1787885632.637719 fallocate 0x03cef3 thru 0x03d6b4 (0x7c2 bytes) 000717 1787885632.645294 read 0x0009a2 thru 0x00e8ce (0xdf2d bytes) 000718 1787885632.667631 read 0x011bcc thru 0x017180 (0x55b5 bytes) 000719 1787885632.677403 fallocate 0x01e13c thru 0x02917c (0xb041 bytes) 000720 1787885632.681973 read 0x01f5f7 thru 0x022d99 (0x37a3 bytes) 000721 1787885632.687355 fallocate 0x007dc7 thru 0x00ff4a (0x8184 bytes) 000722 1787885632.692727 mapwrite 0x03d729 thru 0x03ffff (0x28d7 bytes) 000723 1787885632.773294 mapread 0x009bbc thru 0x00d5f1 (0x3a36 bytes) 000724 1787885632.774443 read 0x0308b2 thru 0x03b5ab (0xacfa bytes) 000726 1787885632.776173 punch from 0x026ee5 to 0x02b1aa (0x42c6 bytes) 000727 1787885632.781440 mapread 0x02a139 thru 0x0359ba (0xb882 bytes) 000728 1787885632.789850 punch from 0x02ab30 to 0x037983 (0xce54 bytes) 000730 1787885632.795136 trunc from 0x040000 to 0x0340f0 (0xbf0f bytes) 000731 1787885632.807653 punch from 0x033e1e to 0x0340f0 (0x2d3 bytes) 000732 1787885632.816810 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x340f1): Mirror 7: 0xb6896393 (stale) Mirror 8: 0xb6896393 (stale) Mirror 9: 0xb6896393 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340f1): Mirror 7: 0xb6896393 (stale) Mirror 8: 0xb6896393 (stale) Mirror 9: 0xb6896393 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340f1: skipped 000733 1787885633.127798 read 0x02b021 thru 0x0340f0 (0x90d0 bytes) 000734 1787885633.137621 punch from 0x005186 to 0x00f4a0 (0xa31b bytes) 000735 1787885633.142838 mapwrite 0x00e64d thru 0x016ea7 (0x885b bytes) 000736 1787885633.184102 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x340f1): Mirror 8: 0xb6896393 (stale) Mirror 9: 0xb6896393 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340f1): Mirror 8: 0xb6896393 (stale) Mirror 9: 0xb6896393 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340f1: skipped 000737 1787885633.555067 punch from 0x02e092 to 0x0340f0 (0x605f bytes) 000738 1787885633.563289 write 0x02f272 thru 0x03321a (0x3fa9 bytes) 000739 1787885633.592302 write 0x029e0e thru 0x02a723 (0x916 bytes) 000740 1787885633.622262 mapwrite 0x03a988 thru 0x03f24f (0x48c8 bytes) 000741 1787885633.738426 mapwrite 0x03ad03 thru 0x03bf1b (0x1219 bytes) 000742 1787885633.762777 trunc from 0x014e43 to 0x01882b (0x39e9 bytes) 000743 1787885633.783767 fallocate 0x019938 thru 0x019b6f (0x238 bytes) 000744 1787885633.794563 punch from 0x01458a to 0x02400a (0xfa81 bytes) 000745 1787885633.810156 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f250): Mirror 9: 0x4c4cde68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f250): Mirror 9: 0x4c4cde68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f250: skipped 000746 1787885634.185719 trunc from 0x02228f to 0x02ec56 (0xc9c8 bytes) 000747 1787885634.192231 mapread 0x023017 thru 0x030e80 (0xde6a bytes) 000748 1787885634.242012 trunc from 0x03f250 to 0x01e688 (0x20bc7 bytes) 000749 1787885634.277857 punch from 0x01d17a to 0x01e688 (0x150f bytes) 000750 1787885634.283966 trunc from 0x010014 to 0x0178e5 (0x78d2 bytes) 000752 1787885634.290530 trunc from 0x009a23 to 0x00c687 (0x2c65 bytes) 000753 1787885634.297829 read 0x0142bf thru 0x014f75 (0xcb7 bytes) 000754 1787885634.308439 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000755 1787885634.436472 punch from 0x000f05 to 0x009425 (0x8521 bytes) 000756 1787885634.442672 trunc from 0x01e689 to 0x01cfa6 (0x16e2 bytes) 000757 1787885634.460763 trunc from 0x0150fb to 0x01cfa6 (0x7eac bytes) 000759 1787885634.466762 trunc from 0x0076a1 to 0x013fd2 (0xc932 bytes) 000760 1787885634.472224 read 0x018a6a thru 0x01aff1 (0x2588 bytes) 000761 1787885634.477853 punch from 0x00a293 to 0x00f566 (0x52d4 bytes) 000762 1787885634.482751 read 0x00ae46 thru 0x017e06 (0xcfc1 bytes) 000763 1787885634.502537 mapread 0x019fdc thru 0x01cfa6 (0x2fcb bytes) 000764 1787885634.503400 mapwrite 0x001cf5 thru 0x010edb (0xf1e7 bytes) 000765 1787885634.530553 trunc from 0x01cfa7 to 0x03dff0 (0x2104a bytes) 000766 1787885634.544398 read 0x028d52 thru 0x03203e (0x92ed bytes) 000767 1787885634.554804 fallocate 0x0157b9 thru 0x022777 (0xcfbf bytes) 000768 1787885634.560092 punch from 0x014cb7 to 0x01e842 (0x9b8c bytes) 000769 1787885634.565612 punch from 0x03ba42 to 0x03dff0 (0x25af bytes) 000770 1787885634.570451 fallocate 0x0045ec thru 0x00dae5 (0x94fa bytes) 000771 1787885634.575410 fallocate 0x0341cf thru 0x03c35d (0x818f bytes) 000772 1787885634.580853 trunc from 0x01cf0f to 0x025a6c (0x8b5e bytes) 000773 1787885634.585519 trunc from 0x03dff1 to 0x002324 (0x3bccc bytes) 000774 1787885634.605689 read 0x0013dd thru 0x002324 (0xf48 bytes) 000775 1787885634.606251 punch from 0x000c74 to 0x00117c (0x509 bytes) 000780 1787885634.616117 write 0x020ec7 thru 0x0243eb (0x3525 bytes) 000781 1787885634.618281 trunc from 0x0243ec to 0x01d7c8 (0x6c23 bytes) 000782 1787885634.635045 mapread 0x015979 thru 0x01d7c8 (0x7e50 bytes) 000783 1787885634.644827 trunc from 0x01d7c9 to 0x00a505 (0x132c3 bytes) 000784 1787885634.658900 punch from 0x001d50 to 0x00a505 (0x87b6 bytes) 000786 1787885634.665351 punch from 0x0024e7 to 0x00a505 (0x801f bytes) 000787 1787885634.670032 fallocate 0x033ed1 thru 0x03ffff (0xc12f bytes) 000788 1787885634.674375 trunc from 0x00a506 to 0x039114 (0x2ec0f bytes) 000789 1787885634.691886 read 0x01c0f6 thru 0x02aa93 (0xe99e bytes) 000790 1787885634.706136 mapread 0x035bb9 thru 0x036ed6 (0x131e bytes) 000791 1787885634.707514 trunc from 0x0367e6 to 0x039114 (0x292f bytes) 000792 1787885634.716258 MIRROR_EXTEND to 2 mirrors 000793 1787885635.508397 write 0x007b3b thru 0x011704 (0x9bca bytes) 000794 1787885635.597216 trunc from 0x0074e9 to 0x00b474 (0x3f8c bytes) 000795 1787885635.637532 write 0x027043 thru 0x027ad1 (0xa8f bytes) 000796 1787885635.645412 write 0x00197e thru 0x00d953 (0xbfd6 bytes) 000797 1787885635.665970 punch from 0x02c547 to 0x039114 (0xcbce bytes) 000798 1787885635.688712 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000799 1787885635.960523 trunc from 0x039115 to 0x018dd5 (0x2033f bytes) 000800 1787885635.987364 mapwrite 0x032241 thru 0x039614 (0x73d4 bytes) 000801 1787885636.040813 mapwrite 0x03deb9 thru 0x03ffff (0x2147 bytes) 000802 1787885636.076355 trunc from 0x040000 to 0x016257 (0x29da8 bytes) 000803 1787885636.095851 read 0x003e71 thru 0x006e76 (0x3006 bytes) 000804 1787885636.103345 read 0x00e505 thru 0x010ccd (0x27c9 bytes) 000805 1787885636.110630 trunc from 0x010aae to 0x01427a (0x37cd bytes) 000806 1787885636.116460 trunc from 0x014d8b to 0x016257 (0x14cd bytes) 000807 1787885636.120881 read 0x008141 thru 0x0126af (0xa56f bytes) 000808 1787885636.129009 fallocate 0x0320f3 thru 0x034138 (0x2046 bytes) 000809 1787885636.133716 fallocate 0x02095f thru 0x02b8e6 (0xaf88 bytes) 000810 1787885636.138236 mapwrite 0x01bfd4 thru 0x024b70 (0x8b9d bytes) 000812 1787885636.166856 write 0x02ef2e thru 0x036da8 (0x7e7b bytes) 000813 1787885636.168629 trunc from 0x027fc6 to 0x035fe1 (0xe01c bytes) 000814 1787885636.181296 read 0x02f70f thru 0x036da8 (0x769a bytes) 000815 1787885636.190463 fallocate 0x026087 thru 0x02fac0 (0x9a3a bytes) 000816 1787885636.195669 trunc from 0x017f83 to 0x02223c (0xa2ba bytes) 000817 1787885636.200421 write 0x00e248 thru 0x016fcc (0x8d85 bytes) 000818 1787885636.204265 trunc from 0x036da9 to 0x01825e (0x1eb4a bytes) 000819 1787885636.218407 read 0x0028bc thru 0x011027 (0xe76c bytes) 000820 1787885636.219071 mapwrite 0x00ed9a thru 0x0182c9 (0x9530 bytes) 000821 1787885636.259416 fallocate 0x002cad thru 0x007244 (0x4598 bytes) 000824 1787885636.268534 mapread 0x00cffa thru 0x0118b3 (0x48ba bytes) 000825 1787885636.270798 mapread 0x00a713 thru 0x0124d1 (0x7dbf bytes) 000826 1787885636.271737 read 0x0064a7 thru 0x015fcb (0xfb25 bytes) 000827 1787885636.272978 mapread 0x000c47 thru 0x0081c3 (0x757d bytes) 000828 1787885636.274438 punch from 0x00c9a7 to 0x0168af (0x9f09 bytes) 000829 1787885636.280844 write 0x005be4 thru 0x00dd81 (0x819e bytes) 000831 1787885636.294102 mapwrite 0x02b2c4 thru 0x02eb85 (0x38c2 bytes) 000832 1787885636.344399 mapread 0x0131e4 thru 0x017a37 (0x4854 bytes) 000833 1787885636.345359 mapread 0x002f25 thru 0x011ccc (0xeda8 bytes) 000834 1787885636.347447 mapread 0x01e74c thru 0x02c1db (0xda90 bytes) 000835 1787885636.348634 trunc from 0x00e3dc to 0x0114e8 (0x310d bytes) 000836 1787885636.352730 mapwrite 0x0220fc thru 0x0309e1 (0xe8e6 bytes) 000837 1787885636.391322 trunc from 0x02eeaf to 0x0309e1 (0x1b33 bytes) 000838 1787885636.397350 fallocate 0x03cbf0 thru 0x03ffff (0x3410 bytes) 000840 1787885636.402724 trunc from 0x040000 to 0x033237 (0xcdc8 bytes) 000841 1787885636.415559 trunc from 0x00c982 to 0x011911 (0x4f90 bytes) 000843 1787885636.421017 read 0x026826 thru 0x0281fa (0x19d5 bytes) 000844 1787885636.421591 mapread 0x0007e8 thru 0x009448 (0x8c61 bytes) 000845 1787885636.422406 mapwrite 0x036ffb thru 0x03ffff (0x9005 bytes) 000846 1787885636.476579 trunc from 0x009578 to 0x017041 (0xdaca bytes) 000848 1787885636.495388 MIRROR_EXTEND to 2 mirrors 000849 1787885637.204038 mapwrite 0x0097f4 thru 0x00993d (0x14a bytes) 000850 1787885637.306529 fallocate 0x009c87 thru 0x0131cc (0x9546 bytes) 000851 1787885637.314753 trunc from 0x018a92 to 0x01ef8e (0x64fd bytes) 000853 1787885637.331502 fallocate 0x02f34a thru 0x02fb87 (0x83e bytes) 000854 1787885637.355736 mapwrite 0x0237c3 thru 0x02c1cd (0x8a0b bytes) 000855 1787885637.464306 punch from 0x01c6a4 to 0x01e9c9 (0x2326 bytes) 000856 1787885637.483807 trunc from 0x040000 to 0x000600 (0x3f9ff bytes) 000858 1787885637.587388 mapread 0x0003d9 thru 0x000600 (0x228 bytes) 000859 1787885637.608088 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000860 1787885637.792776 punch from 0x000401 to 0x000600 (0x200 bytes) 000861 1787885637.803610 MIRROR_EXTEND to 2 mirrors 000862 1787885638.299881 punch from 0x0000c1 to 0x000600 (0x540 bytes) 000863 1787885638.334862 trunc from 0x00047f to 0x000600 (0x182 bytes) 000864 1787885638.340366 mapread 0x000293 thru 0x000600 (0x36e bytes) 000866 1787885638.347668 mapread 0x00052f thru 0x000600 (0xd2 bytes) 000867 1787885638.351089 mapwrite 0x007e26 thru 0x008bb7 (0xd92 bytes) 000868 1787885638.412842 mapread 0x001aad thru 0x008bb7 (0x710b bytes) 000869 1787885638.427209 read 0x0075c2 thru 0x008bb7 (0x15f6 bytes) 000871 1787885638.429322 write 0x00c125 thru 0x01b6ce (0xf5aa bytes) 000872 1787885638.431745 punch from 0x002154 to 0x00d71b (0xb5c8 bytes) 000873 1787885638.447373 write 0x030d50 thru 0x0323a9 (0x165a bytes) 000874 1787885638.450067 mapread 0x007046 thru 0x00a783 (0x373e bytes) 000875 1787885638.467312 trunc from 0x00db32 to 0x01da55 (0xff24 bytes) 000876 1787885638.472324 punch from 0x001b92 to 0x00bbe4 (0xa053 bytes) 000877 1787885638.476861 trunc from 0x01ce85 to 0x01ef56 (0x20d2 bytes) 000879 1787885638.481935 mapread 0x028f41 thru 0x03099c (0x7a5c bytes) 000880 1787885638.491928 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x323aa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323aa): Mirror 2: 0xf6eba0d9 (stale) CRC-32 checksum value for chunk [0, 0x323aa): Mirror 1: 0x80eb11a2 Mirror 3: 0x80eb11a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323aa: skipped 000881 1787885639.435006 punch from 0x02aa8e to 0x0323a9 (0x791c bytes) 000882 1787885639.484610 write 0x017bc0 thru 0x01ffc6 (0x8407 bytes) 000883 1787885639.498042 fallocate 0x01b3f6 thru 0x029c1d (0xe828 bytes) 000884 1787885639.503394 read 0x006561 thru 0x008904 (0x23a4 bytes) 000885 1787885639.508188 mapread 0x00a95d thru 0x00c402 (0x1aa6 bytes) 000886 1787885639.527129 write 0x00f30b thru 0x01d9da (0xe6d0 bytes) 000887 1787885639.528720 trunc from 0x00c3be to 0x018e6d (0xcab0 bytes) 000888 1787885639.546811 trunc from 0x0011fe to 0x002f3d (0x1d40 bytes) 000889 1787885639.551661 trunc from 0x017035 to 0x01ffaf (0x8f7b bytes) 000890 1787885639.558419 punch from 0x02538d to 0x029dd3 (0x4a47 bytes) 000892 1787885639.568050 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x323aa): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323aa): Mirror 2: 0xf6eba0d9 (stale) Mirror 3: 0x80eb11a2 (stale) CRC-32 checksum value for chunk [0, 0x323aa): Mirror 1: 0x2cb177bc Mirror 4: 0x2cb177bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323aa: skipped 000893 1787885640.607728 mapwrite 0x035826 thru 0x037281 (0x1a5c bytes) 000894 1787885640.715866 mapwrite 0x03c7de thru 0x03ffff (0x3822 bytes) 000896 1787885640.762328 mapwrite 0x00450e thru 0x00d429 (0x8f1c bytes) 000897 1787885640.792141 trunc from 0x01c6e9 to 0x027ed6 (0xb7ee bytes) 000898 1787885640.798465 mapwrite 0x0185ad thru 0x024e68 (0xc8bc bytes) 000899 1787885640.841622 punch from 0x02f4dc to 0x030ae0 (0x1605 bytes) 000900 1787885640.845344 punch from 0x0048e8 to 0x0055d3 (0xcec bytes) 000901 1787885640.850348 read 0x029b32 thru 0x03932d (0xf7fc bytes) 000902 1787885640.858097 fallocate 0x002a52 thru 0x005c8e (0x323d bytes) 000903 1787885640.863541 punch from 0x005f7d to 0x00ae21 (0x4ea5 bytes) 000904 1787885640.873069 MIRROR_RESYNC 4 mirrors 000905 1787885642.865375 write 0x007daa thru 0x015f35 (0xe18c bytes) 000906 1787885642.920831 fallocate 0x03629d thru 0x03ffff (0x9d63 bytes) 000907 1787885642.928379 fallocate 0x02edf1 thru 0x03b4e7 (0xc6f7 bytes) 000908 1787885642.932788 mapread 0x015242 thru 0x02391d (0xe6dc bytes) 000909 1787885642.975642 trunc from 0x03a621 to 0x03ffff (0x59df bytes) 000910 1787885642.983109 punch from 0x011783 to 0x01a8b0 (0x912e bytes) 000911 1787885643.000379 mapread 0x024997 thru 0x026f90 (0x25fa bytes) 000912 1787885643.001961 punch from 0x03e11d to 0x03ffff (0x1ee3 bytes) 000913 1787885643.007052 mapread 0x01605c thru 0x0183f4 (0x2399 bytes) 000914 1787885643.018429 trunc from 0x00e16a to 0x00e321 (0x1b8 bytes) 000915 1787885643.023753 write 0x0374a5 thru 0x03a1a2 (0x2cfe bytes) 000916 1787885643.037990 MIRROR_RESYNC 4 mirrors 000917 1787885644.709335 trunc from 0x040000 to 0x010f2d (0x2f0d2 bytes) 000918 1787885644.756588 mapwrite 0x02ec2d thru 0x03cf48 (0xe31c bytes) 000919 1787885644.890874 mapwrite 0x019809 thru 0x01bfa3 (0x279b bytes) 000920 1787885644.921007 write 0x01e719 thru 0x02d56e (0xee56 bytes) 000921 1787885644.927694 write 0x029bf3 thru 0x02d456 (0x3864 bytes) 000922 1787885644.928888 punch from 0x021f9b to 0x025c6d (0x3cd3 bytes) 000923 1787885644.962322 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf49): Mirror 3: 0x3ae226da (stale) Mirror 4: 0x3ae226da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf49): Mirror 3: 0x3ae226da (stale) Mirror 4: 0x3ae226da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf49: skipped 000924 1787885645.334547 trunc from 0x03716b to 0x037857 (0x6ed bytes) 000925 1787885645.346978 mapread 0x003b4a thru 0x00de0b (0xa2c2 bytes) 000927 1787885645.382855 write 0x032c22 thru 0x03ffff (0xd3de bytes) 000928 1787885645.402031 punch from 0x03d981 to 0x03ffff (0x267f bytes) 000929 1787885645.426719 write 0x009150 thru 0x00eef4 (0x5da5 bytes) 000930 1787885645.432554 mapwrite 0x01de7e thru 0x0280d8 (0xa25b bytes) 000931 1787885645.486797 mapread 0x0377bd thru 0x0382cb (0xb0f bytes) 000932 1787885645.487961 mapwrite 0x00783b thru 0x00e95b (0x7121 bytes) 000933 1787885645.514286 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa7a908c2 (stale) Mirror 4: 0xa7a908c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26fcd8d4 Mirror 5: 0x26fcd8d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000934 1787885646.437280 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa7a908c2 (stale) Mirror 4: 0xa7a908c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26fcd8d4 Mirror 5: 0x26fcd8d4 Mirror 6: 0x26fcd8d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000935 1787885647.682172 mapwrite 0x014589 thru 0x020c86 (0xc6fe bytes) 000936 1787885647.832964 mapread 0x0235b8 thru 0x024b2d (0x1576 bytes) 000937 1787885647.835546 write 0x03ff86 thru 0x03ffff (0x7a bytes) 000938 1787885647.848231 write 0x029960 thru 0x034d5d (0xb3fe bytes) 000939 1787885647.858934 write 0x03d562 thru 0x03ffff (0x2a9e bytes) 000940 1787885647.865156 trunc from 0x040000 to 0x037257 (0x8da8 bytes) 000941 1787885647.913291 read 0x00131e thru 0x010f13 (0xfbf6 bytes) 000942 1787885647.926974 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37258): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37258): Mirror 3: 0x5cdad855 (stale) Mirror 4: 0x5cdad855 (stale) Mirror 5: 0x3c6d689a (stale) Mirror 6: 0x3c6d689a (stale) CRC-32 checksum value for chunk [0, 0x37258): Mirror 1: 0x90005b87 Mirror 7: 0x90005b87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37258: skipped 000943 1787885649.133848 punch from 0x0308ed to 0x031df3 (0x1507 bytes) 000945 1787885649.207602 mapread 0x01c81d thru 0x01d14c (0x930 bytes) 000946 1787885649.225256 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x37258): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37258): Mirror 3: 0x5cdad855 (stale) Mirror 4: 0x5cdad855 (stale) Mirror 5: 0x3c6d689a (stale) Mirror 6: 0x3c6d689a (stale) Mirror 7: 0x90005b87 (stale) CRC-32 checksum value for chunk [0, 0x37258): Mirror 1: 0xd8032eba Mirror 8: 0xd8032eba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37258: skipped 000948 1787885650.425306 write 0x003da3 thru 0x00cf19 (0x9177 bytes) 000953 1787885650.480322 write 0x01fe99 thru 0x02cf87 (0xd0ef bytes) 000954 1787885650.491252 fallocate 0x02a1ce thru 0x03051d (0x6350 bytes) 000955 1787885650.497053 fallocate 0x023b3d thru 0x032a4a (0xef0e bytes) 000956 1787885650.502254 fallocate 0x002dd8 thru 0x004cb9 (0x1ee2 bytes) 000957 1787885650.509276 write 0x03d33c thru 0x03ffff (0x2cc4 bytes) 000958 1787885650.510584 punch from 0x003ead to 0x00c59b (0x86ef bytes) 000959 1787885650.531653 write 0x0127cb thru 0x01dc22 (0xb458 bytes) 000960 1787885650.556979 read 0x03de50 thru 0x03ffff (0x21b0 bytes) 000961 1787885650.558226 punch from 0x014f32 to 0x01888f (0x395e bytes) 000962 1787885650.573985 punch from 0x005527 to 0x013fe5 (0xeabf bytes) 000963 1787885650.581545 trunc from 0x00a08a to 0x00b26b (0x11e2 bytes) 000964 1787885650.588737 trunc from 0x040000 to 0x03661b (0x99e4 bytes) 000965 1787885650.615223 write 0x037d75 thru 0x03ffff (0x828b bytes) 000966 1787885650.619397 fallocate 0x03512b thru 0x03ffff (0xaed5 bytes) 000967 1787885650.626281 read 0x03a1db thru 0x03ffff (0x5e25 bytes) 000968 1787885650.629354 fallocate 0x00aa42 thru 0x00f6de (0x4c9d bytes) 000969 1787885650.636604 trunc from 0x040000 to 0x0168e9 (0x29716 bytes) 000970 1787885650.658752 mapwrite 0x0128bd thru 0x01f12f (0xc873 bytes) 000971 1787885650.793329 mapwrite 0x03372f thru 0x03f5ee (0xbec0 bytes) 000972 1787885650.876197 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f5ef): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5ef): Mirror 3: 0x800f32aa (stale) Mirror 4: 0x800f32aa (stale) Mirror 5: 0x3c8518f6 (stale) Mirror 6: 0x3c8518f6 (stale) Mirror 7: 0x435b40e2 (stale) Mirror 8: 0x15f05ce6 (stale) CRC-32 checksum value for chunk [0, 0x3f5ef): Mirror 1: 0xaf7a880d Mirror 9: 0xaf7a880d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5ef: skipped 000973 1787885652.167221 punch from 0x026067 to 0x02c48b (0x6425 bytes) 000974 1787885652.214150 write 0x02a456 thru 0x02f7ba (0x5365 bytes) 000975 1787885652.227172 mapread 0x034c30 thru 0x03f5ee (0xa9bf bytes) 000977 1787885652.254146 mapread 0x01b048 thru 0x022497 (0x7450 bytes) 000978 1787885652.267813 mapwrite 0x038cf6 thru 0x03ffff (0x730a bytes) 000979 1787885652.300391 mapread 0x012292 thru 0x019664 (0x73d3 bytes) 000980 1787885652.309276 trunc from 0x035fe9 to 0x03ffff (0xa017 bytes) 000981 1787885652.315427 trunc from 0x040000 to 0x0070e0 (0x38f1f bytes) 000982 1787885652.337240 read 0x006ad9 thru 0x0070e0 (0x608 bytes) 000983 1787885652.343858 punch from 0x006301 to 0x0070e0 (0xde0 bytes) 000984 1787885652.349301 fallocate 0x01abc8 thru 0x026ce0 (0xc119 bytes) 000986 1787885652.355193 read 0x004b00 thru 0x0070e0 (0x25e1 bytes) 000987 1787885652.363798 mapread 0x004b2c thru 0x0070e0 (0x25b5 bytes) 000988 1787885652.364988 punch from 0x002051 to 0x0070e0 (0x5090 bytes) 000989 1787885652.374606 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x70e1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x70e1): Mirror 3: 0x5286ed9b (stale) Mirror 4: 0x5286ed9b (stale) Mirror 5: 0x5286ed9b (stale) Mirror 6: 0x5286ed9b (stale) Mirror 7: 0x5286ed9b (stale) Mirror 8: 0x5286ed9b (stale) Mirror 9: 0x3751d242 (stale) CRC-32 checksum value for chunk [0, 0x70e1): Mirror 1: 0x96adc3e6 Mirror 10: 0x96adc3e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x70e1: skipped 000990 1787885653.376804 fallocate 0x00e2f7 thru 0x01b1dd (0xcee7 bytes) 000992 1787885653.464074 trunc from 0x0173d1 to 0x01b1dd (0x3e0d bytes) 000993 1787885653.474418 trunc from 0x015ab5 to 0x01b1dd (0x5729 bytes) 000994 1787885653.479009 mapwrite 0x035614 thru 0x03c524 (0x6f11 bytes) 000995 1787885653.631740 MIRROR_RESYNC 9 mirrors 000996 1787885655.208552 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b5:0x0]: Invalid argument (22) 000997 1787885655.794383 read 0x026319 thru 0x02e756 (0x843e bytes) 000998 1787885655.805300 punch from 0x037d17 to 0x03b834 (0x3b1e bytes) 000999 1787885655.866190 write 0x018829 thru 0x018ef3 (0x6cb bytes) 001000 1787885655.890349 trunc from 0x03c525 to 0x026bfa (0x1592a bytes) All operations completed A-OK! PASS 70a (97s) == sanity-flr test 71: check mirror extend parent fid ==== 22:54:21 (1787885661) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1fa:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 22:54:31 (1787885671) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227166 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111813 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 22:54:40 (1787885680) Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre2 Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7717152 .. Extending file size to 837184 .. Extending file size to 2782400 .. Extending file size to 4929472 .. Extending file size to 5414688 .. Extending file size to 7793216 .. Extending file size to 8565984 .. Extending file size to 8421920 .. Extending file size to 9113984 .. Extending file size to 8963520 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9129696 .. Extending file size to 9193376 .. Extending file size to 9212448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61416 61417 61418 61419 61421 192.168.206.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg657-client.virtnet /mnt/lustre2 (opts:) 192.168.206.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg657-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:55:59 (1787885759) Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre2 Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1361792 .. Extending file size to 7975904 .. Extending file size to 1841600 .. Extending file size to 8312736 .. Extending file size to 8864832 .. Extending file size to 8696256 .. Extending file size to 9046592 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62873 62874 62875 62877 62879 done 192.168.206.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg657-client.virtnet /mnt/lustre2 (opts:) 192.168.206.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg657-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:57:20 (1787885840) Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24484 3582536 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18796 3588224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43280 7170760 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24484 3582536 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18796 3588224 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43280 7170760 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161639 s, 13.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156809 s, 13.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x19d2eb22 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg657-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:57:38 (1787885858) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124618 s, 16.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 22:57:46 (1787885866) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:57:54 (1787885874) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.099356 s, 21.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x253:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x254:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:58:01 (1787885881) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0839882 s, 25.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x252:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x253:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x255:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x252:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x253:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x255:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x256:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:58:09 (1787885889) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142214 s, 14.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:58:42 (1787885922) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg657-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812561 s, 12.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x254:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x258:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x257:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x254:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x258:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x257:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x255:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:59:31 (1787885971) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg657-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111414 s, 18.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:00:21 (1787886021) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg657-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg657-server: Pool lustre.test_204f created Adding targets to pool oleg657-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg657-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112901 s, 18.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg657-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg657-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg657-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:01:36 (1787886096) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:01:43 (1787886103) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:01:52 (1787886112) oleg657-server: Pool lustre.test_206 created Adding targets to pool oleg657-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg657-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg657-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg657-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg657-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:02:24 (1787886144) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950788 s, 11.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231724 s, 45.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa14f2ff4 Mirror 3: 0xa14f2ff4 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:02:37 (1787886157) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:02:41 (1787886161) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:02:45 (1787886165) Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.079658 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xec0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1dd0000 Read this (4 bytes): BBBB 192.168.206.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg657-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:03:25 (1787886205) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.032681 s, 251 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:03:33 (1787886213) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104986 s, 10.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611226 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 23:03:36.000000000 -0400 Modify: 2026-08-27 23:03:36.000000000 -0400 Change: 2026-08-27 23:03:37.000000000 -0400 Birth: 2026-08-27 23:03:35.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:03:44 (1787886224) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629713 s, 16.7 MB/s oleg657-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg657-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg657-server: O/280000401/d17/131: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:04:08 (1787886248) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.218496 s, 187 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0295104 s, 139 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 23:04 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 23:04 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:04:16 (1787886256) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:04:24 (1787886264) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1821 sec =========== 23:04:33 (1787886273) === sanity-flr: start cleanup 23:04:35 (1787886275) === === sanity-flr: finish cleanup 23:04:44 (1787886284) ===