-----============= acceptance-small: sanity-flr ============----- Wed Aug 12 06:02:43 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg120-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:03:03 (1786528983) === oleg120-client.virtnet: executing check_config_client /mnt/lustre oleg120-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg120-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff913bca758800.idle_timeout=debug osc.lustre-OST0001-osc-ffff913bca758800.idle_timeout=debug disable quota as required oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:03:50 (1786529030) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8249] == sanity-flr test 0a: lfs mirror create with -N option == 06:03:52 (1786529032) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:04:10 (1786529050) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:04:14 (1786529054) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:04:19 (1786529059) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:04:47 (1786529087) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:04:50 (1786529090) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:04:54 (1786529094) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff913bca758800.stats=clear osc.lustre-OST0001-osc-ffff913bca758800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:05:03 (1786529103) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:05:16 (1786529116) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:05:25 (1786529125) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:05:29 (1786529129) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:05:38 (1786529138) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:05:47 (1786529147) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:05:55 (1786529155) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:06:03 (1786529163) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:06:13 (1786529173) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:06:26 (1786529186) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:06:34 (1786529194) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.347875 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107111 s, 9.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:06:46 (1786529206) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.301243 s, 24.4 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:06:56 (1786529216) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:07:07 (1786529227) Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg120-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg120-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (132s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:09:19 (1786529359) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.589219 s, 12.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:10:10 (1786529410) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg120-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:11:35 (1786529495) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.25931 s, 16.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 12 06:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.180s user 0m0.000s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff913bca758800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.077s user 0m0.000s sys 0m0.180s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg120-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg120-server oleg120-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.520s user 0m0.000s sys 0m0.189s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg120-server: oleg120-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg120-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg120-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff913bca758800.ost_server_uuid in FULL state after 0 sec PASS 33b (106s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:13:21 (1786529601) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:13:25 (1786529605) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:13:29 (1786529609) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:13:33 (1786529613) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 06:13:40 (1786529620) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.717077 s, 17.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:13:54 (1786529634) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509062 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245406 s, 42.7 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:14:05 (1786529645) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:14:16 (1786529656) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000922851 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00385321 s, 266 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00708596 s, 145 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00621934 s, 165 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 06:14:29 (1786529669) create test files with size 15734363 .. 28167d338940ae2e605a52bd6585c0f4 - ce39fbe1b8daddcbb5d538630a556e30 - ce92ee388d9f7a26c7072973766719d3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg120-client.virtnet /mnt/lustre (opts:) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 06:14:54 (1786529694) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:15:10 (1786529710) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:15:19 (1786529719) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112323 s, 9.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204464 s, 51.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 06:15:28 (1786529728) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13391 s, 15.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.407253 s, 10.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247928 s, 8.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 06:15:43 (1786529743) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:15:47 (1786529747) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:15:51 (1786529751) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363004 s, 28.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 06:15:59 (1786529759) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.46825 s, 6.7 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:16:11 (1786529771) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:16:23 (1786529783) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.4786 s, 21.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg120-client.virtnet /mnt/lustre (opts:) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:16:37 (1786529797) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff913bd8e1a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.444134 s, 23.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg120-client.virtnet /mnt/lustre (opts:) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff913bd0ec6800.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:16:57 (1786529817) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:17:38 (1786529858) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:18:22 (1786529902) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:18:27 (1786529907) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.395177 s, 8.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.212877 s, 14.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x2e0cd6f7 (nosync,stale) Mirror 2: 0x2e0cd6f7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4b7fe518558cb5e17125d4a76d65ebf1 - mirror 1 checksum: 7f90ac2a7ab56a4a85c2ea868284eb2d - mirror 2 checksum: 4b7fe518558cb5e17125d4a76d65ebf1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T06:18:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T06:18:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4b7fe518558cb5e17125d4a76d65ebf1 - mirror 1 checksum: 7f90ac2a7ab56a4a85c2ea868284eb2d - mirror 2 checksum: 7f90ac2a7ab56a4a85c2ea868284eb2d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T06:18:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T06:18:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 48 (14s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:18:41 (1786529921) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442681 s, 23.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:18:50 (1786529930) Failing mds1 on oleg120-server Stopping /mnt/lustre-mds1 (opts:) on oleg120-server 06:19:02 (1786529942) shut down facet: mds1 facet_host: oleg120-server facet_failover_host: oleg120-server Failover mds1 to oleg120-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 06:19:35 (1786529975) targets are mounted 06:19:35 (1786529975) facet_failover done oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347412 s, 30.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:19:54 (1786529994) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.047855 s, 428 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0165078 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:20:07 (1786530007) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0717912 s, 285 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0140839 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:20:18 (1786530018) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0476535 s, 344 kB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:20:30 (1786530030) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0914974 s, 224 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 50e: truncate(0) swaps layout when primary mirror is inactive ========================================================== 06:20:40 (1786530040) SKIP: sanity-flr test_50e needs >= 4 OSTs SKIP 50e (5s) == sanity-flr test 50f: truncate(0) ignores inactive non-primary mirror ========================================================== 06:20:45 (1786530045) SKIP: sanity-flr test_50f needs >= 4 OSTs SKIP 50f (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:20:49 (1786530049) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0426037 s, 96.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0531461 s, 77.1 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:20:59 (1786530059) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:21:23 (1786530083) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (35s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:21:57 (1786530117) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:22:25 (1786530145) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:22:52 (1786530172) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 06:23:02 (1786530182) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58271 mirror split pid 58272 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58271 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58272 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 06:24:11 (1786530251) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 254 skipping zero length zero range fallocating to largest ever: 0x975c 000002 1786530254.368629 fallocate 0x0056c6 thru 0x00975b (0x4096 bytes) 000003 1786530254.375486 read 0x006fbe thru 0x00975b (0x279e bytes) 000004 1786530254.385441 trunc from 0x0012b0 to 0x00975b (0x84ac bytes) fallocating to largest ever: 0x3e4c4 000006 1786530254.394140 fallocate 0x038d46 thru 0x03e4c3 (0x577e bytes) 000007 1786530254.418916 trunc from 0x00956c to 0x00975b (0x1f0 bytes) 000008 1786530254.428677 trunc from 0x006955 to 0x007309 (0x9b5 bytes) 000009 1786530254.444806 mapread 0x004d4d thru 0x00975b (0x4a0f bytes) 000010 1786530254.479362 read 0x0046f7 thru 0x00975b (0x5065 bytes) 000013 1786530254.500611 punch from 0x007869 to 0x00975b (0x1ef3 bytes) 000014 1786530254.522479 read 0x004dc7 thru 0x0077d8 (0x2a12 bytes) 000016 1786530254.543339 read 0x0056d8 thru 0x00975b (0x4084 bytes) 000017 1786530254.544436 punch from 0x0006a6 to 0x0087cb (0x8126 bytes) fallocating to largest ever: 0x40000 000018 1786530254.553539 fallocate 0x037a83 thru 0x03ffff (0x857d bytes) 000019 1786530254.569440 fallocate 0x0348a8 thru 0x03ffff (0xb758 bytes) 000020 1786530254.588183 fallocate 0x01be50 thru 0x0279c3 (0xbb74 bytes) 000021 1786530254.621458 mapwrite 0x03a1ba thru 0x03ffff (0x5e46 bytes) 000024 1786530254.689922 trunc from 0x02c3c5 to 0x030044 (0x3c80 bytes) 000025 1786530254.697112 punch from 0x02f490 to 0x03c360 (0xced1 bytes) 000026 1786530254.711238 fallocate 0x024456 thru 0x024d9a (0x945 bytes) 000030 1786530254.721557 mapread 0x001b07 thru 0x0057f4 (0x3cee bytes) 000031 1786530254.723430 trunc from 0x036bc6 to 0x03ffa7 (0x93e2 bytes) 000032 1786530254.734689 trunc from 0x0067c7 to 0x00ad39 (0x4573 bytes) 000034 1786530254.748744 trunc from 0x03f453 to 0x03ffff (0xbad bytes) 000035 1786530254.772268 fallocate 0x034127 thru 0x03e063 (0x9f3d bytes) 000036 1786530254.786897 MIRROR_EXTEND to 2 mirrors 000038 1786530255.772201 trunc from 0x040000 to 0x017cf8 (0x28307 bytes) 000039 1786530255.868887 mapread 0x00b133 thru 0x00c10e (0xfdc bytes) 000040 1786530255.881412 trunc from 0x017cf9 to 0x007c3d (0x100bb bytes) 000041 1786530256.002993 read 0x006518 thru 0x007c3d (0x1726 bytes) 000042 1786530256.048891 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7c3e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7c3e): Mirror 2: 0x66f50e6b (stale) CRC-32 checksum value for chunk [0, 0x7c3e): Mirror 1: 0x66f50e6b Mirror 3: 0x66f50e6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7c3e: skipped 000043 1786530257.302343 read 0x003f39 thru 0x007c3d (0x3d05 bytes) 000044 1786530257.329327 fallocate 0x0117ab thru 0x0151d4 (0x3a2a bytes) 000045 1786530257.448749 read 0x001d9c thru 0x0085eb (0x6850 bytes) 000046 1786530257.477515 punch from 0x006536 to 0x013928 (0xd3f3 bytes) 000047 1786530257.492769 fallocate 0x01d863 thru 0x02c2e6 (0xea84 bytes) 000048 1786530257.524820 fallocate 0x0034f1 thru 0x009b3c (0x664c bytes) 000049 1786530257.543822 punch from 0x011475 to 0x0151d4 (0x3d60 bytes) 000050 1786530257.554746 trunc from 0x0008c2 to 0x00e971 (0xe0b0 bytes) 000051 1786530257.561382 trunc from 0x00d39e to 0x012a3a (0x569d bytes) 000052 1786530257.570353 write 0x0032c4 thru 0x005213 (0x1f50 bytes) 000053 1786530257.588337 mapwrite 0x00d53c thru 0x01c2d0 (0xed95 bytes) 000054 1786530257.745484 punch from 0x00a4bb to 0x012011 (0x7b57 bytes) 000055 1786530257.757525 mapwrite 0x00eb15 thru 0x01e88c (0xfd78 bytes) 000056 1786530257.888816 read 0x01333d thru 0x01ac18 (0x78dc bytes) 000057 1786530257.889645 read 0x0180b0 thru 0x01e88c (0x67dd bytes) 000058 1786530257.890717 write 0x00326f thru 0x006f35 (0x3cc7 bytes) 000059 1786530257.916510 MIRROR_RESYNC 3 mirrors 000060 1786530259.596482 punch from 0x016cb9 to 0x01d8d1 (0x6c19 bytes) 000061 1786530259.669505 mapwrite 0x00dbc9 thru 0x01ae86 (0xd2be bytes) 000062 1786530259.808546 fallocate 0x02fcc1 thru 0x03072d (0xa6d bytes) 000063 1786530259.822477 mapwrite 0x012d3a thru 0x0228e2 (0xfba9 bytes) 000065 1786530259.886168 trunc from 0x0082fd to 0x00e2ef (0x5ff3 bytes) 000066 1786530259.903245 fallocate 0x0154af thru 0x01a4e7 (0x5039 bytes) 000067 1786530259.919029 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3072e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3072e): Mirror 2: 0xaa42e620 (stale) Mirror 3: 0xaa42e620 (stale) CRC-32 checksum value for chunk [0, 0x3072e): Mirror 1: 0x248ffb4 Mirror 4: 0x248ffb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3072e: skipped 000068 1786530261.648682 trunc from 0x03072e to 0x034b14 (0x43e7 bytes) 000069 1786530261.761806 write 0x012a32 thru 0x01555d (0x2b2c bytes) 000070 1786530261.789971 read 0x02dcc6 thru 0x034b14 (0x6e4f bytes) 000071 1786530261.850720 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34b15): Mirror 3: 0x1f767504 (stale) Mirror 4: 0x887959c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b15): Mirror 3: 0x1f767504 (stale) Mirror 4: 0x887959c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b15: skipped 000072 1786530262.490306 trunc from 0x034b15 to 0x027d14 (0xce00 bytes) 000074 1786530262.522128 mapread 0x01814b thru 0x026b1c (0xe9d2 bytes) 000075 1786530262.577810 mapwrite 0x03ae04 thru 0x03ffff (0x51fc bytes) 000076 1786530262.706843 mapwrite 0x006d35 thru 0x01617c (0xf448 bytes) 000077 1786530262.795150 read 0x01637e thru 0x0208fe (0xa581 bytes) 000079 1786530262.796473 trunc from 0x040000 to 0x01bbdc (0x24423 bytes) 000080 1786530262.837001 mapwrite 0x02be84 thru 0x03731e (0xb49b bytes) 000081 1786530263.022209 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3731f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3731f): Mirror 3: 0xd4d0a3c6 (stale) Mirror 4: 0x3c24190e (stale) CRC-32 checksum value for chunk [0, 0x3731f): Mirror 1: 0x5448cab7 Mirror 5: 0x5448cab7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3731f: skipped 000082 1786530264.886924 fallocate 0x016394 thru 0x01efd5 (0x8c42 bytes) 000083 1786530264.983314 read 0x00b796 thru 0x01890e (0xd179 bytes) 000084 1786530264.996550 fallocate 0x00926e thru 0x015138 (0xbecb bytes) 000085 1786530265.005372 write 0x023168 thru 0x02f49f (0xc338 bytes) 000086 1786530265.022011 mapwrite 0x02ea13 thru 0x036256 (0x7844 bytes) 000087 1786530265.089650 write 0x0226d4 thru 0x028fdb (0x6908 bytes) 000088 1786530265.097242 read 0x00eec3 thru 0x01e772 (0xf8b0 bytes) 000089 1786530265.123877 write 0x00e357 thru 0x01096d (0x2617 bytes) 000090 1786530265.132184 MIRROR_RESYNC 4 mirrors 000092 1786530266.844086 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) 000094 1786530267.488881 trunc from 0x03731f to 0x02f85c (0x7ac2 bytes) 000095 1786530267.602062 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f85d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f85d): Mirror 4: 0x3b29c2cf (stale) Mirror 5: 0x3b29c2cf (stale) CRC-32 checksum value for chunk [0, 0x2f85d): Mirror 1: 0x3b29c2cf Mirror 6: 0x3b29c2cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f85d: skipped 000096 1786530269.089967 punch from 0x002033 to 0x008a36 (0x6a04 bytes) 000097 1786530269.174889 read 0x022623 thru 0x02c9f4 (0xa3d2 bytes) 000098 1786530269.197855 punch from 0x00dc12 to 0x01a7e7 (0xcbd6 bytes) 000099 1786530269.205853 trunc from 0x02f85d to 0x0268b7 (0x8fa5 bytes) 000100 1786530269.243604 trunc from 0x008f42 to 0x0178be (0xe97d bytes) 000101 1786530269.250407 punch from 0x018a7d to 0x022811 (0x9d95 bytes) 000102 1786530269.256515 trunc from 0x0268b8 to 0x01b8a2 (0xb015 bytes) 000103 1786530269.273820 fallocate 0x010b92 thru 0x012634 (0x1aa3 bytes) 000104 1786530269.280364 write 0x02f822 thru 0x035a01 (0x61e0 bytes) 000106 1786530269.290204 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35a02): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a02): Mirror 4: 0xe7c62b80 (stale) Mirror 5: 0xe7c62b80 (stale) Mirror 6: 0xfb416316 (stale) CRC-32 checksum value for chunk [0, 0x35a02): Mirror 1: 0x6731682 Mirror 7: 0x6731682 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a02: skipped 000107 1786530270.730899 mapread 0x034234 thru 0x035a01 (0x17ce bytes) 000108 1786530270.814957 fallocate 0x03056d thru 0x03aaf1 (0xa585 bytes) 000109 1786530270.879753 MIRROR_RESYNC 5 mirrors 000110 1786530272.789541 trunc from 0x03aaf2 to 0x036aef (0x4002 bytes) 000111 1786530272.828329 mapwrite 0x03d899 thru 0x03ffff (0x2767 bytes) 000113 1786530272.938898 write 0x010c3e thru 0x015f20 (0x52e3 bytes) 000114 1786530272.963451 trunc from 0x01b485 to 0x024ce6 (0x9862 bytes) 000115 1786530272.983856 trunc from 0x040000 to 0x01b470 (0x24b8f bytes) 000116 1786530273.011686 punch from 0x00d9ff to 0x014843 (0x6e45 bytes) 000118 1786530273.033341 punch from 0x008795 to 0x00f9fd (0x7269 bytes) 000119 1786530273.040474 mapwrite 0x038cc5 thru 0x039c47 (0xf83 bytes) 000120 1786530273.124958 trunc from 0x00d055 to 0x01464a (0x75f6 bytes) 000121 1786530273.138336 mapread 0x02ed1d thru 0x032b6c (0x3e50 bytes) 000122 1786530273.155528 trunc from 0x039c48 to 0x03eedf (0x5298 bytes) 000123 1786530273.216802 fallocate 0x023179 thru 0x02c1f1 (0x9079 bytes) 000124 1786530273.229039 trunc from 0x011174 to 0x018a15 (0x78a2 bytes) 000125 1786530273.245344 punch from 0x01db2b to 0x02aa92 (0xcf68 bytes) 000126 1786530273.264011 trunc from 0x03eee0 to 0x0170cb (0x27e14 bytes) 000127 1786530273.324388 MIRROR_RESYNC 5 mirrors 000128 1786530275.122438 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) 000129 1786530275.791126 trunc from 0x010edf to 0x013754 (0x2876 bytes) 000130 1786530275.845443 write 0x017dbe thru 0x01a810 (0x2a53 bytes) 000131 1786530275.858905 fallocate 0x00d4c8 thru 0x012e4f (0x5988 bytes) 000132 1786530275.868889 fallocate 0x01d3dd thru 0x01e139 (0xd5d bytes) 000133 1786530275.875542 read 0x0158c9 thru 0x01710b (0x1843 bytes) 000134 1786530275.889745 MIRROR_RESYNC 4 mirrors 000135 1786530277.400405 fallocate 0x03097c thru 0x03c039 (0xb6be bytes) 000136 1786530277.484772 trunc from 0x00202f to 0x00beda (0x9eac bytes) 000137 1786530277.507592 write 0x02cf11 thru 0x0383b4 (0xb4a4 bytes) 000138 1786530277.509746 fallocate 0x019357 thru 0x01c44a (0x30f4 bytes) 000139 1786530277.532663 write 0x030b35 thru 0x03981a (0x8ce6 bytes) 000140 1786530277.545029 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3981b): Mirror 6: 0x6d9cfad3 (stale) Mirror 7: 0x6d9cfad3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3981b): Mirror 6: 0x6d9cfad3 (stale) Mirror 7: 0x6d9cfad3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3981b: skipped 000141 1786530278.235274 MIRROR_RESYNC 3 mirrors 000142 1786530279.341325 mapwrite 0x02737f thru 0x02f5f7 (0x8279 bytes) 000143 1786530279.488910 punch from 0x0255a0 to 0x030573 (0xafd4 bytes) 000144 1786530279.503206 mapread 0x009602 thru 0x013a94 (0xa493 bytes) 000145 1786530279.540019 trunc from 0x02b402 to 0x03981a (0xe419 bytes) 000146 1786530279.549802 write 0x017418 thru 0x0248b0 (0xd499 bytes) 000148 1786530279.551910 punch from 0x02e5d0 to 0x031851 (0x3282 bytes) 000149 1786530279.559045 fallocate 0x000721 thru 0x004696 (0x3f76 bytes) 000150 1786530279.574341 trunc from 0x03981b to 0x01db8a (0x1bc90 bytes) 000152 1786530279.613583 punch from 0x00091e to 0x003a24 (0x3107 bytes) 000153 1786530279.621903 trunc from 0x01db8b to 0x036d28 (0x1919e bytes) 000154 1786530279.642294 punch from 0x009bbe to 0x011474 (0x78b7 bytes) 000155 1786530279.653359 trunc from 0x021104 to 0x02e46f (0xd36c bytes) 000156 1786530279.664328 mapread 0x0216d2 thru 0x02c188 (0xaab7 bytes) 000157 1786530279.697868 mapwrite 0x003382 thru 0x006f26 (0x3ba5 bytes) 000158 1786530279.790693 trunc from 0x036d29 to 0x02b889 (0xb49f bytes) 000160 1786530279.828897 trunc from 0x02b88a to 0x01866d (0x1321c bytes) 000161 1786530279.874232 trunc from 0x00bbbc to 0x01866d (0xcab2 bytes) 000162 1786530279.882366 write 0x003e8c thru 0x01308f (0xf204 bytes) 000164 1786530279.894882 trunc from 0x01866e to 0x009ad6 (0xeb97 bytes) 000165 1786530279.923891 read 0x0099df thru 0x009ad6 (0xf8 bytes) 000166 1786530279.927830 fallocate 0x026d51 thru 0x031ba9 (0xae59 bytes) 000167 1786530279.936745 mapread 0x00688f thru 0x009ad6 (0x3248 bytes) 000168 1786530279.939794 trunc from 0x0083ca to 0x009ad6 (0x170d bytes) 000169 1786530279.962782 mapread 0x001730 thru 0x009ad6 (0x83a7 bytes) 000170 1786530279.993072 trunc from 0x001ad1 to 0x009ad6 (0x8006 bytes) 000171 1786530280.004773 fallocate 0x0218de thru 0x02d1fd (0xb920 bytes) 000172 1786530280.022406 trunc from 0x00995a to 0x009ad6 (0x17d bytes) 000173 1786530280.031877 write 0x031625 thru 0x03ffff (0xe9db bytes) 000174 1786530280.044240 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd79f3e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd79f3e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000175 1786530280.503820 mapwrite 0x034df1 thru 0x03ffff (0xb20f bytes) 000176 1786530280.614126 read 0x03635c thru 0x03ffff (0x9ca4 bytes) 000177 1786530280.621923 mapwrite 0x00a3e4 thru 0x013607 (0x9224 bytes) 000178 1786530280.696888 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd79f3e7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84996337 Mirror 8: 0x84996337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000179 1786530282.260980 trunc from 0x03204f to 0x03bb15 (0x9ac7 bytes) 000181 1786530282.334535 punch from 0x0327a5 to 0x03287b (0xd7 bytes) 000182 1786530282.342269 write 0x02c6f4 thru 0x030f63 (0x4870 bytes) 000183 1786530282.366319 write 0x02a356 thru 0x034a44 (0xa6ef bytes) 000184 1786530282.382193 mapread 0x0003c9 thru 0x00c02a (0xbc62 bytes) 000185 1786530282.414456 read 0x011edc thru 0x01b364 (0x9489 bytes) 000186 1786530282.433538 trunc from 0x03b1c4 to 0x03bdb2 (0xbef bytes) 000187 1786530282.442271 fallocate 0x001bfb thru 0x00476a (0x2b70 bytes) 000188 1786530282.448847 read 0x00a068 thru 0x017295 (0xd22e bytes) 000189 1786530282.450783 mapwrite 0x0261ae thru 0x02dab6 (0x7909 bytes) 000190 1786530282.493581 mapwrite 0x01ca3a thru 0x0263e1 (0x99a8 bytes) 000191 1786530282.522223 MIRROR_RESYNC 3 mirrors 000192 1786530284.029352 read 0x01c836 thru 0x02833b (0xbb06 bytes) 000193 1786530284.039569 trunc from 0x040000 to 0x02cbca (0x13435 bytes) 000195 1786530284.087581 write 0x03a2ca thru 0x03ffff (0x5d36 bytes) 000196 1786530284.089756 fallocate 0x02870a thru 0x02f1e0 (0x6ad7 bytes) 000197 1786530284.096653 fallocate 0x013155 thru 0x0190bb (0x5f67 bytes) 000198 1786530284.102122 trunc from 0x040000 to 0x037e4d (0x81b2 bytes) 000199 1786530284.128796 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37e4e): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e4e): Mirror 7: 0x2460499 (stale) Mirror 8: 0x2460499 (stale) CRC-32 checksum value for chunk [0, 0x37e4e): Mirror 1: 0x5ce55859 Mirror 9: 0x5ce55859 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e4e: skipped 000200 1786530285.375869 mapread 0x012ae4 thru 0x016d1d (0x423a bytes) 000201 1786530285.442754 read 0x00160d thru 0x003720 (0x2114 bytes) 000202 1786530285.462872 mapread 0x02e932 thru 0x037e4d (0x951c bytes) 000203 1786530285.520472 mapread 0x0204a6 thru 0x02bb69 (0xb6c4 bytes) 000204 1786530285.569121 trunc from 0x00f26c to 0x014031 (0x4dc6 bytes) 000205 1786530285.642643 MIRROR_RESYNC 4 mirrors 000206 1786530287.438560 trunc from 0x037e4e to 0x036dff (0x104e bytes) 000207 1786530287.568216 trunc from 0x033769 to 0x036dff (0x3697 bytes) 000208 1786530287.594742 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36e00): Mirror 8: 0x4e2c2267 (stale) Mirror 9: 0x4e2c2267 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36e00): Mirror 8: 0x4e2c2267 (stale) Mirror 9: 0x4e2c2267 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36e00: skipped 000209 1786530288.101236 fallocate 0x02809b thru 0x035700 (0xd666 bytes) 000210 1786530288.110434 fallocate 0x02dd66 thru 0x02ebba (0xe55 bytes) 000211 1786530288.138622 MIRROR_RESYNC 3 mirrors 000212 1786530289.525611 mapwrite 0x031dd2 thru 0x03d008 (0xb237 bytes) 000213 1786530289.774082 fallocate 0x01fad2 thru 0x02b004 (0xb533 bytes) 000214 1786530289.789184 trunc from 0x02f398 to 0x03d008 (0xdc71 bytes) 000215 1786530289.803710 trunc from 0x01c8f7 to 0x0248c2 (0x7fcc bytes) 000216 1786530289.813511 fallocate 0x012737 thru 0x02066c (0xdf36 bytes) 000217 1786530289.844211 trunc from 0x00e178 to 0x0178c7 (0x9750 bytes) 000218 1786530289.861817 punch from 0x021e02 to 0x02fc00 (0xddff bytes) 000219 1786530289.896213 fallocate 0x03d850 thru 0x03ffff (0x27b0 bytes) 000221 1786530289.917100 mapwrite 0x0253a0 thru 0x0328a1 (0xd502 bytes) 000222 1786530290.129667 MIRROR_RESYNC 3 mirrors 000223 1786530291.678722 mapread 0x02a8ee thru 0x02f1d9 (0x48ec bytes) 000224 1786530291.729159 trunc from 0x01ce34 to 0x0277b9 (0xa986 bytes) 000225 1786530291.804173 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d009): Mirror 9: 0x51c6337a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d009): Mirror 9: 0x51c6337a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d009: skipped 000226 1786530292.286399 mapwrite 0x00c1e0 thru 0x01b70a (0xf52b bytes) 000227 1786530292.350659 punch from 0x01264f to 0x014e94 (0x2846 bytes) 000228 1786530292.356878 write 0x005e24 thru 0x008f62 (0x313f bytes) 000229 1786530292.365455 mapwrite 0x03b7a1 thru 0x03ffff (0x485f bytes) 000231 1786530292.409738 mapread 0x00abfe thru 0x00b8b8 (0xcbb bytes) 000232 1786530292.431998 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) 000233 1786530292.595462 punch from 0x031298 to 0x032c27 (0x1990 bytes) 000234 1786530292.611636 MIRROR_EXTEND to 2 mirrors 000235 1786530293.857496 mapwrite 0x01b077 thru 0x023957 (0x88e1 bytes) 000236 1786530293.971550 trunc from 0x02d474 to 0x032f37 (0x5ac4 bytes) 000238 1786530293.978700 read 0x00bfc2 thru 0x018302 (0xc341 bytes) 000239 1786530293.989088 read 0x01f047 thru 0x0217f2 (0x27ac bytes) 000240 1786530293.990469 fallocate 0x00fba4 thru 0x014193 (0x45f0 bytes) 000241 1786530293.996579 write 0x03769e thru 0x03ffff (0x8962 bytes) 000242 1786530294.015557 trunc from 0x040000 to 0x0077aa (0x38855 bytes) 000243 1786530294.047710 mapread 0x00639d thru 0x0077aa (0x140e bytes) 000244 1786530294.052193 fallocate 0x01c25a thru 0x01fd2c (0x3ad3 bytes) 000245 1786530294.059710 fallocate 0x01fab5 thru 0x02e6f1 (0xec3d bytes) 000246 1786530294.065697 mapread 0x000d73 thru 0x00c60d (0xb89b bytes) 000247 1786530294.096419 mapwrite 0x033566 thru 0x03ffff (0xca9a bytes) 000248 1786530294.217777 punch from 0x005da6 to 0x00a26c (0x44c7 bytes) 000250 1786530294.227346 fallocate 0x0072e2 thru 0x010a20 (0x973f bytes) 000251 1786530294.234571 mapread 0x02635f thru 0x031f19 (0xbbbb bytes) 000252 1786530294.298222 mapread 0x009e44 thru 0x00c067 (0x2224 bytes) 000253 1786530294.319709 mapread 0x00e8e0 thru 0x010bd6 (0x22f7 bytes) 000255 1786530294.346696 trunc from 0x01e1b9 to 0x02395f (0x57a7 bytes) 000256 1786530294.357463 read 0x030dc0 thru 0x03398b (0x2bcc bytes) 000257 1786530294.375761 write 0x03822d thru 0x03ced2 (0x4ca6 bytes) 000259 1786530294.380192 write 0x039eef thru 0x03acf1 (0xe03 bytes) 000260 1786530294.382703 trunc from 0x040000 to 0x014091 (0x2bf6e bytes) 000261 1786530294.413399 trunc from 0x014092 to 0x02caa0 (0x18a0f bytes) 000262 1786530294.459808 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2caa1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2caa1): Mirror 2: 0x1a141046 (stale) CRC-32 checksum value for chunk [0, 0x2caa1): Mirror 1: 0xff99c87d Mirror 3: 0xff99c87d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2caa1: skipped 000263 1786530295.649592 MIRROR_RESYNC 3 mirrors 000264 1786530296.644121 fallocate 0x0108fa thru 0x01d545 (0xcc4c bytes) 000266 1786530296.701105 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2caa1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2caa1): Mirror 2: 0xff99c87d (stale) Mirror 3: 0xff99c87d (stale) CRC-32 checksum value for chunk [0, 0x2caa1): Mirror 1: 0xff99c87d Mirror 4: 0xff99c87d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2caa1: skipped 000267 1786530298.009183 mapwrite 0x007263 thru 0x0121ee (0xaf8c bytes) 000270 1786530298.251659 fallocate 0x037af1 thru 0x03e4f2 (0x6a02 bytes) 000271 1786530298.261746 punch from 0x025728 to 0x027792 (0x206b bytes) 000272 1786530298.269149 write 0x0308ef thru 0x03cda7 (0xc4b9 bytes) 000273 1786530298.280888 read 0x000329 thru 0x006a37 (0x670f bytes) 000274 1786530298.298605 trunc from 0x03cda8 to 0x01d9b6 (0x1f3f1 bytes) 000275 1786530298.331711 punch from 0x002e82 to 0x008879 (0x59f8 bytes) 000276 1786530298.336721 fallocate 0x03b798 thru 0x03ffff (0x4868 bytes) 000277 1786530298.355214 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000278 1786530298.868389 mapread 0x018500 thru 0x01b059 (0x2b5a bytes) 000279 1786530298.909904 punch from 0x03b85e to 0x03f6e3 (0x3e86 bytes) 000280 1786530298.915903 trunc from 0x00e1e0 to 0x00fdd2 (0x1bf3 bytes) 000281 1786530298.934856 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf3201c6 Mirror 5: 0xaf3201c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000282 1786530300.259582 trunc from 0x040000 to 0x018818 (0x277e7 bytes) 000283 1786530300.364574 mapwrite 0x037af2 thru 0x03af55 (0x3464 bytes) 000285 1786530300.483671 mapread 0x01e6eb thru 0x02852f (0x9e45 bytes) 000286 1786530300.524721 punch from 0x034642 to 0x03af55 (0x6914 bytes) 000287 1786530300.547486 write 0x02c5cf thru 0x037330 (0xad62 bytes) 000288 1786530300.580778 trunc from 0x026c11 to 0x034fa9 (0xe399 bytes) 000289 1786530300.625870 punch from 0x023db6 to 0x02e5a0 (0xa7eb bytes) 000290 1786530300.634811 mapread 0x01bc3e thru 0x029339 (0xd6fc bytes) 000291 1786530300.655275 punch from 0x02fc52 to 0x03af55 (0xb304 bytes) 000292 1786530300.663605 write 0x02c2ef thru 0x0384bc (0xc1ce bytes) 000293 1786530300.675696 mapread 0x030b26 thru 0x03a592 (0x9a6d bytes) 000294 1786530300.685722 trunc from 0x03af56 to 0x02c6a9 (0xe8ac bytes) 000295 1786530300.732283 trunc from 0x02c6aa to 0x01e816 (0xde93 bytes) 000296 1786530300.777559 fallocate 0x019967 thru 0x022cfe (0x9398 bytes) 000297 1786530300.797766 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e817): Mirror 2: 0x3c2f7fa (stale) Mirror 4: 0x3c2f7fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e817): Mirror 2: 0x3c2f7fa (stale) Mirror 4: 0x3c2f7fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e817: skipped 000298 1786530301.420741 mapwrite 0x0331d1 thru 0x03e629 (0xb459 bytes) 000299 1786530301.515819 fallocate 0x037739 thru 0x03ffff (0x88c7 bytes) 000300 1786530301.524517 write 0x00852d thru 0x01403c (0xbb10 bytes) 000301 1786530301.563667 mapread 0x03c1bd thru 0x03e629 (0x246d bytes) 000302 1786530301.564724 trunc from 0x03d4d8 to 0x03e629 (0x1152 bytes) 000303 1786530301.573051 trunc from 0x01ec8b to 0x0251cc (0x6542 bytes) 000305 1786530301.581304 fallocate 0x0388f3 thru 0x03ffff (0x770d bytes) 000306 1786530301.586287 fallocate 0x02c775 thru 0x02da4c (0x12d8 bytes) 000308 1786530301.594695 read 0x017a0c thru 0x01e936 (0x6f2b bytes) 000310 1786530301.605721 mapwrite 0x00e532 thru 0x010fa1 (0x2a70 bytes) 000311 1786530301.646393 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000312 1786530302.150249 mapwrite 0x02ef4a thru 0x03b505 (0xc5bc bytes) 000314 1786530302.251112 fallocate 0x00324c thru 0x004be4 (0x1999 bytes) 000315 1786530302.262808 trunc from 0x011b4c to 0x013902 (0x1db7 bytes) 000316 1786530302.272618 read 0x0140b5 thru 0x0234ae (0xf3fa bytes) 000317 1786530302.293841 read 0x03171f thru 0x03d995 (0xc277 bytes) 000318 1786530302.294763 fallocate 0x01f5bc thru 0x020d32 (0x1777 bytes) 000319 1786530302.300875 read 0x000f10 thru 0x010574 (0xf665 bytes) 000320 1786530302.315399 fallocate 0x00995b thru 0x00de48 (0x44ee bytes) 000321 1786530302.323488 write 0x015bab thru 0x0239ae (0xde04 bytes) 000324 1786530302.331444 fallocate 0x00aafb thru 0x01774c (0xcc52 bytes) 000325 1786530302.338958 fallocate 0x02dfdb thru 0x037553 (0x9579 bytes) 000326 1786530302.345635 trunc from 0x040000 to 0x0258dd (0x1a722 bytes) 000327 1786530302.363358 trunc from 0x0258de to 0x0252ed (0x5f0 bytes) 000328 1786530302.380745 write 0x00cf42 thru 0x01ab36 (0xdbf5 bytes) 000329 1786530302.382418 read 0x0180d0 thru 0x0252ed (0xd21e bytes) 000330 1786530302.389430 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) 000331 1786530302.568635 write 0x01f87b thru 0x02369d (0x3e23 bytes) 000332 1786530302.613449 fallocate 0x02d3cf thru 0x03767c (0xa2ae bytes) 000333 1786530302.627398 trunc from 0x0252ee to 0x017cb5 (0xd638 bytes) 000334 1786530302.668424 fallocate 0x02c28c thru 0x03071c (0x4491 bytes) 000335 1786530302.684953 trunc from 0x03071d to 0x03e851 (0xe135 bytes) 000336 1786530302.737718 mapread 0x026ee9 thru 0x027305 (0x41d bytes) 000337 1786530302.769822 MIRROR_EXTEND to 2 mirrors 000338 1786530304.043710 mapread 0x00698b thru 0x007c60 (0x12d6 bytes) 000339 1786530304.090664 trunc from 0x03e852 to 0x0352ed (0x9564 bytes) 000340 1786530304.178558 trunc from 0x0352ee to 0x0005f6 (0x34cf7 bytes) 000342 1786530304.199720 punch from 0x000344 to 0x0005f6 (0x2b3 bytes) 000344 1786530304.206455 mapwrite 0x022c9c thru 0x02ef51 (0xc2b6 bytes) 000346 1786530304.273598 mapwrite 0x018977 thru 0x01d5cc (0x4c56 bytes) 000347 1786530304.336596 read 0x02ca10 thru 0x02ef51 (0x2542 bytes) 000349 1786530304.339725 mapread 0x01293b thru 0x01aee7 (0x85ad bytes) 000350 1786530304.353795 punch from 0x00060d to 0x0059b8 (0x53ac bytes) 000351 1786530304.368709 mapwrite 0x00150d thru 0x003692 (0x2186 bytes) 000352 1786530304.465660 fallocate 0x03af1b thru 0x03ffff (0x50e5 bytes) 000354 1786530304.498701 trunc from 0x040000 to 0x017ffb (0x28004 bytes) 000355 1786530304.532509 punch from 0x01504b to 0x017ffb (0x2fb1 bytes) 000356 1786530304.554720 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x17ffc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ffc): Mirror 2: 0xa6b7990c (stale) CRC-32 checksum value for chunk [0, 0x17ffc): Mirror 1: 0xebbca681 Mirror 3: 0xebbca681 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ffc: skipped 000357 1786530306.614143 trunc from 0x017ffc to 0x0058a3 (0x12758 bytes) 000359 1786530306.753531 read 0x000d0f thru 0x0058a3 (0x4b95 bytes) 000360 1786530306.767424 fallocate 0x0126c3 thru 0x01a24e (0x7b8c bytes) 000361 1786530306.775228 write 0x0223d2 thru 0x022bc5 (0x7f4 bytes) 000362 1786530306.781481 read 0x009aeb thru 0x00c9fc (0x2f12 bytes) 000363 1786530306.809508 mapread 0x01b2f6 thru 0x022bc5 (0x78d0 bytes) 000364 1786530306.813962 trunc from 0x022bc6 to 0x039625 (0x16a60 bytes) 000365 1786530306.914569 write 0x01d9d2 thru 0x02bc50 (0xe27f bytes) 000366 1786530306.928501 read 0x022361 thru 0x028cda (0x697a bytes) 000367 1786530306.929307 trunc from 0x039626 to 0x014c0b (0x24a1a bytes) 000368 1786530306.957067 punch from 0x000b0e to 0x00b947 (0xae3a bytes) 000369 1786530306.981882 write 0x03c7f2 thru 0x03ffff (0x380e bytes) 000370 1786530306.993010 mapwrite 0x00ca72 thru 0x018de0 (0xc36f bytes) 000372 1786530307.133195 write 0x01a101 thru 0x01add1 (0xcd1 bytes) 000373 1786530307.134742 write 0x00ad95 thru 0x015dac (0xb018 bytes) 000374 1786530307.136337 punch from 0x03a872 to 0x03c13e (0x18cd bytes) 000375 1786530307.154103 write 0x02ceaa thru 0x037c66 (0xadbd bytes) 000376 1786530307.172038 trunc from 0x040000 to 0x02945f (0x16ba0 bytes) 000377 1786530307.202859 punch from 0x007492 to 0x00e011 (0x6b80 bytes) 000378 1786530307.263040 MIRROR_RESYNC 3 mirrors 000379 1786530308.712609 read 0x015e5d thru 0x020f2e (0xb0d2 bytes) 000380 1786530308.726816 read 0x003148 thru 0x0041c6 (0x107f bytes) 000381 1786530308.747392 fallocate 0x01c539 thru 0x02030c (0x3dd4 bytes) 000383 1786530308.781479 read 0x01d617 thru 0x01e79b (0x1185 bytes) 000384 1786530308.795353 fallocate 0x0034fb thru 0x005ed4 (0x29da bytes) 000385 1786530308.802566 trunc from 0x029460 to 0x003699 (0x25dc6 bytes) 000386 1786530308.833875 trunc from 0x00369a to 0x01c730 (0x19097 bytes) 000387 1786530308.861727 fallocate 0x031648 thru 0x03e272 (0xcc2b bytes) 000388 1786530308.873443 mapread 0x0261e6 thru 0x02e622 (0x843d bytes) 000389 1786530308.909623 mapread 0x02985a thru 0x02fd2f (0x64d6 bytes) 000390 1786530308.914727 trunc from 0x03e273 to 0x02b12e (0x13144 bytes) 000392 1786530308.945017 punch from 0x003fee to 0x00fa73 (0xba86 bytes) 000393 1786530308.951849 trunc from 0x02b12f to 0x018f5a (0x121d4 bytes) 000395 1786530309.005424 fallocate 0x016159 thru 0x022da8 (0xcc50 bytes) 000396 1786530309.034259 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22da9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22da9): Mirror 2: 0xe1890a1e (stale) Mirror 3: 0xe1890a1e (stale) CRC-32 checksum value for chunk [0, 0x22da9): Mirror 1: 0xb2e54d91 Mirror 4: 0xb2e54d91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22da9: skipped 000397 1786530310.722730 trunc from 0x022da9 to 0x03ad7b (0x17fd3 bytes) 000398 1786530310.863462 trunc from 0x03ad7c to 0x0067d9 (0x345a2 bytes) 000399 1786530310.926205 trunc from 0x005fbe to 0x0067d9 (0x81c bytes) 000400 1786530310.971789 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x67da): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x67da): Mirror 2: 0x1ed00ec4 (stale) Mirror 3: 0x1ed00ec4 (stale) Mirror 4: 0x1ed00ec4 (stale) CRC-32 checksum value for chunk [0, 0x67da): Mirror 1: 0x1ed00ec4 Mirror 5: 0x1ed00ec4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x67da: skipped 000401 1786530312.361110 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x67da): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x67da): Mirror 2: 0x1ed00ec4 (stale) Mirror 3: 0x1ed00ec4 (stale) Mirror 4: 0x1ed00ec4 (stale) CRC-32 checksum value for chunk [0, 0x67da): Mirror 1: 0x1ed00ec4 Mirror 5: 0x1ed00ec4 Mirror 6: 0x1ed00ec4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x67da: skipped 000403 1786530314.227716 write 0x0374c0 thru 0x03ffff (0x8b40 bytes) 000404 1786530314.367165 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa138fde1 (stale) Mirror 3: 0xa138fde1 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7e473ee7 Mirror 7: 0x7e473ee7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000405 1786530316.386252 write 0x0157c5 thru 0x018f83 (0x37bf bytes) 000406 1786530316.450896 punch from 0x029cdf to 0x0382d9 (0xe5fb bytes) 000407 1786530316.459418 fallocate 0x0223c2 thru 0x025ce2 (0x3921 bytes) 000408 1786530316.465648 write 0x00c655 thru 0x00fb5a (0x3506 bytes) 000409 1786530316.477420 mapwrite 0x000403 thru 0x003b6e (0x376c bytes) 000411 1786530316.519089 mapread 0x02908e thru 0x02a707 (0x167a bytes) 000412 1786530316.529960 trunc from 0x040000 to 0x026178 (0x19e87 bytes) 000413 1786530316.559245 mapread 0x01c464 thru 0x026178 (0x9d15 bytes) 000414 1786530316.572785 mapwrite 0x032d60 thru 0x0349c8 (0x1c69 bytes) 000415 1786530316.666516 mapwrite 0x007c97 thru 0x0106cf (0x8a39 bytes) 000416 1786530316.743299 mapwrite 0x0339bf thru 0x03ffff (0xc641 bytes) 000417 1786530316.894361 read 0x0056dd thru 0x00c8a3 (0x71c7 bytes) 000418 1786530316.895731 trunc from 0x01078d to 0x0176e8 (0x6f5c bytes) 000419 1786530316.917659 read 0x030c04 thru 0x031fdf (0x13dc bytes) 000420 1786530316.918576 trunc from 0x040000 to 0x00b6ef (0x34910 bytes) 000421 1786530316.959686 trunc from 0x00b6f0 to 0x03b9ad (0x302be bytes) 000422 1786530317.034657 fallocate 0x03d4c7 thru 0x03ffff (0x2b39 bytes) 000423 1786530317.051980 punch from 0x00f4e1 to 0x01c2af (0xcdcf bytes) 000424 1786530317.073603 read 0x005625 thru 0x013eec (0xe8c8 bytes) 000425 1786530317.135776 write 0x029cca thru 0x02d827 (0x3b5e bytes) 000426 1786530317.143850 read 0x035271 thru 0x03b9ad (0x673d bytes) 000427 1786530317.155753 MIRROR_RESYNC 7 mirrors 000428 1786530320.451245 mapread 0x02c69b thru 0x037a0e (0xb374 bytes) 000429 1786530320.523774 mapread 0x015dd3 thru 0x01f052 (0x9280 bytes) 000430 1786530320.547631 fallocate 0x03f3d2 thru 0x03ffff (0xc2e bytes) 000431 1786530320.579864 mapread 0x036d82 thru 0x03ffff (0x927e bytes) 000432 1786530320.597488 write 0x01902b thru 0x0230e9 (0xa0bf bytes) 000433 1786530320.632270 mapwrite 0x0282d3 thru 0x02e4ed (0x621b bytes) 000434 1786530320.661302 trunc from 0x040000 to 0x018f13 (0x270ec bytes) 000435 1786530320.684226 mapwrite 0x0029f6 thru 0x00f68a (0xcc95 bytes) 000436 1786530320.720230 trunc from 0x018f14 to 0x007be4 (0x1132f bytes) 000437 1786530320.742170 mapread 0x0030c9 thru 0x007be4 (0x4b1c bytes) 000438 1786530320.743473 punch from 0x002e78 to 0x0061cb (0x3354 bytes) 000439 1786530320.767193 trunc from 0x005e31 to 0x007be4 (0x1db4 bytes) 000440 1786530320.794142 read 0x007697 thru 0x007be4 (0x54e bytes) 000441 1786530320.818702 trunc from 0x007be5 to 0x00f7c4 (0x7be0 bytes) 000443 1786530320.862020 mapwrite 0x028122 thru 0x02d36d (0x524c bytes) 000444 1786530320.981071 mapread 0x008076 thru 0x010079 (0x8004 bytes) 000445 1786530320.997300 trunc from 0x02d36e to 0x02b700 (0x1c6d bytes) 000446 1786530321.015403 trunc from 0x02b701 to 0x01f4b1 (0xc24f bytes) 000447 1786530321.033196 mapread 0x01ea4d thru 0x01f4b1 (0xa65 bytes) 000448 1786530321.041584 write 0x000e84 thru 0x002b12 (0x1c8f bytes) 000449 1786530321.046324 read 0x00beaa thru 0x00c29b (0x3f2 bytes) 000451 1786530321.052546 MIRROR_RESYNC 7 mirrors 000452 1786530323.232412 read 0x01a901 thru 0x01f4b1 (0x4bb1 bytes) 000453 1786530323.250410 trunc from 0x01f4b2 to 0x00878b (0x16d26 bytes) 000454 1786530323.356202 read 0x005790 thru 0x00878b (0x2ffc bytes) 000456 1786530323.380224 mapread 0x002eb4 thru 0x00878b (0x58d8 bytes) 000457 1786530323.387589 MIRROR_RESYNC 7 mirrors 000458 1786530325.477820 mapread 0x0058a3 thru 0x00878b (0x2ee9 bytes) 000459 1786530325.527392 trunc from 0x0019de to 0x005965 (0x3f88 bytes) 000460 1786530325.572545 fallocate 0x01aaf9 thru 0x023a61 (0x8f69 bytes) 000461 1786530325.580083 punch from 0x0027f4 to 0x00878b (0x5f98 bytes) 000463 1786530325.588420 write 0x00694e thru 0x007387 (0xa3a bytes) 000464 1786530325.603235 trunc from 0x001edc to 0x006f53 (0x5078 bytes) 000465 1786530325.616847 trunc from 0x001e6b to 0x00878b (0x6921 bytes) 000466 1786530325.623133 write 0x01f4d1 thru 0x0237d0 (0x4300 bytes) 000468 1786530325.626550 punch from 0x0155c9 to 0x01f1ad (0x9be5 bytes) 000469 1786530325.649593 write 0x03e35b thru 0x03ffff (0x1ca5 bytes) 000470 1786530325.653104 trunc from 0x040000 to 0x03d4fd (0x2b02 bytes) 000471 1786530325.684287 write 0x006221 thru 0x0074ad (0x128d bytes) 000472 1786530325.702817 punch from 0x0005db to 0x00a2e7 (0x9d0d bytes) 000473 1786530325.725209 write 0x024616 thru 0x02efb9 (0xa9a4 bytes) 000475 1786530325.748056 fallocate 0x013a9c thru 0x019dba (0x631f bytes) 000476 1786530325.758506 trunc from 0x02fe8f to 0x03d4fd (0xd66f bytes) 000477 1786530325.768280 trunc from 0x016258 to 0x017473 (0x121c bytes) 000478 1786530325.773903 punch from 0x0112b3 to 0x01278b (0x14d9 bytes) 000479 1786530325.796217 read 0x028341 thru 0x038038 (0xfcf8 bytes) 000480 1786530325.815762 mapread 0x00ae0b thru 0x00ccea (0x1ee0 bytes) 000481 1786530325.849276 punch from 0x039b51 to 0x03d4fd (0x39ad bytes) 000482 1786530325.858222 trunc from 0x033cc2 to 0x03be76 (0x81b5 bytes) 000483 1786530325.884900 punch from 0x012f17 to 0x017bfa (0x4ce4 bytes) 000484 1786530325.891591 punch from 0x00b303 to 0x01aef2 (0xfbf0 bytes) 000485 1786530325.906522 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d4fe): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4fe): Mirror 2: 0x51b270ae (stale) Mirror 3: 0x51b270ae (stale) Mirror 4: 0x51b270ae (stale) Mirror 5: 0x51b270ae (stale) Mirror 6: 0x51b270ae (stale) Mirror 7: 0x51b270ae (stale) CRC-32 checksum value for chunk [0, 0x3d4fe): Mirror 1: 0x2942beb9 Mirror 8: 0x2942beb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4fe: skipped 000487 1786530327.565211 read 0x00d91c thru 0x019c79 (0xc35e bytes) 000488 1786530327.642810 mapwrite 0x00a086 thru 0x00f088 (0x5003 bytes) 000489 1786530327.886251 punch from 0x02de0d to 0x03531f (0x7513 bytes) 000490 1786530327.897058 mapread 0x03b316 thru 0x03d4fd (0x21e8 bytes) 000491 1786530327.919597 fallocate 0x031c79 thru 0x035fa8 (0x4330 bytes) 000493 1786530327.947149 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4fe): Mirror 3: 0x51b270ae (stale) Mirror 4: 0x51b270ae (stale) Mirror 5: 0x51b270ae (stale) Mirror 6: 0x51b270ae (stale) Mirror 7: 0x51b270ae (stale) Mirror 8: 0x2942beb9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4fe): Mirror 3: 0x51b270ae (stale) Mirror 4: 0x51b270ae (stale) Mirror 5: 0x51b270ae (stale) Mirror 6: 0x51b270ae (stale) Mirror 7: 0x51b270ae (stale) Mirror 8: 0x2942beb9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4fe: skipped 000494 1786530328.763663 write 0x0262ac thru 0x02f2d2 (0x9027 bytes) 000497 1786530328.785186 mapread 0x01934e thru 0x01a258 (0xf0b bytes) 000498 1786530328.798259 trunc from 0x019c95 to 0x020661 (0x69cd bytes) 000499 1786530328.806169 trunc from 0x03d4fe to 0x03268b (0xae72 bytes) 000500 1786530328.831590 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3268c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3268c): Mirror 3: 0xd0d7a43d (stale) Mirror 4: 0xd0d7a43d (stale) Mirror 5: 0xd0d7a43d (stale) Mirror 6: 0xd0d7a43d (stale) Mirror 7: 0xd0d7a43d (stale) Mirror 8: 0xb16fc084 (stale) CRC-32 checksum value for chunk [0, 0x3268c): Mirror 1: 0x82108174 Mirror 9: 0x82108174 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3268c: skipped 000501 1786530330.463890 trunc from 0x03268c to 0x014c88 (0x1da03 bytes) 000502 1786530330.557362 trunc from 0x014c89 to 0x00aa68 (0xa220 bytes) 000503 1786530330.589647 mapwrite 0x028fd4 thru 0x0370c4 (0xe0f1 bytes) 000504 1786530330.695793 mapread 0x01d8ee thru 0x027622 (0x9d35 bytes) 000505 1786530330.719511 trunc from 0x019c33 to 0x026ec8 (0xd296 bytes) 000506 1786530330.730731 read 0x01a865 thru 0x0266aa (0xbe46 bytes) 000507 1786530330.749009 mapread 0x013774 thru 0x01b597 (0x7e24 bytes) 000508 1786530330.771466 punch from 0x02dcd9 to 0x030f83 (0x32ab bytes) 000509 1786530330.782965 trunc from 0x002763 to 0x00ca89 (0xa327 bytes) 000511 1786530330.795518 trunc from 0x021362 to 0x024edb (0x3b7a bytes) 000512 1786530330.810177 trunc from 0x018890 to 0x023e09 (0xb57a bytes) 000514 1786530330.823227 trunc from 0x0370c5 to 0x03e1a5 (0x70e1 bytes) 000515 1786530330.851074 write 0x008097 thru 0x00a4fc (0x2466 bytes) 000516 1786530330.880769 mapwrite 0x009bfa thru 0x011013 (0x741a bytes) 000517 1786530330.941149 trunc from 0x03e1a6 to 0x013beb (0x2a5ba bytes) 000519 1786530330.976226 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13bec): Mirror 3: 0xf96f6735 (stale) Mirror 4: 0xf96f6735 (stale) Mirror 5: 0xf96f6735 (stale) Mirror 6: 0xf96f6735 (stale) Mirror 7: 0xf96f6735 (stale) Mirror 9: 0x864b3ded (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13bec): Mirror 3: 0xf96f6735 (stale) Mirror 4: 0xf96f6735 (stale) Mirror 5: 0xf96f6735 (stale) Mirror 6: 0xf96f6735 (stale) Mirror 7: 0xf96f6735 (stale) Mirror 9: 0x864b3ded (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13bec: skipped 000520 1786530331.724706 fallocate 0x022dcf thru 0x025389 (0x25bb bytes) 000521 1786530331.739910 read 0x0034d8 thru 0x00aaef (0x7618 bytes) 000522 1786530331.762230 trunc from 0x013bec to 0x0339e8 (0x1fdfd bytes) 000523 1786530331.837894 read 0x0027ba thru 0x00bff4 (0x983b bytes) 000524 1786530331.856520 trunc from 0x0339e9 to 0x02e782 (0x5266 bytes) 000526 1786530331.902570 fallocate 0x0248ed thru 0x024edc (0x5f0 bytes) 000527 1786530331.915458 read 0x0228a6 thru 0x0265be (0x3d19 bytes) 000528 1786530331.916154 trunc from 0x02e783 to 0x02c2d7 (0x24ab bytes) 000529 1786530331.967607 read 0x00db31 thru 0x0159d0 (0x7ea0 bytes) 000530 1786530331.970597 read 0x00f57b thru 0x00f6b1 (0x137 bytes) 000531 1786530331.981905 mapwrite 0x03f7f3 thru 0x03ffff (0x80d bytes) 000532 1786530332.102012 trunc from 0x01417d to 0x01a03c (0x5ec0 bytes) 000533 1786530332.114044 mapwrite 0x032ec0 thru 0x038e96 (0x5fd7 bytes) 000534 1786530332.168881 mapwrite 0x020678 thru 0x020b47 (0x4d0 bytes) 000536 1786530332.197290 punch from 0x02a7e8 to 0x03198a (0x71a3 bytes) 000537 1786530332.210880 write 0x00655c thru 0x01213f (0xbbe4 bytes) 000538 1786530332.213892 punch from 0x007b27 to 0x016680 (0xeb5a bytes) 000539 1786530332.235828 fallocate 0x0146a8 thru 0x01f72a (0xb083 bytes) 000540 1786530332.241542 mapwrite 0x01dbdd thru 0x025483 (0x78a7 bytes) 000541 1786530332.267381 MIRROR_RESYNC 7 mirrors 000542 1786530335.851265 trunc from 0x040000 to 0x006eda (0x39125 bytes) 000543 1786530335.890814 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x6edb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6edb): Mirror 3: 0xdde7a159 (stale) Mirror 4: 0xdde7a159 (stale) Mirror 5: 0xdde7a159 (stale) Mirror 6: 0xdde7a159 (stale) Mirror 7: 0xdde7a159 (stale) Mirror 9: 0xdde7a159 (stale) CRC-32 checksum value for chunk [0, 0x6edb): Mirror 1: 0xdde7a159 Mirror 10: 0xdde7a159 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6edb: skipped 000544 1786530336.740481 read 0x004029 thru 0x006eda (0x2eb2 bytes) 000545 1786530336.751323 punch from 0x004327 to 0x006eda (0x2bb4 bytes) 000546 1786530336.789101 mapread 0x004edf thru 0x006eda (0x1ffc bytes) 000547 1786530336.815954 trunc from 0x0054ad to 0x006eda (0x1a2e bytes) 000548 1786530336.821775 mapwrite 0x028091 thru 0x029fe2 (0x1f52 bytes) 000549 1786530336.866723 punch from 0x009635 to 0x017874 (0xe240 bytes) 000550 1786530336.873002 trunc from 0x024e85 to 0x026d3c (0x1eb8 bytes) 000551 1786530336.878849 read 0x02841f thru 0x029fe2 (0x1bc4 bytes) 000552 1786530336.879458 read 0x00a195 thru 0x0129e6 (0x8852 bytes) 000554 1786530336.892917 punch from 0x0045a9 to 0x0071d4 (0x2c2c bytes) 000556 1786530336.899167 read 0x01ab2e thru 0x01d235 (0x2708 bytes) 000557 1786530336.900030 read 0x01eaa8 thru 0x029fe2 (0xb53b bytes) 000558 1786530336.901431 write 0x01745f thru 0x0190da (0x1c7c bytes) 000559 1786530336.902888 trunc from 0x029fe3 to 0x0233ac (0x6c36 bytes) 000560 1786530336.929319 trunc from 0x0233ad to 0x01babb (0x78f1 bytes) 000561 1786530336.949353 punch from 0x0169ca to 0x017e94 (0x14cb bytes) 000563 1786530336.963345 mapread 0x0021a2 thru 0x00f362 (0xd1c1 bytes) 000564 1786530336.974481 fallocate 0x021ae1 thru 0x023e61 (0x2381 bytes) 000565 1786530336.980372 trunc from 0x010f3b to 0x01a386 (0x944c bytes) 000566 1786530336.984729 trunc from 0x023e62 to 0x0123f0 (0x11a71 bytes) 000567 1786530336.999826 trunc from 0x0123f1 to 0x027d84 (0x15994 bytes) 000568 1786530337.019057 punch from 0x004510 to 0x0121b5 (0xdca6 bytes) 000569 1786530337.024048 trunc from 0x027d85 to 0x000b09 (0x2727b bytes) 000570 1786530337.047874 write 0x022d55 thru 0x02ce0d (0xa0b9 bytes) 000571 1786530337.050645 mapread 0x025cf6 thru 0x02ba2c (0x5d37 bytes) 000572 1786530337.061678 read 0x005156 thru 0x01341c (0xe2c7 bytes) 000573 1786530337.075697 trunc from 0x00dc5c to 0x010179 (0x251e bytes) 000574 1786530337.092406 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ce0e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce0e): Mirror 3: 0xc816bb0c (stale) Mirror 4: 0xc816bb0c (stale) Mirror 5: 0xc816bb0c (stale) Mirror 6: 0xc816bb0c (stale) Mirror 7: 0xc816bb0c (stale) Mirror 9: 0xc816bb0c (stale) Mirror 10: 0xe07b6be0 (stale) CRC-32 checksum value for chunk [0, 0x2ce0e): Mirror 1: 0x19b5b948 Mirror 11: 0x19b5b948 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce0e: skipped 000575 1786530338.100446 read 0x00e32e thru 0x01117f (0x2e52 bytes) 000576 1786530338.121541 write 0x023135 thru 0x02492f (0x17fb bytes) 000577 1786530338.200011 fallocate 0x0380ba thru 0x03ffff (0x7f46 bytes) 000578 1786530338.207229 trunc from 0x02f38b to 0x03cbde (0xd854 bytes) 000579 1786530338.213360 fallocate 0x00f465 thru 0x01cd38 (0xd8d4 bytes) 000580 1786530338.220449 read 0x017d35 thru 0x01903e (0x130a bytes) 000581 1786530338.249552 read 0x01db36 thru 0x023ee3 (0x63ae bytes) 000582 1786530338.250351 mapread 0x0274c8 thru 0x0348e8 (0xd421 bytes) 000583 1786530338.252788 write 0x021a8d thru 0x02ad67 (0x92db bytes) 000584 1786530338.260748 trunc from 0x040000 to 0x0264d0 (0x19b2f bytes) 000585 1786530338.356321 mapread 0x019431 thru 0x01caae (0x367e bytes) 000586 1786530338.365174 read 0x0213cb thru 0x0264d0 (0x5106 bytes) 000587 1786530338.368494 trunc from 0x0264d1 to 0x00192c (0x24ba4 bytes) 000588 1786530338.394662 write 0x016079 thru 0x021d10 (0xbc98 bytes) 000589 1786530338.396469 trunc from 0x021d11 to 0x022e9d (0x118d bytes) 000590 1786530338.423974 mapread 0x00fc73 thru 0x0119d4 (0x1d62 bytes) 000591 1786530338.446818 read 0x0108f7 thru 0x01a516 (0x9c20 bytes) 000592 1786530338.448310 trunc from 0x022e9e to 0x012745 (0x10758 bytes) 000593 1786530338.473407 read 0x0111ac thru 0x012745 (0x159a bytes) 000594 1786530338.474447 fallocate 0x03a79f thru 0x03ffff (0x5861 bytes) 000595 1786530338.483392 punch from 0x03fcd3 to 0x03ffff (0x32d bytes) 000596 1786530338.491765 mapwrite 0x039d75 thru 0x03fafc (0x5d88 bytes) 000597 1786530338.545034 trunc from 0x00dc40 to 0x016828 (0x8be9 bytes) 000598 1786530338.556167 write 0x000ea5 thru 0x00f2b3 (0xe40f bytes) 000599 1786530338.576426 trunc from 0x040000 to 0x0344f4 (0xbb0b bytes) 000600 1786530338.605784 fallocate 0x02dd5e thru 0x03902a (0xb2cd bytes) 000601 1786530338.617438 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x344f5): Mirror 4: 0x427822c (stale) Mirror 5: 0x427822c (stale) Mirror 6: 0x427822c (stale) Mirror 7: 0x427822c (stale) Mirror 9: 0x427822c (stale) Mirror 10: 0xd19229a9 (stale) Mirror 11: 0x3991e3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x344f5): Mirror 4: 0x427822c (stale) Mirror 5: 0x427822c (stale) Mirror 6: 0x427822c (stale) Mirror 7: 0x427822c (stale) Mirror 9: 0x427822c (stale) Mirror 10: 0xd19229a9 (stale) Mirror 11: 0x3991e3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x344f5: skipped 000602 1786530339.662546 trunc from 0x033a0b to 0x0344f4 (0xaea bytes) 000603 1786530339.668779 trunc from 0x033016 to 0x0344f4 (0x14df bytes) 000604 1786530339.673846 write 0x036e10 thru 0x038be6 (0x1dd7 bytes) 000605 1786530339.681240 mapread 0x0361d5 thru 0x038be6 (0x2a12 bytes) 000606 1786530339.683063 read 0x00d3c5 thru 0x00eab1 (0x16ed bytes) 000607 1786530339.692110 mapwrite 0x01e464 thru 0x02923a (0xadd7 bytes) 000608 1786530339.760757 write 0x003026 thru 0x0086d9 (0x56b4 bytes) 000609 1786530339.779600 write 0x0094fb thru 0x00c004 (0x2b0a bytes) 000610 1786530339.780911 mapread 0x013626 thru 0x01f6bd (0xc098 bytes) 000611 1786530339.794105 punch from 0x01b453 to 0x01d003 (0x1bb1 bytes) 000612 1786530339.799884 read 0x037a87 thru 0x038be6 (0x1160 bytes) 000613 1786530339.801318 mapread 0x0198a0 thru 0x025ed9 (0xc63a bytes) 000616 1786530339.814225 mapread 0x010ba9 thru 0x017391 (0x67e9 bytes) 000617 1786530339.821056 MIRROR_RESYNC 8 mirrors 000618 1786530341.945289 punch from 0x011ca2 to 0x0183f4 (0x6753 bytes) 000619 1786530341.990690 write 0x00c109 thru 0x00d319 (0x1211 bytes) 000620 1786530342.002390 write 0x01b432 thru 0x01ee2c (0x39fb bytes) 000621 1786530342.024531 mapread 0x02164e thru 0x02b111 (0x9ac4 bytes) 000622 1786530342.052765 mapread 0x00cb37 thru 0x00ebb8 (0x2082 bytes) 000623 1786530342.068766 trunc from 0x000b28 to 0x00cbd9 (0xc0b2 bytes) 000624 1786530342.099842 mapread 0x016b26 thru 0x0185ae (0x1a89 bytes) 000628 1786530342.136830 mapread 0x00d6a3 thru 0x01b487 (0xdde5 bytes) 000629 1786530342.140251 trunc from 0x038be7 to 0x007578 (0x3166e bytes) 000630 1786530342.186575 mapread 0x006e6a thru 0x007578 (0x70f bytes) 000631 1786530342.199858 write 0x03d745 thru 0x03ffff (0x28bb bytes) 000632 1786530342.201107 read 0x02c951 thru 0x035796 (0x8e46 bytes) 000634 1786530342.210369 fallocate 0x023ead thru 0x03092a (0xca7e bytes) 000635 1786530342.245903 MIRROR_RESYNC 8 mirrors 000636 1786530344.073857 read 0x037765 thru 0x03cdb3 (0x564f bytes) 000637 1786530344.083686 trunc from 0x027dae to 0x0360a1 (0xe2f4 bytes) 000639 1786530344.121684 mapwrite 0x034d8b thru 0x03ffff (0xb275 bytes) 000640 1786530344.170605 trunc from 0x040000 to 0x017fdc (0x28023 bytes) 000641 1786530344.188842 mapread 0x00b15b thru 0x0103e8 (0x528e bytes) 000642 1786530344.200847 fallocate 0x0347c8 thru 0x0356ed (0xf26 bytes) 000643 1786530344.206816 trunc from 0x01a5bf to 0x020fbd (0x69ff bytes) 000644 1786530344.212014 write 0x035070 thru 0x035299 (0x22a bytes) 000645 1786530344.217230 read 0x004ce4 thru 0x013c51 (0xef6e bytes) 000646 1786530344.218127 write 0x00e0dd thru 0x013be6 (0x5b0a bytes) 000647 1786530344.219617 trunc from 0x0356ee to 0x018b00 (0x1cbed bytes) 000648 1786530344.241502 mapread 0x014727 thru 0x018b00 (0x43da bytes) 000649 1786530344.251656 trunc from 0x00e5a8 to 0x013eb6 (0x590f bytes) 000650 1786530344.266733 mapwrite 0x036501 thru 0x03ffff (0x9aff bytes) 000651 1786530344.330267 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x66730fa7 (stale) Mirror 6: 0x66730fa7 (stale) Mirror 7: 0x66730fa7 (stale) Mirror 9: 0x66730fa7 (stale) Mirror 10: 0x66730fa7 (stale) Mirror 11: 0x66730fa7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x66730fa7 (stale) Mirror 6: 0x66730fa7 (stale) Mirror 7: 0x66730fa7 (stale) Mirror 9: 0x66730fa7 (stale) Mirror 10: 0x66730fa7 (stale) Mirror 11: 0x66730fa7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000652 1786530345.181845 fallocate 0x01c53b thru 0x021bf1 (0x56b7 bytes) 000653 1786530345.195859 mapwrite 0x00cbca thru 0x00f7b4 (0x2beb bytes) 000654 1786530345.271763 trunc from 0x016a19 to 0x01859a (0x1b82 bytes) 000655 1786530345.276674 write 0x000626 thru 0x007aef (0x74ca bytes) 000656 1786530345.304425 punch from 0x02db39 to 0x03aec3 (0xd38b bytes) 000658 1786530345.313911 trunc from 0x0169ee to 0x023b03 (0xd116 bytes) 000659 1786530345.325271 write 0x0232e4 thru 0x02e64b (0xb368 bytes) 000660 1786530345.338828 punch from 0x03fc12 to 0x03ffff (0x3ee bytes) 000661 1786530345.346266 read 0x0396b7 thru 0x03ffff (0x6949 bytes) 000662 1786530345.362026 fallocate 0x0374a8 thru 0x03ffff (0x8b58 bytes) 000663 1786530345.377228 trunc from 0x010947 to 0x0186d9 (0x7d93 bytes) 000664 1786530345.399155 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x66730fa7 (stale) Mirror 6: 0x66730fa7 (stale) Mirror 7: 0x66730fa7 (stale) Mirror 9: 0x66730fa7 (stale) Mirror 10: 0x66730fa7 (stale) Mirror 11: 0x66730fa7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7458a08c Mirror 12: 0x7458a08c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000665 1786530347.312861 read 0x0072a2 thru 0x0161eb (0xef4a bytes) 000666 1786530347.348543 mapread 0x020486 thru 0x02d813 (0xd38e bytes) 000667 1786530347.418436 mapwrite 0x002f2e thru 0x009c0b (0x6cde bytes) 000668 1786530347.666889 mapwrite 0x002a60 thru 0x005a82 (0x3023 bytes) 000669 1786530347.717646 write 0x029d68 thru 0x02cd81 (0x301a bytes) 000670 1786530347.739117 mapwrite 0x02ce34 thru 0x02d367 (0x534 bytes) 000671 1786530347.788564 punch from 0x03962e to 0x03ffff (0x69d2 bytes) 000672 1786530347.800582 write 0x001023 thru 0x00e197 (0xd175 bytes) 000675 1786530347.811195 mapwrite 0x00c4c7 thru 0x00d994 (0x14ce bytes) 000678 1786530347.862129 fallocate 0x032e9f thru 0x03ffff (0xd161 bytes) 000679 1786530347.872775 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x66730fa7 (stale) Mirror 6: 0x66730fa7 (stale) Mirror 7: 0x66730fa7 (stale) Mirror 9: 0x66730fa7 (stale) Mirror 10: 0x66730fa7 (stale) Mirror 11: 0x66730fa7 (stale) Mirror 12: 0x7458a08c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdcf26e8e Mirror 13: 0xdcf26e8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000680 1786530349.646147 trunc from 0x040000 to 0x007f55 (0x380aa bytes) 000681 1786530349.779217 punch from 0x0072e0 to 0x007f55 (0xc76 bytes) 000682 1786530349.797142 punch from 0x00383d to 0x007f55 (0x4719 bytes) 000683 1786530349.817230 fallocate 0x0378eb thru 0x03ffff (0x8715 bytes) 000684 1786530349.848271 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x7f56): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7f56): Mirror 5: 0xf50a1c55 (stale) Mirror 6: 0xf50a1c55 (stale) Mirror 7: 0xf50a1c55 (stale) Mirror 9: 0xf50a1c55 (stale) Mirror 10: 0xf50a1c55 (stale) Mirror 11: 0xf50a1c55 (stale) Mirror 12: 0xb6a051a6 (stale) Mirror 13: 0xfa913f43 (stale) CRC-32 checksum value for chunk [0, 0x7f56): Mirror 1: 0xea5f50c4 Mirror 14: 0xea5f50c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7f56: skipped 000686 1786530351.056493 mapread 0x005cce thru 0x007f55 (0x2288 bytes) 000687 1786530351.116804 fallocate 0x02c0f2 thru 0x03851a (0xc429 bytes) 000688 1786530351.169409 punch from 0x005e7c to 0x006b09 (0xc8e bytes) 000689 1786530351.181185 punch from 0x002b7e to 0x007f55 (0x53d8 bytes) 000690 1786530351.191117 trunc from 0x007f56 to 0x01663e (0xe6e9 bytes) 000692 1786530351.234568 punch from 0x016625 to 0x01663e (0x1a bytes) 000693 1786530351.251279 write 0x0242aa thru 0x02f842 (0xb599 bytes) 000695 1786530351.254270 mapwrite 0x0215ec thru 0x02c30f (0xad24 bytes) 000697 1786530351.316467 MIRROR_RESYNC 10 mirrors 000698 1786530353.459886 fallocate 0x034778 thru 0x03ffff (0xb888 bytes) 000699 1786530353.517311 fallocate 0x0324e1 thru 0x036018 (0x3b38 bytes) 000700 1786530353.531429 trunc from 0x040000 to 0x01e7cc (0x21833 bytes) 000701 1786530353.560759 mapwrite 0x020396 thru 0x025d42 (0x59ad bytes) 000702 1786530353.694156 write 0x0023e0 thru 0x00be85 (0x9aa6 bytes) 000703 1786530353.739723 punch from 0x015188 to 0x01f13a (0x9fb3 bytes) 000704 1786530353.747222 read 0x01beb9 thru 0x020764 (0x48ac bytes) 000705 1786530353.777709 trunc from 0x025d43 to 0x02940d (0x36cb bytes) 000706 1786530353.834107 mapread 0x0198df thru 0x0238f4 (0xa016 bytes) 000707 1786530353.864365 trunc from 0x02940e to 0x00a395 (0x1f078 bytes) 000708 1786530353.902158 mapread 0x002284 thru 0x00a395 (0x8112 bytes) 000709 1786530353.903701 mapread 0x0069ae thru 0x00a395 (0x39e8 bytes) 000710 1786530353.917910 trunc from 0x00a396 to 0x030b8a (0x267f5 bytes) 000711 1786530353.998408 trunc from 0x030b8b to 0x02f4b0 (0x16da bytes) 000712 1786530354.050913 punch from 0x0291a5 to 0x02b179 (0x1fd5 bytes) 000713 1786530354.063327 punch from 0x00e2e8 to 0x017477 (0x9190 bytes) 000714 1786530354.075530 fallocate 0x039fe2 thru 0x03ffff (0x601e bytes) 000715 1786530354.093070 read 0x029abd thru 0x02f4b0 (0x59f4 bytes) 000717 1786530354.133646 read 0x01f468 thru 0x02dad6 (0xe66f bytes) 000718 1786530354.134556 trunc from 0x01ed3f to 0x02c1d8 (0xd49a bytes) 000719 1786530354.151110 punch from 0x016783 to 0x017963 (0x11e1 bytes) 000720 1786530354.166173 read 0x017ccd thru 0x018816 (0xb4a bytes) 000721 1786530354.207458 MIRROR_RESYNC 10 mirrors 000722 1786530356.060173 punch from 0x01b586 to 0x02b56d (0xffe8 bytes) 000723 1786530356.122089 punch from 0x015c85 to 0x0189fc (0x2d78 bytes) 000724 1786530356.132546 trunc from 0x0089e5 to 0x00c806 (0x3e22 bytes) 000725 1786530356.141457 punch from 0x007fdd to 0x01093a (0x895e bytes) 000727 1786530356.149529 read 0x0141cd thru 0x019b38 (0x596c bytes) 000728 1786530356.160443 trunc from 0x02f4b1 to 0x001513 (0x2df9d bytes) 000729 1786530356.190162 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1514): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1514): Mirror 5: 0x5fb0a429 (stale) Mirror 6: 0x5fb0a429 (stale) Mirror 7: 0x5fb0a429 (stale) Mirror 9: 0x5fb0a429 (stale) Mirror 10: 0x5fb0a429 (stale) Mirror 11: 0x5fb0a429 (stale) Mirror 12: 0x5fb0a429 (stale) Mirror 13: 0x5fb0a429 (stale) Mirror 14: 0x5fb0a429 (stale) CRC-32 checksum value for chunk [0, 0x1514): Mirror 1: 0x5fb0a429 Mirror 15: 0x5fb0a429 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1514: skipped 000730 1786530357.064629 trunc from 0x000d32 to 0x001513 (0x7e2 bytes) 000731 1786530357.110288 trunc from 0x001514 to 0x00a3f2 (0x8edf bytes) 000732 1786530357.163810 read 0x004548 thru 0x009123 (0x4bdc bytes) 000733 1786530357.172765 read 0x002fb5 thru 0x008dc2 (0x5e0e bytes) 000734 1786530357.180643 trunc from 0x00a3f3 to 0x01dbac (0x137ba bytes) 000735 1786530357.207787 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dbad): Mirror 5: 0x2ab1beee (stale) Mirror 6: 0x2ab1beee (stale) Mirror 7: 0x2ab1beee (stale) Mirror 10: 0x2ab1beee (stale) Mirror 11: 0x2ab1beee (stale) Mirror 12: 0x2ab1beee (stale) Mirror 13: 0x2ab1beee (stale) Mirror 14: 0x2ab1beee (stale) Mirror 15: 0xe65a6998 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbad): Mirror 5: 0x2ab1beee (stale) Mirror 6: 0x2ab1beee (stale) Mirror 7: 0x2ab1beee (stale) Mirror 10: 0x2ab1beee (stale) Mirror 11: 0x2ab1beee (stale) Mirror 12: 0x2ab1beee (stale) Mirror 13: 0x2ab1beee (stale) Mirror 14: 0x2ab1beee (stale) Mirror 15: 0xe65a6998 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbad: skipped 000736 1786530357.744246 read 0x00f2d6 thru 0x017ea4 (0x8bcf bytes) 000738 1786530357.754911 fallocate 0x038d06 thru 0x03ffff (0x72fa bytes) 000739 1786530357.759679 punch from 0x023929 to 0x02660b (0x2ce3 bytes) 000740 1786530357.764545 write 0x031ef4 thru 0x039dc3 (0x7ed0 bytes) 000741 1786530357.779705 write 0x00f034 thru 0x011023 (0x1ff0 bytes) 000742 1786530357.780666 mapwrite 0x036b5d thru 0x03ffff (0x94a3 bytes) 000743 1786530357.819906 trunc from 0x040000 to 0x033830 (0xc7cf bytes) 000744 1786530357.848983 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33831): Mirror 6: 0x596cc509 (stale) Mirror 7: 0x596cc509 (stale) Mirror 10: 0x596cc509 (stale) Mirror 11: 0x596cc509 (stale) Mirror 12: 0x596cc509 (stale) Mirror 13: 0x596cc509 (stale) Mirror 14: 0x596cc509 (stale) Mirror 15: 0x378fba3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33831): Mirror 6: 0x596cc509 (stale) Mirror 7: 0x596cc509 (stale) Mirror 10: 0x596cc509 (stale) Mirror 11: 0x596cc509 (stale) Mirror 12: 0x596cc509 (stale) Mirror 13: 0x596cc509 (stale) Mirror 14: 0x596cc509 (stale) Mirror 15: 0x378fba3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33831: skipped 000745 1786530358.506678 mapwrite 0x02fd3f thru 0x031483 (0x1745 bytes) 000747 1786530358.547004 read 0x00c4a1 thru 0x0102e2 (0x3e42 bytes) 000748 1786530358.556812 mapwrite 0x03e73f thru 0x03ffff (0x18c1 bytes) 000749 1786530358.611464 mapwrite 0x016290 thru 0x021081 (0xadf2 bytes) 000750 1786530358.660883 mapread 0x038d6f thru 0x03ffff (0x7291 bytes) 000751 1786530358.663657 punch from 0x03cd46 to 0x03e2b8 (0x1573 bytes) 000752 1786530358.670669 fallocate 0x006477 thru 0x0065dc (0x166 bytes) 000753 1786530358.677758 write 0x02ca5f thru 0x03348b (0x6a2d bytes) 000756 1786530358.681530 mapread 0x025fa2 thru 0x034a9d (0xeafc bytes) 000757 1786530358.699739 trunc from 0x040000 to 0x00a5bd (0x35a42 bytes) 000758 1786530358.720858 read 0x000ede thru 0x007141 (0x6264 bytes) 000759 1786530358.736063 trunc from 0x0062f0 to 0x00772a (0x143b bytes) 000760 1786530358.743011 read 0x005b87 thru 0x009546 (0x39c0 bytes) 000761 1786530358.749619 trunc from 0x00a5be to 0x0063e4 (0x41d9 bytes) 000762 1786530358.770905 write 0x000082 thru 0x00c532 (0xc4b1 bytes) 000763 1786530358.774639 write 0x03e636 thru 0x03ffff (0x19ca bytes) 000764 1786530358.776206 trunc from 0x040000 to 0x03b10c (0x4ef3 bytes) 000765 1786530358.800099 fallocate 0x02d50b thru 0x0359c5 (0x84bb bytes) 000766 1786530358.809494 write 0x00bfc2 thru 0x00e6c0 (0x26ff bytes) 000767 1786530358.823860 fallocate 0x02f9c0 thru 0x03ca6a (0xd0ab bytes) 000768 1786530358.832460 write 0x034f0a thru 0x03ffff (0xb0f6 bytes) 000769 1786530358.834268 trunc from 0x040000 to 0x029565 (0x16a9a bytes) 000770 1786530358.860543 trunc from 0x029566 to 0x02801e (0x1547 bytes) 000771 1786530358.879096 trunc from 0x002f76 to 0x01252c (0xf5b7 bytes) 000772 1786530358.900751 read 0x026a6f thru 0x02801e (0x15b0 bytes) 000773 1786530358.901787 fallocate 0x0364e0 thru 0x03ffff (0x9b20 bytes) 000774 1786530358.917780 punch from 0x016494 to 0x022d8d (0xc8fa bytes) 000775 1786530358.933862 fallocate 0x0214d7 thru 0x027996 (0x64c0 bytes) 000776 1786530358.953659 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2801f): Mirror 6: 0xfa8a9af2 (stale) Mirror 10: 0xfa8a9af2 (stale) Mirror 11: 0xfa8a9af2 (stale) Mirror 12: 0xfa8a9af2 (stale) Mirror 13: 0xfa8a9af2 (stale) Mirror 14: 0xfa8a9af2 (stale) Mirror 15: 0x3be1b294 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2801f): Mirror 6: 0xfa8a9af2 (stale) Mirror 10: 0xfa8a9af2 (stale) Mirror 11: 0xfa8a9af2 (stale) Mirror 12: 0xfa8a9af2 (stale) Mirror 13: 0xfa8a9af2 (stale) Mirror 14: 0xfa8a9af2 (stale) Mirror 15: 0x3be1b294 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2801f: skipped 000777 1786530359.951478 read 0x023590 thru 0x02801e (0x4a8f bytes) 000778 1786530359.966644 fallocate 0x0292a5 thru 0x034771 (0xb4cd bytes) 000779 1786530359.975759 read 0x0089bc thru 0x016792 (0xddd7 bytes) 000780 1786530359.993327 mapwrite 0x0199be thru 0x01b89d (0x1ee0 bytes) 000781 1786530360.026282 mapwrite 0x0154af thru 0x019449 (0x3f9b bytes) 000782 1786530360.085274 write 0x0109e2 thru 0x01c75d (0xbd7c bytes) 000783 1786530360.105626 read 0x01baad thru 0x02801e (0xc572 bytes) 000784 1786530360.106373 mapread 0x00bacd thru 0x00d452 (0x1986 bytes) 000785 1786530360.155895 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2801f): Mirror 6: 0xfa8a9af2 (stale) Mirror 10: 0xfa8a9af2 (stale) Mirror 11: 0xfa8a9af2 (stale) Mirror 12: 0xfa8a9af2 (stale) Mirror 13: 0xfa8a9af2 (stale) Mirror 15: 0x3be1b294 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2801f): Mirror 6: 0xfa8a9af2 (stale) Mirror 10: 0xfa8a9af2 (stale) Mirror 11: 0xfa8a9af2 (stale) Mirror 12: 0xfa8a9af2 (stale) Mirror 13: 0xfa8a9af2 (stale) Mirror 15: 0x3be1b294 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2801f: skipped 000786 1786530360.881432 trunc from 0x0011a8 to 0x01043d (0xf296 bytes) 000787 1786530360.895682 trunc from 0x019a58 to 0x02801e (0xe5c7 bytes) 000788 1786530360.902678 read 0x00640a thru 0x00731b (0xf12 bytes) 000789 1786530360.909908 mapwrite 0x03d63e thru 0x03ffff (0x29c2 bytes) 000790 1786530361.003471 fallocate 0x0236c3 thru 0x028c05 (0x5543 bytes) 000791 1786530361.015659 mapread 0x002cb6 thru 0x00ebfc (0xbf47 bytes) 000792 1786530361.038272 mapwrite 0x0313a2 thru 0x03aab7 (0x9716 bytes) 000793 1786530361.127371 write 0x023851 thru 0x0300df (0xc88f bytes) 000794 1786530361.144256 fallocate 0x00c8be thru 0x0143a4 (0x7ae7 bytes) 000795 1786530361.153614 trunc from 0x040000 to 0x004625 (0x3b9da bytes) 000796 1786530361.194577 read 0x000b33 thru 0x004625 (0x3af3 bytes) 000797 1786530361.208798 mapread 0x0004d1 thru 0x004625 (0x4155 bytes) 000799 1786530361.217578 mapread 0x004227 thru 0x004625 (0x3ff bytes) 000800 1786530361.237344 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x4626): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4626): Mirror 6: 0x95fa5e50 (stale) Mirror 10: 0x95fa5e50 (stale) Mirror 11: 0x95fa5e50 (stale) Mirror 12: 0x95fa5e50 (stale) Mirror 13: 0x95fa5e50 (stale) Mirror 15: 0x1edf8fa (stale) CRC-32 checksum value for chunk [0, 0x4626): Mirror 1: 0x2756ddac Mirror 16: 0x2756ddac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4626: skipped 000801 1786530362.097019 mapread 0x00144e thru 0x004625 (0x31d8 bytes) 000802 1786530362.127411 write 0x039f1d thru 0x03ffff (0x60e3 bytes) 000803 1786530362.177280 read 0x00f8c0 thru 0x0140fb (0x483c bytes) 000804 1786530362.193243 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa90b08e6 (stale) Mirror 10: 0xa90b08e6 (stale) Mirror 12: 0xa90b08e6 (stale) Mirror 13: 0xa90b08e6 (stale) Mirror 15: 0xe3e16686 (stale) Mirror 16: 0x5eff8b4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa90b08e6 (stale) Mirror 10: 0xa90b08e6 (stale) Mirror 12: 0xa90b08e6 (stale) Mirror 13: 0xa90b08e6 (stale) Mirror 15: 0xe3e16686 (stale) Mirror 16: 0x5eff8b4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000805 1786530362.772881 read 0x028bd7 thru 0x036351 (0xd77b bytes) 000806 1786530362.799712 punch from 0x00a7b1 to 0x0102f5 (0x5b45 bytes) 000807 1786530362.808011 trunc from 0x038d79 to 0x03ffff (0x7287 bytes) 000808 1786530362.831368 write 0x031863 thru 0x03ffff (0xe79d bytes) 000809 1786530362.844889 write 0x0027f1 thru 0x010535 (0xdd45 bytes) 000810 1786530362.863784 punch from 0x012f28 to 0x021591 (0xe66a bytes) 000811 1786530362.876780 write 0x00be19 thru 0x011ae6 (0x5cce bytes) 000812 1786530362.897328 write 0x020ad0 thru 0x02f958 (0xee89 bytes) 000813 1786530362.914022 mapread 0x034fe5 thru 0x03ffff (0xb01b bytes) 000814 1786530362.925394 trunc from 0x040000 to 0x01555d (0x2aaa2 bytes) 000815 1786530362.982892 trunc from 0x00bc0b to 0x00e1c9 (0x25bf bytes) 000816 1786530363.013591 punch from 0x00e7b8 to 0x01555d (0x6da6 bytes) 000817 1786530363.029071 punch from 0x014846 to 0x01555d (0xd18 bytes) 000818 1786530363.049718 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1555e): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1555e): Mirror 6: 0xf7b72e37 (stale) Mirror 10: 0xf7b72e37 (stale) Mirror 12: 0xf7b72e37 (stale) Mirror 13: 0xf7b72e37 (stale) Mirror 15: 0x337ea074 (stale) Mirror 16: 0xb9e125a3 (stale) CRC-32 checksum value for chunk [0, 0x1555e): Mirror 1: 0x6b04f2fa Mirror 17: 0x6b04f2fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1555e: skipped 000820 1786530364.367293 fallocate 0x0114a4 thru 0x019441 (0x7f9e bytes) 000821 1786530364.431512 trunc from 0x019442 to 0x01eb47 (0x5706 bytes) 000823 1786530364.476577 fallocate 0x014aad thru 0x022fcf (0xe523 bytes) 000824 1786530364.484092 write 0x005676 thru 0x00e29f (0x8c2a bytes) 000825 1786530364.503430 trunc from 0x00d579 to 0x01a0d4 (0xcb5c bytes) 000826 1786530364.548424 write 0x008889 thru 0x00f290 (0x6a08 bytes) 000828 1786530364.560417 write 0x03598c thru 0x03ffff (0xa674 bytes) 000829 1786530364.564754 mapread 0x037d1a thru 0x03ffff (0x82e6 bytes) 000830 1786530364.566844 trunc from 0x040000 to 0x0371fe (0x8e01 bytes) 000831 1786530364.608554 punch from 0x034fce to 0x0371fe (0x2231 bytes) 000832 1786530364.626366 mapread 0x0177c0 thru 0x01892a (0x116b bytes) 000833 1786530364.642481 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x371ff): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x371ff): Mirror 6: 0xb3a53f66 (stale) Mirror 10: 0xb3a53f66 (stale) Mirror 12: 0xb3a53f66 (stale) Mirror 13: 0xb3a53f66 (stale) Mirror 15: 0x2863d735 (stale) Mirror 16: 0x6acba138 (stale) Mirror 17: 0x28c0db93 (stale) CRC-32 checksum value for chunk [0, 0x371ff): Mirror 1: 0x8e41f76b Mirror 18: 0x8e41f76b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x371ff: skipped 000834 1786530366.215296 trunc from 0x00387e to 0x011b9e (0xe321 bytes) 000835 1786530366.321512 fallocate 0x0221a1 thru 0x0257d0 (0x3630 bytes) 000836 1786530366.329486 read 0x031c0f thru 0x036645 (0x4a37 bytes) 000837 1786530366.345599 write 0x02a6e7 thru 0x0313fb (0x6d15 bytes) 000838 1786530366.356147 trunc from 0x03297b to 0x033d63 (0x13e9 bytes) 000839 1786530366.361449 fallocate 0x03e857 thru 0x03eeba (0x664 bytes) 000840 1786530366.368426 punch from 0x02b7f6 to 0x0371fe (0xba09 bytes) 000841 1786530366.386951 read 0x00f430 thru 0x01d2d2 (0xdea3 bytes) 000842 1786530366.411217 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x371ff): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x371ff): Mirror 6: 0xb3a53f66 (stale) Mirror 10: 0xb3a53f66 (stale) Mirror 12: 0xb3a53f66 (stale) Mirror 13: 0xb3a53f66 (stale) Mirror 15: 0x2863d735 (stale) Mirror 16: 0x6acba138 (stale) Mirror 17: 0x28c0db93 (stale) Mirror 18: 0x8e41f76b (stale) CRC-32 checksum value for chunk [0, 0x371ff): Mirror 1: 0xe7971917 Mirror 19: 0xe7971917 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x371ff: skipped 000843 1786530367.448232 write 0x00580d thru 0x006498 (0xc8c bytes) 000844 1786530367.515660 punch from 0x02b9d1 to 0x036eca (0xb4fa bytes) 000845 1786530367.520856 punch from 0x0095fc to 0x0119fe (0x8403 bytes) 000846 1786530367.530150 MIRROR_RESYNC 10 mirrors 000847 1786530370.783533 mapwrite 0x03f844 thru 0x03f8ee (0xab bytes) 000848 1786530370.900840 MIRROR_RESYNC 10 mirrors 000849 1786530374.685209 mapwrite 0x03602d thru 0x0391fe (0x31d2 bytes) 000850 1786530374.815366 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8ef): Mirror 6: 0x5400bd7b (stale) Mirror 10: 0x5400bd7b (stale) Mirror 12: 0x5400bd7b (stale) Mirror 13: 0x5400bd7b (stale) Mirror 15: 0x5400bd7b (stale) Mirror 16: 0x5400bd7b (stale) Mirror 17: 0x5400bd7b (stale) Mirror 19: 0x5400bd7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8ef): Mirror 6: 0x5400bd7b (stale) Mirror 10: 0x5400bd7b (stale) Mirror 12: 0x5400bd7b (stale) Mirror 13: 0x5400bd7b (stale) Mirror 15: 0x5400bd7b (stale) Mirror 16: 0x5400bd7b (stale) Mirror 17: 0x5400bd7b (stale) Mirror 19: 0x5400bd7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8ef: skipped 000851 1786530375.417062 mapread 0x0046fc thru 0x00a577 (0x5e7c bytes) 000852 1786530375.450746 write 0x01d1b4 thru 0x01e56e (0x13bb bytes) 000853 1786530375.466720 trunc from 0x03a01b to 0x03f8ee (0x58d4 bytes) 000854 1786530375.475247 mapwrite 0x01e633 thru 0x027f03 (0x98d1 bytes) 000855 1786530375.530334 mapread 0x02d09b thru 0x03517d (0x80e3 bytes) 000856 1786530375.560447 mapread 0x02be49 thru 0x0375a6 (0xb75e bytes) 000857 1786530375.588455 mapread 0x0260bc thru 0x02e35b (0x82a0 bytes) 000858 1786530375.589922 mapread 0x016fbe thru 0x01ac0f (0x3c52 bytes) 000859 1786530375.603683 mapwrite 0x03b6f7 thru 0x03ffff (0x4909 bytes) 000860 1786530375.673679 trunc from 0x040000 to 0x01424c (0x2bdb3 bytes) 000861 1786530375.720663 mapread 0x00ce5d thru 0x0138db (0x6a7f bytes) 000863 1786530375.726997 mapwrite 0x0035a9 thru 0x00dfeb (0xaa43 bytes) 000864 1786530375.773518 write 0x01d2e0 thru 0x028d60 (0xba81 bytes) 000865 1786530375.786716 punch from 0x01e2d0 to 0x025c7a (0x79ab bytes) 000866 1786530375.833788 MIRROR_RESYNC 9 mirrors 000867 1786530378.189745 read 0x025eb8 thru 0x028d60 (0x2ea9 bytes) 000868 1786530378.203228 MIRROR_EXTEND to 10 mirrors 000869 1786530379.024041 punch from 0x00b990 to 0x0155dc (0x9c4d bytes) 000870 1786530379.068823 read 0x014f29 thru 0x019b71 (0x4c49 bytes) 000871 1786530379.079541 trunc from 0x016b90 to 0x01790a (0xd7b bytes) 000873 1786530379.086300 mapwrite 0x034b29 thru 0x036d02 (0x21da bytes) 000874 1786530379.150451 write 0x01e046 thru 0x02a5ae (0xc569 bytes) 000875 1786530379.171112 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d03): Mirror 6: 0x7e4fe9a8 (stale) Mirror 10: 0x7e4fe9a8 (stale) Mirror 12: 0x7e4fe9a8 (stale) Mirror 13: 0x7e4fe9a8 (stale) Mirror 15: 0x7e4fe9a8 (stale) Mirror 17: 0x7e4fe9a8 (stale) Mirror 19: 0x7e4fe9a8 (stale) Mirror 20: 0x7e4fe9a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d03): Mirror 6: 0x7e4fe9a8 (stale) Mirror 10: 0x7e4fe9a8 (stale) Mirror 12: 0x7e4fe9a8 (stale) Mirror 13: 0x7e4fe9a8 (stale) Mirror 15: 0x7e4fe9a8 (stale) Mirror 17: 0x7e4fe9a8 (stale) Mirror 19: 0x7e4fe9a8 (stale) Mirror 20: 0x7e4fe9a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d03: skipped 000877 1786530379.866239 mapwrite 0x0346e0 thru 0x03ffff (0xb920 bytes) 000878 1786530379.940763 punch from 0x02911f to 0x02bef4 (0x2dd6 bytes) 000879 1786530379.952701 mapwrite 0x002375 thru 0x002964 (0x5f0 bytes) 000880 1786530379.991134 punch from 0x038f06 to 0x03cfeb (0x40e6 bytes) 000881 1786530380.004286 punch from 0x00860a to 0x008715 (0x10c bytes) 000882 1786530380.011562 trunc from 0x040000 to 0x034e8d (0xb172 bytes) 000883 1786530380.046659 mapwrite 0x01c3cf thru 0x02380c (0x743e bytes) 000884 1786530380.086407 read 0x029ac8 thru 0x034e8d (0xb3c6 bytes) 000886 1786530380.107004 fallocate 0x023f4f thru 0x033c4f (0xfd01 bytes) 000887 1786530380.114266 trunc from 0x000092 to 0x00f78b (0xf6fa bytes) 000888 1786530380.125539 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x34e8e): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e8e): Mirror 6: 0x307d400d (stale) Mirror 10: 0x307d400d (stale) Mirror 12: 0x307d400d (stale) Mirror 13: 0x307d400d (stale) Mirror 15: 0x307d400d (stale) Mirror 17: 0x307d400d (stale) Mirror 19: 0x307d400d (stale) Mirror 20: 0x307d400d (stale) CRC-32 checksum value for chunk [0, 0x34e8e): Mirror 1: 0xbb389d2c Mirror 21: 0xbb389d2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e8e: skipped 000889 1786530381.326275 fallocate 0x006768 thru 0x00f6d1 (0x8f6a bytes) 000890 1786530381.438672 fallocate 0x02a36d thru 0x0391f4 (0xee88 bytes) 000891 1786530381.447024 fallocate 0x0371d4 thru 0x03e752 (0x757f bytes) 000892 1786530381.456674 fallocate 0x030a67 thru 0x031316 (0x8b0 bytes) 000893 1786530381.465443 fallocate 0x03db18 thru 0x03ffff (0x24e8 bytes) 000894 1786530381.490390 write 0x00ef20 thru 0x01c477 (0xd558 bytes) 000895 1786530381.524662 mapwrite 0x008fa0 thru 0x00f725 (0x6786 bytes) 000896 1786530381.614228 punch from 0x00e1ed to 0x010793 (0x25a7 bytes) 000898 1786530381.621356 punch from 0x010c9f to 0x01a11a (0x947c bytes) 000899 1786530381.631855 trunc from 0x004ace to 0x00b19f (0x66d2 bytes) 000901 1786530381.648841 mapread 0x01330b thru 0x01c1ae (0x8ea4 bytes) 000902 1786530381.668417 punch from 0x017ece to 0x01faad (0x7be0 bytes) 000904 1786530381.676669 trunc from 0x022b75 to 0x02e1f8 (0xb684 bytes) 000905 1786530381.684078 fallocate 0x00e968 thru 0x01ce98 (0xe531 bytes) 000906 1786530381.690287 mapread 0x00f3e1 thru 0x010168 (0xd88 bytes) 000907 1786530381.691812 write 0x021ddb thru 0x023f5f (0x2185 bytes) 000908 1786530381.713016 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xefea48b4 (stale) Mirror 12: 0xefea48b4 (stale) Mirror 13: 0xefea48b4 (stale) Mirror 15: 0xefea48b4 (stale) Mirror 17: 0xefea48b4 (stale) Mirror 19: 0xefea48b4 (stale) Mirror 20: 0xefea48b4 (stale) Mirror 21: 0x92c48952 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xefea48b4 (stale) Mirror 12: 0xefea48b4 (stale) Mirror 13: 0xefea48b4 (stale) Mirror 15: 0xefea48b4 (stale) Mirror 17: 0xefea48b4 (stale) Mirror 19: 0xefea48b4 (stale) Mirror 20: 0xefea48b4 (stale) Mirror 21: 0x92c48952 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000909 1786530382.597623 write 0x003a0e thru 0x00c5fd (0x8bf0 bytes) 000910 1786530382.621668 read 0x036956 thru 0x03e304 (0x79af bytes) 000911 1786530382.633486 punch from 0x02deb5 to 0x03c86d (0xe9b9 bytes) 000912 1786530382.640316 mapread 0x009e8e thru 0x015ffe (0xc171 bytes) 000913 1786530382.666876 fallocate 0x00fb5a thru 0x01bd9a (0xc241 bytes) 000914 1786530382.681594 mapread 0x02fd28 thru 0x032532 (0x280b bytes) 000915 1786530382.693282 fallocate 0x01a689 thru 0x02a1ef (0xfb67 bytes) 000916 1786530382.700388 punch from 0x006e78 to 0x00f9d7 (0x8b60 bytes) 000917 1786530382.727258 read 0x02aa33 thru 0x02dd82 (0x3350 bytes) 000918 1786530382.730287 mapread 0x005574 thru 0x00a4eb (0x4f78 bytes) 000919 1786530382.756278 trunc from 0x040000 to 0x00937c (0x36c83 bytes) 000920 1786530382.822286 mapwrite 0x018f04 thru 0x0205a3 (0x76a0 bytes) 000921 1786530382.993730 trunc from 0x00c600 to 0x00dec6 (0x18c7 bytes) 000922 1786530383.010667 mapread 0x00c0d3 thru 0x0173fe (0xb32c bytes) 000924 1786530383.146386 trunc from 0x0205a4 to 0x005820 (0x1ad83 bytes) 000927 1786530383.184623 mapread 0x0005a5 thru 0x004798 (0x41f4 bytes) 000929 1786530383.192157 punch from 0x002809 to 0x005820 (0x3018 bytes) 000930 1786530383.203537 punch from 0x00055e to 0x00385e (0x3301 bytes) 000932 1786530383.226628 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x5821): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5821): Mirror 10: 0xf175a396 (stale) Mirror 12: 0xf175a396 (stale) Mirror 13: 0xf175a396 (stale) Mirror 15: 0xf175a396 (stale) Mirror 17: 0xf175a396 (stale) Mirror 19: 0xf175a396 (stale) Mirror 20: 0xf175a396 (stale) Mirror 21: 0xae80f071 (stale) CRC-32 checksum value for chunk [0, 0x5821): Mirror 1: 0xae80f071 Mirror 22: 0xae80f071 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5821: skipped 000933 1786530384.485691 read 0x000ea5 thru 0x005820 (0x497c bytes) 000934 1786530384.514634 punch from 0x001303 to 0x004581 (0x327f bytes) 000935 1786530384.613023 punch from 0x003ef2 to 0x005820 (0x192f bytes) 000936 1786530384.620603 write 0x031034 thru 0x03af3e (0x9f0b bytes) 000937 1786530384.632342 fallocate 0x03220e thru 0x0380ee (0x5ee1 bytes) 000939 1786530384.648265 mapwrite 0x01726e thru 0x024f9a (0xdd2d bytes) 000941 1786530384.732381 read 0x01a088 thru 0x01e422 (0x439b bytes) 000942 1786530384.734481 mapread 0x01eaf7 thru 0x02d3b5 (0xe8bf bytes) 000943 1786530384.756551 mapwrite 0x037539 thru 0x03ffff (0x8ac7 bytes) 000944 1786530384.880185 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xefea48b4 (stale) Mirror 12: 0xefea48b4 (stale) Mirror 13: 0xefea48b4 (stale) Mirror 15: 0xefea48b4 (stale) Mirror 17: 0xefea48b4 (stale) Mirror 19: 0xefea48b4 (stale) Mirror 20: 0xefea48b4 (stale) Mirror 21: 0x92c48952 (stale) Mirror 22: 0x242d5ff7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x37eb2dc4 Mirror 23: 0x37eb2dc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000945 1786530386.239260 mapread 0x039af1 thru 0x03ffff (0x650f bytes) 000946 1786530386.277876 punch from 0x03a32e to 0x03ffff (0x5cd2 bytes) 000947 1786530386.352176 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xefea48b4 (stale) Mirror 13: 0xefea48b4 (stale) Mirror 15: 0xefea48b4 (stale) Mirror 17: 0xefea48b4 (stale) Mirror 19: 0xefea48b4 (stale) Mirror 20: 0xefea48b4 (stale) Mirror 21: 0x92c48952 (stale) Mirror 22: 0x242d5ff7 (stale) Mirror 23: 0x37eb2dc4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xefea48b4 (stale) Mirror 13: 0xefea48b4 (stale) Mirror 15: 0xefea48b4 (stale) Mirror 17: 0xefea48b4 (stale) Mirror 19: 0xefea48b4 (stale) Mirror 20: 0xefea48b4 (stale) Mirror 21: 0x92c48952 (stale) Mirror 22: 0x242d5ff7 (stale) Mirror 23: 0x37eb2dc4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000948 1786530386.894533 mapwrite 0x03bfca thru 0x03ffff (0x4036 bytes) 000949 1786530386.952502 punch from 0x0291a6 to 0x033ac4 (0xa91f bytes) 000950 1786530386.982894 trunc from 0x01e6fd to 0x022699 (0x3f9d bytes) 000951 1786530387.003311 trunc from 0x040000 to 0x02128b (0x1ed74 bytes) 000952 1786530387.032826 write 0x0341d9 thru 0x03cf51 (0x8d79 bytes) 000953 1786530387.040280 read 0x00e2e1 thru 0x017e68 (0x9b88 bytes) 000954 1786530387.083856 punch from 0x039a03 to 0x03cf51 (0x354f bytes) 000956 1786530387.125522 punch from 0x029382 to 0x0317a6 (0x8425 bytes) 000957 1786530387.146323 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf52): Mirror 12: 0x3dd9ee5c (stale) Mirror 13: 0x3dd9ee5c (stale) Mirror 15: 0x3dd9ee5c (stale) Mirror 17: 0x3dd9ee5c (stale) Mirror 20: 0x3dd9ee5c (stale) Mirror 21: 0x2b790c5a (stale) Mirror 22: 0xe487cfc4 (stale) Mirror 23: 0x7a4dccb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf52): Mirror 12: 0x3dd9ee5c (stale) Mirror 13: 0x3dd9ee5c (stale) Mirror 15: 0x3dd9ee5c (stale) Mirror 17: 0x3dd9ee5c (stale) Mirror 20: 0x3dd9ee5c (stale) Mirror 21: 0x2b790c5a (stale) Mirror 22: 0xe487cfc4 (stale) Mirror 23: 0x7a4dccb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf52: skipped 000958 1786530388.051855 read 0x028f2b thru 0x029b76 (0xc4c bytes) 000959 1786530388.071462 trunc from 0x03cf52 to 0x02c585 (0x109cc bytes) 000960 1786530388.097761 punch from 0x02a521 to 0x02c585 (0x2065 bytes) 000961 1786530388.104103 write 0x005bfa thru 0x00e172 (0x8579 bytes) 000962 1786530388.116633 fallocate 0x035689 thru 0x039920 (0x4298 bytes) 000963 1786530388.123064 write 0x00f2d5 thru 0x012fe6 (0x3d12 bytes) 000964 1786530388.135365 fallocate 0x0328c7 thru 0x03b351 (0x8a8b bytes) 000966 1786530388.143481 trunc from 0x0301f9 to 0x036514 (0x631c bytes) 000967 1786530388.162376 write 0x02c119 thru 0x02f6ba (0x35a2 bytes) 000968 1786530388.201632 trunc from 0x016196 to 0x01954b (0x33b6 bytes) 000969 1786530388.210057 punch from 0x0142c0 to 0x01e572 (0xa2b3 bytes) 000970 1786530388.226747 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3b352): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b352): Mirror 12: 0xf9749cf6 (stale) Mirror 13: 0xf9749cf6 (stale) Mirror 15: 0xf9749cf6 (stale) Mirror 17: 0xf9749cf6 (stale) Mirror 20: 0xf9749cf6 (stale) Mirror 21: 0x64267734 (stale) Mirror 22: 0xab861b4c (stale) Mirror 23: 0x20d9e03d (stale) CRC-32 checksum value for chunk [0, 0x3b352): Mirror 1: 0xfd764f9b Mirror 24: 0xfd764f9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b352: skipped 000971 1786530389.948921 mapread 0x009de6 thru 0x019daa (0xffc5 bytes) 000973 1786530390.104270 mapread 0x028308 thru 0x035332 (0xd02b bytes) 000975 1786530390.232561 mapread 0x006417 thru 0x00fe39 (0x9a23 bytes) 000976 1786530390.268903 mapread 0x0153b2 thru 0x01b412 (0x6061 bytes) 000977 1786530390.294000 read 0x0011ad thru 0x00f767 (0xe5bb bytes) 000978 1786530390.315838 trunc from 0x00f57c to 0x014ef7 (0x597c bytes) 000979 1786530390.380354 read 0x01c035 thru 0x0256d4 (0x96a0 bytes) 000980 1786530390.396069 read 0x01e2aa thru 0x02af51 (0xcca8 bytes) 000981 1786530390.397136 mapread 0x0363e7 thru 0x03b351 (0x4f6b bytes) 000983 1786530390.399859 read 0x004cf5 thru 0x0096f9 (0x4a05 bytes) 000987 1786530390.405146 punch from 0x02853a to 0x035983 (0xd44a bytes) 000988 1786530390.410618 fallocate 0x00c2dd thru 0x00e47b (0x219f bytes) 000989 1786530390.418066 trunc from 0x02b038 to 0x02bc55 (0xc1e bytes) 000990 1786530390.424283 mapread 0x0105cb thru 0x01f231 (0xec67 bytes) 000991 1786530390.426076 read 0x0383a1 thru 0x0390f9 (0xd59 bytes) 000992 1786530390.428289 fallocate 0x0145b3 thru 0x01ce3d (0x888b bytes) 000993 1786530390.433629 trunc from 0x03b352 to 0x011c35 (0x2971c bytes) 000994 1786530390.498132 mapwrite 0x01b215 thru 0x024c06 (0x99f2 bytes) 000996 1786530390.571221 mapread 0x015030 thru 0x02478f (0xf760 bytes) 000997 1786530390.585053 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c07): Mirror 12: 0x7a51c28e (stale) Mirror 15: 0x7a51c28e (stale) Mirror 17: 0x7a51c28e (stale) Mirror 20: 0x7a51c28e (stale) Mirror 21: 0xc00cd242 (stale) Mirror 22: 0xec4fb670 (stale) Mirror 23: 0x89f7ecf5 (stale) Mirror 24: 0xf0a5eda6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c07): Mirror 12: 0x7a51c28e (stale) Mirror 15: 0x7a51c28e (stale) Mirror 17: 0x7a51c28e (stale) Mirror 20: 0x7a51c28e (stale) Mirror 21: 0xc00cd242 (stale) Mirror 22: 0xec4fb670 (stale) Mirror 23: 0x89f7ecf5 (stale) Mirror 24: 0xf0a5eda6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c07: skipped 000998 1786530391.267364 mapwrite 0x0222f6 thru 0x02cb56 (0xa861 bytes) 000999 1786530391.345495 trunc from 0x02ca29 to 0x02cb56 (0x12e bytes) 001000 1786530391.353464 fallocate 0x02ba91 thru 0x0351d5 (0x9745 bytes) All operations completed A-OK! PASS 70a (147s) == sanity-flr test 71: check mirror extend parent fid ==== 06:26:38 (1786530398) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x190:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (17s) == sanity-flr test 72: check append on FLR file ========== 06:26:55 (1786530415) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.353 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156126 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (13s) == sanity-flr test 200a: stress test ===================== 06:27:09 (1786530429) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6740448 .. Extending file size to 2435200 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6333152 .. done Extending file size to 7959168 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6944704 .. Extending file size to 7161696 .. Extending file size to 8477536 .. Extending file size to 8273536 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8370432 .. Extending file size to 9305856 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9125248 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61940 61941 61943 61945 61946 done 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) 192.168.201.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:28:35 (1786530515) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4004160 .. Extending file size to 442176 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 720032 .. failed Extending file size to 5205856 .. Extending file size to 5674304 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8815200 .. Extending file size to 7484704 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8484992 .. Extending file size to 9078368 .. Extending file size to 9182752 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63260 63261 63263 63264 63266 done 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) 192.168.201.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (93s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:30:08 (1786530608) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2216 1269088 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17756 3589264 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25492 3581528 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43248 7170792 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2216 1269088 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17756 3589264 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25492 3581528 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43248 7170792 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146895 s, 14.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31143 s, 6.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4dd06786 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) PASS 200c (25s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:30:36 (1786530636) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171765 s, 12.2 MB/s PASS 202 (13s) == sanity-flr test 203: mirror file preserve mirror ID === 06:30:49 (1786530649) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } PASS 203 (14s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:31:03 (1786530663) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148702 s, 14.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1df:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:31:14 (1786530674) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129302 s, 16.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } PASS 204b (12s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:31:27 (1786530687) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168483 s, 12.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } PASS 204c (39s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:32:06 (1786530726) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg120-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194152 s, 5.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e1:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:33:06 (1786530786) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg120-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254896 s, 8.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:34:00 (1786530840) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg120-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg120-server: Pool lustre.test_204f created Adding targets to pool oleg120-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg120-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125808 s, 16.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg120-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg120-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg120-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:35:21 (1786530921) PASS 205a (13s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:35:35 (1786530935) PASS 205b (13s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:35:49 (1786530949) oleg120-server: Pool lustre.test_206 created Adding targets to pool oleg120-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg120-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg120-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg120-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg120-server: Pool lustre.test_206 destroyed PASS 206 (35s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:36:23 (1786530983) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726902 s, 14.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625671 s, 16.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbca4902a Mirror 3: 0xbca4902a PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:36:35 (1786530995) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (7s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:36:43 (1786531003) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (7s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:36:49 (1786531009) Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0317528 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1990000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2550000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2550000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2550000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x990000 Read this (4 bytes): BBBB 192.168.201.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg120-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:37:34 (1786531054) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0374159 s, 219 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:37:44 (1786531064) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836548 s, 12.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-12 06:37:46.000000000 -0400 Modify: 2026-08-12 06:37:46.000000000 -0400 Change: 2026-08-12 06:37:48.000000000 -0400 Birth: 2026-08-12 06:37:46.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:37:54 (1786531074) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429021 s, 24.4 MB/s oleg120-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg120-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg120-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:38:21 (1786531101) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.373645 s, 110 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0270495 s, 151 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 12 06:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 12 06:38 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:38:31 (1786531111) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:38:41 (1786531121) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2167 sec =========== 06:38:52 (1786531132) === sanity-flr: start cleanup 06:38:54 (1786531134) === === sanity-flr: finish cleanup 06:39:03 (1786531143) ===