-----============= acceptance-small: sanity-flr ============----- Fri Jul 24 19:29:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg353-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:29:54 (1784935794) === oleg353-client.virtnet: executing check_config_client /mnt/lustre oleg353-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg353-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff933203413800.idle_timeout=debug osc.lustre-OST0001-osc-ffff933203413800.idle_timeout=debug disable quota as required oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 19:30:34 (1784935834) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8218] == sanity-flr test 0a: lfs mirror create with -N option == 19:30:36 (1784935836) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:30:52 (1784935852) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:30:55 (1784935855) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:30:57 (1784935857) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:31:23 (1784935883) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:31:26 (1784935886) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:31:30 (1784935890) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff933203413800.stats=clear osc.lustre-OST0001-osc-ffff933203413800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:31:38 (1784935898) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:31:48 (1784935908) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:31:55 (1784935915) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:31:59 (1784935919) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:32:07 (1784935927) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 19:32:16 (1784935936) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:32:24 (1784935944) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:32:32 (1784935952) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:32:40 (1784935960) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:32:50 (1784935970) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:32:57 (1784935977) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.17099 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115384 s, 9.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:33:07 (1784935987) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.624868 s, 33.6 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:33:16 (1784935996) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:33:29 (1784936009) Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg353-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (135s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:35:44 (1784936144) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201004 s, 10.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:36:26 (1784936186) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:37:38 (1784936258) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.96765 s, 17.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 24 19:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.963s user 0m0.000s sys 0m0.144s Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff933203413800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.147s user 0m0.002s sys 0m0.184s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.400s user 0m0.000s sys 0m0.188s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff933203413800.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:39:10 (1784936350) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:39:13 (1784936353) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:39:17 (1784936357) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 19:39:20 (1784936360) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 19:39:27 (1784936367) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.469758 s, 26.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:39:38 (1784936378) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027552 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067533 s, 15.5 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:39:47 (1784936387) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:39:59 (1784936399) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00257661 s, 397 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00290685 s, 352 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00166129 s, 616 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00502803 s, 204 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 19:40:09 (1784936409) create test files with size 15744668 .. ba2908683244efae5e4f60f4c928a0e6 - c2f1e81ace6c4a76a8faea82a43222ca - c5669ececdd2246ecf17f8276a71f33b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 19:40:33 (1784936433) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:40:48 (1784936448) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:40:54 (1784936454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549487 s, 19.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312639 s, 33.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 19:41:02 (1784936462) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0820934 s, 25.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.269012 s, 15.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154638 s, 13.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 19:41:15 (1784936475) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 19:41:18 (1784936478) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:41:21 (1784936481) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351541 s, 29.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 19:41:28 (1784936488) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.404677 s, 7.8 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:41:38 (1784936498) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:41:48 (1784936508) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333487 s, 31.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:41:59 (1784936519) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff933218bf9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.230464 s, 45.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff93321865c000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:42:14 (1784936534) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:42:49 (1784936569) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:43:22 (1784936602) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:43:25 (1784936605) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.219189 s, 14.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.128305 s, 24.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe59f6ae5 (nosync,stale) Mirror 2: 0xe59f6ae5 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0a213f870172b416eba98bcca88662d4 - mirror 1 checksum: 4c0eec1fc8c3f253948e9e2a791fe04c - mirror 2 checksum: 0a213f870172b416eba98bcca88662d4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-24T19:43:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-24T19:43:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0a213f870172b416eba98bcca88662d4 - mirror 1 checksum: 4c0eec1fc8c3f253948e9e2a791fe04c - mirror 2 checksum: 4c0eec1fc8c3f253948e9e2a791fe04c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-24T19:43:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-24T19:43:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe5:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 19:43:34 (1784936614) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311055 s, 33.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:43:42 (1784936622) Failing mds1 on oleg353-server Stopping /mnt/lustre-mds1 (opts:) on oleg353-server 19:43:55 (1784936635) shut down facet: mds1 facet_host: oleg353-server facet_failover_host: oleg353-server Failover mds1 to oleg353-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 19:44:29 (1784936669) targets are mounted 19:44:29 (1784936669) facet_failover done oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282581 s, 37.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe7:0x0] } PASS 50A (68s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:44:50 (1784936690) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0458734 s, 446 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0234829 s, 872 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:45:03 (1784936703) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0548036 s, 374 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0103717 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:45:13 (1784936713) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0113008 s, 1.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:45:21 (1784936721) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.072786 s, 281 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:45:29 (1784936729) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0221383 s, 185 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0244669 s, 167 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:45:36 (1784936736) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:45:59 (1784936759) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:46:27 (1784936787) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:46:50 (1784936810) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:47:15 (1784936835) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 19:47:22 (1784936842) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57571 mirror split pid 57572 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57571 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57572 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 19:48:29 (1784936909) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6911 000001 1784936912.026105 write 0x014199 thru 0x015c08 (0x1a70 bytes) zero_range to largest ever: 0x15c09 000002 1784936912.043929 trunc from 0x010788 to 0x015c08 (0x5481 bytes) 000003 1784936912.066950 punch from 0x011404 to 0x015c08 (0x4805 bytes) 000004 1784936912.074700 mapwrite 0x0356b7 thru 0x03ffff (0xa949 bytes) 000005 1784936912.168917 mapread 0x01f2da thru 0x02dca6 (0xe9cd bytes) 000006 1784936912.218346 write 0x02b8c1 thru 0x02ff9e (0x46de bytes) 000007 1784936912.224635 punch from 0x0000db to 0x006213 (0x6139 bytes) fallocating to largest ever: 0x30556 000008 1784936912.248809 fallocate 0x0209f7 thru 0x030555 (0xfb5f bytes) 000010 1784936912.262533 mapwrite 0x028dd3 thru 0x02a8ec (0x1b1a bytes) 000011 1784936912.290093 punch from 0x01a7e2 to 0x01ad08 (0x527 bytes) 000012 1784936912.296053 fallocate 0x00a69f thru 0x0150bb (0xaa1d bytes) 000013 1784936912.305856 fallocate 0x015549 thru 0x01cc04 (0x76bc bytes) 000014 1784936912.317142 trunc from 0x021108 to 0x02da37 (0xc930 bytes) fallocating to largest ever: 0x40000 000016 1784936912.332404 fallocate 0x03ffde thru 0x03ffff (0x22 bytes) 000017 1784936912.354323 trunc from 0x040000 to 0x022a45 (0x1d5ba bytes) 000019 1784936912.377875 trunc from 0x022a46 to 0x01eb02 (0x3f43 bytes) 000020 1784936912.407229 write 0x013601 thru 0x01931c (0x5d1c bytes) 000023 1784936912.415834 mapwrite 0x028df3 thru 0x02c9d9 (0x3be7 bytes) 000025 1784936912.457971 fallocate 0x03b1f2 thru 0x03ffff (0x4e0e bytes) 000027 1784936912.466814 trunc from 0x02c9da to 0x02607b (0x695e bytes) 000028 1784936912.480766 trunc from 0x02607c to 0x026b91 (0xb16 bytes) 000029 1784936912.504180 read 0x00f009 thru 0x011d75 (0x2d6d bytes) 000031 1784936912.517592 mapread 0x01ee73 thru 0x026b91 (0x7d1f bytes) 000032 1784936912.529656 write 0x01ad50 thru 0x022ebc (0x816d bytes) 000033 1784936912.531805 mapread 0x005840 thru 0x00a50b (0x4ccc bytes) 000036 1784936912.547132 punch from 0x01c9c6 to 0x02335e (0x6999 bytes) 000038 1784936912.575406 read 0x00a3b5 thru 0x013b01 (0x974d bytes) 000040 1784936912.596344 read 0x000669 thru 0x0046e6 (0x407e bytes) 000041 1784936912.606276 MIRROR_EXTEND to 2 mirrors 000042 1784936913.550797 mapread 0x006fc5 thru 0x009592 (0x25ce bytes) 000043 1784936913.596998 trunc from 0x026b92 to 0x026f53 (0x3c2 bytes) 000044 1784936913.654296 write 0x0248f6 thru 0x02ee38 (0xa543 bytes) 000046 1784936913.662172 write 0x0082da thru 0x0109d9 (0x8700 bytes) 000047 1784936913.675025 mapwrite 0x027715 thru 0x036fff (0xf8eb bytes) 000048 1784936913.729138 write 0x02c00b thru 0x030547 (0x453d bytes) 000050 1784936913.732164 punch from 0x023f82 to 0x025257 (0x12d6 bytes) 000052 1784936913.737877 trunc from 0x030b06 to 0x036fff (0x64fa bytes) 000053 1784936913.750815 read 0x029ec0 thru 0x02afc2 (0x1103 bytes) 000054 1784936913.751961 punch from 0x011307 to 0x01675b (0x5455 bytes) 000055 1784936913.761336 trunc from 0x037000 to 0x03582a (0x17d5 bytes) 000056 1784936913.790216 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3582b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3582b): Mirror 2: 0x1153d849 (stale) CRC-32 checksum value for chunk [0, 0x3582b): Mirror 1: 0x43b9d3b Mirror 3: 0x43b9d3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3582b: skipped 000057 1784936915.091265 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3582b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3582b): Mirror 2: 0x1153d849 (stale) CRC-32 checksum value for chunk [0, 0x3582b): Mirror 1: 0x43b9d3b Mirror 3: 0x43b9d3b Mirror 4: 0x43b9d3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3582b: skipped 000058 1784936916.281108 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3582b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3582b): Mirror 2: 0x1153d849 (stale) CRC-32 checksum value for chunk [0, 0x3582b): Mirror 1: 0x43b9d3b Mirror 3: 0x43b9d3b Mirror 4: 0x43b9d3b Mirror 5: 0x43b9d3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3582b: skipped 000059 1784936917.948154 fallocate 0x02b3af thru 0x0300ce (0x4d20 bytes) 000060 1784936918.023503 trunc from 0x03582b to 0x029969 (0xbec1 bytes) 000063 1784936918.084792 read 0x01ae99 thru 0x02402c (0x9194 bytes) 000064 1784936918.102956 trunc from 0x01f361 to 0x029969 (0xa609 bytes) 000065 1784936918.124400 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2996a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2996a): Mirror 2: 0x4da1f662 (stale) Mirror 3: 0xe7b42aba (stale) Mirror 4: 0xe7b42aba (stale) Mirror 5: 0xe7b42aba (stale) CRC-32 checksum value for chunk [0, 0x2996a): Mirror 1: 0x8cff42af Mirror 6: 0x8cff42af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2996a: skipped 000066 1784936919.355671 read 0x004b0e thru 0x00db2b (0x901e bytes) 000067 1784936919.402534 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2996a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2996a): Mirror 2: 0x4da1f662 (stale) Mirror 3: 0xe7b42aba (stale) Mirror 4: 0xe7b42aba (stale) Mirror 5: 0xe7b42aba (stale) CRC-32 checksum value for chunk [0, 0x2996a): Mirror 1: 0x8cff42af Mirror 6: 0x8cff42af Mirror 7: 0x8cff42af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2996a: skipped 000068 1784936920.635922 read 0x02278d thru 0x023015 (0x889 bytes) 000069 1784936920.646806 read 0x019213 thru 0x01eeeb (0x5cd9 bytes) 000070 1784936920.701959 write 0x00e241 thru 0x0195ac (0xb36c bytes) 000071 1784936920.874388 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2996a): Mirror 3: 0xe7b42aba (stale) Mirror 4: 0xe7b42aba (stale) Mirror 5: 0xe7b42aba (stale) Mirror 6: 0x8cff42af (stale) Mirror 7: 0x8cff42af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2996a): Mirror 3: 0xe7b42aba (stale) Mirror 4: 0xe7b42aba (stale) Mirror 5: 0xe7b42aba (stale) Mirror 6: 0x8cff42af (stale) Mirror 7: 0x8cff42af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2996a: skipped 000072 1784936921.707419 mapwrite 0x008fed thru 0x009626 (0x63a bytes) 000073 1784936921.752530 read 0x01f868 thru 0x025daf (0x6548 bytes) 000074 1784936921.766307 read 0x004fc2 thru 0x0143d6 (0xf415 bytes) 000075 1784936921.779353 fallocate 0x022f00 thru 0x02e7b9 (0xb8ba bytes) 000076 1784936921.789099 punch from 0x001852 to 0x010d5c (0xf50b bytes) 000077 1784936921.807317 write 0x0287e7 thru 0x02c2bf (0x3ad9 bytes) 000078 1784936921.823489 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c2c0): Mirror 4: 0xe15361c0 (stale) Mirror 5: 0xe15361c0 (stale) Mirror 6: 0xa43c5c98 (stale) Mirror 7: 0xa43c5c98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c2c0): Mirror 4: 0xe15361c0 (stale) Mirror 5: 0xe15361c0 (stale) Mirror 6: 0xa43c5c98 (stale) Mirror 7: 0xa43c5c98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c2c0: skipped 000079 1784936922.335751 mapwrite 0x036a3d thru 0x03ffff (0x95c3 bytes) 000081 1784936922.444244 write 0x019957 thru 0x0259c5 (0xc06f bytes) 000082 1784936922.458010 fallocate 0x01086e thru 0x011409 (0xb9c bytes) 000083 1784936922.467385 trunc from 0x01049e to 0x01d131 (0xcc94 bytes) 000084 1784936922.497550 trunc from 0x040000 to 0x00889e (0x37761 bytes) 000085 1784936922.520463 mapread 0x006dd8 thru 0x00889e (0x1ac7 bytes) 000086 1784936922.532045 trunc from 0x00889f to 0x0032b1 (0x55ed bytes) 000088 1784936922.552395 fallocate 0x00e9b7 thru 0x017498 (0x8ae2 bytes) 000089 1784936922.561473 write 0x0086d8 thru 0x00fa03 (0x732c bytes) 000090 1784936922.571390 mapwrite 0x0019dc thru 0x0059c0 (0x3fe5 bytes) 000091 1784936922.595878 trunc from 0x017499 to 0x0345e1 (0x1d149 bytes) 000092 1784936922.608796 mapread 0x0040d7 thru 0x010575 (0xc49f bytes) 000093 1784936922.610299 mapread 0x008f13 thru 0x00fe6c (0x6f5a bytes) 000095 1784936922.614012 read 0x025833 thru 0x02f9d7 (0xa1a5 bytes) 000096 1784936922.626833 MIRROR_RESYNC 5 mirrors 000097 1784936924.215354 MIRROR_RESYNC 5 mirrors 000098 1784936924.789978 trunc from 0x02a989 to 0x02aa5b (0xd3 bytes) 000099 1784936924.855063 read 0x0285bc thru 0x02fffb (0x7a40 bytes) 000100 1784936924.883035 punch from 0x02e8e8 to 0x0345e1 (0x5cfa bytes) 000101 1784936924.889510 mapread 0x012315 thru 0x01bb85 (0x9871 bytes) 000102 1784936924.909290 mapread 0x015aed thru 0x01be5c (0x6370 bytes) 000103 1784936924.910504 mapread 0x00ea08 thru 0x018986 (0x9f7f bytes) 000104 1784936924.912738 mapread 0x004d2a thru 0x00ea9f (0x9d76 bytes) 000105 1784936924.923877 trunc from 0x0345e2 to 0x0266ac (0xdf35 bytes) 000106 1784936924.951889 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x266ad): Mirror 4: 0xa178d974 (stale) Mirror 5: 0xa178d974 (stale) Mirror 7: 0xa178d974 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266ad): Mirror 4: 0xa178d974 (stale) Mirror 5: 0xa178d974 (stale) Mirror 7: 0xa178d974 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266ad: skipped 000107 1784936925.534948 trunc from 0x01ec6c to 0x02080a (0x1b9f bytes) 000108 1784936925.543041 read 0x014368 thru 0x016299 (0x1f32 bytes) 000109 1784936925.552556 punch from 0x00d2aa to 0x0147fb (0x7552 bytes) 000110 1784936925.570671 trunc from 0x01a8ed to 0x0266ac (0xbdc0 bytes) 000111 1784936925.576841 trunc from 0x01732b to 0x01c403 (0x50d9 bytes) 000112 1784936925.588472 read 0x0045ee thru 0x00a1b6 (0x5bc9 bytes) 000113 1784936925.616901 write 0x014325 thru 0x0207e5 (0xc4c1 bytes) 000114 1784936925.626445 punch from 0x01be9a to 0x0266ac (0xa813 bytes) 000115 1784936925.686000 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x266ad): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266ad): Mirror 4: 0xa178d974 (stale) Mirror 5: 0xa178d974 (stale) Mirror 7: 0xa178d974 (stale) CRC-32 checksum value for chunk [0, 0x266ad): Mirror 1: 0xcec4fdad Mirror 8: 0xcec4fdad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266ad: skipped 000116 1784936927.543896 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x266ad): Mirror 5: 0xa178d974 (stale) Mirror 7: 0xa178d974 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266ad): Mirror 5: 0xa178d974 (stale) Mirror 7: 0xa178d974 (stale) CRC-32 checksum value for chunk [0, 0x266ad): Mirror 1: 0xcec4fdad Mirror 8: 0xcec4fdad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266ad: skipped 000117 1784936928.691396 fallocate 0x0217bd thru 0x02b410 (0x9c54 bytes) 000119 1784936928.865805 MIRROR_RESYNC 4 mirrors 000121 1784936930.355477 MIRROR_EXTEND to 5 mirrors 000122 1784936931.362772 write 0x02e7a5 thru 0x039f55 (0xb7b1 bytes) 000123 1784936931.439068 trunc from 0x039f56 to 0x0196bd (0x20898 bytes) 000124 1784936931.508502 write 0x003c27 thru 0x00fa18 (0xbdf2 bytes) 000125 1784936931.576377 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x196be): Mirror 7: 0xf1225f93 (stale) Mirror 8: 0xf1225f93 (stale) Mirror 9: 0xf1225f93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x196be): Mirror 7: 0xf1225f93 (stale) Mirror 8: 0xf1225f93 (stale) Mirror 9: 0xf1225f93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x196be: skipped 000126 1784936932.059921 trunc from 0x0196be to 0x01f48f (0x5dd2 bytes) 000127 1784936932.087560 fallocate 0x018a90 thru 0x021034 (0x85a5 bytes) 000128 1784936932.091233 fallocate 0x02a639 thru 0x02a673 (0x3b bytes) 000129 1784936932.094968 mapread 0x016613 thru 0x01a3a3 (0x3d91 bytes) 000130 1784936932.122191 trunc from 0x02a674 to 0x022ccc (0x79a7 bytes) 000131 1784936932.161925 write 0x0175a1 thru 0x01a56e (0x2fce bytes) 000132 1784936932.163161 punch from 0x009370 to 0x015e16 (0xcaa7 bytes) 000133 1784936932.167968 read 0x01d8b5 thru 0x0209aa (0x30f6 bytes) 000134 1784936932.168828 write 0x00e754 thru 0x01bc1e (0xd4cb bytes) 000135 1784936932.173927 fallocate 0x01117d thru 0x020d42 (0xfbc6 bytes) 000136 1784936932.178920 punch from 0x00fd64 to 0x015c55 (0x5ef2 bytes) 000137 1784936932.192481 fallocate 0x00305b thru 0x0060e6 (0x308c bytes) 000138 1784936932.196917 fallocate 0x033650 thru 0x03ffff (0xc9b0 bytes) 000139 1784936932.202667 trunc from 0x040000 to 0x02bd98 (0x14267 bytes) 000140 1784936932.221850 mapread 0x01774d thru 0x01ab36 (0x33ea bytes) 000141 1784936932.223126 read 0x01c924 thru 0x020ee1 (0x45be bytes) 000142 1784936932.225012 write 0x005b96 thru 0x012306 (0xc771 bytes) 000143 1784936932.246315 mapread 0x00f187 thru 0x0183a8 (0x9222 bytes) 000145 1784936932.248738 punch from 0x02ad90 to 0x02bd98 (0x1009 bytes) 000146 1784936932.253978 trunc from 0x02a6f4 to 0x02b053 (0x960 bytes) 000147 1784936932.259022 fallocate 0x009ac6 thru 0x01295c (0x8e97 bytes) 000148 1784936932.264489 mapread 0x005bf8 thru 0x007abc (0x1ec5 bytes) 000149 1784936932.266451 mapwrite 0x01c3f8 thru 0x026ca5 (0xa8ae bytes) 000151 1784936932.322693 write 0x000bfd thru 0x010799 (0xfb9d bytes) 000152 1784936932.332235 fallocate 0x006e50 thru 0x011ca1 (0xae52 bytes) 000153 1784936932.342625 trunc from 0x01d29e to 0x01ecde (0x1a41 bytes) 000154 1784936932.355458 mapread 0x000e43 thru 0x00de4b (0xd009 bytes) 000155 1784936932.385483 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2bd99): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd99): Mirror 7: 0x1d22fea0 (stale) Mirror 8: 0x1d22fea0 (stale) Mirror 9: 0x1d22fea0 (stale) CRC-32 checksum value for chunk [0, 0x2bd99): Mirror 1: 0xfda85c03 Mirror 10: 0xfda85c03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd99: skipped 000156 1784936933.811765 punch from 0x029101 to 0x02a0c5 (0xfc5 bytes) 000157 1784936933.874700 mapread 0x017e98 thru 0x020f51 (0x90ba bytes) 000159 1784936933.927344 trunc from 0x02bd99 to 0x00ede0 (0x1cfb8 bytes) 000160 1784936934.004711 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xede1): Mirror 8: 0x7835955a (stale) Mirror 9: 0x7835955a (stale) Mirror 10: 0xd6564010 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xede1): Mirror 8: 0x7835955a (stale) Mirror 9: 0x7835955a (stale) Mirror 10: 0xd6564010 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xede1: skipped 000161 1784936934.430758 fallocate 0x01e9c9 thru 0x02095d (0x1f95 bytes) 000162 1784936934.436875 mapwrite 0x0367fa thru 0x03e7af (0x7fb6 bytes) 000164 1784936934.517913 trunc from 0x03e7b0 to 0x004af8 (0x39cb7 bytes) 000165 1784936934.558849 punch from 0x001ff7 to 0x004af8 (0x2b02 bytes) 000166 1784936934.566371 read 0x000a64 thru 0x004af8 (0x4095 bytes) 000167 1784936934.582852 fallocate 0x00fcd6 thru 0x013f25 (0x4250 bytes) 000168 1784936934.592337 read 0x002b87 thru 0x004af8 (0x1f72 bytes) 000169 1784936934.601045 fallocate 0x014859 thru 0x022d05 (0xe4ad bytes) 000170 1784936934.612266 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x4af9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4af9): Mirror 8: 0xdf7c11cf (stale) Mirror 9: 0xdf7c11cf (stale) Mirror 10: 0x62acd0c1 (stale) CRC-32 checksum value for chunk [0, 0x4af9): Mirror 1: 0x88a51065 Mirror 11: 0x88a51065 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4af9: skipped 000171 1784936936.158268 MIRROR_RESYNC 5 mirrors 000174 1784936937.131064 mapwrite 0x03f546 thru 0x03ffff (0xaba bytes) 000175 1784936937.249581 mapread 0x02c6a9 thru 0x02d970 (0x12c8 bytes) 000176 1784936937.282560 mapread 0x002de1 thru 0x004aed (0x1d0d bytes) 000177 1784936937.301928 fallocate 0x01b7fa thru 0x029139 (0xd940 bytes) 000178 1784936937.329667 punch from 0x026fe3 to 0x032746 (0xb764 bytes) 000180 1784936937.340011 fallocate 0x02648b thru 0x02e519 (0x808f bytes) 000181 1784936937.347317 read 0x01cfe8 thru 0x02553b (0x8554 bytes) 000182 1784936937.360786 mapread 0x02aa2b thru 0x036d41 (0xc317 bytes) 000183 1784936937.393702 punch from 0x03c2cd to 0x03ffff (0x3d33 bytes) 000185 1784936937.401559 trunc from 0x040000 to 0x02fec0 (0x1013f bytes) 000187 1784936937.432050 mapread 0x028483 thru 0x02fec0 (0x7a3e bytes) 000188 1784936937.439412 read 0x005805 thru 0x0118e8 (0xc0e4 bytes) 000190 1784936937.455839 mapwrite 0x003b60 thru 0x00c15a (0x85fb bytes) 000191 1784936937.486837 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fec1): Mirror 9: 0x4c00b151 (stale) Mirror 10: 0x4c00b151 (stale) Mirror 11: 0x4c00b151 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fec1): Mirror 9: 0x4c00b151 (stale) Mirror 10: 0x4c00b151 (stale) Mirror 11: 0x4c00b151 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fec1: skipped 000193 1784936937.916194 fallocate 0x004a88 thru 0x004b91 (0x10a bytes) 000195 1784936937.921520 trunc from 0x02fec1 to 0x02bd0c (0x41b4 bytes) 000196 1784936937.940203 trunc from 0x006a62 to 0x00898a (0x1f29 bytes) 000197 1784936937.954760 mapread 0x01c98d thru 0x02b755 (0xedc9 bytes) 000198 1784936937.991303 trunc from 0x02bd0d to 0x030650 (0x4944 bytes) 000199 1784936938.027371 trunc from 0x01d26c to 0x020581 (0x3316 bytes) 000200 1784936938.033255 mapwrite 0x03df51 thru 0x03ffff (0x20af bytes) 000201 1784936938.068288 trunc from 0x040000 to 0x03b096 (0x4f69 bytes) 000202 1784936938.088695 mapread 0x00e0ce thru 0x0113ca (0x32fd bytes) 000203 1784936938.108002 fallocate 0x0067fb thru 0x007c23 (0x1429 bytes) 000204 1784936938.113161 write 0x020b21 thru 0x02776c (0x6c4c bytes) 000205 1784936938.124390 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b097): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b097): Mirror 9: 0xf6d6f5ec (stale) Mirror 10: 0xf6d6f5ec (stale) Mirror 11: 0xf6d6f5ec (stale) CRC-32 checksum value for chunk [0, 0x3b097): Mirror 1: 0x3fe974b9 Mirror 12: 0x3fe974b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b097: skipped 000206 1784936939.700749 mapwrite 0x0360ec thru 0x03727c (0x1191 bytes) 000207 1784936939.851743 read 0x03a509 thru 0x03b096 (0xb8e bytes) 000208 1784936939.852861 mapwrite 0x00ad10 thru 0x0155fe (0xa8ef bytes) 000209 1784936939.892429 punch from 0x022600 to 0x024a77 (0x2478 bytes) 000210 1784936939.901095 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b097): Mirror 9: 0xf6d6f5ec (stale) Mirror 10: 0xf6d6f5ec (stale) Mirror 12: 0x3fe974b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b097): Mirror 9: 0xf6d6f5ec (stale) Mirror 10: 0xf6d6f5ec (stale) Mirror 12: 0x3fe974b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b097: skipped 000211 1784936940.211983 mapread 0x00ae4d thru 0x013af1 (0x8ca5 bytes) 000212 1784936940.245715 MIRROR_RESYNC 4 mirrors 000213 1784936942.518440 fallocate 0x02cb1e thru 0x02ddc3 (0x12a6 bytes) 000214 1784936942.555362 trunc from 0x03b097 to 0x0309b6 (0xa6e0 bytes) 000215 1784936942.588169 punch from 0x00c962 to 0x0143fe (0x7a9d bytes) 000216 1784936942.596516 fallocate 0x02ab2b thru 0x035326 (0xa7fc bytes) 000217 1784936942.611405 mapwrite 0x035aed thru 0x03c533 (0x6a47 bytes) 000218 1784936942.685198 mapwrite 0x0323c2 thru 0x036d43 (0x4982 bytes) 000219 1784936942.725538 punch from 0x0314e6 to 0x03748f (0x5faa bytes) 000220 1784936942.734026 mapwrite 0x03f86d thru 0x03ffff (0x793 bytes) 000221 1784936942.789740 fallocate 0x0049ba thru 0x00d38e (0x89d5 bytes) 000222 1784936942.797916 punch from 0x03a7f5 to 0x03f281 (0x4a8d bytes) 000223 1784936942.817146 read 0x03e61f thru 0x03ffff (0x19e1 bytes) 000224 1784936942.828171 fallocate 0x039741 thru 0x03ffff (0x68bf bytes) 000225 1784936942.842844 mapread 0x013179 thru 0x017dff (0x4c87 bytes) 000226 1784936942.873516 trunc from 0x0125ee to 0x01950e (0x6f21 bytes) 000227 1784936942.893540 punch from 0x006783 to 0x010205 (0x9a83 bytes) 000228 1784936942.917845 trunc from 0x040000 to 0x004632 (0x3b9cd bytes) 000229 1784936942.950088 mapread 0x000b98 thru 0x004632 (0x3a9b bytes) 000230 1784936942.963761 trunc from 0x004633 to 0x036560 (0x31f2e bytes) 000231 1784936943.001180 mapwrite 0x038756 thru 0x03be4b (0x36f6 bytes) 000233 1784936943.117490 mapread 0x00ce00 thru 0x019f96 (0xd197 bytes) 000234 1784936943.162132 punch from 0x0071e1 to 0x00bcaf (0x4acf bytes) 000235 1784936943.169860 punch from 0x033d36 to 0x03547b (0x1746 bytes) 000236 1784936943.180910 MIRROR_RESYNC 4 mirrors 000237 1784936944.107849 mapwrite 0x00b115 thru 0x0108db (0x57c7 bytes) 000238 1784936944.229133 mapwrite 0x00b2a5 thru 0x00d1ea (0x1f46 bytes) 000239 1784936944.288828 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3be4c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be4c): Mirror 9: 0x8b9f373f (stale) Mirror 10: 0x8b9f373f (stale) Mirror 12: 0x8b9f373f (stale) CRC-32 checksum value for chunk [0, 0x3be4c): Mirror 1: 0x40439e6a Mirror 13: 0x40439e6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be4c: skipped 000240 1784936945.432032 fallocate 0x029018 thru 0x030897 (0x7880 bytes) 000241 1784936945.477786 punch from 0x000b91 to 0x0027c1 (0x1c31 bytes) 000242 1784936945.484403 punch from 0x01b823 to 0x01dade (0x22bc bytes) 000243 1784936945.493813 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3be4c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be4c): Mirror 9: 0x8b9f373f (stale) Mirror 10: 0x8b9f373f (stale) Mirror 12: 0x8b9f373f (stale) Mirror 13: 0x40439e6a (stale) CRC-32 checksum value for chunk [0, 0x3be4c): Mirror 1: 0x4c025154 Mirror 14: 0x4c025154 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be4c: skipped 000244 1784936946.569923 write 0x03a147 thru 0x03ffff (0x5eb9 bytes) 000245 1784936946.624419 punch from 0x03bb09 to 0x03d1b4 (0x16ac bytes) 000246 1784936946.639673 mapread 0x00c57f thru 0x00fb2f (0x35b1 bytes) 000247 1784936946.660195 trunc from 0x012139 to 0x01550a (0x33d2 bytes) 000248 1784936946.667342 punch from 0x03908c to 0x03f690 (0x6605 bytes) 000249 1784936946.672804 write 0x03f0cb thru 0x03ffff (0xf35 bytes) 000250 1784936946.683130 MIRROR_RESYNC 6 mirrors 000251 1784936948.998285 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) 000252 1784936949.577219 MIRROR_RESYNC 5 mirrors 000253 1784936950.234449 mapwrite 0x01d1e5 thru 0x02a122 (0xcf3e bytes) 000254 1784936950.391118 mapwrite 0x00759f thru 0x00dc53 (0x66b5 bytes) 000255 1784936950.444041 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x86ef7d7f (stale) Mirror 12: 0x86ef7d7f (stale) Mirror 14: 0x86ef7d7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x86ef7d7f (stale) Mirror 12: 0x86ef7d7f (stale) Mirror 14: 0x86ef7d7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000256 1784936950.948175 mapwrite 0x038135 thru 0x038465 (0x331 bytes) 000257 1784936950.980148 punch from 0x02ce7b to 0x03224a (0x53d0 bytes) 000258 1784936950.991762 MIRROR_RESYNC 4 mirrors 000259 1784936953.011000 fallocate 0x01390c thru 0x01da23 (0xa118 bytes) 000260 1784936953.043173 fallocate 0x03221f thru 0x0339a4 (0x1786 bytes) 000261 1784936953.051939 read 0x00b89c thru 0x018fa8 (0xd70d bytes) 000263 1784936953.070522 punch from 0x0089ce to 0x014e0f (0xc442 bytes) 000264 1784936953.079347 fallocate 0x003f74 thru 0x00a1e8 (0x6275 bytes) 000265 1784936953.096791 punch from 0x021b51 to 0x02bc84 (0xa134 bytes) 000266 1784936953.103524 mapread 0x01f173 thru 0x02d15a (0xdfe8 bytes) 000267 1784936953.144768 mapread 0x003a1f thru 0x00c411 (0x89f3 bytes) 000269 1784936953.155374 fallocate 0x016309 thru 0x01d201 (0x6ef9 bytes) 000270 1784936953.161755 mapread 0x01669c thru 0x023240 (0xcba5 bytes) 000271 1784936953.175562 mapread 0x015b71 thru 0x01d621 (0x7ab1 bytes) 000272 1784936953.178733 trunc from 0x00829d to 0x013c55 (0xb9b9 bytes) 000273 1784936953.186203 write 0x02681e thru 0x02843b (0x1c1e bytes) 000274 1784936953.188755 trunc from 0x024db8 to 0x02eac4 (0x9d0d bytes) 000276 1784936953.200793 read 0x010384 thru 0x010457 (0xd4 bytes) 000277 1784936953.216127 write 0x007e7a thru 0x00fa8f (0x7c16 bytes) 000278 1784936953.220585 mapwrite 0x01fd55 thru 0x023ef7 (0x41a3 bytes) 000279 1784936953.245535 punch from 0x006d5f to 0x00811d (0x13bf bytes) 000280 1784936953.252768 trunc from 0x00e864 to 0x00f7dd (0xf7a bytes) 000281 1784936953.258634 trunc from 0x02e7bf to 0x03c33b (0xdb7d bytes) 000282 1784936953.266083 read 0x012f2c thru 0x02004c (0xd121 bytes) 000283 1784936953.268723 mapwrite 0x0003b4 thru 0x002bd8 (0x2825 bytes) 000284 1784936953.286686 trunc from 0x0128f1 to 0x012d8c (0x49c bytes) 000285 1784936953.292710 punch from 0x0093f2 to 0x00f1e6 (0x5df5 bytes) 000286 1784936953.302775 mapwrite 0x00d8d2 thru 0x016702 (0x8e31 bytes) 000287 1784936953.332815 fallocate 0x0016f4 thru 0x0089da (0x72e7 bytes) 000288 1784936953.340933 punch from 0x02fbe5 to 0x034848 (0x4c64 bytes) 000289 1784936953.347137 punch from 0x01fcd6 to 0x02cd66 (0xd091 bytes) 000290 1784936953.355197 trunc from 0x040000 to 0x0170d5 (0x28f2a bytes) 000291 1784936953.374331 fallocate 0x03db69 thru 0x03ffff (0x2497 bytes) 000292 1784936953.379809 read 0x006fd1 thru 0x00cec7 (0x5ef7 bytes) 000294 1784936953.381891 punch from 0x006010 to 0x00f072 (0x9063 bytes) 000295 1784936953.387437 trunc from 0x0170d6 to 0x03ed50 (0x27c7b bytes) 000296 1784936953.405375 fallocate 0x01417d thru 0x014671 (0x4f5 bytes) 000297 1784936953.410469 trunc from 0x03759c to 0x03ed50 (0x77b5 bytes) 000298 1784936953.415716 punch from 0x01c43f to 0x0240cb (0x7c8d bytes) 000299 1784936953.419770 punch from 0x010e34 to 0x01cf02 (0xc0cf bytes) 000300 1784936953.425933 fallocate 0x01e4bd thru 0x025419 (0x6f5d bytes) 000301 1784936953.430396 mapread 0x00f00f thru 0x0133ab (0x439d bytes) 000302 1784936953.442177 punch from 0x03ad2f to 0x03c1ad (0x147f bytes) 000303 1784936953.446633 trunc from 0x03ed51 to 0x03cd62 (0x1fee bytes) 000304 1784936953.466471 punch from 0x03c941 to 0x03cd62 (0x422 bytes) 000305 1784936953.476527 punch from 0x02eb31 to 0x035d22 (0x71f2 bytes) 000306 1784936953.488562 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 12: 0x14a43339 (stale) Mirror 14: 0x14a43339 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 12: 0x14a43339 (stale) Mirror 14: 0x14a43339 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd63: skipped 000307 1784936953.804844 write 0x023c41 thru 0x024ad8 (0xe98 bytes) 000308 1784936953.821506 fallocate 0x005ae6 thru 0x01462d (0xeb48 bytes) 000309 1784936953.831208 read 0x02abd3 thru 0x031df0 (0x721e bytes) 000310 1784936953.844430 mapwrite 0x0315e5 thru 0x03c961 (0xb37d bytes) 000311 1784936953.882380 punch from 0x02c048 to 0x035e97 (0x9e50 bytes) 000312 1784936953.887493 trunc from 0x01cbe6 to 0x021cd2 (0x50ed bytes) 000313 1784936953.892819 write 0x02d436 thru 0x0339c7 (0x6592 bytes) 000314 1784936953.904966 fallocate 0x0303f8 thru 0x0335fc (0x3205 bytes) 000315 1784936953.910285 trunc from 0x0382ed to 0x039e18 (0x1b2c bytes) 000316 1784936953.915846 trunc from 0x015626 to 0x0209f3 (0xb3ce bytes) 000317 1784936953.921560 mapwrite 0x0339ec thru 0x03ffff (0xc614 bytes) 000318 1784936953.973892 read 0x0361f9 thru 0x03ffff (0x9e07 bytes) 000319 1784936953.974924 punch from 0x03759b to 0x03ffff (0x8a65 bytes) 000320 1784936953.982549 trunc from 0x03a821 to 0x03ffff (0x57df bytes) 000321 1784936953.986738 write 0x00d124 thru 0x017dd1 (0xacae bytes) 000322 1784936954.000949 punch from 0x004648 to 0x0051dd (0xb96 bytes) 000323 1784936954.013874 punch from 0x01d3a3 to 0x02d212 (0xfe70 bytes) 000324 1784936954.039148 write 0x011d12 thru 0x01ef4a (0xd239 bytes) 000325 1784936954.058339 mapread 0x02ce38 thru 0x031a76 (0x4c3f bytes) 000326 1784936954.081080 write 0x00f316 thru 0x01c50f (0xd1fa bytes) 000327 1784936954.085164 read 0x02b7c6 thru 0x034230 (0x8a6b bytes) 000328 1784936954.099803 trunc from 0x040000 to 0x01a4aa (0x25b55 bytes) 000330 1784936954.125074 MIRROR_RESYNC 3 mirrors 000331 1784936955.100165 read 0x0025b9 thru 0x010fc9 (0xea11 bytes) 000332 1784936955.121984 write 0x0266df thru 0x032035 (0xb957 bytes) 000333 1784936955.162111 read 0x02c1e8 thru 0x032035 (0x5e4e bytes) 000334 1784936955.163138 punch from 0x02273f to 0x028ddc (0x669e bytes) 000335 1784936955.180575 trunc from 0x032036 to 0x039bbd (0x7b88 bytes) 000336 1784936955.220574 trunc from 0x02ab65 to 0x032de2 (0x827e bytes) 000337 1784936955.226547 punch from 0x01ef42 to 0x02b8c1 (0xc980 bytes) 000338 1784936955.237088 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39bbe): Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39bbe): Mirror 12: 0x8146ecda (stale) Mirror 14: 0x8146ecda (stale) CRC-32 checksum value for chunk [0, 0x39bbe): Mirror 1: 0x8146ecda Mirror 15: 0x8146ecda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39bbe: skipped 000339 1784936956.720854 mapwrite 0x028d20 thru 0x034c51 (0xbf32 bytes) 000340 1784936956.936625 trunc from 0x039bbe to 0x031827 (0x8396 bytes) 000341 1784936956.994554 punch from 0x007fc0 to 0x0097c2 (0x1803 bytes) 000342 1784936957.003939 trunc from 0x031828 to 0x02f90a (0x1f1d bytes) 000343 1784936957.044511 mapread 0x011e7b thru 0x016e84 (0x500a bytes) 000344 1784936957.061581 fallocate 0x02fb38 thru 0x03a6db (0xaba4 bytes) 000345 1784936957.070493 fallocate 0x033bd0 thru 0x03ffff (0xc430 bytes) 000346 1784936957.082850 punch from 0x020689 to 0x02bd4b (0xb6c3 bytes) 000347 1784936957.091502 fallocate 0x026bf1 thru 0x036409 (0xf819 bytes) 000348 1784936957.103113 mapread 0x031099 thru 0x03dcd0 (0xcc38 bytes) 000349 1784936957.129591 mapwrite 0x020bbb thru 0x0281dd (0x7623 bytes) 000350 1784936957.169873 trunc from 0x040000 to 0x010292 (0x2fd6d bytes) 000351 1784936957.194567 fallocate 0x035a5b thru 0x03ffff (0xa5a5 bytes) 000352 1784936957.203569 trunc from 0x010293 to 0x03cadd (0x2c84b bytes) 000353 1784936957.233755 trunc from 0x03cade to 0x0060f6 (0x369e7 bytes) 000354 1784936957.271948 mapwrite 0x016280 thru 0x01ec01 (0x8982 bytes) 000355 1784936957.390714 mapread 0x0016c1 thru 0x006c1c (0x555c bytes) 000356 1784936957.404738 write 0x02cf3e thru 0x03b681 (0xe744 bytes) 000357 1784936957.414785 MIRROR_RESYNC 4 mirrors 000358 1784936958.544328 fallocate 0x030436 thru 0x031f0a (0x1ad5 bytes) 000359 1784936958.633286 trunc from 0x03b682 to 0x0225e5 (0x1909c bytes) 000360 1784936958.655980 punch from 0x01dc12 to 0x0225e5 (0x49d4 bytes) 000361 1784936958.662917 trunc from 0x009344 to 0x00bb54 (0x2811 bytes) 000362 1784936958.676921 trunc from 0x01c115 to 0x0225e5 (0x64d1 bytes) 000363 1784936958.685856 mapwrite 0x02f165 thru 0x033ae3 (0x497f bytes) 000364 1784936958.837561 trunc from 0x033ae4 to 0x0287d1 (0xb312 bytes) 000365 1784936958.885160 trunc from 0x0287d2 to 0x005a13 (0x22dbe bytes) 000367 1784936958.919067 write 0x0348b0 thru 0x03ffff (0xb750 bytes) 000368 1784936958.925101 trunc from 0x040000 to 0x01cceb (0x23314 bytes) 000369 1784936958.948718 trunc from 0x00f518 to 0x017023 (0x7b0c bytes) 000370 1784936958.960735 punch from 0x00631c to 0x016102 (0xfde7 bytes) 000371 1784936958.970933 write 0x034e15 thru 0x03a9f5 (0x5be1 bytes) 000372 1784936958.975013 trunc from 0x03a9f6 to 0x008f9c (0x31a59 bytes) 000373 1784936959.007805 write 0x021940 thru 0x026a3d (0x50fe bytes) 000374 1784936959.009442 trunc from 0x00edbf to 0x016051 (0x7293 bytes) 000375 1784936959.014427 trunc from 0x026a3e to 0x027916 (0xed9 bytes) 000376 1784936959.042400 mapread 0x01f787 thru 0x027916 (0x8190 bytes) 000377 1784936959.057782 fallocate 0x0340e9 thru 0x0376aa (0x35c2 bytes) 000379 1784936959.067007 trunc from 0x010671 to 0x015d2b (0x56bb bytes) 000380 1784936959.075734 write 0x028b74 thru 0x02e4bf (0x594c bytes) 000381 1784936959.088686 write 0x0388c4 thru 0x03ffff (0x773c bytes) 000382 1784936959.093305 punch from 0x034aad to 0x03ffff (0xb553 bytes) 000383 1784936959.112346 punch from 0x00d312 to 0x019080 (0xbd6f bytes) 000384 1784936959.117386 mapwrite 0x01bdb5 thru 0x0267ef (0xaa3b bytes) 000385 1784936959.145412 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x407872f (stale) Mirror 14: 0x407872f (stale) Mirror 15: 0x407872f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8e5297bf Mirror 16: 0x8e5297bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000386 1784936960.100031 read 0x029dcc thru 0x02e521 (0x4756 bytes) 000387 1784936960.118169 trunc from 0x00a35f to 0x00ec1b (0x48bd bytes) 000388 1784936960.158998 punch from 0x008991 to 0x00ee13 (0x6483 bytes) 000389 1784936960.164704 trunc from 0x040000 to 0x00b0fb (0x34f04 bytes) 000390 1784936960.218051 trunc from 0x0019c3 to 0x00824c (0x688a bytes) 000391 1784936960.225125 read 0x0099d9 thru 0x00b0fb (0x1723 bytes) 000392 1784936960.237330 trunc from 0x00b0fc to 0x02ebef (0x23af4 bytes) 000393 1784936960.259132 write 0x02bc0f thru 0x02eaf5 (0x2ee7 bytes) 000394 1784936960.271029 mapread 0x0117a6 thru 0x01abc5 (0x9420 bytes) 000395 1784936960.273507 mapread 0x0186a0 thru 0x02696b (0xe2cc bytes) 000396 1784936960.275556 fallocate 0x039dc2 thru 0x03ffff (0x623e bytes) 000397 1784936960.282862 punch from 0x01a523 to 0x022661 (0x813f bytes) 000399 1784936960.289866 punch from 0x015a22 to 0x0201fa (0xa7d9 bytes) 000400 1784936960.295130 punch from 0x0011e4 to 0x00da90 (0xc8ad bytes) 000401 1784936960.301291 mapwrite 0x023d3e thru 0x028b30 (0x4df3 bytes) 000402 1784936960.319360 read 0x00653f thru 0x008b4b (0x260d bytes) 000403 1784936960.332992 read 0x03a0bb thru 0x03ffff (0x5f45 bytes) 000404 1784936960.334046 punch from 0x00f6d5 to 0x016228 (0x6b54 bytes) 000405 1784936960.345289 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x407872f (stale) Mirror 14: 0x407872f (stale) Mirror 15: 0x407872f (stale) Mirror 16: 0x8e5297bf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad6f6d19 Mirror 17: 0xad6f6d19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000406 1784936961.481023 write 0x001333 thru 0x008486 (0x7154 bytes) 000407 1784936961.574815 fallocate 0x005af9 thru 0x00f56f (0x9a77 bytes) 000408 1784936961.587261 mapwrite 0x0071df thru 0x011e9b (0xacbd bytes) 000409 1784936961.679507 mapread 0x02e5cb thru 0x030638 (0x206e bytes) 000411 1784936961.710361 mapwrite 0x030c90 thru 0x0363ab (0x571c bytes) 000412 1784936961.751072 trunc from 0x03970d to 0x03ffff (0x68f3 bytes) 000413 1784936961.758301 trunc from 0x040000 to 0x0248d3 (0x1b72c bytes) 000415 1784936961.820525 MIRROR_RESYNC 6 mirrors 000416 1784936963.416499 read 0x0244f8 thru 0x0248d3 (0x3dc bytes) 000417 1784936963.422244 mapread 0x01681b thru 0x017cbf (0x14a5 bytes) 000418 1784936963.435770 write 0x00524f thru 0x00d162 (0x7f14 bytes) 000419 1784936963.491652 punch from 0x01da6a to 0x0248d3 (0x6e6a bytes) 000420 1784936963.504702 fallocate 0x015ea4 thru 0x01e356 (0x84b3 bytes) 000421 1784936963.516453 mapwrite 0x01371a thru 0x01bf44 (0x882b bytes) 000422 1784936963.564654 punch from 0x01c78a to 0x0248d3 (0x814a bytes) 000423 1784936963.589737 MIRROR_RESYNC 6 mirrors 000425 1784936965.347389 fallocate 0x01dcf0 thru 0x024ea9 (0x71ba bytes) 000426 1784936965.382322 trunc from 0x009432 to 0x011173 (0x7d42 bytes) 000427 1784936965.394583 fallocate 0x028e29 thru 0x02962b (0x803 bytes) 000428 1784936965.400361 read 0x000e56 thru 0x008447 (0x75f2 bytes) 000429 1784936965.409469 mapwrite 0x03f742 thru 0x03ffff (0x8be bytes) 000430 1784936965.444239 trunc from 0x01973d to 0x0266a2 (0xcf66 bytes) 000432 1784936965.450763 punch from 0x023e1e to 0x032f6f (0xf152 bytes) 000433 1784936965.455398 write 0x0055c5 thru 0x014b72 (0xf5ae bytes) 000434 1784936965.467376 MIRROR_RESYNC 6 mirrors 000436 1784936966.794666 write 0x0054be thru 0x00af70 (0x5ab3 bytes) 000437 1784936966.865283 write 0x03d721 thru 0x03ffff (0x28df bytes) 000438 1784936966.876073 trunc from 0x01d5db to 0x02c941 (0xf367 bytes) 000439 1784936966.893102 MIRROR_RESYNC 6 mirrors 000440 1784936968.323509 mapread 0x01d520 thru 0x02bb8d (0xe66e bytes) 000441 1784936968.372919 write 0x0198c8 thru 0x01ad0f (0x1448 bytes) 000442 1784936968.460455 trunc from 0x040000 to 0x028dcf (0x17230 bytes) 000443 1784936968.512145 mapwrite 0x02b176 thru 0x033db3 (0x8c3e bytes) 000444 1784936968.586527 trunc from 0x033db4 to 0x00838c (0x2ba27 bytes) 000446 1784936968.608663 trunc from 0x004508 to 0x00838c (0x3e85 bytes) 000447 1784936968.617030 write 0x03a9cb thru 0x03ffff (0x5635 bytes) 000448 1784936968.620586 mapwrite 0x033368 thru 0x03a0a0 (0x6d39 bytes) 000450 1784936968.667731 mapwrite 0x019dfa thru 0x0201a4 (0x63ab bytes) 000452 1784936968.713591 read 0x01eb83 thru 0x027609 (0x8a87 bytes) 000453 1784936968.728545 trunc from 0x040000 to 0x01f83c (0x207c3 bytes) 000454 1784936968.755243 trunc from 0x00e1fa to 0x01cbc5 (0xe9cc bytes) 000458 1784936968.766868 punch from 0x0192a6 to 0x01f83c (0x6597 bytes) 000459 1784936968.773654 write 0x03b575 thru 0x03ffff (0x4a8b bytes) 000460 1784936968.776210 punch from 0x03a96c to 0x03e0a6 (0x373b bytes) 000461 1784936968.804222 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd3a72447 (stale) Mirror 14: 0xd3a72447 (stale) Mirror 16: 0xd3a72447 (stale) Mirror 17: 0xd3a72447 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd3a72447 (stale) Mirror 14: 0xd3a72447 (stale) Mirror 16: 0xd3a72447 (stale) Mirror 17: 0xd3a72447 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000462 1784936969.313663 punch from 0x01e08f to 0x022249 (0x41bb bytes) 000467 1784936969.328794 trunc from 0x040000 to 0x00546a (0x3ab95 bytes) 000468 1784936969.385996 fallocate 0x031aaf thru 0x031b18 (0x6a bytes) 000469 1784936969.400306 mapwrite 0x035ea5 thru 0x03ffff (0xa15b bytes) 000470 1784936969.554443 mapwrite 0x03df79 thru 0x03ffff (0x2087 bytes) 000471 1784936969.587558 write 0x024b86 thru 0x026650 (0x1acb bytes) 000472 1784936969.612946 trunc from 0x040000 to 0x039a07 (0x65f8 bytes) 000473 1784936969.639968 mapread 0x032e99 thru 0x039a07 (0x6b6f bytes) 000474 1784936969.648658 mapwrite 0x032737 thru 0x036ad3 (0x439d bytes) 000475 1784936969.696760 punch from 0x017ab3 to 0x01b335 (0x3883 bytes) 000476 1784936969.707012 write 0x01945f thru 0x0232ce (0x9e70 bytes) 000477 1784936969.724788 mapread 0x00f634 thru 0x0161d9 (0x6ba6 bytes) 000478 1784936969.749109 mapread 0x012a4b thru 0x01ce85 (0xa43b bytes) 000479 1784936969.763788 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39a08): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a08): Mirror 12: 0x5cc76fb9 (stale) Mirror 14: 0x5cc76fb9 (stale) Mirror 16: 0x5cc76fb9 (stale) Mirror 17: 0x5cc76fb9 (stale) CRC-32 checksum value for chunk [0, 0x39a08): Mirror 1: 0xef4b15eb Mirror 18: 0xef4b15eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a08: skipped 000480 1784936971.158313 trunc from 0x0073f3 to 0x00dcf7 (0x6905 bytes) 000481 1784936971.213989 fallocate 0x03f741 thru 0x03ffff (0x8bf bytes) 000482 1784936971.221196 trunc from 0x040000 to 0x039a14 (0x65eb bytes) 000483 1784936971.266497 mapwrite 0x01773b thru 0x01a67d (0x2f43 bytes) 000484 1784936971.342139 read 0x020023 thru 0x02cd3a (0xcd18 bytes) 000485 1784936971.357115 mapwrite 0x005314 thru 0x011967 (0xc654 bytes) 000486 1784936971.416949 read 0x032b19 thru 0x034dae (0x2296 bytes) 000487 1784936971.442969 trunc from 0x039a15 to 0x037773 (0x22a1 bytes) 000488 1784936971.467248 write 0x0096c4 thru 0x00f95f (0x629c bytes) 000489 1784936971.468943 trunc from 0x037774 to 0x00ab3b (0x2cc38 bytes) 000492 1784936971.509510 trunc from 0x0033ff to 0x0056d6 (0x22d8 bytes) 000494 1784936971.522423 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xab3c): Mirror 12: 0x64b4b93f (stale) Mirror 14: 0x64b4b93f (stale) Mirror 17: 0x64b4b93f (stale) Mirror 18: 0xb0e73424 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xab3c): Mirror 12: 0x64b4b93f (stale) Mirror 14: 0x64b4b93f (stale) Mirror 17: 0x64b4b93f (stale) Mirror 18: 0xb0e73424 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xab3c: skipped 000495 1784936971.927111 fallocate 0x0313e8 thru 0x038cfe (0x7917 bytes) 000496 1784936971.937089 read 0x017539 thru 0x01b0d7 (0x3b9f bytes) 000497 1784936971.965082 MIRROR_RESYNC 5 mirrors 000498 1784936973.653035 MIRROR_EXTEND to 6 mirrors 000499 1784936974.701986 fallocate 0x00afff thru 0x0138c7 (0x88c9 bytes) 000500 1784936974.764719 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38cff): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cff): Mirror 12: 0xe2849d3d (stale) Mirror 14: 0xe2849d3d (stale) Mirror 17: 0xe2849d3d (stale) Mirror 18: 0xe2849d3d (stale) Mirror 19: 0xe2849d3d (stale) CRC-32 checksum value for chunk [0, 0x38cff): Mirror 1: 0xe2849d3d Mirror 20: 0xe2849d3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cff: skipped 000502 1784936976.139561 trunc from 0x0306ae to 0x036c07 (0x655a bytes) 000503 1784936976.197375 mapwrite 0x00e2e1 thru 0x01c250 (0xdf70 bytes) 000504 1784936976.266799 MIRROR_RESYNC 7 mirrors 000505 1784936978.362268 read 0x013dd6 thru 0x019786 (0x59b1 bytes) 000507 1784936978.376135 write 0x0205f7 thru 0x02ac10 (0xa61a bytes) 000508 1784936978.476278 trunc from 0x038cff to 0x0070d1 (0x31c2d bytes) 000509 1784936978.532368 trunc from 0x0070d2 to 0x01e488 (0x173b7 bytes) 000510 1784936978.567983 punch from 0x00bcba to 0x016945 (0xac8c bytes) 000511 1784936978.587578 trunc from 0x01e489 to 0x007230 (0x17258 bytes) 000512 1784936978.641505 MIRROR_RESYNC 7 mirrors 000513 1784936980.110156 read 0x000f5b thru 0x007230 (0x62d6 bytes) 000514 1784936980.123409 write 0x023f05 thru 0x02d087 (0x9183 bytes) 000515 1784936980.173192 mapread 0x004961 thru 0x014733 (0xfdd3 bytes) 000516 1784936980.224090 mapwrite 0x03ad2f thru 0x03ffff (0x52d1 bytes) 000517 1784936980.366309 fallocate 0x037f10 thru 0x03ffff (0x80f0 bytes) 000518 1784936980.386750 fallocate 0x01b656 thru 0x029ab5 (0xe460 bytes) 000519 1784936980.401409 fallocate 0x00fb51 thru 0x014cc3 (0x5173 bytes) 000523 1784936980.426442 read 0x000d94 thru 0x00563f (0x48ac bytes) 000524 1784936980.444221 fallocate 0x0136d2 thru 0x02334b (0xfc7a bytes) 000525 1784936980.456483 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2f1d9fe (stale) Mirror 14: 0x2f1d9fe (stale) Mirror 17: 0x2f1d9fe (stale) Mirror 18: 0x2f1d9fe (stale) Mirror 19: 0x2f1d9fe (stale) Mirror 20: 0x2f1d9fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd690cb61 Mirror 21: 0xd690cb61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000526 1784936981.971489 punch from 0x02646c to 0x027424 (0xfb9 bytes) 000527 1784936982.021093 trunc from 0x040000 to 0x02f689 (0x10976 bytes) 000528 1784936982.069245 mapwrite 0x00f68f thru 0x01f5c3 (0xff35 bytes) 000529 1784936982.128011 MIRROR_RESYNC 8 mirrors 000530 1784936985.016005 mapwrite 0x03300e thru 0x03ffff (0xcff2 bytes) 000531 1784936985.193107 read 0x03dd4f thru 0x03ffff (0x22b1 bytes) 000533 1784936985.194450 fallocate 0x0285fe thru 0x037010 (0xea13 bytes) 000534 1784936985.216185 fallocate 0x01b816 thru 0x01e0e5 (0x28d0 bytes) 000535 1784936985.252516 MIRROR_RESYNC 8 mirrors 000537 1784936988.145352 read 0x022421 thru 0x02ec5c (0xc83c bytes) 000538 1784936988.166078 write 0x0379d8 thru 0x03c1f5 (0x481e bytes) 000539 1784936988.255427 trunc from 0x040000 to 0x03b56b (0x4a94 bytes) 000540 1784936988.312464 MIRROR_RESYNC 8 mirrors 000541 1784936991.390925 mapread 0x02613c thru 0x030173 (0xa038 bytes) 000542 1784936991.420101 trunc from 0x03b1bd to 0x03b56b (0x3af bytes) 000544 1784936991.449281 mapwrite 0x00179d thru 0x006267 (0x4acb bytes) 000545 1784936991.528646 mapwrite 0x03a897 thru 0x03ffff (0x5769 bytes) 000546 1784936991.608307 mapread 0x035b6e thru 0x03ffff (0xa492 bytes) 000547 1784936991.610232 write 0x039d47 thru 0x03ffff (0x62b9 bytes) 000548 1784936991.612417 punch from 0x036349 to 0x036756 (0x40e bytes) 000549 1784936991.619319 punch from 0x02ceb1 to 0x03c495 (0xf5e5 bytes) 000550 1784936991.649463 mapwrite 0x028c4b thru 0x02ba9c (0x2e52 bytes) 000551 1784936991.680942 trunc from 0x040000 to 0x001787 (0x3e878 bytes) 000552 1784936991.702859 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1788): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1788): Mirror 12: 0x27c67c48 (stale) Mirror 14: 0x27c67c48 (stale) Mirror 17: 0x27c67c48 (stale) Mirror 18: 0x27c67c48 (stale) Mirror 19: 0x27c67c48 (stale) Mirror 20: 0x27c67c48 (stale) Mirror 21: 0x27c67c48 (stale) CRC-32 checksum value for chunk [0, 0x1788): Mirror 1: 0x27c67c48 Mirror 22: 0x27c67c48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1788: skipped 000553 1784936992.620880 mapread 0x0010b1 thru 0x001787 (0x6d7 bytes) 000554 1784936992.644496 mapread 0x000129 thru 0x001787 (0x165f bytes) 000556 1784936992.695093 read 0x000a44 thru 0x001787 (0xd44 bytes) 000557 1784936992.707726 write 0x023f8e thru 0x02e57e (0xa5f1 bytes) 000558 1784936992.790028 mapread 0x00d70c thru 0x01001c (0x2911 bytes) 000559 1784936992.800384 mapread 0x0213e3 thru 0x025caa (0x48c8 bytes) 000560 1784936992.812173 mapread 0x028fa3 thru 0x02d46f (0x44cd bytes) 000561 1784936992.819390 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e57f): Mirror 12: 0xb7449a6f (stale) Mirror 17: 0xb7449a6f (stale) Mirror 18: 0xb7449a6f (stale) Mirror 19: 0xb7449a6f (stale) Mirror 20: 0xb7449a6f (stale) Mirror 21: 0xb7449a6f (stale) Mirror 22: 0x5f2fd0df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e57f): Mirror 12: 0xb7449a6f (stale) Mirror 17: 0xb7449a6f (stale) Mirror 18: 0xb7449a6f (stale) Mirror 19: 0xb7449a6f (stale) Mirror 20: 0xb7449a6f (stale) Mirror 21: 0xb7449a6f (stale) Mirror 22: 0x5f2fd0df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e57f: skipped 000562 1784936993.415358 mapwrite 0x0135f6 thru 0x018819 (0x5224 bytes) 000564 1784936993.446267 trunc from 0x007513 to 0x00824a (0xd38 bytes) 000565 1784936993.452259 mapread 0x0077f0 thru 0x00d5a2 (0x5db3 bytes) 000566 1784936993.467783 trunc from 0x02e57f to 0x00d13e (0x21440 bytes) 000567 1784936993.491370 trunc from 0x00af25 to 0x00d13e (0x221a bytes) 000568 1784936993.502272 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd13f): Mirror 12: 0x425724eb (stale) Mirror 17: 0x425724eb (stale) Mirror 18: 0x425724eb (stale) Mirror 19: 0x425724eb (stale) Mirror 21: 0x425724eb (stale) Mirror 22: 0xec480cc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd13f): Mirror 12: 0x425724eb (stale) Mirror 17: 0x425724eb (stale) Mirror 18: 0x425724eb (stale) Mirror 19: 0x425724eb (stale) Mirror 21: 0x425724eb (stale) Mirror 22: 0xec480cc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd13f: skipped 000570 1784936993.882156 trunc from 0x00d13f to 0x008db3 (0x438b bytes) 000571 1784936993.914148 punch from 0x005f6e to 0x008db3 (0x2e46 bytes) 000572 1784936993.922325 mapread 0x0045e0 thru 0x008db3 (0x47d4 bytes) 000573 1784936993.945998 trunc from 0x002495 to 0x008db3 (0x691f bytes) 000574 1784936993.949377 trunc from 0x001d1b to 0x008db3 (0x7099 bytes) 000575 1784936993.956109 fallocate 0x00a63f thru 0x011260 (0x6c22 bytes) 000576 1784936993.965354 fallocate 0x00faab thru 0x013cba (0x4210 bytes) 000577 1784936993.980029 trunc from 0x008db4 to 0x03d5b2 (0x347ff bytes) 000578 1784936994.011457 mapwrite 0x02c44f thru 0x02e6db (0x228d bytes) 000579 1784936994.033857 write 0x015be2 thru 0x018b43 (0x2f62 bytes) 000580 1784936994.046739 MIRROR_RESYNC 7 mirrors 000581 1784936996.297121 read 0x023bd8 thru 0x02fce4 (0xc10d bytes) 000582 1784936996.310278 trunc from 0x03d5b3 to 0x039036 (0x457c bytes) 000583 1784936996.359345 mapread 0x021e61 thru 0x02dd4b (0xbeeb bytes) 000585 1784936996.390314 fallocate 0x00f7e3 thru 0x019e17 (0xa635 bytes) 000586 1784936996.395593 trunc from 0x01344b to 0x020d12 (0xd8c8 bytes) 000587 1784936996.400535 trunc from 0x039037 to 0x02ca63 (0xc5d3 bytes) 000588 1784936996.427682 MIRROR_RESYNC 7 mirrors 000589 1784936997.785206 write 0x018e51 thru 0x01f82a (0x69da bytes) 000590 1784936997.830925 mapread 0x022143 thru 0x02ca63 (0xa921 bytes) 000591 1784936997.853238 fallocate 0x01a7d2 thru 0x028cdf (0xe50e bytes) 000592 1784936997.859837 mapread 0x005711 thru 0x00b8ac (0x619c bytes) 000593 1784936997.874455 trunc from 0x02ca64 to 0x006d12 (0x25d51 bytes) 000594 1784936997.907430 mapread 0x000237 thru 0x00306b (0x2e35 bytes) 000595 1784936997.920035 read 0x0021b8 thru 0x006d12 (0x4b5b bytes) 000596 1784936997.923162 trunc from 0x002387 to 0x004408 (0x2082 bytes) 000597 1784936997.938791 read 0x005fde thru 0x006d12 (0xd35 bytes) 000598 1784936997.939587 mapread 0x00656e thru 0x006d12 (0x7a5 bytes) 000599 1784936997.940689 mapread 0x006aa3 thru 0x006d12 (0x270 bytes) 000600 1784936997.946597 trunc from 0x006d13 to 0x0041ca (0x2b48 bytes) 000601 1784936997.985224 punch from 0x002390 to 0x0041ca (0x1e3b bytes) 000602 1784936997.990173 read 0x002b83 thru 0x0041ca (0x1648 bytes) 000603 1784936997.999344 trunc from 0x0041cb to 0x02471c (0x20552 bytes) 000604 1784936998.020513 punch from 0x012f48 to 0x01a961 (0x7a1a bytes) 000605 1784936998.027221 trunc from 0x021a3f to 0x02471c (0x2cde bytes) 000606 1784936998.033149 mapread 0x0066d5 thru 0x012173 (0xba9f bytes) 000607 1784936998.045341 mapread 0x02032b thru 0x02471c (0x43f2 bytes) 000608 1784936998.046919 punch from 0x019e5f to 0x02471c (0xa8be bytes) 000609 1784936998.054339 fallocate 0x0133ee thru 0x016b7c (0x378f bytes) 000611 1784936998.061322 trunc from 0x00145a to 0x002bc0 (0x1767 bytes) 000612 1784936998.071373 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2471d): Mirror 12: 0x78d3790b (stale) Mirror 17: 0x78d3790b (stale) Mirror 19: 0x78d3790b (stale) Mirror 21: 0x78d3790b (stale) Mirror 22: 0x78d3790b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2471d): Mirror 12: 0x78d3790b (stale) Mirror 17: 0x78d3790b (stale) Mirror 19: 0x78d3790b (stale) Mirror 21: 0x78d3790b (stale) Mirror 22: 0x78d3790b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2471d: skipped 000614 1784936998.711074 write 0x02528c thru 0x0341d3 (0xef48 bytes) 000615 1784936998.726446 mapwrite 0x0246e6 thru 0x02bd63 (0x767e bytes) 000616 1784936998.764416 read 0x01ce13 thru 0x0235b3 (0x67a1 bytes) 000617 1784936998.833721 punch from 0x02cc3c to 0x0341d3 (0x7598 bytes) 000618 1784936998.866119 fallocate 0x0256df thru 0x025c72 (0x594 bytes) 000619 1784936998.876855 read 0x008c15 thru 0x017013 (0xe3ff bytes) 000620 1784936998.901140 read 0x02834a thru 0x02f46b (0x7122 bytes) 000621 1784936998.905124 fallocate 0x03f319 thru 0x03ffff (0xce7 bytes) 000622 1784936998.935823 trunc from 0x0341d4 to 0x004242 (0x2ff91 bytes) 000623 1784936999.038516 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x4243): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4243): Mirror 12: 0x13b81657 (stale) Mirror 17: 0x13b81657 (stale) Mirror 19: 0x13b81657 (stale) Mirror 21: 0x13b81657 (stale) Mirror 22: 0x13b81657 (stale) CRC-32 checksum value for chunk [0, 0x4243): Mirror 1: 0xf3fa0bcd Mirror 23: 0xf3fa0bcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4243: skipped 000624 1784937000.638819 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4243): Mirror 12: 0x13b81657 (stale) Mirror 17: 0x13b81657 (stale) Mirror 19: 0x13b81657 (stale) Mirror 21: 0x13b81657 (stale) Mirror 22: 0x13b81657 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4243): Mirror 12: 0x13b81657 (stale) Mirror 17: 0x13b81657 (stale) Mirror 19: 0x13b81657 (stale) Mirror 21: 0x13b81657 (stale) Mirror 22: 0x13b81657 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4243: skipped 000626 1784937001.172413 mapwrite 0x03e3e4 thru 0x03ffff (0x1c1c bytes) 000627 1784937001.247103 read 0x02a4e5 thru 0x038781 (0xe29d bytes) 000628 1784937001.260012 mapread 0x037fde thru 0x03ffff (0x8022 bytes) 000629 1784937001.274370 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1aa4ee47 (stale) Mirror 17: 0x1aa4ee47 (stale) Mirror 19: 0x1aa4ee47 (stale) Mirror 21: 0x1aa4ee47 (stale) Mirror 22: 0x1aa4ee47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd50260b2 Mirror 23: 0xd50260b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000631 1784937002.684300 trunc from 0x040000 to 0x03135a (0xeca5 bytes) 000632 1784937002.800971 mapread 0x0259bf thru 0x03135a (0xb99c bytes) 000633 1784937002.871825 mapwrite 0x03be49 thru 0x03ffff (0x41b7 bytes) 000634 1784937002.966591 trunc from 0x040000 to 0x00167d (0x3e982 bytes) 000635 1784937002.998563 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x167e): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x167e): Mirror 12: 0x37abbc9b (stale) Mirror 17: 0x37abbc9b (stale) Mirror 19: 0x37abbc9b (stale) Mirror 21: 0x37abbc9b (stale) Mirror 22: 0x37abbc9b (stale) Mirror 23: 0x83a255ec (stale) CRC-32 checksum value for chunk [0, 0x167e): Mirror 1: 0x83a255ec Mirror 24: 0x83a255ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x167e: skipped 000636 1784937004.177392 MIRROR_RESYNC 8 mirrors 000637 1784937005.538596 trunc from 0x00167e to 0x00209c (0xa1f bytes) 000638 1784937005.621202 mapread 0x001681 thru 0x00209c (0xa1c bytes) 000639 1784937005.642459 fallocate 0x03dc7c thru 0x03ffff (0x2384 bytes) 000640 1784937005.661458 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x209d): Mirror 17: 0xca75dc37 (stale) Mirror 19: 0xca75dc37 (stale) Mirror 21: 0xca75dc37 (stale) Mirror 22: 0xca75dc37 (stale) Mirror 23: 0xca75dc37 (stale) Mirror 24: 0xca75dc37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x209d): Mirror 17: 0xca75dc37 (stale) Mirror 19: 0xca75dc37 (stale) Mirror 21: 0xca75dc37 (stale) Mirror 22: 0xca75dc37 (stale) Mirror 23: 0xca75dc37 (stale) Mirror 24: 0xca75dc37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x209d: skipped 000641 1784937006.048182 punch from 0x001ad3 to 0x00209c (0x5ca bytes) 000642 1784937006.052959 trunc from 0x001fbd to 0x00209c (0xe0 bytes) 000643 1784937006.057257 mapwrite 0x00d5d0 thru 0x014532 (0x6f63 bytes) 000645 1784937006.161918 mapwrite 0x02a3a2 thru 0x0362c4 (0xbf23 bytes) 000646 1784937006.263634 write 0x012823 thru 0x01844f (0x5c2d bytes) 000647 1784937006.303023 MIRROR_RESYNC 7 mirrors 000648 1784937008.122516 fallocate 0x03144b thru 0x036775 (0x532b bytes) 000649 1784937008.164315 read 0x011b5c thru 0x02040b (0xe8b0 bytes) 000650 1784937008.179704 trunc from 0x0362c5 to 0x028beb (0xd6d9 bytes) 000651 1784937008.199983 trunc from 0x0110e5 to 0x018779 (0x7695 bytes) 000652 1784937008.207465 read 0x016be3 thru 0x01fc97 (0x90b5 bytes) 000653 1784937008.221323 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x28bec): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28bec): Mirror 17: 0xc8391fc1 (stale) Mirror 19: 0xc8391fc1 (stale) Mirror 21: 0xc8391fc1 (stale) Mirror 22: 0xc8391fc1 (stale) Mirror 23: 0xc8391fc1 (stale) Mirror 24: 0xc8391fc1 (stale) CRC-32 checksum value for chunk [0, 0x28bec): Mirror 1: 0x5d43d24d Mirror 25: 0x5d43d24d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28bec: skipped 000654 1784937009.424917 fallocate 0x01ffee thru 0x0200ff (0x112 bytes) 000655 1784937009.498672 trunc from 0x028bec to 0x032991 (0x9da6 bytes) 000656 1784937009.560301 mapwrite 0x02b818 thru 0x031a3d (0x6226 bytes) 000658 1784937009.623922 punch from 0x031f76 to 0x032991 (0xa1c bytes) 000659 1784937009.638984 MIRROR_RESYNC 8 mirrors 000660 1784937012.746497 write 0x00c34b thru 0x0152df (0x8f95 bytes) 000662 1784937012.802039 write 0x00d831 thru 0x01d7bf (0xff8f bytes) 000663 1784937012.811508 trunc from 0x02be29 to 0x02c211 (0x3e9 bytes) 000664 1784937012.818053 read 0x007777 thru 0x01498f (0xd219 bytes) 000665 1784937012.832566 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32992): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32992): Mirror 17: 0x371b696d (stale) Mirror 19: 0x371b696d (stale) Mirror 21: 0x371b696d (stale) Mirror 22: 0x371b696d (stale) Mirror 23: 0x371b696d (stale) Mirror 24: 0x371b696d (stale) Mirror 25: 0x371b696d (stale) CRC-32 checksum value for chunk [0, 0x32992): Mirror 1: 0x7d2365f4 Mirror 26: 0x7d2365f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32992: skipped 000666 1784937014.470341 mapread 0x017e4d thru 0x02327b (0xb42f bytes) 000668 1784937014.532538 fallocate 0x03f653 thru 0x03ffff (0x9ad bytes) 000669 1784937014.587002 write 0x036427 thru 0x03ffff (0x9bd9 bytes) 000670 1784937014.596532 read 0x019e12 thru 0x026ea3 (0xd092 bytes) 000671 1784937014.614277 trunc from 0x00df1f to 0x00f903 (0x19e5 bytes) 000672 1784937014.621481 fallocate 0x0035ae thru 0x0094b8 (0x5f0b bytes) 000673 1784937014.634724 write 0x012e54 thru 0x0184d1 (0x567e bytes) 000674 1784937014.656948 read 0x03bc24 thru 0x03ffff (0x43dc bytes) 000675 1784937014.669496 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc2b0881c (stale) Mirror 19: 0xc2b0881c (stale) Mirror 21: 0xc2b0881c (stale) Mirror 22: 0xc2b0881c (stale) Mirror 23: 0xc2b0881c (stale) Mirror 24: 0xc2b0881c (stale) Mirror 25: 0xc2b0881c (stale) Mirror 26: 0x47a16b19 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb813116 Mirror 27: 0xbb813116 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000676 1784937016.395532 trunc from 0x026076 to 0x02d845 (0x77d0 bytes) 000677 1784937016.450226 punch from 0x03b8b7 to 0x03ffff (0x4749 bytes) 000679 1784937016.460262 write 0x018eee thru 0x021854 (0x8967 bytes) 000680 1784937016.480756 trunc from 0x0333f0 to 0x03c4d1 (0x90e2 bytes) 000681 1784937016.487483 trunc from 0x013257 to 0x01c7de (0x9588 bytes) 000682 1784937016.511512 trunc from 0x040000 to 0x008951 (0x376ae bytes) 000683 1784937016.592494 write 0x02020c thru 0x02af86 (0xad7b bytes) 000685 1784937016.603156 fallocate 0x010db5 thru 0x01a721 (0x996d bytes) 000686 1784937016.622105 trunc from 0x006039 to 0x00837c (0x2344 bytes) 000687 1784937016.627229 write 0x02eab2 thru 0x03b1cf (0xc71e bytes) 000688 1784937016.631051 fallocate 0x01f335 thru 0x025e51 (0x6b1d bytes) 000689 1784937016.642950 read 0x01014a thru 0x018bab (0x8a62 bytes) 000690 1784937016.673291 mapwrite 0x020329 thru 0x0275da (0x72b2 bytes) 000691 1784937016.730958 write 0x01ec3f thru 0x0299cf (0xad91 bytes) 000692 1784937016.744440 trunc from 0x03b1d0 to 0x027a7d (0x13752 bytes) 000693 1784937016.785168 write 0x005bd1 thru 0x013822 (0xdc52 bytes) 000694 1784937016.794978 fallocate 0x02fb92 thru 0x032a38 (0x2ea7 bytes) 000695 1784937016.810089 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 17: 0x2db0e698 (stale) Mirror 19: 0x2db0e698 (stale) Mirror 21: 0x2db0e698 (stale) Mirror 22: 0x2db0e698 (stale) Mirror 23: 0x2db0e698 (stale) Mirror 24: 0x2db0e698 (stale) Mirror 25: 0x2db0e698 (stale) Mirror 26: 0x792dd97e (stale) Mirror 27: 0xe2963bf4 (stale) CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 1: 0xccbe7fae Mirror 28: 0xccbe7fae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a7e: skipped 000697 1784937018.276900 mapread 0x0203ce thru 0x02345e (0x3091 bytes) 000698 1784937018.352456 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 17: 0x2db0e698 (stale) Mirror 19: 0x2db0e698 (stale) Mirror 22: 0x2db0e698 (stale) Mirror 23: 0x2db0e698 (stale) Mirror 24: 0x2db0e698 (stale) Mirror 25: 0x2db0e698 (stale) Mirror 26: 0x792dd97e (stale) Mirror 27: 0xe2963bf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 17: 0x2db0e698 (stale) Mirror 19: 0x2db0e698 (stale) Mirror 22: 0x2db0e698 (stale) Mirror 23: 0x2db0e698 (stale) Mirror 24: 0x2db0e698 (stale) Mirror 25: 0x2db0e698 (stale) Mirror 26: 0x792dd97e (stale) Mirror 27: 0xe2963bf4 (stale) CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 1: 0xccbe7fae Mirror 28: 0xccbe7fae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a7e: skipped 000699 1784937019.555566 read 0x010dbc thru 0x01ed55 (0xdf9a bytes) 000700 1784937019.607873 fallocate 0x033566 thru 0x039fb3 (0x6a4e bytes) 000701 1784937019.709388 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 17: 0x2db0e698 (stale) Mirror 19: 0x2db0e698 (stale) Mirror 23: 0x2db0e698 (stale) Mirror 24: 0x2db0e698 (stale) Mirror 25: 0x2db0e698 (stale) Mirror 26: 0x792dd97e (stale) Mirror 27: 0xe2963bf4 (stale) Mirror 28: 0xccbe7fae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a7e): Mirror 17: 0x2db0e698 (stale) Mirror 19: 0x2db0e698 (stale) Mirror 23: 0x2db0e698 (stale) Mirror 24: 0x2db0e698 (stale) Mirror 25: 0x2db0e698 (stale) Mirror 26: 0x792dd97e (stale) Mirror 27: 0xe2963bf4 (stale) Mirror 28: 0xccbe7fae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a7e: skipped 000703 1784937020.400132 write 0x033653 thru 0x03fff1 (0xc99f bytes) 000705 1784937020.409059 read 0x019391 thru 0x01d288 (0x3ef8 bytes) 000706 1784937020.419268 fallocate 0x00da93 thru 0x011edd (0x444b bytes) 000707 1784937020.424739 fallocate 0x00278d thru 0x009d8f (0x7603 bytes) 000708 1784937020.430446 punch from 0x00a623 to 0x00de7c (0x385a bytes) 000709 1784937020.440455 MIRROR_RESYNC 9 mirrors 000710 1784937024.066532 mapwrite 0x0311f1 thru 0x03c8c6 (0xb6d6 bytes) 000711 1784937024.197185 read 0x005bca thru 0x0081d5 (0x260c bytes) 000712 1784937024.215438 MIRROR_RESYNC 9 mirrors 000713 1784937027.895901 MIRROR_EXTEND to 10 mirrors 000714 1784937029.425148 trunc from 0x01f84a to 0x0244ad (0x4c64 bytes) 000715 1784937029.488029 mapread 0x010c2d thru 0x017f0c (0x72e0 bytes) 000716 1784937029.532560 write 0x02e9b3 thru 0x03a498 (0xbae6 bytes) 000717 1784937029.558425 mapread 0x038ed6 thru 0x03fff1 (0x711c bytes) 000719 1784937029.589190 mapwrite 0x01519c thru 0x018c6d (0x3ad2 bytes) 000720 1784937029.661547 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3fff2): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fff2): Mirror 17: 0xa439385f (stale) Mirror 19: 0xa439385f (stale) Mirror 23: 0xa439385f (stale) Mirror 24: 0xa439385f (stale) Mirror 25: 0xa439385f (stale) Mirror 26: 0xa439385f (stale) Mirror 27: 0xa439385f (stale) Mirror 28: 0xa439385f (stale) Mirror 29: 0xa439385f (stale) CRC-32 checksum value for chunk [0, 0x3fff2): Mirror 1: 0x1a1ddd0b Mirror 30: 0x1a1ddd0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fff2: skipped 000722 1784937031.653932 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fff2): Mirror 17: 0xa439385f (stale) Mirror 19: 0xa439385f (stale) Mirror 23: 0xa439385f (stale) Mirror 25: 0xa439385f (stale) Mirror 26: 0xa439385f (stale) Mirror 27: 0xa439385f (stale) Mirror 28: 0xa439385f (stale) Mirror 29: 0xa439385f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fff2): Mirror 17: 0xa439385f (stale) Mirror 19: 0xa439385f (stale) Mirror 23: 0xa439385f (stale) Mirror 25: 0xa439385f (stale) Mirror 26: 0xa439385f (stale) Mirror 27: 0xa439385f (stale) Mirror 28: 0xa439385f (stale) Mirror 29: 0xa439385f (stale) CRC-32 checksum value for chunk [0, 0x3fff2): Mirror 1: 0x1a1ddd0b Mirror 30: 0x1a1ddd0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fff2: skipped 000723 1784937033.393424 mapwrite 0x0074f2 thru 0x01189b (0xa3aa bytes) 000724 1784937033.596918 mapwrite 0x02c91c thru 0x03c8e1 (0xffc6 bytes) 000725 1784937033.716189 trunc from 0x03fff2 to 0x02ff0b (0x100e6 bytes) 000726 1784937033.775284 write 0x022917 thru 0x0271e1 (0x48cb bytes) 000727 1784937033.796401 trunc from 0x02df38 to 0x02ff0b (0x1fd4 bytes) 000728 1784937033.810325 fallocate 0x0063ae thru 0x01636a (0xffbd bytes) 000729 1784937033.819295 trunc from 0x02ff0c to 0x029133 (0x6dd8 bytes) 000730 1784937033.837404 mapwrite 0x0140b8 thru 0x022fa6 (0xeeef bytes) 000731 1784937033.876328 trunc from 0x01d5a0 to 0x0243e4 (0x6e45 bytes) 000733 1784937033.884312 mapread 0x023636 thru 0x029133 (0x5afe bytes) 000734 1784937033.897080 write 0x03805f thru 0x03a242 (0x21e4 bytes) 000735 1784937033.899313 trunc from 0x03a243 to 0x01afa4 (0x1f29e bytes) 000736 1784937033.920240 mapwrite 0x020ef4 thru 0x021d1c (0xe29 bytes) 000737 1784937033.998410 read 0x0038ee thru 0x003b46 (0x259 bytes) 000738 1784937034.013207 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x21d1d): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21d1d): Mirror 17: 0x712371d4 (stale) Mirror 19: 0x712371d4 (stale) Mirror 23: 0x712371d4 (stale) Mirror 25: 0x712371d4 (stale) Mirror 26: 0x712371d4 (stale) Mirror 27: 0x712371d4 (stale) Mirror 28: 0x712371d4 (stale) Mirror 29: 0x712371d4 (stale) Mirror 30: 0xc271024e (stale) CRC-32 checksum value for chunk [0, 0x21d1d): Mirror 1: 0x2f23fc1e Mirror 31: 0x2f23fc1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21d1d: skipped 000739 1784937035.496303 mapwrite 0x02ff23 thru 0x0381ef (0x82cd bytes) 000740 1784937035.664568 MIRROR_RESYNC 11 mirrors 000741 1784937040.048385 fallocate 0x01b5ce thru 0x02a9bc (0xf3ef bytes) 000742 1784937040.132786 read 0x0213b2 thru 0x0248d2 (0x3521 bytes) 000743 1784937040.152732 trunc from 0x028288 to 0x02d41a (0x5193 bytes) 000744 1784937040.170053 mapread 0x00703f thru 0x010970 (0x9932 bytes) 000745 1784937040.211395 mapwrite 0x009c1f thru 0x00c551 (0x2933 bytes) 000746 1784937040.257937 trunc from 0x0236e5 to 0x0323b3 (0xeccf bytes) 000747 1784937040.280579 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x381f0): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381f0): Mirror 17: 0xfb75f83e (stale) Mirror 19: 0xfb75f83e (stale) Mirror 23: 0xfb75f83e (stale) Mirror 25: 0xfb75f83e (stale) Mirror 26: 0xfb75f83e (stale) Mirror 27: 0xfb75f83e (stale) Mirror 28: 0xfb75f83e (stale) Mirror 29: 0xfb75f83e (stale) Mirror 30: 0xfb75f83e (stale) Mirror 31: 0xfb75f83e (stale) CRC-32 checksum value for chunk [0, 0x381f0): Mirror 1: 0x9624a1f2 Mirror 32: 0x9624a1f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381f0: skipped 000748 1784937042.057338 trunc from 0x0381f0 to 0x00bc1b (0x2c5d4 bytes) 000749 1784937042.285470 trunc from 0x00bc1c to 0x00d1a1 (0x1586 bytes) 000750 1784937042.368651 trunc from 0x00d1a2 to 0x0083cb (0x4dd6 bytes) 000751 1784937042.417325 mapread 0x00627b thru 0x0083cb (0x2151 bytes) 000753 1784937042.479852 mapread 0x005e28 thru 0x007c78 (0x1e51 bytes) 000754 1784937042.483189 trunc from 0x0083cc to 0x028766 (0x2039b bytes) 000755 1784937042.514751 mapread 0x00c426 thru 0x01886a (0xc445 bytes) 000756 1784937042.564649 trunc from 0x006064 to 0x00ed07 (0x8ca4 bytes) 000757 1784937042.579274 trunc from 0x00cfdb to 0x015e10 (0x8e36 bytes) 000758 1784937042.594056 mapread 0x01d62f thru 0x01e922 (0x12f4 bytes) 000759 1784937042.642794 fallocate 0x02514d thru 0x027f03 (0x2db7 bytes) 000760 1784937042.652118 punch from 0x0133e9 to 0x0232d7 (0xfeef bytes) 000761 1784937042.661306 read 0x015051 thru 0x01b9f9 (0x69a9 bytes) 000762 1784937042.687296 read 0x000deb thru 0x0034a7 (0x26bd bytes) 000763 1784937042.716272 trunc from 0x028767 to 0x00eb39 (0x19c2d bytes) 000764 1784937042.757470 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xeb3a): Mirror 17: 0x4da159e5 (stale) Mirror 23: 0x4da159e5 (stale) Mirror 25: 0x4da159e5 (stale) Mirror 26: 0x4da159e5 (stale) Mirror 27: 0x4da159e5 (stale) Mirror 28: 0x4da159e5 (stale) Mirror 29: 0x4da159e5 (stale) Mirror 30: 0x4da159e5 (stale) Mirror 31: 0x4da159e5 (stale) Mirror 32: 0x1e055c85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeb3a): Mirror 17: 0x4da159e5 (stale) Mirror 23: 0x4da159e5 (stale) Mirror 25: 0x4da159e5 (stale) Mirror 26: 0x4da159e5 (stale) Mirror 27: 0x4da159e5 (stale) Mirror 28: 0x4da159e5 (stale) Mirror 29: 0x4da159e5 (stale) Mirror 30: 0x4da159e5 (stale) Mirror 31: 0x4da159e5 (stale) Mirror 32: 0x1e055c85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeb3a: skipped 000765 1784937043.630799 trunc from 0x0029db to 0x004af0 (0x2116 bytes) 000767 1784937043.641466 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xeb3a): Mirror 17: 0x4da159e5 (stale) Mirror 23: 0x4da159e5 (stale) Mirror 25: 0x4da159e5 (stale) Mirror 26: 0x4da159e5 (stale) Mirror 27: 0x4da159e5 (stale) Mirror 28: 0x4da159e5 (stale) Mirror 29: 0x4da159e5 (stale) Mirror 30: 0x4da159e5 (stale) Mirror 31: 0x4da159e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeb3a): Mirror 17: 0x4da159e5 (stale) Mirror 23: 0x4da159e5 (stale) Mirror 25: 0x4da159e5 (stale) Mirror 26: 0x4da159e5 (stale) Mirror 27: 0x4da159e5 (stale) Mirror 28: 0x4da159e5 (stale) Mirror 29: 0x4da159e5 (stale) Mirror 30: 0x4da159e5 (stale) Mirror 31: 0x4da159e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeb3a: skipped 000768 1784937044.581300 mapwrite 0x032bcc thru 0x03983b (0x6c70 bytes) 000769 1784937044.705822 punch from 0x010cbf to 0x01920f (0x8551 bytes) 000770 1784937044.727573 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3983c): Mirror 17: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3983c): Mirror 17: 0x93e85f77 (stale) Mirror 23: 0x93e85f77 (stale) Mirror 25: 0x93e85f77 (stale) Mirror 26: 0x93e85f77 (stale) Mirror 27: 0x93e85f77 (stale) Mirror 28: 0x93e85f77 (stale) Mirror 29: 0x93e85f77 (stale) Mirror 30: 0x93e85f77 (stale) Mirror 31: 0x93e85f77 (stale) CRC-32 checksum value for chunk [0, 0x3983c): Mirror 1: 0xdd61a133 Mirror 32: 0xdd61a133 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3983c: skipped 000771 1784937046.985121 trunc from 0x03983c to 0x03a21c (0x9e1 bytes) 000773 1784937047.130273 mapread 0x0383a7 thru 0x03a21c (0x1e76 bytes) 000774 1784937047.163490 MIRROR_RESYNC 11 mirrors 000775 1784937051.201778 mapread 0x024f3e thru 0x027505 (0x25c8 bytes) 000776 1784937051.254681 mapwrite 0x032c9e thru 0x037819 (0x4b7c bytes) 000777 1784937051.480022 punch from 0x02db31 to 0x03921b (0xb6eb bytes) 000778 1784937051.498763 trunc from 0x00f3e4 to 0x0118c9 (0x24e6 bytes) 000779 1784937051.516504 trunc from 0x0028c1 to 0x0116ac (0xedec bytes) 000780 1784937051.537234 trunc from 0x03a21d to 0x015b5f (0x246bd bytes) 000781 1784937051.587211 mapread 0x00434a thru 0x00dd38 (0x99ef bytes) 000782 1784937051.605950 mapwrite 0x016fb8 thru 0x020338 (0x9381 bytes) 000783 1784937051.817069 MIRROR_RESYNC 11 mirrors 000785 1784937054.762750 fallocate 0x016585 thru 0x01fe5e (0x98da bytes) 000786 1784937054.813678 punch from 0x006949 to 0x00d00b (0x66c3 bytes) 000787 1784937054.833931 punch from 0x00e5bf to 0x0151e0 (0x6c22 bytes) 000788 1784937054.856513 read 0x0121bf thru 0x01d65c (0xb49e bytes) 000789 1784937054.886468 fallocate 0x03c706 thru 0x03ffff (0x38fa bytes) 000790 1784937054.896917 punch from 0x00fddf to 0x01232e (0x2550 bytes) 000791 1784937054.917770 mapread 0x001dc9 thru 0x00fcc8 (0xdf00 bytes) 000792 1784937054.994268 punch from 0x006bce to 0x015ebf (0xf2f2 bytes) 000793 1784937055.005689 fallocate 0x01b4e9 thru 0x023487 (0x7f9f bytes) 000794 1784937055.015044 punch from 0x00aae9 to 0x017b19 (0xd031 bytes) 000795 1784937055.039005 trunc from 0x023488 to 0x0162f3 (0xd194 bytes) 000796 1784937055.073879 fallocate 0x0186fc thru 0x022da2 (0xa6a7 bytes) 000797 1784937055.091897 mapread 0x00be85 thru 0x010f2d (0x50a9 bytes) 000798 1784937055.113984 trunc from 0x0162f4 to 0x034808 (0x1e515 bytes) 000799 1784937055.136444 trunc from 0x034809 to 0x0062ab (0x2e55d bytes) 000800 1784937055.193076 write 0x01e97c thru 0x02e4a1 (0xfb26 bytes) 000801 1784937055.202150 mapread 0x029587 thru 0x02e4a1 (0x4f1b bytes) 000803 1784937055.204568 write 0x00a0af thru 0x00bc9c (0x1bee bytes) 000804 1784937055.259195 MIRROR_RESYNC 11 mirrors 000805 1784937058.815470 read 0x00a004 thru 0x00d868 (0x3865 bytes) 000806 1784937058.824128 fallocate 0x014221 thru 0x0210f2 (0xced2 bytes) 000807 1784937058.862934 trunc from 0x01c425 to 0x02575d (0x9339 bytes) 000808 1784937058.873835 trunc from 0x01c60d to 0x028748 (0xc13c bytes) 000809 1784937058.893257 MIRROR_RESYNC 11 mirrors 000810 1784937061.512769 read 0x020425 thru 0x02e382 (0xdf5e bytes) 000811 1784937061.539010 punch from 0x00ab30 to 0x01179e (0x6c6f bytes) 000812 1784937061.588593 trunc from 0x02e4a2 to 0x035d17 (0x7876 bytes) 000813 1784937061.637182 read 0x017e7c thru 0x01a3ec (0x2571 bytes) 000814 1784937061.662220 trunc from 0x0081e3 to 0x013b76 (0xb994 bytes) 000815 1784937061.683148 read 0x0079e1 thru 0x00c850 (0x4e70 bytes) 000816 1784937061.694495 fallocate 0x0015f2 thru 0x00393e (0x234d bytes) 000817 1784937061.712948 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35d18): Mirror 17: 0x1042a8ac (stale) Mirror 23: 0x1042a8ac (stale) Mirror 26: 0x1042a8ac (stale) Mirror 27: 0x1042a8ac (stale) Mirror 28: 0x1042a8ac (stale) Mirror 29: 0x1042a8ac (stale) Mirror 30: 0x1042a8ac (stale) Mirror 31: 0x1042a8ac (stale) Mirror 32: 0x1042a8ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d18): Mirror 17: 0x1042a8ac (stale) Mirror 23: 0x1042a8ac (stale) Mirror 26: 0x1042a8ac (stale) Mirror 27: 0x1042a8ac (stale) Mirror 28: 0x1042a8ac (stale) Mirror 29: 0x1042a8ac (stale) Mirror 30: 0x1042a8ac (stale) Mirror 31: 0x1042a8ac (stale) Mirror 32: 0x1042a8ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d18: skipped 000818 1784937062.485237 trunc from 0x035d18 to 0x00fdbc (0x25f5b bytes) 000820 1784937062.532019 mapwrite 0x00e14a thru 0x00e8bd (0x774 bytes) 000821 1784937062.567444 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfdbd): Mirror 17: 0xe2a3de27 (stale) Mirror 26: 0xe2a3de27 (stale) Mirror 27: 0xe2a3de27 (stale) Mirror 28: 0xe2a3de27 (stale) Mirror 29: 0xe2a3de27 (stale) Mirror 30: 0xe2a3de27 (stale) Mirror 31: 0xe2a3de27 (stale) Mirror 32: 0xe2a3de27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfdbd): Mirror 17: 0xe2a3de27 (stale) Mirror 26: 0xe2a3de27 (stale) Mirror 27: 0xe2a3de27 (stale) Mirror 28: 0xe2a3de27 (stale) Mirror 29: 0xe2a3de27 (stale) Mirror 30: 0xe2a3de27 (stale) Mirror 31: 0xe2a3de27 (stale) Mirror 32: 0xe2a3de27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfdbd: skipped 000822 1784937063.331475 trunc from 0x00fdbd to 0x004d2e (0xb08e bytes) 000823 1784937063.354176 punch from 0x000334 to 0x003433 (0x3100 bytes) 000824 1784937063.363393 punch from 0x001e4b to 0x004d2e (0x2ee4 bytes) 000825 1784937063.370026 fallocate 0x01723c thru 0x01912f (0x1ef4 bytes) 000826 1784937063.376204 fallocate 0x0191a1 thru 0x023c45 (0xaaa5 bytes) 000827 1784937063.382157 punch from 0x017a03 to 0x01912f (0x172d bytes) 000828 1784937063.387734 mapread 0x00f62e thru 0x010f12 (0x18e5 bytes) 000829 1784937063.404451 trunc from 0x019130 to 0x01925b (0x12c bytes) 000830 1784937063.418855 trunc from 0x000eeb to 0x001630 (0x746 bytes) 000831 1784937063.423805 fallocate 0x03ac6e thru 0x03ffff (0x5392 bytes) 000832 1784937063.438235 mapwrite 0x00826e thru 0x013d20 (0xbab3 bytes) 000833 1784937063.496173 mapread 0x000794 thru 0x0039a4 (0x3211 bytes) 000834 1784937063.503262 mapread 0x002184 thru 0x005e10 (0x3c8d bytes) 000835 1784937063.505257 write 0x00e05a thru 0x011167 (0x310e bytes) 000836 1784937063.508006 mapread 0x00a9e1 thru 0x01205a (0x767a bytes) 000837 1784937063.510202 punch from 0x009978 to 0x00fb27 (0x61b0 bytes) 000838 1784937063.529862 mapread 0x018fe9 thru 0x01925b (0x273 bytes) 000841 1784937063.538314 read 0x006db5 thru 0x016d18 (0xff64 bytes) 000842 1784937063.566057 fallocate 0x02560f thru 0x02c0f0 (0x6ae2 bytes) 000843 1784937063.609305 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c0f1): Mirror 17: 0x3a3ca93c (stale) Mirror 26: 0x3a3ca93c (stale) Mirror 27: 0x3a3ca93c (stale) Mirror 28: 0x3a3ca93c (stale) Mirror 29: 0x3a3ca93c (stale) Mirror 30: 0x3a3ca93c (stale) Mirror 31: 0x3a3ca93c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c0f1): Mirror 17: 0x3a3ca93c (stale) Mirror 26: 0x3a3ca93c (stale) Mirror 27: 0x3a3ca93c (stale) Mirror 28: 0x3a3ca93c (stale) Mirror 29: 0x3a3ca93c (stale) Mirror 30: 0x3a3ca93c (stale) Mirror 31: 0x3a3ca93c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c0f1: skipped 000845 1784937064.201818 write 0x02b658 thru 0x03af36 (0xf8df bytes) 000847 1784937064.226196 trunc from 0x03614c to 0x03af36 (0x4deb bytes) 000848 1784937064.249901 mapread 0x027f19 thru 0x036ea2 (0xef8a bytes) 000849 1784937064.282107 trunc from 0x03af37 to 0x01ab36 (0x20400 bytes) 000850 1784937064.306179 read 0x019f9f thru 0x01ab36 (0xb98 bytes) 000851 1784937064.313245 fallocate 0x00ab0e thru 0x00bca8 (0x119b bytes) 000852 1784937064.318906 fallocate 0x039af2 thru 0x03efc9 (0x54d8 bytes) 000853 1784937064.325875 mapwrite 0x01edd7 thru 0x024412 (0x563c bytes) 000854 1784937064.381002 read 0x00c9a0 thru 0x01a433 (0xda94 bytes) 000855 1784937064.398089 mapread 0x003ce6 thru 0x00c603 (0x891e bytes) 000856 1784937064.404920 trunc from 0x024413 to 0x00e316 (0x160fc bytes) 000857 1784937064.422334 fallocate 0x00ee47 thru 0x019088 (0xa242 bytes) 000858 1784937064.437855 fallocate 0x03fd09 thru 0x03ffff (0x2f7 bytes) 000859 1784937064.452658 MIRROR_RESYNC 8 mirrors 000860 1784937066.499586 mapread 0x0160fb thru 0x019088 (0x2f8e bytes) 000861 1784937066.527830 read 0x002a07 thru 0x011106 (0xe700 bytes) 000862 1784937066.537728 trunc from 0x0023a2 to 0x00a4f3 (0x8152 bytes) 000863 1784937066.598566 read 0x017014 thru 0x019088 (0x2075 bytes) 000864 1784937066.611952 trunc from 0x019089 to 0x01b475 (0x23ed bytes) 000865 1784937066.646979 punch from 0x014256 to 0x01b475 (0x7220 bytes) 000866 1784937066.656559 fallocate 0x035a9e thru 0x03ce39 (0x739c bytes) 000867 1784937066.667999 trunc from 0x01b476 to 0x02430b (0x8e96 bytes) 000868 1784937066.699034 write 0x007665 thru 0x012668 (0xb004 bytes) 000869 1784937066.701871 read 0x0216f8 thru 0x02430b (0x2c14 bytes) 000870 1784937066.719099 trunc from 0x00e91f to 0x012803 (0x3ee5 bytes) 000871 1784937066.751651 write 0x03ea40 thru 0x03ffff (0x15c0 bytes) 000872 1784937066.765150 fallocate 0x016732 thru 0x01f51d (0x8dec bytes) 000873 1784937066.777982 fallocate 0x002db0 thru 0x00c8a1 (0x9af2 bytes) 000874 1784937066.785325 punch from 0x0244e5 to 0x02a4d8 (0x5ff4 bytes) 000875 1784937066.800060 mapread 0x01086d thru 0x012405 (0x1b99 bytes) 000876 1784937066.837204 write 0x019500 thru 0x026042 (0xcb43 bytes) 000877 1784937066.840111 mapread 0x01fdab thru 0x026502 (0x6758 bytes) 000878 1784937066.841504 fallocate 0x027096 thru 0x0349d5 (0xd940 bytes) 000879 1784937066.848485 read 0x00cad8 thru 0x00eaad (0x1fd6 bytes) 000880 1784937066.863220 write 0x03557c thru 0x03c57f (0x7004 bytes) 000881 1784937066.864694 mapread 0x00f116 thru 0x0130bf (0x3faa bytes) 000882 1784937066.866305 trunc from 0x03d07a to 0x03ffff (0x2f86 bytes) 000883 1784937066.907383 read 0x032754 thru 0x033679 (0xf26 bytes) 000884 1784937066.908215 fallocate 0x0037c4 thru 0x00bc91 (0x84ce bytes) 000885 1784937066.915183 mapwrite 0x0066c8 thru 0x011d01 (0xb63a bytes) 000886 1784937066.980941 punch from 0x01a229 to 0x0264a6 (0xc27e bytes) 000887 1784937066.996871 read 0x015b4f thru 0x0219d0 (0xbe82 bytes) 000888 1784937067.034756 trunc from 0x008512 to 0x00b32c (0x2e1b bytes) 000889 1784937067.046923 write 0x00f551 thru 0x013c44 (0x46f4 bytes) 000890 1784937067.058386 write 0x03471b thru 0x03599b (0x1281 bytes) 000891 1784937067.063156 punch from 0x027406 to 0x03671d (0xf318 bytes) 000892 1784937067.104849 trunc from 0x040000 to 0x031553 (0xeaac bytes) 000893 1784937067.140096 read 0x0235d2 thru 0x029f3a (0x6969 bytes) 000894 1784937067.148999 trunc from 0x031554 to 0x01b825 (0x15d2e bytes) 000895 1784937067.184838 write 0x00e51d thru 0x014bb3 (0x6697 bytes) 000897 1784937067.190795 write 0x032488 thru 0x033beb (0x1764 bytes) 000898 1784937067.197253 fallocate 0x033e03 thru 0x03ffff (0xc1fd bytes) 000899 1784937067.208739 mapwrite 0x02d25f thru 0x02fc3a (0x29dc bytes) 000900 1784937067.335419 trunc from 0x040000 to 0x0381a7 (0x7e58 bytes) 000901 1784937067.367100 punch from 0x01b373 to 0x01ce3e (0x1acc bytes) 000902 1784937067.397249 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381a8): Mirror 17: 0x833e246c (stale) Mirror 26: 0x833e246c (stale) Mirror 27: 0x833e246c (stale) Mirror 29: 0x833e246c (stale) Mirror 30: 0x833e246c (stale) Mirror 31: 0x833e246c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381a8): Mirror 17: 0x833e246c (stale) Mirror 26: 0x833e246c (stale) Mirror 27: 0x833e246c (stale) Mirror 29: 0x833e246c (stale) Mirror 30: 0x833e246c (stale) Mirror 31: 0x833e246c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381a8: skipped 000903 1784937068.133393 punch from 0x00e596 to 0x01022a (0x1c95 bytes) 000904 1784937068.140049 fallocate 0x017e77 thru 0x021afa (0x9c84 bytes) 000905 1784937068.147891 trunc from 0x0381a8 to 0x03f2c4 (0x711d bytes) 000906 1784937068.193079 trunc from 0x00209a to 0x00a935 (0x889c bytes) 000907 1784937068.212111 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f2c5): Mirror 17: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c5): Mirror 17: 0x2eba7d9d (stale) Mirror 26: 0x2eba7d9d (stale) Mirror 27: 0x2eba7d9d (stale) Mirror 29: 0x2eba7d9d (stale) Mirror 30: 0x2eba7d9d (stale) Mirror 31: 0x2eba7d9d (stale) CRC-32 checksum value for chunk [0, 0x3f2c5): Mirror 1: 0x6b99adb9 Mirror 32: 0x6b99adb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c5: skipped 000908 1784937069.670197 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f2c5): Mirror 17: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c5): Mirror 17: 0x2eba7d9d (stale) Mirror 26: 0x2eba7d9d (stale) Mirror 27: 0x2eba7d9d (stale) Mirror 29: 0x2eba7d9d (stale) Mirror 30: 0x2eba7d9d (stale) Mirror 31: 0x2eba7d9d (stale) CRC-32 checksum value for chunk [0, 0x3f2c5): Mirror 1: 0x6b99adb9 Mirror 32: 0x6b99adb9 Mirror 33: 0x6b99adb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c5: skipped 000909 1784937071.223150 trunc from 0x03f2c5 to 0x0112e3 (0x2dfe1 bytes) 000910 1784937071.307558 trunc from 0x00ad29 to 0x00e7eb (0x3ac3 bytes) 000911 1784937071.319921 read 0x008c0d thru 0x0112e3 (0x86d7 bytes) 000912 1784937071.360908 fallocate 0x01fd3a thru 0x028e9a (0x9161 bytes) 000913 1784937071.373155 trunc from 0x00d3ef to 0x0112e3 (0x3ef5 bytes) 000914 1784937071.392770 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x112e4): Mirror 17: 0xaa735154 (stale) Mirror 26: 0xaa735154 (stale) Mirror 27: 0xaa735154 (stale) Mirror 29: 0xaa735154 (stale) Mirror 30: 0xaa735154 (stale) Mirror 32: 0x44718e58 (stale) Mirror 33: 0x44718e58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x112e4): Mirror 17: 0xaa735154 (stale) Mirror 26: 0xaa735154 (stale) Mirror 27: 0xaa735154 (stale) Mirror 29: 0xaa735154 (stale) Mirror 30: 0xaa735154 (stale) Mirror 32: 0x44718e58 (stale) Mirror 33: 0x44718e58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x112e4: skipped 000915 1784937071.981550 mapwrite 0x023772 thru 0x02ed96 (0xb625 bytes) 000916 1784937072.104165 write 0x014772 thru 0x01ea6c (0xa2fb bytes) 000917 1784937072.133995 trunc from 0x00f073 to 0x0157b1 (0x673f bytes) 000918 1784937072.157279 mapread 0x010deb thru 0x011c7c (0xe92 bytes) 000919 1784937072.176558 trunc from 0x02ed97 to 0x003d0d (0x2b089 bytes) 000920 1784937072.198187 punch from 0x000749 to 0x003d0d (0x35c5 bytes) 000921 1784937072.211122 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d0e): Mirror 17: 0xfed3d070 (stale) Mirror 26: 0xfed3d070 (stale) Mirror 29: 0xfed3d070 (stale) Mirror 30: 0xfed3d070 (stale) Mirror 32: 0xfed3d070 (stale) Mirror 33: 0xfed3d070 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0e): Mirror 17: 0xfed3d070 (stale) Mirror 26: 0xfed3d070 (stale) Mirror 29: 0xfed3d070 (stale) Mirror 30: 0xfed3d070 (stale) Mirror 32: 0xfed3d070 (stale) Mirror 33: 0xfed3d070 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0e: skipped 000922 1784937072.740115 mapwrite 0x03cd60 thru 0x03ffff (0x32a0 bytes) 000923 1784937072.854342 punch from 0x03b5bf to 0x03eb2b (0x356d bytes) 000924 1784937072.865015 trunc from 0x007f2e to 0x01507d (0xd150 bytes) 000925 1784937072.871375 mapwrite 0x026249 thru 0x0357e8 (0xf5a0 bytes) 000926 1784937072.969948 mapread 0x0168f0 thru 0x018eba (0x25cb bytes) 000928 1784937072.990534 mapwrite 0x03d3af thru 0x03ffff (0x2c51 bytes) 000929 1784937073.016444 trunc from 0x0092f8 to 0x00e335 (0x503e bytes) 000930 1784937073.026041 punch from 0x01149d to 0x01c7d7 (0xb33b bytes) 000931 1784937073.032731 mapread 0x013fe9 thru 0x020765 (0xc77d bytes) 000932 1784937073.074430 read 0x02d149 thru 0x035690 (0x8548 bytes) 000933 1784937073.075566 trunc from 0x040000 to 0x008c12 (0x373ed bytes) 000934 1784937073.106983 write 0x03cd57 thru 0x03ed1d (0x1fc7 bytes) 000935 1784937073.108315 mapread 0x02117b thru 0x0259db (0x4861 bytes) 000937 1784937073.125414 fallocate 0x03064b thru 0x031834 (0x11ea bytes) 000938 1784937073.137678 mapwrite 0x0388c5 thru 0x03df6c (0x56a8 bytes) 000939 1784937073.184703 mapwrite 0x011bcc thru 0x018bf7 (0x702c bytes) 000940 1784937073.220272 write 0x001393 thru 0x003332 (0x1fa0 bytes) 000941 1784937073.231936 trunc from 0x03ed1e to 0x0201c8 (0x1eb55 bytes) 000942 1784937073.277584 punch from 0x014e4e to 0x01fe59 (0xb00c bytes) 000943 1784937073.286031 mapread 0x01e574 thru 0x01ec5e (0x6eb bytes) 000944 1784937073.295132 write 0x03fe82 thru 0x03ffff (0x17e bytes) 000945 1784937073.296586 punch from 0x003231 to 0x00617d (0x2f4d bytes) 000946 1784937073.319070 fallocate 0x01f0b9 thru 0x02ac17 (0xbb5f bytes) 000947 1784937073.334108 trunc from 0x019a5e to 0x01c339 (0x28dc bytes) 000948 1784937073.347391 fallocate 0x00f77c thru 0x016bf0 (0x7475 bytes) 000949 1784937073.365661 read 0x01fb5c thru 0x023c0c (0x40b1 bytes) 000950 1784937073.387377 trunc from 0x040000 to 0x010134 (0x2fecb bytes) 000951 1784937073.422541 mapread 0x00c180 thru 0x010134 (0x3fb5 bytes) 000952 1784937073.423709 mapwrite 0x038a98 thru 0x03ffff (0x7568 bytes) 000953 1784937073.568966 write 0x02a788 thru 0x037f90 (0xd809 bytes) 000954 1784937073.589238 mapwrite 0x033222 thru 0x03ab75 (0x7954 bytes) 000955 1784937073.613761 trunc from 0x00bfbc to 0x01b0e9 (0xf12e bytes) 000956 1784937073.620026 trunc from 0x040000 to 0x016e55 (0x291aa bytes) 000957 1784937073.638107 read 0x00b5c9 thru 0x016e55 (0xb88d bytes) 000958 1784937073.648576 fallocate 0x00a70e thru 0x014d05 (0xa5f8 bytes) 000959 1784937073.653701 mapwrite 0x00a65b thru 0x016c95 (0xc63b bytes) 000960 1784937073.676411 trunc from 0x00433c to 0x00c97a (0x863f bytes) 000962 1784937073.685630 fallocate 0x011c89 thru 0x01e4a7 (0xc81f bytes) 000963 1784937073.690201 trunc from 0x0124c4 to 0x019e5b (0x7998 bytes) 000964 1784937073.699689 fallocate 0x003998 thru 0x0060c8 (0x2731 bytes) 000965 1784937073.723328 write 0x038588 thru 0x03ffff (0x7a78 bytes) 000966 1784937073.729155 trunc from 0x033a13 to 0x03db04 (0xa0f2 bytes) 000967 1784937073.769473 write 0x035011 thru 0x03ffff (0xafef bytes) 000968 1784937073.796945 punch from 0x035b7c to 0x03ffff (0xa484 bytes) 000970 1784937073.825777 read 0x0180c5 thru 0x02145a (0x9396 bytes) 000971 1784937073.827364 write 0x03fc3a thru 0x03ffff (0x3c6 bytes) 000972 1784937073.851192 write 0x033fdc thru 0x036eff (0x2f24 bytes) 000973 1784937073.853541 punch from 0x021101 to 0x02e1af (0xd0af bytes) 000975 1784937073.868762 trunc from 0x01333e to 0x019890 (0x6553 bytes) 000976 1784937073.877747 mapread 0x016b67 thru 0x019126 (0x25c0 bytes) 000977 1784937073.921682 fallocate 0x0238b6 thru 0x025cb0 (0x23fb bytes) 000978 1784937073.930803 punch from 0x02593d to 0x02aaab (0x516f bytes) 000979 1784937073.946594 trunc from 0x0386ec to 0x03ffff (0x7914 bytes) 000980 1784937073.973197 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x9c9f2b74 (stale) Mirror 26: 0x9c9f2b74 (stale) Mirror 29: 0x9c9f2b74 (stale) Mirror 30: 0x9c9f2b74 (stale) Mirror 32: 0xbdb8c82e (stale) Mirror 33: 0xbdb8c82e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdc746c32 Mirror 34: 0xdc746c32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000982 1784937075.431805 mapwrite 0x029412 thru 0x036d41 (0xd930 bytes) 000983 1784937075.685975 punch from 0x036e39 to 0x03dc0b (0x6dd3 bytes) 000984 1784937075.694960 read 0x03407f thru 0x03f8f7 (0xb879 bytes) 000985 1784937075.709923 trunc from 0x02e6b2 to 0x03a962 (0xc2b1 bytes) 000986 1784937075.716208 punch from 0x019cf1 to 0x020e27 (0x7137 bytes) 000987 1784937075.724550 write 0x028a71 thru 0x036b05 (0xe095 bytes) 000988 1784937075.739657 mapwrite 0x01078a thru 0x01b899 (0xb110 bytes) 000989 1784937075.799571 mapread 0x02177e thru 0x024aea (0x336d bytes) 000990 1784937075.813053 mapread 0x01c5ab thru 0x0209ff (0x4455 bytes) 000992 1784937075.819590 mapwrite 0x01f196 thru 0x02db49 (0xe9b4 bytes) 000993 1784937075.844768 trunc from 0x02faa1 to 0x03d178 (0xd6d8 bytes) 000995 1784937075.853525 mapread 0x02907d thru 0x036de1 (0xdd65 bytes) 000996 1784937075.869375 mapwrite 0x03dce8 thru 0x03dd74 (0x8d bytes) 000997 1784937075.893797 read 0x01094f thru 0x0147ed (0x3e9f bytes) 000998 1784937075.894483 punch from 0x02496e to 0x02f1cc (0xa85f bytes) 000999 1784937075.902065 trunc from 0x02378b to 0x02c8f0 (0x9166 bytes) 001000 1784937075.910924 trunc from 0x040000 to 0x03a0d6 (0x5f29 bytes) All operations completed A-OK! PASS 70a (172s) == sanity-flr test 71: check mirror extend parent fid ==== 19:51:21 (1784937081) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x9b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 19:51:32 (1784937092) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.350063 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139302 s, 7.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x11e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 19:51:44 (1784937104) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6020640 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7904224 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8962240 .. Extending file size to 7952672 .. Extending file size to 8531552 .. done Extending file size to 9010272 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61392 61393 61396 61398 61400 done 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) 192.168.203.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:53:12 (1784937192) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8200960 .. Extending file size to 881984 .. failed Extending file size to 6306976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6918624 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7458464 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8869312 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8414592 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8910752 .. Waiting 62665 62666 62667 62670 62674 failed 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) 192.168.203.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:54:36 (1784937276) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2508 1268796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24052 3582968 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18492 3588528 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42544 7171496 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2508 1268796 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24052 3582968 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18492 3588528 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42544 7171496 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138508 s, 15.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194695 s, 10.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9be16cdf (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:54:59 (1784937299) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0852651 s, 24.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:55:08 (1784937308) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xaa:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:55:17 (1784937317) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0989969 s, 21.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:55:26 (1784937326) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161768 s, 13.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:55:36 (1784937336) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.256035 s, 8.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x211:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x211:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:56:11 (1784937371) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943573 s, 11.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x214:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x213:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x214:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x213:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:57:03 (1784937423) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x215:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110593 s, 19.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x215:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x215:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:58:02 (1784937482) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg353-server: Pool lustre.test_204f created Adding targets to pool oleg353-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg353-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160298 s, 13.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x212:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x212:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg353-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg353-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg353-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:59:22 (1784937562) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:59:29 (1784937569) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:59:37 (1784937577) oleg353-server: Pool lustre.test_206 created Adding targets to pool oleg353-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg353-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg353-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg353-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg353-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:00:02 (1784937602) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657435 s, 15.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268763 s, 39.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x91c63751 Mirror 3: 0x91c63751 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:00:12 (1784937612) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:00:15 (1784937615) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:00:18 (1784937618) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0405329 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ac0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1c50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1c50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1c50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x710000 Read this (4 bytes): BBBB 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:00:58 (1784937658) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0474991 s, 172 kB/s the very first read read with warm cache PASS 209b (11s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:01:09 (1784937669) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06575 s, 15.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc3 lmm_fid: [0x200000405:0xc3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc3 lmm_fid: [0x200000405:0xc3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x160:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611331 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-24 20:01:11.000000000 -0400 Modify: 2026-07-24 20:01:11.000000000 -0400 Change: 2026-07-24 20:01:13.000000000 -0400 Birth: 2026-07-24 20:01:12.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:01:18 (1784937678) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446851 s, 23.5 MB/s oleg353-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg353-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg353-server: O/280000401/d30/13e: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:01:40 (1784937700) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.209887 s, 195 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0322641 s, 127 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 24 20:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 24 20:01 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:01:48 (1784937708) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:01:55 (1784937715) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1945 sec =========== 20:02:03 (1784937723) === sanity-flr: start cleanup 20:02:05 (1784937725) === === sanity-flr: finish cleanup 20:02:14 (1784937734) ===