-----============= acceptance-small: sanity-flr ============----- Sat Jul 4 00:07:30 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg206-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 00:07:51 (1783138071) === oleg206-client.virtnet: executing check_config_client /mnt/lustre oleg206-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg206-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b17c7c0a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b17c7c0a000.idle_timeout=debug disable quota as required oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 00:08:42 (1783138122) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8241] == sanity-flr test 0a: lfs mirror create with -N option == 00:08:44 (1783138124) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:09:01 (1783138141) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:09:06 (1783138146) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:09:10 (1783138150) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:09:34 (1783138174) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:09:37 (1783138177) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 00:09:40 (1783138180) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b17c7c0a000.stats=clear osc.lustre-OST0001-osc-ffff8b17c7c0a000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:09:49 (1783138189) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:10:02 (1783138202) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:10:11 (1783138211) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:10:15 (1783138215) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:10:24 (1783138224) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 00:10:34 (1783138234) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:10:42 (1783138242) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:10:50 (1783138250) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:10:59 (1783138259) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:11:09 (1783138269) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:11:16 (1783138276) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.37458 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835633 s, 12.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:11:25 (1783138285) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.34277 s, 33.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:11:34 (1783138294) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:11:46 (1783138306) Stopping /mnt/lustre-ost1 (opts:) on oleg206-server oleg206-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg206-server: oleg206-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg206-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg206-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg206-server oleg206-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg206-server: oleg206-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg206-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (146s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:14:12 (1783138452) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.694824 s, 16.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg206-server oleg206-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg206-server: oleg206-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg206-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:15:02 (1783138502) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg206-server oleg206-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg206-server: oleg206-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg206-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg206-server oleg206-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg206-server: oleg206-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg206-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:16:30 (1783138590) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.40021 s, 15.4 MB/s -rw-r--r-- 1 root root 52428800 Jul 4 00:16 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.387s user 0m0.001s sys 0m0.148s Stopping /mnt/lustre-ost1 (opts:) on oleg206-server oleg206-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b17c7c0a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.065s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg206-server: oleg206-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg206-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg206-server oleg206-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.283s user 0m0.001s sys 0m0.190s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg206-server: oleg206-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg206-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg206-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b17c7c0a000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:18:10 (1783138690) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:18:13 (1783138693) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:18:17 (1783138697) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 00:18:20 (1783138700) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 00:18:27 (1783138707) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.576776 s, 21.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:18:41 (1783138721) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372546 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347876 s, 30.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:18:48 (1783138728) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:18:58 (1783138738) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00334623 s, 306 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00316248 s, 324 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138195 s, 741 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000817443 s, 1.3 MB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 00:19:10 (1783138750) create test files with size 15753567 .. 3e7690d5ffb5c60214c8bb7210f18ac8 - 2cb6c7dca04797ba9f4efca2ff2113c8 - c3dc69a2c9e4e61cefcaa126e9ccab9a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg206-client.virtnet /mnt/lustre (opts:) Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 00:19:38 (1783138778) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:19:52 (1783138792) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:19:59 (1783138799) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490032 s, 21.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350927 s, 29.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 00:20:07 (1783138807) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0746984 s, 28.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.247209 s, 17.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122076 s, 17.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 00:20:20 (1783138820) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 00:20:23 (1783138823) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:20:27 (1783138827) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368693 s, 28.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 00:20:37 (1783138837) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.251852 s, 12.5 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:20:45 (1783138845) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:20:55 (1783138855) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.346445 s, 30.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg206-client.virtnet /mnt/lustre (opts:) Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:21:05 (1783138865) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8b17c7e35800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.258875 s, 40.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg206-client.virtnet /mnt/lustre (opts:) Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8b17e1515000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:21:18 (1783138878) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:21:54 (1783138914) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:22:29 (1783138949) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 00:22:32 (1783138952) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241939 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.15358 s, 20.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3ebccca4 (nosync,stale) Mirror 2: 0x3ebccca4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4d905f5ed9782f320cd239ae9001b006 - mirror 1 checksum: d87f3d2b24f91696f6c93a22a88b3e37 - mirror 2 checksum: 4d905f5ed9782f320cd239ae9001b006 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-04T00:22:35-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-04T00:22:35-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4d905f5ed9782f320cd239ae9001b006 - mirror 1 checksum: d87f3d2b24f91696f6c93a22a88b3e37 - mirror 2 checksum: d87f3d2b24f91696f6c93a22a88b3e37 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-04T00:22:35-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-04T00:22:35-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 00:22:42 (1783138962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282448 s, 37.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:22:49 (1783138969) Failing mds1 on oleg206-server Stopping /mnt/lustre-mds1 (opts:) on oleg206-server 00:23:01 (1783138981) shut down facet: mds1 facet_host: oleg206-server facet_failover_host: oleg206-server Failover mds1 to oleg206-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 00:23:34 (1783139014) targets are mounted 00:23:34 (1783139014) facet_failover done oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331764 s, 31.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:23:50 (1783139030) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0702438 s, 292 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0384727 s, 532 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:24:00 (1783139040) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0472811 s, 433 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0103646 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:24:11 (1783139051) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00569277 s, 2.9 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:24:21 (1783139061) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0492729 s, 416 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:24:30 (1783139070) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0061651 s, 664 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0250204 s, 164 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:24:38 (1783139078) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:24:57 (1783139097) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:25:24 (1783139124) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:25:48 (1783139148) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:26:12 (1783139172) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 00:26:21 (1783139181) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57419 mirror split pid 57420 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57419 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57420 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 00:27:29 (1783139249) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 9251 skipping zero size read skipping zero length zero range fallocating to largest ever: 0x3cf05 000003 1783139252.004587 fallocate 0x036f50 thru 0x03cf04 (0x5fb5 bytes) fallocating to largest ever: 0x3eb7c 000005 1783139252.026321 fallocate 0x033f8a thru 0x03eb7b (0xabf2 bytes) 000006 1783139252.041497 fallocate 0x02324b thru 0x02838b (0x5141 bytes) 000007 1783139252.065749 fallocate 0x020ffb thru 0x024f81 (0x3f87 bytes) 000008 1783139252.074209 trunc from 0x01cd64 to 0x027cf3 (0xaf90 bytes) 000009 1783139252.081952 fallocate 0x00f282 thru 0x016037 (0x6db6 bytes) 000010 1783139252.107347 mapwrite 0x00152e thru 0x0092c5 (0x7d98 bytes) 000011 1783139252.201788 mapread 0x01e25c thru 0x0279a8 (0x974d bytes) 000013 1783139252.259627 fallocate 0x006038 thru 0x009aef (0x3ab8 bytes) 000014 1783139252.267661 trunc from 0x02838c to 0x006abf (0x218cc bytes) 000015 1783139252.299632 mapread 0x003a52 thru 0x006abf (0x306e bytes) 000016 1783139252.309216 write 0x017446 thru 0x01a7d8 (0x3393 bytes) 000017 1783139252.324473 write 0x00b91b thru 0x00f972 (0x4058 bytes) 000018 1783139252.348064 trunc from 0x00ae19 to 0x015545 (0xa72d bytes) 000019 1783139252.374255 read 0x00cbed thru 0x012c88 (0x609c bytes) 000020 1783139252.398418 trunc from 0x01a7d9 to 0x00c793 (0xe045 bytes) 000021 1783139252.431206 trunc from 0x00c794 to 0x0259fd (0x1926a bytes) 000022 1783139252.462699 read 0x01e444 thru 0x0218f3 (0x34b0 bytes) 000023 1783139252.481575 mapread 0x002820 thru 0x005ed7 (0x36b8 bytes) 000024 1783139252.489708 mapread 0x00c4fe thru 0x01a8ff (0xe402 bytes) 000025 1783139252.494553 read 0x01cc2d thru 0x0259fd (0x8dd1 bytes) 000026 1783139252.496624 mapread 0x002dbc thru 0x00d2bc (0xa501 bytes) 000028 1783139252.502444 trunc from 0x0259fe to 0x01dcff (0x7cfe bytes) 000030 1783139252.534719 punch from 0x005ec1 to 0x014a5e (0xeb9e bytes) 000031 1783139252.543444 fallocate 0x0010ff thru 0x002a6a (0x196c bytes) 000032 1783139252.556663 MIRROR_EXTEND to 2 mirrors 000033 1783139254.034303 read 0x014cd0 thru 0x01dcff (0x9030 bytes) 000034 1783139254.093758 MIRROR_RESYNC 2 mirrors 000035 1783139254.805273 MIRROR_EXTEND to 3 mirrors 000036 1783139256.059510 MIRROR_RESYNC 3 mirrors 000037 1783139256.631509 write 0x010d99 thru 0x01f1ae (0xe416 bytes) 000039 1783139256.697332 trunc from 0x01f1af to 0x017631 (0x7b7d bytes) 000040 1783139256.733898 write 0x018c90 thru 0x02313b (0xa4ac bytes) 000041 1783139256.735627 read 0x01f553 thru 0x02313b (0x3be9 bytes) 000042 1783139256.738474 mapwrite 0x004eaf thru 0x009ecc (0x501e bytes) 000043 1783139256.818858 fallocate 0x017bd1 thru 0x027836 (0xfc66 bytes) 000045 1783139256.846609 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2313c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2313c): Mirror 2: 0x5969d7cd (stale) Mirror 3: 0x5969d7cd (stale) CRC-32 checksum value for chunk [0, 0x2313c): Mirror 1: 0x780fdfc6 Mirror 4: 0x780fdfc6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2313c: skipped 000047 1783139258.439561 write 0x03acbb thru 0x03ffff (0x5345 bytes) 000048 1783139258.542401 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc3606b23 (stale) Mirror 3: 0xc3606b23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc3606b23 (stale) Mirror 3: 0xc3606b23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000049 1783139259.071650 read 0x02e2e8 thru 0x03a469 (0xc182 bytes) 000050 1783139259.093775 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc3606b23 (stale) Mirror 3: 0xc3606b23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xafc9eb5d Mirror 4: 0xafc9eb5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000051 1783139260.376442 fallocate 0x002ebd thru 0x00d67f (0xa7c3 bytes) 000052 1783139260.440423 trunc from 0x040000 to 0x00f12b (0x30ed4 bytes) 000053 1783139260.478356 MIRROR_RESYNC 4 mirrors 000054 1783139262.005656 trunc from 0x00cb04 to 0x00f12b (0x2628 bytes) 000055 1783139262.053405 trunc from 0x00f12c to 0x03d32e (0x2e203 bytes) 000056 1783139262.115297 fallocate 0x003e63 thru 0x00bcbe (0x7e5c bytes) 000057 1783139262.123593 punch from 0x021c09 to 0x0220e2 (0x4da bytes) 000058 1783139262.134374 punch from 0x01544d to 0x016f47 (0x1afb bytes) 000059 1783139262.141989 write 0x0040c4 thru 0x007362 (0x329f bytes) 000060 1783139262.163197 mapwrite 0x0268a2 thru 0x03151e (0xac7d bytes) 000061 1783139262.276781 trunc from 0x03d32f to 0x028ca4 (0x1468a bytes) 000062 1783139262.329866 read 0x01730a thru 0x019046 (0x1d3d bytes) 000063 1783139262.348606 fallocate 0x0080b9 thru 0x00ff9e (0x7ee6 bytes) 000064 1783139262.371278 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28ca5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ca5): Mirror 2: 0x278f3fac (stale) Mirror 3: 0x278f3fac (stale) Mirror 4: 0x278f3fac (stale) CRC-32 checksum value for chunk [0, 0x28ca5): Mirror 1: 0xbf83712d Mirror 5: 0xbf83712d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ca5: skipped 000065 1783139263.640007 read 0x027342 thru 0x028ca4 (0x1963 bytes) fallocating to largest ever: 0x40000 000066 1783139263.644874 fallocate 0x03980c thru 0x03ffff (0x67f4 bytes) 000067 1783139263.719483 trunc from 0x028ca5 to 0x00d036 (0x1bc6e bytes) 000068 1783139263.754651 read 0x00b6ca thru 0x00d036 (0x196d bytes) 000069 1783139263.766897 fallocate 0x0030ad thru 0x012014 (0xef68 bytes) 000070 1783139263.773103 mapread 0x0066ee thru 0x00d036 (0x6949 bytes) 000071 1783139263.774272 punch from 0x002ba8 to 0x007f52 (0x53ab bytes) 000072 1783139263.781148 mapwrite 0x02a7f5 thru 0x037a91 (0xd29d bytes) 000073 1783139263.842707 read 0x025e9e thru 0x0266c1 (0x824 bytes) 000074 1783139263.844574 write 0x00b712 thru 0x01a736 (0xf025 bytes) 000075 1783139263.854920 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37a92): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a92): Mirror 2: 0x447ee4ce (stale) Mirror 3: 0x447ee4ce (stale) Mirror 4: 0x447ee4ce (stale) Mirror 5: 0x66be5fad (stale) CRC-32 checksum value for chunk [0, 0x37a92): Mirror 1: 0x34c6246c Mirror 6: 0x34c6246c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a92: skipped 000076 1783139265.157827 write 0x01f8b3 thru 0x02ee51 (0xf59f bytes) 000077 1783139265.227982 fallocate 0x039be5 thru 0x03ffff (0x641b bytes) 000078 1783139265.237040 mapread 0x03e0a3 thru 0x03ffff (0x1f5d bytes) 000079 1783139265.250334 mapwrite 0x015d6d thru 0x01a2cb (0x455f bytes) 000080 1783139265.297548 mapwrite 0x017aec thru 0x01b554 (0x3a69 bytes) 000081 1783139265.319116 punch from 0x000540 to 0x00a86a (0xa32b bytes) 000082 1783139265.324764 trunc from 0x040000 to 0x017ca6 (0x28359 bytes) 000084 1783139265.348403 trunc from 0x017ca7 to 0x0328f5 (0x1ac4f bytes) 000086 1783139265.369601 trunc from 0x0328f6 to 0x033f0e (0x1619 bytes) 000087 1783139265.389463 mapwrite 0x030d95 thru 0x0387ad (0x7a19 bytes) 000088 1783139265.450458 write 0x00327f thru 0x006c98 (0x3a1a bytes) 000089 1783139265.464218 trunc from 0x0387ae to 0x035cc7 (0x2ae6 bytes) 000091 1783139265.491407 write 0x035b6d thru 0x03e886 (0x8d1a bytes) 000092 1783139265.494596 read 0x02efb9 thru 0x03aae9 (0xbb31 bytes) 000093 1783139265.496322 mapread 0x00ab23 thru 0x0128ae (0x7d8c bytes) 000095 1783139265.510737 mapwrite 0x00c829 thru 0x0146f2 (0x7eca bytes) 000096 1783139265.554435 trunc from 0x03e887 to 0x03af49 (0x393d bytes) 000097 1783139265.594247 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 2: 0xb220408f (stale) Mirror 3: 0xb220408f (stale) Mirror 4: 0xb220408f (stale) Mirror 5: 0xa1bdc02a (stale) Mirror 6: 0xe8568c56 (stale) CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 1: 0x6a731d2a Mirror 7: 0x6a731d2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af4a: skipped 000098 1783139266.795633 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 2: 0xb220408f (stale) Mirror 3: 0xb220408f (stale) Mirror 4: 0xb220408f (stale) Mirror 6: 0xe8568c56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 2: 0xb220408f (stale) Mirror 3: 0xb220408f (stale) Mirror 4: 0xb220408f (stale) Mirror 6: 0xe8568c56 (stale) CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 1: 0x6a731d2a Mirror 7: 0x6a731d2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af4a: skipped 000099 1783139267.640640 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 2: 0xb220408f (stale) Mirror 3: 0xb220408f (stale) Mirror 4: 0xb220408f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 2: 0xb220408f (stale) Mirror 3: 0xb220408f (stale) Mirror 4: 0xb220408f (stale) CRC-32 checksum value for chunk [0, 0x3af4a): Mirror 1: 0x6a731d2a Mirror 7: 0x6a731d2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af4a: skipped 000100 1783139268.316289 fallocate 0x00d787 thru 0x01d144 (0xf9be bytes) 000101 1783139268.419089 punch from 0x03366a to 0x03add6 (0x776d bytes) 000102 1783139268.428632 punch from 0x0272ab to 0x0339cb (0xc721 bytes) 000103 1783139268.451613 MIRROR_RESYNC 5 mirrors 000104 1783139270.250220 mapwrite 0x000af6 thru 0x0105d0 (0xfadb bytes) 000105 1783139270.511974 fallocate 0x011d76 thru 0x019807 (0x7a92 bytes) 000106 1783139270.525305 punch from 0x01d567 to 0x01f646 (0x20e0 bytes) 000107 1783139270.546272 mapwrite 0x018d4b thru 0x0236b1 (0xa967 bytes) 000108 1783139270.615489 trunc from 0x036bfe to 0x03af49 (0x434c bytes) 000109 1783139270.641260 trunc from 0x0216c1 to 0x02d9cd (0xc30d bytes) 000110 1783139270.661708 trunc from 0x03af4a to 0x02cfc6 (0xdf83 bytes) 000111 1783139270.704763 write 0x036148 thru 0x03ffff (0x9eb8 bytes) 000112 1783139270.706658 mapread 0x028301 thru 0x035729 (0xd429 bytes) 000113 1783139270.761418 write 0x01f0bd thru 0x01f5ab (0x4ef bytes) 000116 1783139270.766779 fallocate 0x026af6 thru 0x0323d6 (0xb8e1 bytes) 000117 1783139270.779138 trunc from 0x040000 to 0x027c71 (0x1838e bytes) 000118 1783139270.798494 punch from 0x01d92c to 0x01fd0a (0x23df bytes) 000119 1783139270.809154 read 0x024827 thru 0x027c71 (0x344b bytes) 000120 1783139270.826416 mapread 0x00b3df thru 0x013e52 (0x8a74 bytes) 000121 1783139270.827791 write 0x009cd2 thru 0x00f1b7 (0x54e6 bytes) 000123 1783139270.835484 read 0x012a0c thru 0x01fd00 (0xd2f5 bytes) 000124 1783139270.844026 write 0x0121a5 thru 0x01a0a2 (0x7efe bytes) 000125 1783139270.845523 read 0x0176df thru 0x019a7c (0x239e bytes) 000126 1783139270.846529 fallocate 0x032f98 thru 0x03b956 (0x89bf bytes) 000127 1783139270.853714 fallocate 0x01acef thru 0x0252da (0xa5ec bytes) 000128 1783139270.860836 fallocate 0x0385c4 thru 0x03ffff (0x7a3c bytes) 000129 1783139270.868623 mapwrite 0x00e90d thru 0x017da6 (0x949a bytes) 000130 1783139270.893157 read 0x03aa81 thru 0x03b46f (0x9ef bytes) 000131 1783139270.907058 trunc from 0x040000 to 0x0368e8 (0x9717 bytes) 000132 1783139270.932522 punch from 0x000e2f to 0x00e38f (0xd561 bytes) 000133 1783139270.940739 mapwrite 0x02369c thru 0x027ab5 (0x441a bytes) 000135 1783139270.968436 fallocate 0x020c81 thru 0x02d498 (0xc818 bytes) 000137 1783139270.976924 punch from 0x019926 to 0x023dd4 (0xa4af bytes) 000138 1783139270.984440 read 0x02535a thru 0x02bd33 (0x69da bytes) 000140 1783139270.986773 fallocate 0x00f4c0 thru 0x0168e8 (0x7429 bytes) 000141 1783139271.000240 read 0x010fdc thru 0x011dfe (0xe23 bytes) 000142 1783139271.000991 fallocate 0x01d429 thru 0x0255e4 (0x81bc bytes) 000143 1783139271.013737 write 0x02b172 thru 0x038092 (0xcf21 bytes) 000144 1783139271.021244 punch from 0x02bc78 to 0x034b78 (0x8f01 bytes) 000145 1783139271.063376 write 0x02edbb thru 0x030b27 (0x1d6d bytes) 000146 1783139271.078141 fallocate 0x01c81e thru 0x021f2d (0x5710 bytes) 000147 1783139271.086567 write 0x027ecf thru 0x0358f4 (0xda26 bytes) 000148 1783139271.088294 mapwrite 0x017182 thru 0x02641b (0xf29a bytes) 000149 1783139271.134251 punch from 0x014e4d to 0x017033 (0x21e7 bytes) 000150 1783139271.156995 MIRROR_RESYNC 5 mirrors 000151 1783139272.848589 fallocate 0x032b65 thru 0x036940 (0x3ddc bytes) 000152 1783139272.900623 mapwrite 0x03965b thru 0x03ffff (0x69a5 bytes) 000153 1783139272.961225 fallocate 0x02b33c thru 0x030228 (0x4eed bytes) 000154 1783139272.968168 mapwrite 0x03f557 thru 0x03ffff (0xaa9 bytes) 000155 1783139272.981328 punch from 0x0091b0 to 0x016e30 (0xdc81 bytes) 000157 1783139272.995199 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xae90efdb (stale) Mirror 3: 0xae90efdb (stale) Mirror 4: 0xae90efdb (stale) Mirror 7: 0xae90efdb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x28be02b4 Mirror 8: 0x28be02b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000158 1783139274.442436 read 0x00f564 thru 0x011584 (0x2021 bytes) 000159 1783139274.458281 fallocate 0x02dcbb thru 0x0395da (0xb920 bytes) 000160 1783139274.530802 punch from 0x01b739 to 0x024f76 (0x983e bytes) 000161 1783139274.539818 punch from 0x027dbb to 0x034c2b (0xce71 bytes) 000162 1783139274.545675 fallocate 0x01ea20 thru 0x029ae2 (0xb0c3 bytes) 000163 1783139274.554106 fallocate 0x014e20 thru 0x015fd3 (0x11b4 bytes) 000164 1783139274.560660 read 0x031c93 thru 0x039e8f (0x81fd bytes) 000165 1783139274.577209 trunc from 0x040000 to 0x001f40 (0x3e0bf bytes) 000167 1783139274.607758 read 0x000334 thru 0x001f40 (0x1c0d bytes) 000169 1783139274.609751 read 0x00076a thru 0x001f40 (0x17d7 bytes) 000171 1783139274.612764 trunc from 0x001f41 to 0x022c82 (0x20d42 bytes) 000172 1783139274.631674 mapread 0x00ccbf thru 0x00f917 (0x2c59 bytes) 000173 1783139274.652871 mapwrite 0x0255ba thru 0x02dacc (0x8513 bytes) 000174 1783139274.700406 mapread 0x02d36a thru 0x02dacc (0x763 bytes) 000175 1783139274.701669 trunc from 0x010a6d to 0x01e94f (0xdee3 bytes) 000176 1783139274.708550 fallocate 0x01b927 thru 0x028107 (0xc7e1 bytes) 000178 1783139274.717304 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2dacd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dacd): Mirror 2: 0xc127e5c6 (stale) Mirror 3: 0xc127e5c6 (stale) Mirror 4: 0xc127e5c6 (stale) Mirror 7: 0xc127e5c6 (stale) Mirror 8: 0x1af116a8 (stale) CRC-32 checksum value for chunk [0, 0x2dacd): Mirror 1: 0x9b5490d Mirror 9: 0x9b5490d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dacd: skipped 000179 1783139275.934699 trunc from 0x02dacd to 0x0322c9 (0x47fd bytes) 000180 1783139276.005602 trunc from 0x0322ca to 0x019205 (0x190c4 bytes) 000181 1783139276.030268 trunc from 0x006f5e to 0x007c3f (0xce2 bytes) 000182 1783139276.036223 trunc from 0x019206 to 0x0171a0 (0x2065 bytes) 000183 1783139276.055163 fallocate 0x0390d3 thru 0x03ffff (0x6f2d bytes) 000184 1783139276.061073 mapwrite 0x0388c8 thru 0x03ffff (0x7738 bytes) 000185 1783139276.134251 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xae90efdb (stale) Mirror 3: 0xae90efdb (stale) Mirror 4: 0xae90efdb (stale) Mirror 7: 0xae90efdb (stale) Mirror 8: 0x28be02b4 (stale) Mirror 9: 0x37563709 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x76078000 Mirror 10: 0x76078000 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000186 1783139277.445645 mapread 0x00ccbd thru 0x014d8b (0x80cf bytes) 000187 1783139277.491722 MIRROR_RESYNC 8 mirrors 000188 1783139279.338408 trunc from 0x040000 to 0x0020bd (0x3df42 bytes) 000189 1783139279.491997 read 0x001072 thru 0x0020bd (0x104c bytes) 000190 1783139279.500262 punch from 0x0004bf to 0x0020bd (0x1bff bytes) 000191 1783139279.523175 punch from 0x00173a to 0x0020bd (0x984 bytes) 000192 1783139279.532810 write 0x027935 thru 0x02a017 (0x26e3 bytes) 000193 1783139279.544126 fallocate 0x00a4db thru 0x018d28 (0xe84e bytes) 000194 1783139279.560080 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a018): Mirror 2: 0x22f77bc (stale) Mirror 3: 0x22f77bc (stale) Mirror 4: 0x22f77bc (stale) Mirror 7: 0x22f77bc (stale) Mirror 8: 0x22f77bc (stale) Mirror 10: 0x22f77bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a018): Mirror 2: 0x22f77bc (stale) Mirror 3: 0x22f77bc (stale) Mirror 4: 0x22f77bc (stale) Mirror 7: 0x22f77bc (stale) Mirror 8: 0x22f77bc (stale) Mirror 10: 0x22f77bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a018: skipped 000195 1783139280.593348 punch from 0x00a05f to 0x0159b8 (0xb95a bytes) 000196 1783139280.604485 read 0x00f1c2 thru 0x010351 (0x1190 bytes) 000197 1783139280.619397 write 0x00234d thru 0x003420 (0x10d4 bytes) 000198 1783139280.642867 read 0x018364 thru 0x0276c0 (0xf35d bytes) 000199 1783139280.677385 punch from 0x00cfbe to 0x016540 (0x9583 bytes) 000200 1783139280.693128 fallocate 0x016a5b thru 0x01ed00 (0x82a6 bytes) 000201 1783139280.704528 trunc from 0x02a018 to 0x0284cd (0x1b4a bytes) 000202 1783139280.736600 fallocate 0x0265bc thru 0x032836 (0xc27b bytes) 000203 1783139280.753909 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x284ce): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x284ce): Mirror 2: 0xdd8f7bea (stale) Mirror 3: 0xdd8f7bea (stale) Mirror 4: 0xdd8f7bea (stale) Mirror 7: 0xdd8f7bea (stale) Mirror 8: 0xdd8f7bea (stale) Mirror 10: 0xdd8f7bea (stale) CRC-32 checksum value for chunk [0, 0x284ce): Mirror 1: 0xdd884543 Mirror 11: 0xdd884543 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x284ce: skipped 000204 1783139282.538410 trunc from 0x0284ce to 0x013483 (0x1504a bytes) 000205 1783139282.613765 mapwrite 0x030bd4 thru 0x03ac04 (0xa031 bytes) 000207 1783139282.713497 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ac05): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac05): Mirror 2: 0x2a82d594 (stale) Mirror 3: 0x2a82d594 (stale) Mirror 4: 0x2a82d594 (stale) Mirror 7: 0x2a82d594 (stale) Mirror 8: 0x2a82d594 (stale) Mirror 10: 0x2a82d594 (stale) Mirror 11: 0x2bed3e80 (stale) CRC-32 checksum value for chunk [0, 0x3ac05): Mirror 1: 0x77913131 Mirror 12: 0x77913131 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac05: skipped 000208 1783139284.236330 mapread 0x023470 thru 0x025da7 (0x2938 bytes) 000209 1783139284.285097 mapread 0x0181fc thru 0x018c38 (0xa3d bytes) 000210 1783139284.345360 read 0x015a05 thru 0x019e2c (0x4428 bytes) 000211 1783139284.357137 mapread 0x00cbf8 thru 0x00cd0b (0x114 bytes) 000212 1783139284.396703 mapwrite 0x03f6db thru 0x03ffff (0x925 bytes) 000213 1783139284.552188 read 0x002872 thru 0x007cc1 (0x5450 bytes) 000214 1783139284.590677 MIRROR_RESYNC 9 mirrors 000215 1783139287.136089 punch from 0x0326f6 to 0x034c0a (0x2515 bytes) 000216 1783139287.198411 read 0x01db3f thru 0x028b59 (0xb01b bytes) 000217 1783139287.214275 read 0x033030 thru 0x039205 (0x61d6 bytes) 000218 1783139287.229613 mapwrite 0x03dfbc thru 0x03ffff (0x2044 bytes) 000219 1783139287.242627 trunc from 0x040000 to 0x036121 (0x9ede bytes) 000220 1783139287.266252 punch from 0x0284fd to 0x036121 (0xdc25 bytes) 000221 1783139287.273139 write 0x033741 thru 0x035831 (0x20f1 bytes) 000222 1783139287.294845 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x36122): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36122): Mirror 2: 0xbb88b820 (stale) Mirror 3: 0xbb88b820 (stale) Mirror 4: 0xbb88b820 (stale) Mirror 7: 0xbb88b820 (stale) Mirror 8: 0xbb88b820 (stale) Mirror 10: 0xbb88b820 (stale) Mirror 11: 0xbb88b820 (stale) Mirror 12: 0xbb88b820 (stale) CRC-32 checksum value for chunk [0, 0x36122): Mirror 1: 0x3a831b72 Mirror 13: 0x3a831b72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36122: skipped 000223 1783139288.206631 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36122): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36122): Mirror 2: 0xbb88b820 (stale) Mirror 3: 0xbb88b820 (stale) Mirror 4: 0xbb88b820 (stale) Mirror 7: 0xbb88b820 (stale) Mirror 8: 0xbb88b820 (stale) Mirror 10: 0xbb88b820 (stale) Mirror 11: 0xbb88b820 (stale) Mirror 12: 0xbb88b820 (stale) CRC-32 checksum value for chunk [0, 0x36122): Mirror 1: 0x3a831b72 Mirror 13: 0x3a831b72 Mirror 14: 0x3a831b72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36122: skipped 000224 1783139289.442063 read 0x015559 thru 0x01a12c (0x4bd4 bytes) 000225 1783139289.469066 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36122): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36122): Mirror 2: 0xbb88b820 (stale) Mirror 3: 0xbb88b820 (stale) Mirror 4: 0xbb88b820 (stale) Mirror 7: 0xbb88b820 (stale) Mirror 8: 0xbb88b820 (stale) Mirror 10: 0xbb88b820 (stale) Mirror 11: 0xbb88b820 (stale) Mirror 12: 0xbb88b820 (stale) CRC-32 checksum value for chunk [0, 0x36122): Mirror 1: 0x3a831b72 Mirror 13: 0x3a831b72 Mirror 14: 0x3a831b72 Mirror 15: 0x3a831b72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36122: skipped 000226 1783139290.931828 write 0x03fee5 thru 0x03ffff (0x11b bytes) 000227 1783139290.999371 trunc from 0x040000 to 0x00155c (0x3eaa3 bytes) 000228 1783139291.084376 fallocate 0x0061e5 thru 0x00bc42 (0x5a5e bytes) 000229 1783139291.091346 read 0x000faf thru 0x00155c (0x5ae bytes) 000230 1783139291.098105 mapread 0x00101c thru 0x00155c (0x541 bytes) 000231 1783139291.099997 trunc from 0x00155d to 0x02224e (0x20cf2 bytes) 000233 1783139291.120053 trunc from 0x00a417 to 0x017e2c (0xda16 bytes) 000234 1783139291.125118 trunc from 0x01a809 to 0x01e9aa (0x41a2 bytes) 000235 1783139291.136089 MIRROR_RESYNC 12 mirrors 000236 1783139293.237012 fallocate 0x03de4e thru 0x03ffff (0x21b2 bytes) 000237 1783139293.283424 MIRROR_RESYNC 12 mirrors 000238 1783139295.068813 mapread 0x00786c thru 0x00f19d (0x7932 bytes) 000239 1783139295.105362 MIRROR_EXTEND to 13 mirrors 000240 1783139296.191449 trunc from 0x010bc7 to 0x01f896 (0xecd0 bytes) 000241 1783139296.278839 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2224f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2224f): Mirror 2: 0xe537b1ba (stale) Mirror 3: 0xe537b1ba (stale) Mirror 4: 0xe537b1ba (stale) Mirror 7: 0xe537b1ba (stale) Mirror 8: 0xe537b1ba (stale) Mirror 10: 0xe537b1ba (stale) Mirror 11: 0xe537b1ba (stale) Mirror 12: 0xe537b1ba (stale) Mirror 13: 0xe537b1ba (stale) Mirror 14: 0xe537b1ba (stale) Mirror 15: 0xe537b1ba (stale) Mirror 16: 0xe537b1ba (stale) CRC-32 checksum value for chunk [0, 0x2224f): Mirror 1: 0xe537b1ba Mirror 17: 0xe537b1ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2224f: skipped 000242 1783139297.650110 fallocate 0x03f061 thru 0x03ffff (0xf9f bytes) 000243 1783139297.758040 write 0x03fb6c thru 0x03ffff (0x494 bytes) 000244 1783139297.789526 mapwrite 0x03159e thru 0x037af6 (0x6559 bytes) 000245 1783139297.894738 punch from 0x00b1c5 to 0x01957e (0xe3ba bytes) 000246 1783139297.904492 read 0x039c27 thru 0x03ffff (0x63d9 bytes) 000247 1783139297.927969 write 0x03e1b8 thru 0x03ffff (0x1e48 bytes) 000248 1783139297.952435 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7a88ab4e Mirror 18: 0x7a88ab4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000249 1783139299.125900 mapwrite 0x01a0ae thru 0x0207b6 (0x6709 bytes) 000250 1783139299.291374 MIRROR_SPLIT mirror 9 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 18: 0x7a88ab4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 18: 0x7a88ab4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000251 1783139300.174541 write 0x0266a5 thru 0x0363f8 (0xfd54 bytes) 000253 1783139300.204721 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 18: 0x7a88ab4e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x411d6a25 Mirror 19: 0x411d6a25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000254 1783139301.979411 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 18: 0x7a88ab4e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x411d6a25 Mirror 19: 0x411d6a25 Mirror 20: 0x411d6a25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1783139303.338492 mapwrite 0x010f21 thru 0x01bb2d (0xac0d bytes) 000256 1783139303.638235 trunc from 0x023198 to 0x0303ff (0xd268 bytes) 000257 1783139303.652177 mapwrite 0x0284b4 thru 0x0346cf (0xc21c bytes) 000258 1783139303.754466 mapread 0x01981a thru 0x025616 (0xbdfd bytes) 000259 1783139303.757232 read 0x02b54b thru 0x037b54 (0xc60a bytes) 000260 1783139303.758364 write 0x0152e7 thru 0x01f85f (0xa579 bytes) 000262 1783139303.766338 mapwrite 0x033eea thru 0x03e499 (0xa5b0 bytes) 000263 1783139303.815919 punch from 0x037ca3 to 0x03ffff (0x835d bytes) 000264 1783139303.823488 fallocate 0x001058 thru 0x0044b3 (0x345c bytes) 000265 1783139303.830199 fallocate 0x0199b4 thru 0x02984b (0xfe98 bytes) 000268 1783139303.842826 punch from 0x005c5c to 0x0155ca (0xf96f bytes) 000269 1783139303.848811 mapwrite 0x020b53 thru 0x0278d8 (0x6d86 bytes) 000270 1783139303.892694 read 0x02c1a3 thru 0x03907e (0xcedc bytes) 000272 1783139303.917325 punch from 0x02e4b9 to 0x0333cf (0x4f17 bytes) 000273 1783139303.935327 trunc from 0x040000 to 0x019a4f (0x265b0 bytes) 000274 1783139303.994232 write 0x0167c3 thru 0x0178b4 (0x10f2 bytes) 000275 1783139303.995516 trunc from 0x019a50 to 0x03fccc (0x2627d bytes) 000276 1783139304.022393 punch from 0x0111ef to 0x0162d0 (0x50e2 bytes) 000277 1783139304.041547 read 0x013d2e thru 0x02362e (0xf901 bytes) 000278 1783139304.065468 write 0x02f36e thru 0x03b152 (0xbde5 bytes) 000279 1783139304.067673 write 0x007b6a thru 0x00926f (0x1706 bytes) 000280 1783139304.077283 write 0x039bf8 thru 0x03ffff (0x6408 bytes) 000281 1783139304.085757 write 0x0072d4 thru 0x00d6ae (0x63db bytes) 000282 1783139304.087376 mapread 0x022398 thru 0x02a7b1 (0x841a bytes) 000283 1783139304.088704 trunc from 0x02e7d1 to 0x0373ab (0x8bdb bytes) 000284 1783139304.139401 read 0x00c2d6 thru 0x00d9c1 (0x16ec bytes) 000285 1783139304.149066 mapwrite 0x00806b thru 0x013cdf (0xbc75 bytes) 000287 1783139304.190672 trunc from 0x040000 to 0x02d30e (0x12cf1 bytes) 000288 1783139304.225741 trunc from 0x02d30f to 0x0241c9 (0x9145 bytes) 000290 1783139304.275914 write 0x01f12c thru 0x0286f4 (0x95c9 bytes) 000291 1783139304.283160 fallocate 0x015cff thru 0x023643 (0xd945 bytes) 000292 1783139304.295987 mapwrite 0x03ab3f thru 0x03e285 (0x3747 bytes) 000293 1783139304.430184 mapread 0x0117f1 thru 0x01ac64 (0x9474 bytes) 000294 1783139304.436243 mapread 0x034b0e thru 0x036884 (0x1d77 bytes) 000295 1783139304.438380 punch from 0x01ff65 to 0x020de9 (0xe85 bytes) 000297 1783139304.448257 punch from 0x029357 to 0x031e44 (0x8aee bytes) 000298 1783139304.456419 write 0x021c08 thru 0x0278eb (0x5ce4 bytes) 000299 1783139304.459720 mapread 0x023415 thru 0x025444 (0x2030 bytes) 000300 1783139304.463482 punch from 0x010459 to 0x016109 (0x5cb1 bytes) 000301 1783139304.474530 mapread 0x03b3b8 thru 0x03e285 (0x2ece bytes) 000302 1783139304.478330 mapread 0x030da5 thru 0x03489b (0x3af7 bytes) 000303 1783139304.492124 fallocate 0x03e936 thru 0x03ffff (0x16ca bytes) 000304 1783139304.500349 mapwrite 0x02a9b6 thru 0x035c85 (0xb2d0 bytes) 000305 1783139304.609081 MIRROR_RESYNC 16 mirrors 000306 1783139310.472491 read 0x01f37a thru 0x02ecd1 (0xf958 bytes) 000307 1783139310.487905 read 0x016532 thru 0x021158 (0xac27 bytes) 000308 1783139310.500750 mapwrite 0x02bf2b thru 0x030568 (0x463e bytes) 000309 1783139310.670435 write 0x011838 thru 0x021259 (0xfa22 bytes) 000310 1783139310.692092 punch from 0x03cb77 to 0x03d327 (0x7b1 bytes) 000311 1783139310.700376 write 0x02cef1 thru 0x02e1ba (0x12ca bytes) 000312 1783139310.710712 MIRROR_RESYNC 16 mirrors 000313 1783139315.523363 punch from 0x005769 to 0x009a06 (0x429e bytes) 000314 1783139315.600978 mapwrite 0x00eca1 thru 0x01da0c (0xed6c bytes) 000315 1783139315.702832 trunc from 0x03e286 to 0x012c2c (0x2b659 bytes) 000316 1783139315.748963 fallocate 0x027416 thru 0x0372a9 (0xfe94 bytes) 000317 1783139315.760782 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12c2d): Mirror 2: 0x93c67983 (stale) Mirror 3: 0x93c67983 (stale) Mirror 4: 0x93c67983 (stale) Mirror 7: 0x93c67983 (stale) Mirror 8: 0x93c67983 (stale) Mirror 10: 0x93c67983 (stale) Mirror 11: 0x93c67983 (stale) Mirror 14: 0x93c67983 (stale) Mirror 15: 0x93c67983 (stale) Mirror 16: 0x93c67983 (stale) Mirror 17: 0x93c67983 (stale) Mirror 18: 0x93c67983 (stale) Mirror 19: 0x93c67983 (stale) Mirror 20: 0x93c67983 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12c2d): Mirror 2: 0x93c67983 (stale) Mirror 3: 0x93c67983 (stale) Mirror 4: 0x93c67983 (stale) Mirror 7: 0x93c67983 (stale) Mirror 8: 0x93c67983 (stale) Mirror 10: 0x93c67983 (stale) Mirror 11: 0x93c67983 (stale) Mirror 14: 0x93c67983 (stale) Mirror 15: 0x93c67983 (stale) Mirror 16: 0x93c67983 (stale) Mirror 17: 0x93c67983 (stale) Mirror 18: 0x93c67983 (stale) Mirror 19: 0x93c67983 (stale) Mirror 20: 0x93c67983 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12c2d: skipped 000318 1783139316.725968 write 0x034542 thru 0x03b87e (0x733d bytes) 000319 1783139316.732739 write 0x00a7c5 thru 0x0175b9 (0xcdf5 bytes) 000320 1783139316.746375 punch from 0x020ee1 to 0x02e77c (0xd89c bytes) 000321 1783139316.753244 read 0x023046 thru 0x03300a (0xffc5 bytes) 000322 1783139316.767255 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b87f): Mirror 2: 0xbea1dbba (stale) Mirror 3: 0xbea1dbba (stale) Mirror 4: 0xbea1dbba (stale) Mirror 7: 0xbea1dbba (stale) Mirror 8: 0xbea1dbba (stale) Mirror 10: 0xbea1dbba (stale) Mirror 14: 0xbea1dbba (stale) Mirror 15: 0xbea1dbba (stale) Mirror 16: 0xbea1dbba (stale) Mirror 17: 0xbea1dbba (stale) Mirror 18: 0xbea1dbba (stale) Mirror 19: 0xbea1dbba (stale) Mirror 20: 0xbea1dbba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b87f): Mirror 2: 0xbea1dbba (stale) Mirror 3: 0xbea1dbba (stale) Mirror 4: 0xbea1dbba (stale) Mirror 7: 0xbea1dbba (stale) Mirror 8: 0xbea1dbba (stale) Mirror 10: 0xbea1dbba (stale) Mirror 14: 0xbea1dbba (stale) Mirror 15: 0xbea1dbba (stale) Mirror 16: 0xbea1dbba (stale) Mirror 17: 0xbea1dbba (stale) Mirror 18: 0xbea1dbba (stale) Mirror 19: 0xbea1dbba (stale) Mirror 20: 0xbea1dbba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b87f: skipped 000323 1783139317.694542 trunc from 0x037acc to 0x03b87e (0x3db3 bytes) 000324 1783139317.704740 mapread 0x0060c0 thru 0x00cd2f (0x6c70 bytes) 000326 1783139317.729580 write 0x016710 thru 0x0231af (0xcaa0 bytes) 000327 1783139317.753421 read 0x005da3 thru 0x015234 (0xf492 bytes) 000328 1783139317.762898 write 0x001145 thru 0x00f096 (0xdf52 bytes) 000329 1783139317.765281 read 0x0193d2 thru 0x01f66d (0x629c bytes) 000330 1783139317.774646 MIRROR_RESYNC 14 mirrors 000332 1783139320.853286 trunc from 0x03b87f to 0x004b8d (0x36cf1 bytes) 000333 1783139320.970665 read 0x00406d thru 0x004b8d (0xb21 bytes) 000334 1783139320.981120 read 0x000470 thru 0x004b8d (0x471e bytes) 000335 1783139321.004884 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x4b8e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4b8e): Mirror 2: 0x98b3fc41 (stale) Mirror 3: 0x98b3fc41 (stale) Mirror 4: 0x98b3fc41 (stale) Mirror 7: 0x98b3fc41 (stale) Mirror 8: 0x98b3fc41 (stale) Mirror 10: 0x98b3fc41 (stale) Mirror 14: 0x98b3fc41 (stale) Mirror 15: 0x98b3fc41 (stale) Mirror 16: 0x98b3fc41 (stale) Mirror 17: 0x98b3fc41 (stale) Mirror 18: 0x98b3fc41 (stale) Mirror 19: 0x98b3fc41 (stale) Mirror 20: 0x98b3fc41 (stale) CRC-32 checksum value for chunk [0, 0x4b8e): Mirror 1: 0x98b3fc41 Mirror 21: 0x98b3fc41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4b8e: skipped 000336 1783139321.988273 write 0x00f3cb thru 0x011398 (0x1fce bytes) 000337 1783139322.035761 punch from 0x00b575 to 0x011398 (0x5e24 bytes) 000338 1783139322.050319 punch from 0x0019c2 to 0x010887 (0xeec6 bytes) 000339 1783139322.058852 trunc from 0x011399 to 0x03de7d (0x2cae5 bytes) 000340 1783139322.090004 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3de7e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de7e): Mirror 2: 0x116898d9 (stale) Mirror 3: 0x116898d9 (stale) Mirror 4: 0x116898d9 (stale) Mirror 7: 0x116898d9 (stale) Mirror 8: 0x116898d9 (stale) Mirror 10: 0x116898d9 (stale) Mirror 14: 0x116898d9 (stale) Mirror 15: 0x116898d9 (stale) Mirror 16: 0x116898d9 (stale) Mirror 17: 0x116898d9 (stale) Mirror 18: 0x116898d9 (stale) Mirror 19: 0x116898d9 (stale) Mirror 20: 0x116898d9 (stale) Mirror 21: 0x28159aba (stale) CRC-32 checksum value for chunk [0, 0x3de7e): Mirror 1: 0xd673401c Mirror 22: 0xd673401c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de7e: skipped 000342 1783139323.404798 trunc from 0x03de7e to 0x03208c (0xbdf1 bytes) 000343 1783139323.475310 write 0x01adf7 thru 0x01c498 (0x16a2 bytes) 000345 1783139323.483854 punch from 0x009da3 to 0x00bed4 (0x2132 bytes) 000346 1783139323.488088 punch from 0x016885 to 0x023e5f (0xd5db bytes) 000347 1783139323.501236 write 0x020b0b thru 0x030b05 (0xfffb bytes) 000348 1783139323.511072 trunc from 0x03208d to 0x03072f (0x195d bytes) 000349 1783139323.532231 fallocate 0x00e4b7 thru 0x019c0f (0xb759 bytes) 000350 1783139323.538409 trunc from 0x030730 to 0x03907d (0x894e bytes) 000351 1783139323.557902 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3907e): Mirror 3: 0x886eac5e (stale) Mirror 4: 0x886eac5e (stale) Mirror 7: 0x886eac5e (stale) Mirror 8: 0x886eac5e (stale) Mirror 10: 0x886eac5e (stale) Mirror 14: 0x886eac5e (stale) Mirror 15: 0x886eac5e (stale) Mirror 16: 0x886eac5e (stale) Mirror 17: 0x886eac5e (stale) Mirror 18: 0x886eac5e (stale) Mirror 19: 0x886eac5e (stale) Mirror 20: 0x886eac5e (stale) Mirror 21: 0x5e90e1ac (stale) Mirror 22: 0x9df6c3c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3907e): Mirror 3: 0x886eac5e (stale) Mirror 4: 0x886eac5e (stale) Mirror 7: 0x886eac5e (stale) Mirror 8: 0x886eac5e (stale) Mirror 10: 0x886eac5e (stale) Mirror 14: 0x886eac5e (stale) Mirror 15: 0x886eac5e (stale) Mirror 16: 0x886eac5e (stale) Mirror 17: 0x886eac5e (stale) Mirror 18: 0x886eac5e (stale) Mirror 19: 0x886eac5e (stale) Mirror 20: 0x886eac5e (stale) Mirror 21: 0x5e90e1ac (stale) Mirror 22: 0x9df6c3c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3907e: skipped 000352 1783139324.345979 read 0x0043a0 thru 0x00e4a7 (0xa108 bytes) 000353 1783139324.381814 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3907e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3907e): Mirror 3: 0x886eac5e (stale) Mirror 4: 0x886eac5e (stale) Mirror 7: 0x886eac5e (stale) Mirror 8: 0x886eac5e (stale) Mirror 10: 0x886eac5e (stale) Mirror 14: 0x886eac5e (stale) Mirror 15: 0x886eac5e (stale) Mirror 16: 0x886eac5e (stale) Mirror 17: 0x886eac5e (stale) Mirror 18: 0x886eac5e (stale) Mirror 19: 0x886eac5e (stale) Mirror 20: 0x886eac5e (stale) Mirror 21: 0x5e90e1ac (stale) Mirror 22: 0x9df6c3c7 (stale) CRC-32 checksum value for chunk [0, 0x3907e): Mirror 1: 0x1242d0a8 Mirror 23: 0x1242d0a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3907e: skipped 000354 1783139325.726726 trunc from 0x03907e to 0x0131ad (0x25ed0 bytes) 000355 1783139325.897155 punch from 0x0073bd to 0x0131ad (0xbdf1 bytes) 000356 1783139325.905182 trunc from 0x0131ae to 0x0237e0 (0x10633 bytes) 000357 1783139325.942023 write 0x03301f thru 0x03eb21 (0xbb03 bytes) 000358 1783139325.951684 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb22): Mirror 3: 0x8d5ac272 (stale) Mirror 4: 0x8d5ac272 (stale) Mirror 7: 0x8d5ac272 (stale) Mirror 8: 0x8d5ac272 (stale) Mirror 10: 0x8d5ac272 (stale) Mirror 14: 0x8d5ac272 (stale) Mirror 15: 0x8d5ac272 (stale) Mirror 16: 0x8d5ac272 (stale) Mirror 18: 0x8d5ac272 (stale) Mirror 19: 0x8d5ac272 (stale) Mirror 20: 0x8d5ac272 (stale) Mirror 21: 0x55db2a6e (stale) Mirror 22: 0xe8487480 (stale) Mirror 23: 0x46e08021 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb22): Mirror 3: 0x8d5ac272 (stale) Mirror 4: 0x8d5ac272 (stale) Mirror 7: 0x8d5ac272 (stale) Mirror 8: 0x8d5ac272 (stale) Mirror 10: 0x8d5ac272 (stale) Mirror 14: 0x8d5ac272 (stale) Mirror 15: 0x8d5ac272 (stale) Mirror 16: 0x8d5ac272 (stale) Mirror 18: 0x8d5ac272 (stale) Mirror 19: 0x8d5ac272 (stale) Mirror 20: 0x8d5ac272 (stale) Mirror 21: 0x55db2a6e (stale) Mirror 22: 0xe8487480 (stale) Mirror 23: 0x46e08021 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb22: skipped 000359 1783139326.772608 mapwrite 0x02cc57 thru 0x03839d (0xb747 bytes) 000360 1783139326.833983 mapwrite 0x0322f3 thru 0x03ffff (0xdd0d bytes) 000361 1783139326.887829 trunc from 0x040000 to 0x02efe1 (0x1101e bytes) 000363 1783139326.909568 punch from 0x022b6c to 0x02cefd (0xa392 bytes) 000364 1783139326.916214 punch from 0x015c26 to 0x023ac6 (0xdea1 bytes) 000365 1783139326.921423 punch from 0x01854a to 0x01aa08 (0x24bf bytes) 000366 1783139326.927040 trunc from 0x02efe2 to 0x01352b (0x1bab6 bytes) 000367 1783139326.942776 punch from 0x003b30 to 0x0114be (0xd98f bytes) 000368 1783139326.947276 trunc from 0x009579 to 0x01352b (0x9fb3 bytes) 000369 1783139326.952165 trunc from 0x00fad0 to 0x01352b (0x3a5c bytes) 000370 1783139326.955847 mapread 0x01111a thru 0x01352b (0x2412 bytes) 000371 1783139326.963413 mapread 0x00ae32 thru 0x01352b (0x86fa bytes) 000372 1783139326.972726 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1352c): Mirror 3: 0xf4759a52 (stale) Mirror 4: 0xf4759a52 (stale) Mirror 7: 0xf4759a52 (stale) Mirror 10: 0xf4759a52 (stale) Mirror 14: 0xf4759a52 (stale) Mirror 15: 0xf4759a52 (stale) Mirror 16: 0xf4759a52 (stale) Mirror 18: 0xf4759a52 (stale) Mirror 19: 0xf4759a52 (stale) Mirror 20: 0xf4759a52 (stale) Mirror 21: 0x84d477d2 (stale) Mirror 22: 0x5b2a705f (stale) Mirror 23: 0x5b2a705f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1352c): Mirror 3: 0xf4759a52 (stale) Mirror 4: 0xf4759a52 (stale) Mirror 7: 0xf4759a52 (stale) Mirror 10: 0xf4759a52 (stale) Mirror 14: 0xf4759a52 (stale) Mirror 15: 0xf4759a52 (stale) Mirror 16: 0xf4759a52 (stale) Mirror 18: 0xf4759a52 (stale) Mirror 19: 0xf4759a52 (stale) Mirror 20: 0xf4759a52 (stale) Mirror 21: 0x84d477d2 (stale) Mirror 22: 0x5b2a705f (stale) Mirror 23: 0x5b2a705f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1352c: skipped 000373 1783139327.850287 fallocate 0x035984 thru 0x038f65 (0x35e2 bytes) 000374 1783139327.858903 trunc from 0x038f66 to 0x00f2db (0x29c8a bytes) 000375 1783139327.905814 MIRROR_RESYNC 14 mirrors 000377 1783139330.263491 trunc from 0x00f2dc to 0x018026 (0x8d4b bytes) 000379 1783139330.313199 trunc from 0x002d56 to 0x0071d2 (0x447d bytes) 000380 1783139330.319467 mapread 0x00dea4 thru 0x0107a9 (0x2906 bytes) 000381 1783139330.341466 read 0x0148ea thru 0x018026 (0x373d bytes) 000382 1783139330.349268 fallocate 0x00619e thru 0x006518 (0x37b bytes) 000383 1783139330.356172 write 0x00ff33 thru 0x01f131 (0xf1ff bytes) 000384 1783139330.368535 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f132): Mirror 3: 0xf2b40ab5 (stale) Mirror 4: 0xf2b40ab5 (stale) Mirror 7: 0xf2b40ab5 (stale) Mirror 10: 0xf2b40ab5 (stale) Mirror 14: 0xf2b40ab5 (stale) Mirror 15: 0xf2b40ab5 (stale) Mirror 16: 0xf2b40ab5 (stale) Mirror 18: 0xf2b40ab5 (stale) Mirror 19: 0xf2b40ab5 (stale) Mirror 21: 0xf2b40ab5 (stale) Mirror 22: 0xf2b40ab5 (stale) Mirror 23: 0xf2b40ab5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f132): Mirror 3: 0xf2b40ab5 (stale) Mirror 4: 0xf2b40ab5 (stale) Mirror 7: 0xf2b40ab5 (stale) Mirror 10: 0xf2b40ab5 (stale) Mirror 14: 0xf2b40ab5 (stale) Mirror 15: 0xf2b40ab5 (stale) Mirror 16: 0xf2b40ab5 (stale) Mirror 18: 0xf2b40ab5 (stale) Mirror 19: 0xf2b40ab5 (stale) Mirror 21: 0xf2b40ab5 (stale) Mirror 22: 0xf2b40ab5 (stale) Mirror 23: 0xf2b40ab5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f132: skipped 000385 1783139331.045916 write 0x0110fc thru 0x011b5d (0xa62 bytes) 000386 1783139331.061155 punch from 0x001091 to 0x004ea7 (0x3e17 bytes) 000387 1783139331.068827 read 0x010b73 thru 0x013095 (0x2523 bytes) 000389 1783139331.081541 punch from 0x007da6 to 0x012bcb (0xae26 bytes) 000391 1783139331.106743 mapread 0x01e4f3 thru 0x01f131 (0xc3f bytes) 000392 1783139331.123539 trunc from 0x004d05 to 0x005642 (0x93e bytes) 000393 1783139331.130640 read 0x00c898 thru 0x010a58 (0x41c1 bytes) 000394 1783139331.146090 mapread 0x01c5f6 thru 0x01f131 (0x2b3c bytes) 000395 1783139331.147187 trunc from 0x01f132 to 0x02caa3 (0xd972 bytes) 000396 1783139331.176790 mapread 0x012fda thru 0x01e940 (0xb967 bytes) 000397 1783139331.182251 trunc from 0x02caa4 to 0x010f34 (0x1bb6f bytes) 000398 1783139331.213349 punch from 0x00ce98 to 0x010a51 (0x3bba bytes) 000399 1783139331.220727 mapwrite 0x012396 thru 0x014ae8 (0x2753 bytes) 000400 1783139331.294453 mapwrite 0x005e1a thru 0x0107e7 (0xa9ce bytes) 000401 1783139331.352280 trunc from 0x014ae9 to 0x022e7a (0xe392 bytes) 000402 1783139331.383122 mapread 0x014c65 thru 0x016202 (0x159e bytes) 000403 1783139331.403145 trunc from 0x020374 to 0x022e7a (0x2b07 bytes) 000404 1783139331.411676 fallocate 0x01c24c thru 0x0264d9 (0xa28e bytes) 000405 1783139331.419157 fallocate 0x03d1fa thru 0x03e96a (0x1771 bytes) 000406 1783139331.427120 write 0x028f8e thru 0x0359d0 (0xca43 bytes) 000407 1783139331.443301 fallocate 0x023c71 thru 0x029873 (0x5c03 bytes) 000409 1783139331.458563 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e96b): Mirror 3: 0x58175870 (stale) Mirror 4: 0x58175870 (stale) Mirror 7: 0x58175870 (stale) Mirror 14: 0x58175870 (stale) Mirror 15: 0x58175870 (stale) Mirror 16: 0x58175870 (stale) Mirror 18: 0x58175870 (stale) Mirror 19: 0x58175870 (stale) Mirror 21: 0x58175870 (stale) Mirror 22: 0x58175870 (stale) Mirror 23: 0x58175870 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e96b): Mirror 3: 0x58175870 (stale) Mirror 4: 0x58175870 (stale) Mirror 7: 0x58175870 (stale) Mirror 14: 0x58175870 (stale) Mirror 15: 0x58175870 (stale) Mirror 16: 0x58175870 (stale) Mirror 18: 0x58175870 (stale) Mirror 19: 0x58175870 (stale) Mirror 21: 0x58175870 (stale) Mirror 22: 0x58175870 (stale) Mirror 23: 0x58175870 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e96b: skipped 000410 1783139332.275018 mapread 0x0363ef thru 0x03ac97 (0x48a9 bytes) 000411 1783139332.315264 fallocate 0x01b12a thru 0x028ed7 (0xddae bytes) 000412 1783139332.323220 trunc from 0x03e96b to 0x02c0a4 (0x128c6 bytes) 000413 1783139332.356234 read 0x004b91 thru 0x0112e9 (0xc759 bytes) 000414 1783139332.376575 trunc from 0x02c0a5 to 0x032bc4 (0x6b20 bytes) 000415 1783139332.399646 fallocate 0x02fac0 thru 0x035e8a (0x63cb bytes) 000416 1783139332.410819 mapread 0x00cce2 thru 0x00fa97 (0x2db6 bytes) 000417 1783139332.414332 write 0x03ac26 thru 0x03b551 (0x92c bytes) 000418 1783139332.419009 trunc from 0x033fc3 to 0x03b551 (0x758f bytes) 000419 1783139332.453665 MIRROR_RESYNC 12 mirrors 000420 1783139337.097602 write 0x021cc3 thru 0x02dcf9 (0xc037 bytes) 000421 1783139337.218100 mapread 0x012913 thru 0x020601 (0xdcef bytes) 000422 1783139337.280763 write 0x00db74 thru 0x016cde (0x916b bytes) 000423 1783139337.285681 fallocate 0x00cc0f thru 0x00f5d1 (0x29c3 bytes) 000424 1783139337.298502 read 0x010e2f thru 0x0137bb (0x298d bytes) 000425 1783139337.300220 trunc from 0x02717e to 0x03686d (0xf6f0 bytes) 000426 1783139337.349523 mapread 0x02498a thru 0x02f862 (0xaed9 bytes) 000427 1783139337.378873 mapwrite 0x02bfe6 thru 0x037646 (0xb661 bytes) 000428 1783139337.443083 punch from 0x024fd2 to 0x02fdfa (0xae29 bytes) 000429 1783139337.453052 mapread 0x023630 thru 0x0282a4 (0x4c75 bytes) 000430 1783139337.468839 mapread 0x019e60 thru 0x01bab5 (0x1c56 bytes) 000431 1783139337.473070 write 0x0208db thru 0x027d76 (0x749c bytes) 000433 1783139337.476766 punch from 0x02e5d4 to 0x0346ff (0x612c bytes) 000434 1783139337.491583 write 0x019c88 thru 0x01f673 (0x59ec bytes) 000435 1783139337.493260 trunc from 0x03b552 to 0x02ac5a (0x108f7 bytes) 000436 1783139337.575242 fallocate 0x01d212 thru 0x0201b4 (0x2fa3 bytes) 000437 1783139337.586535 trunc from 0x02ac5b to 0x01d206 (0xda54 bytes) 000438 1783139337.622549 trunc from 0x01172b to 0x018638 (0x6f0e bytes) 000439 1783139337.638783 trunc from 0x01d207 to 0x02d339 (0x10133 bytes) 000440 1783139337.685913 fallocate 0x031784 thru 0x03cde1 (0xb65e bytes) 000441 1783139337.697248 punch from 0x022b61 to 0x030c8f (0xe12f bytes) 000442 1783139337.738170 MIRROR_RESYNC 12 mirrors 000443 1783139341.336884 MIRROR_EXTEND to 13 mirrors 000444 1783139342.500697 mapread 0x01d362 thru 0x01dfb2 (0xc51 bytes) 000445 1783139342.506640 mapwrite 0x02f80e thru 0x035969 (0x615c bytes) 000446 1783139342.670658 read 0x005b42 thru 0x010e76 (0xb335 bytes) 000447 1783139342.693915 punch from 0x03cd5c to 0x03cde1 (0x86 bytes) 000449 1783139342.708607 trunc from 0x03cde2 to 0x004013 (0x38dce bytes) 000450 1783139342.763561 mapwrite 0x015a7c thru 0x01d306 (0x788b bytes) 000451 1783139342.840250 mapwrite 0x0177dd thru 0x01f934 (0x8158 bytes) 000452 1783139342.909022 trunc from 0x01f935 to 0x003af7 (0x1be3d bytes) 000454 1783139342.946634 read 0x000f20 thru 0x00223c (0x131d bytes) 000455 1783139342.957073 read 0x001b0c thru 0x003af7 (0x1fec bytes) 000456 1783139342.960488 punch from 0x000c5b to 0x003af7 (0x2e9d bytes) 000457 1783139342.971566 fallocate 0x007c40 thru 0x014c92 (0xd053 bytes) 000458 1783139342.979281 mapwrite 0x00911a thru 0x017bb3 (0xea9a bytes) 000459 1783139343.053829 read 0x01598a thru 0x017bb3 (0x222a bytes) 000460 1783139343.054844 fallocate 0x03f2ef thru 0x03ffff (0xd11 bytes) 000461 1783139343.061015 mapread 0x0124a2 thru 0x017bb3 (0x5712 bytes) 000462 1783139343.069258 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x17bb4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17bb4): Mirror 3: 0x5de874a6 (stale) Mirror 4: 0x5de874a6 (stale) Mirror 7: 0x5de874a6 (stale) Mirror 14: 0x5de874a6 (stale) Mirror 15: 0x5de874a6 (stale) Mirror 16: 0x5de874a6 (stale) Mirror 18: 0x5de874a6 (stale) Mirror 19: 0x5de874a6 (stale) Mirror 21: 0x5de874a6 (stale) Mirror 22: 0x5de874a6 (stale) Mirror 23: 0x5de874a6 (stale) Mirror 24: 0x5de874a6 (stale) CRC-32 checksum value for chunk [0, 0x17bb4): Mirror 1: 0xb57eef97 Mirror 25: 0xb57eef97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17bb4: skipped 000463 1783139344.404348 trunc from 0x017bb4 to 0x039255 (0x216a2 bytes) 000464 1783139344.506600 write 0x00ad0f thru 0x00dc54 (0x2f46 bytes) 000465 1783139344.521052 write 0x03cf09 thru 0x03ffff (0x30f7 bytes) 000466 1783139344.523498 mapread 0x032724 thru 0x0356af (0x2f8c bytes) 000467 1783139344.549739 mapwrite 0x01e3c4 thru 0x024b2e (0x676b bytes) 000468 1783139344.596561 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf831e71f (stale) Mirror 4: 0xf831e71f (stale) Mirror 7: 0xf831e71f (stale) Mirror 14: 0xf831e71f (stale) Mirror 16: 0xf831e71f (stale) Mirror 18: 0xf831e71f (stale) Mirror 19: 0xf831e71f (stale) Mirror 21: 0xf831e71f (stale) Mirror 22: 0xf831e71f (stale) Mirror 23: 0xf831e71f (stale) Mirror 24: 0xf831e71f (stale) Mirror 25: 0xb45b5106 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf831e71f (stale) Mirror 4: 0xf831e71f (stale) Mirror 7: 0xf831e71f (stale) Mirror 14: 0xf831e71f (stale) Mirror 16: 0xf831e71f (stale) Mirror 18: 0xf831e71f (stale) Mirror 19: 0xf831e71f (stale) Mirror 21: 0xf831e71f (stale) Mirror 22: 0xf831e71f (stale) Mirror 23: 0xf831e71f (stale) Mirror 24: 0xf831e71f (stale) Mirror 25: 0xb45b5106 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1783139345.375586 fallocate 0x038a05 thru 0x03ffff (0x75fb bytes) 000470 1783139345.390657 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf831e71f (stale) Mirror 4: 0xf831e71f (stale) Mirror 7: 0xf831e71f (stale) Mirror 14: 0xf831e71f (stale) Mirror 16: 0xf831e71f (stale) Mirror 18: 0xf831e71f (stale) Mirror 19: 0xf831e71f (stale) Mirror 21: 0xf831e71f (stale) Mirror 22: 0xf831e71f (stale) Mirror 23: 0xf831e71f (stale) Mirror 24: 0xf831e71f (stale) Mirror 25: 0xb45b5106 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb9e52bfa Mirror 26: 0xb9e52bfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000471 1783139347.048867 trunc from 0x037f75 to 0x03f828 (0x78b4 bytes) 000472 1783139347.124718 fallocate 0x024cd8 thru 0x028036 (0x335f bytes) 000473 1783139347.132659 write 0x0076fe thru 0x0081d2 (0xad5 bytes) 000474 1783139347.150856 write 0x00310e thru 0x0105b6 (0xd4a9 bytes) 000475 1783139347.165869 punch from 0x00dc87 to 0x01a07f (0xc3f9 bytes) 000476 1783139347.185667 fallocate 0x02223f thru 0x02a45c (0x821e bytes) 000477 1783139347.193819 trunc from 0x040000 to 0x024803 (0x1b7fc bytes) 000478 1783139347.220463 fallocate 0x03168a thru 0x032c52 (0x15c9 bytes) 000480 1783139347.228786 trunc from 0x00e5b2 to 0x0178e2 (0x9331 bytes) 000483 1783139347.242160 MIRROR_RESYNC 14 mirrors 000484 1783139350.749802 write 0x001b2a thru 0x009f06 (0x83dd bytes) 000485 1783139350.809723 trunc from 0x00f583 to 0x017e26 (0x88a4 bytes) 000486 1783139350.818629 mapwrite 0x03212c thru 0x03ffff (0xded4 bytes) 000487 1783139350.920686 mapwrite 0x00b6d3 thru 0x00d91e (0x224c bytes) 000488 1783139350.957693 fallocate 0x0057a7 thru 0x011eaa (0xc704 bytes) 000489 1783139350.963781 mapread 0x02afb7 thru 0x0312c3 (0x630d bytes) 000490 1783139351.001588 punch from 0x0296f9 to 0x037855 (0xe15d bytes) 000491 1783139351.012548 read 0x03bec5 thru 0x03ffff (0x413b bytes) 000492 1783139351.016571 trunc from 0x014cdd to 0x014f8d (0x2b1 bytes) 000493 1783139351.028544 punch from 0x000624 to 0x00c1ae (0xbb8b bytes) 000494 1783139351.039659 fallocate 0x014784 thru 0x015103 (0x980 bytes) 000495 1783139351.050771 read 0x037eac thru 0x03ffff (0x8154 bytes) 000496 1783139351.086011 mapread 0x032924 thru 0x03c41a (0x9af7 bytes) 000497 1783139351.087447 fallocate 0x013288 thru 0x0169a9 (0x3722 bytes) 000498 1783139351.093995 read 0x01b474 thru 0x0212ca (0x5e57 bytes) 000499 1783139351.094746 mapread 0x01c0a3 thru 0x02603a (0x9f98 bytes) 000500 1783139351.097247 punch from 0x0182ba to 0x01bc7f (0x39c6 bytes) 000501 1783139351.109224 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xff9fe519 (stale) Mirror 4: 0xff9fe519 (stale) Mirror 7: 0xff9fe519 (stale) Mirror 14: 0xff9fe519 (stale) Mirror 16: 0xff9fe519 (stale) Mirror 18: 0xff9fe519 (stale) Mirror 19: 0xff9fe519 (stale) Mirror 21: 0xff9fe519 (stale) Mirror 22: 0xff9fe519 (stale) Mirror 23: 0xff9fe519 (stale) Mirror 24: 0xff9fe519 (stale) Mirror 25: 0xff9fe519 (stale) Mirror 26: 0xff9fe519 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x89e70330 Mirror 27: 0x89e70330 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000502 1783139352.405476 fallocate 0x031469 thru 0x034c63 (0x37fb bytes) 000503 1783139352.493141 read 0x033151 thru 0x03d36d (0xa21d bytes) 000504 1783139352.519146 write 0x036977 thru 0x036a3f (0xc9 bytes) 000505 1783139352.527040 fallocate 0x003db3 thru 0x00778d (0x39db bytes) 000507 1783139352.546088 mapwrite 0x022815 thru 0x0293ca (0x6bb6 bytes) 000508 1783139352.631132 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xff9fe519 (stale) Mirror 4: 0xff9fe519 (stale) Mirror 7: 0xff9fe519 (stale) Mirror 14: 0xff9fe519 (stale) Mirror 16: 0xff9fe519 (stale) Mirror 18: 0xff9fe519 (stale) Mirror 19: 0xff9fe519 (stale) Mirror 21: 0xff9fe519 (stale) Mirror 22: 0xff9fe519 (stale) Mirror 23: 0xff9fe519 (stale) Mirror 24: 0xff9fe519 (stale) Mirror 25: 0xff9fe519 (stale) Mirror 26: 0xff9fe519 (stale) Mirror 27: 0x89e70330 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd162ceca Mirror 28: 0xd162ceca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000509 1783139354.230622 write 0x026b27 thru 0x0334c6 (0xc9a0 bytes) 000510 1783139354.331764 MIRROR_SPLIT mirror 14 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xff9fe519 (stale) Mirror 4: 0xff9fe519 (stale) Mirror 7: 0xff9fe519 (stale) Mirror 14: 0xff9fe519 (stale) Mirror 16: 0xff9fe519 (stale) Mirror 18: 0xff9fe519 (stale) Mirror 19: 0xff9fe519 (stale) Mirror 21: 0xff9fe519 (stale) Mirror 22: 0xff9fe519 (stale) Mirror 23: 0xff9fe519 (stale) Mirror 24: 0xff9fe519 (stale) Mirror 25: 0xff9fe519 (stale) Mirror 26: 0xff9fe519 (stale) Mirror 28: 0xd162ceca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xff9fe519 (stale) Mirror 4: 0xff9fe519 (stale) Mirror 7: 0xff9fe519 (stale) Mirror 14: 0xff9fe519 (stale) Mirror 16: 0xff9fe519 (stale) Mirror 18: 0xff9fe519 (stale) Mirror 19: 0xff9fe519 (stale) Mirror 21: 0xff9fe519 (stale) Mirror 22: 0xff9fe519 (stale) Mirror 23: 0xff9fe519 (stale) Mirror 24: 0xff9fe519 (stale) Mirror 25: 0xff9fe519 (stale) Mirror 26: 0xff9fe519 (stale) Mirror 28: 0xd162ceca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1783139355.455832 write 0x038b9d thru 0x03d662 (0x4ac6 bytes) 000512 1783139355.479138 mapread 0x002acb thru 0x012299 (0xf7cf bytes) 000513 1783139355.508890 trunc from 0x040000 to 0x013a8c (0x2c573 bytes) 000514 1783139355.533269 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13a8d): Mirror 4: 0xd87871ee (stale) Mirror 7: 0xd87871ee (stale) Mirror 14: 0xd87871ee (stale) Mirror 16: 0xd87871ee (stale) Mirror 18: 0xd87871ee (stale) Mirror 19: 0xd87871ee (stale) Mirror 21: 0xd87871ee (stale) Mirror 22: 0xd87871ee (stale) Mirror 23: 0xd87871ee (stale) Mirror 24: 0xd87871ee (stale) Mirror 25: 0xd87871ee (stale) Mirror 26: 0xd87871ee (stale) Mirror 28: 0xd99dc88d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13a8d): Mirror 4: 0xd87871ee (stale) Mirror 7: 0xd87871ee (stale) Mirror 14: 0xd87871ee (stale) Mirror 16: 0xd87871ee (stale) Mirror 18: 0xd87871ee (stale) Mirror 19: 0xd87871ee (stale) Mirror 21: 0xd87871ee (stale) Mirror 22: 0xd87871ee (stale) Mirror 23: 0xd87871ee (stale) Mirror 24: 0xd87871ee (stale) Mirror 25: 0xd87871ee (stale) Mirror 26: 0xd87871ee (stale) Mirror 28: 0xd99dc88d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13a8d: skipped 000515 1783139356.295605 mapwrite 0x021fe3 thru 0x03104a (0xf068 bytes) 000516 1783139356.438013 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3104b): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3104b): Mirror 4: 0x896c55ec (stale) Mirror 7: 0x896c55ec (stale) Mirror 14: 0x896c55ec (stale) Mirror 16: 0x896c55ec (stale) Mirror 18: 0x896c55ec (stale) Mirror 19: 0x896c55ec (stale) Mirror 21: 0x896c55ec (stale) Mirror 22: 0x896c55ec (stale) Mirror 23: 0x896c55ec (stale) Mirror 24: 0x896c55ec (stale) Mirror 25: 0x896c55ec (stale) Mirror 26: 0x896c55ec (stale) Mirror 28: 0xd40290ba (stale) CRC-32 checksum value for chunk [0, 0x3104b): Mirror 1: 0x474f3be7 Mirror 29: 0x474f3be7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3104b: skipped 000517 1783139357.768471 trunc from 0x00cc94 to 0x01a82f (0xdb9c bytes) 000518 1783139357.837218 trunc from 0x03104b to 0x00d76b (0x238df bytes) 000519 1783139357.913746 read 0x009ace thru 0x00d76b (0x3c9e bytes) 000521 1783139357.929010 mapwrite 0x020c62 thru 0x0256f2 (0x4a91 bytes) 000522 1783139358.034453 trunc from 0x0256f3 to 0x02f094 (0x99a2 bytes) 000523 1783139358.062035 mapread 0x006f62 thru 0x016a4c (0xfaeb bytes) 000525 1783139358.085286 trunc from 0x005413 to 0x00ea5d (0x964b bytes) 000526 1783139358.090649 trunc from 0x01f0ed to 0x02a654 (0xb568 bytes) 000527 1783139358.097519 fallocate 0x0192f2 thru 0x024d2d (0xba3c bytes) 000528 1783139358.108400 MIRROR_RESYNC 15 mirrors 000529 1783139360.674806 MIRROR_RESYNC 15 mirrors 000530 1783139361.812697 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) 000531 1783139362.398580 fallocate 0x023771 thru 0x02e0d2 (0xa962 bytes) 000532 1783139362.445465 read 0x02d152 thru 0x02f094 (0x1f43 bytes) 000533 1783139362.452640 trunc from 0x0107a3 to 0x01ea49 (0xe2a7 bytes) 000534 1783139362.457252 mapwrite 0x01d806 thru 0x024839 (0x7034 bytes) 000535 1783139362.500986 trunc from 0x02f095 to 0x00f943 (0x1f751 bytes) 000536 1783139362.524058 mapread 0x005623 thru 0x00b484 (0x5e62 bytes) 000537 1783139362.538442 punch from 0x00904e to 0x00f943 (0x68f6 bytes) 000539 1783139362.544769 write 0x0350c9 thru 0x03d110 (0x8048 bytes) 000540 1783139362.552213 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d111): Mirror 7: 0x98dfd5ab (stale) Mirror 14: 0x98dfd5ab (stale) Mirror 16: 0x98dfd5ab (stale) Mirror 18: 0x98dfd5ab (stale) Mirror 19: 0x98dfd5ab (stale) Mirror 21: 0x98dfd5ab (stale) Mirror 22: 0x98dfd5ab (stale) Mirror 23: 0x98dfd5ab (stale) Mirror 24: 0x98dfd5ab (stale) Mirror 25: 0x98dfd5ab (stale) Mirror 26: 0x98dfd5ab (stale) Mirror 28: 0x98dfd5ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d111): Mirror 7: 0x98dfd5ab (stale) Mirror 14: 0x98dfd5ab (stale) Mirror 16: 0x98dfd5ab (stale) Mirror 18: 0x98dfd5ab (stale) Mirror 19: 0x98dfd5ab (stale) Mirror 21: 0x98dfd5ab (stale) Mirror 22: 0x98dfd5ab (stale) Mirror 23: 0x98dfd5ab (stale) Mirror 24: 0x98dfd5ab (stale) Mirror 25: 0x98dfd5ab (stale) Mirror 26: 0x98dfd5ab (stale) Mirror 28: 0x98dfd5ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d111: skipped 000541 1783139363.429476 mapwrite 0x025176 thru 0x02c45f (0x72ea bytes) 000543 1783139363.474885 write 0x0083b9 thru 0x00bf0c (0x3b54 bytes) 000544 1783139363.499361 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d111): Mirror 14: 0x98dfd5ab (stale) Mirror 16: 0x98dfd5ab (stale) Mirror 18: 0x98dfd5ab (stale) Mirror 19: 0x98dfd5ab (stale) Mirror 21: 0x98dfd5ab (stale) Mirror 22: 0x98dfd5ab (stale) Mirror 23: 0x98dfd5ab (stale) Mirror 24: 0x98dfd5ab (stale) Mirror 25: 0x98dfd5ab (stale) Mirror 26: 0x98dfd5ab (stale) Mirror 28: 0x98dfd5ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d111): Mirror 14: 0x98dfd5ab (stale) Mirror 16: 0x98dfd5ab (stale) Mirror 18: 0x98dfd5ab (stale) Mirror 19: 0x98dfd5ab (stale) Mirror 21: 0x98dfd5ab (stale) Mirror 22: 0x98dfd5ab (stale) Mirror 23: 0x98dfd5ab (stale) Mirror 24: 0x98dfd5ab (stale) Mirror 25: 0x98dfd5ab (stale) Mirror 26: 0x98dfd5ab (stale) Mirror 28: 0x98dfd5ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d111: skipped 000545 1783139364.449552 write 0x0323cf thru 0x035154 (0x2d86 bytes) 000546 1783139364.469895 trunc from 0x03d111 to 0x001e8f (0x3b281 bytes) 000547 1783139364.557761 trunc from 0x0013ba to 0x001e8f (0xad6 bytes) 000548 1783139364.584426 read 0x001930 thru 0x001e8f (0x560 bytes) 000550 1783139364.607653 trunc from 0x000956 to 0x001e8f (0x153a bytes) 000551 1783139364.621765 trunc from 0x001e90 to 0x01bd03 (0x19e74 bytes) 000552 1783139364.658886 MIRROR_RESYNC 12 mirrors 000553 1783139366.950402 punch from 0x001686 to 0x00d4eb (0xbe66 bytes) 000554 1783139367.007686 mapwrite 0x014412 thru 0x020d64 (0xc953 bytes) 000555 1783139367.156520 MIRROR_RESYNC 12 mirrors 000556 1783139369.216547 trunc from 0x01dcca to 0x020d64 (0x309b bytes) 000557 1783139369.290670 punch from 0x020ba8 to 0x020d64 (0x1bd bytes) 000558 1783139369.302064 mapwrite 0x023d15 thru 0x024fa4 (0x1290 bytes) 000559 1783139369.372772 fallocate 0x03cf76 thru 0x03ffff (0x308a bytes) 000560 1783139369.380911 trunc from 0x01697b to 0x01ac13 (0x4299 bytes) 000562 1783139369.394864 trunc from 0x024fa5 to 0x000b44 (0x24460 bytes) 000563 1783139369.418411 read 0x00058f thru 0x000b44 (0x5b6 bytes) 000564 1783139369.425656 write 0x014a32 thru 0x018de6 (0x43b5 bytes) 000567 1783139369.429326 mapwrite 0x01150e thru 0x01b911 (0xa404 bytes) 000568 1783139369.497309 mapwrite 0x01df91 thru 0x01e3f8 (0x468 bytes) 000569 1783139369.551187 write 0x00685d thru 0x008f98 (0x273c bytes) 000571 1783139369.570849 fallocate 0x024453 thru 0x030c67 (0xc815 bytes) 000572 1783139369.588408 MIRROR_RESYNC 12 mirrors 000573 1783139372.956677 MIRROR_EXTEND to 13 mirrors 000574 1783139374.004663 fallocate 0x03740d thru 0x037b39 (0x72d bytes) 000575 1783139374.047561 trunc from 0x037b3a to 0x0319d3 (0x6166 bytes) 000576 1783139374.073411 read 0x0240e5 thru 0x03122a (0xd146 bytes) 000577 1783139374.084293 trunc from 0x0319d4 to 0x03d00f (0xb63c bytes) 000578 1783139374.101460 trunc from 0x03d010 to 0x00e589 (0x2ea86 bytes) 000579 1783139374.123740 read 0x009e8a thru 0x00bb79 (0x1cf0 bytes) 000580 1783139374.134331 read 0x00d03a thru 0x00e589 (0x1550 bytes) 000581 1783139374.135115 write 0x0172b4 thru 0x01c217 (0x4f64 bytes) 000582 1783139374.138569 write 0x01d834 thru 0x028999 (0xb166 bytes) 000583 1783139374.141510 mapwrite 0x00730c thru 0x013fab (0xcca0 bytes) 000584 1783139374.176678 trunc from 0x02899a to 0x01548a (0x1350f bytes) 000585 1783139374.197433 write 0x029024 thru 0x031ba6 (0x8b83 bytes) 000586 1783139374.202468 punch from 0x003c18 to 0x005b6d (0x1f56 bytes) 000587 1783139374.210051 fallocate 0x03bd30 thru 0x03c015 (0x2e6 bytes) 000588 1783139374.217359 read 0x007c0a thru 0x011013 (0x940a bytes) 000589 1783139374.226344 MIRROR_RESYNC 13 mirrors 000590 1783139376.858084 read 0x01eb3f thru 0x026885 (0x7d47 bytes) 000591 1783139376.867487 read 0x01ca73 thru 0x01e1d0 (0x175e bytes) 000592 1783139376.878557 trunc from 0x031ba7 to 0x03bdbe (0xa218 bytes) 000593 1783139376.916405 write 0x00955b thru 0x00cdd7 (0x387d bytes) 000594 1783139376.929399 trunc from 0x03bdbf to 0x0188b6 (0x23508 bytes) 000595 1783139376.954571 read 0x00157e thru 0x0109c8 (0xf44b bytes) 000596 1783139376.970312 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x188b7): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x188b7): Mirror 14: 0xf37b80dd (stale) Mirror 16: 0xf37b80dd (stale) Mirror 18: 0xf37b80dd (stale) Mirror 19: 0xf37b80dd (stale) Mirror 21: 0xf37b80dd (stale) Mirror 22: 0xf37b80dd (stale) Mirror 23: 0xf37b80dd (stale) Mirror 24: 0xf37b80dd (stale) Mirror 25: 0xf37b80dd (stale) Mirror 26: 0xf37b80dd (stale) Mirror 28: 0xf37b80dd (stale) Mirror 29: 0xf37b80dd (stale) CRC-32 checksum value for chunk [0, 0x188b7): Mirror 1: 0xbbe7653c Mirror 30: 0xbbe7653c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x188b7: skipped 000597 1783139378.056518 trunc from 0x0188b7 to 0x01203c (0x687a bytes) 000598 1783139378.148845 mapwrite 0x008710 thru 0x01022b (0x7b1c bytes) 000599 1783139378.252612 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1203d): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1203d): Mirror 14: 0x559e0a19 (stale) Mirror 16: 0x559e0a19 (stale) Mirror 18: 0x559e0a19 (stale) Mirror 19: 0x559e0a19 (stale) Mirror 21: 0x559e0a19 (stale) Mirror 22: 0x559e0a19 (stale) Mirror 23: 0x559e0a19 (stale) Mirror 24: 0x559e0a19 (stale) Mirror 25: 0x559e0a19 (stale) Mirror 26: 0x559e0a19 (stale) Mirror 28: 0x559e0a19 (stale) Mirror 29: 0x559e0a19 (stale) Mirror 30: 0xbcd42c64 (stale) CRC-32 checksum value for chunk [0, 0x1203d): Mirror 1: 0x32d17c53 Mirror 31: 0x32d17c53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1203d: skipped 000600 1783139379.583266 mapread 0x011a8b thru 0x01203c (0x5b2 bytes) 000601 1783139379.645183 trunc from 0x000efd to 0x00d189 (0xc28d bytes) 000602 1783139379.729786 mapwrite 0x03b7e0 thru 0x03ffff (0x4820 bytes) 000603 1783139379.841326 punch from 0x0201c1 to 0x025611 (0x5451 bytes) 000604 1783139379.846661 trunc from 0x040000 to 0x00a603 (0x359fc bytes) 000605 1783139379.872249 mapwrite 0x023db8 thru 0x02905f (0x52a8 bytes) 000606 1783139379.931893 trunc from 0x0258f7 to 0x02905f (0x3769 bytes) 000607 1783139379.939443 mapwrite 0x0393f8 thru 0x03ffff (0x6c08 bytes) 000608 1783139379.988718 fallocate 0x029e75 thru 0x02c071 (0x21fd bytes) 000609 1783139379.995911 mapwrite 0x015301 thru 0x01ea03 (0x9703 bytes) 000610 1783139380.029508 punch from 0x0337be to 0x0376fa (0x3f3d bytes) 000611 1783139380.039919 MIRROR_RESYNC 15 mirrors 000612 1783139385.210521 mapwrite 0x03a864 thru 0x03ffff (0x579c bytes) 000613 1783139385.307798 mapread 0x01ded2 thru 0x02908b (0xb1ba bytes) 000614 1783139385.326233 punch from 0x01ad35 to 0x026802 (0xbace bytes) 000615 1783139385.331491 mapread 0x02e077 thru 0x033b6a (0x5af4 bytes) 000616 1783139385.340661 read 0x03eaef thru 0x03ffff (0x1511 bytes) 000617 1783139385.341414 write 0x02bbe2 thru 0x02ea90 (0x2eaf bytes) 000619 1783139385.346698 punch from 0x038876 to 0x039d4a (0x14d5 bytes) 000620 1783139385.352458 trunc from 0x040000 to 0x00b8be (0x34741 bytes) 000621 1783139385.375569 punch from 0x008a7b to 0x0092a4 (0x82a bytes) 000622 1783139385.381445 mapread 0x00a332 thru 0x00b8be (0x158d bytes) 000623 1783139385.394721 write 0x02b0ab thru 0x02e1b9 (0x310f bytes) 000624 1783139385.396431 mapread 0x01ec26 thru 0x02804e (0x9429 bytes) 000625 1783139385.410802 trunc from 0x02e1ba to 0x021eb6 (0xc303 bytes) 000626 1783139385.430272 mapread 0x00ce0a thru 0x0103cd (0x35c4 bytes) 000627 1783139385.445160 write 0x01c023 thru 0x02b6d0 (0xf6ae bytes) 000628 1783139385.447857 mapwrite 0x02ce87 thru 0x03aa4a (0xdbc4 bytes) 000629 1783139385.518400 mapread 0x032861 thru 0x03359f (0xd3f bytes) 000630 1783139385.519804 trunc from 0x00115f to 0x001206 (0xa8 bytes) 000631 1783139385.527415 write 0x02946f thru 0x0351a0 (0xbd32 bytes) 000632 1783139385.532445 fallocate 0x0100b5 thru 0x01ee00 (0xed4c bytes) 000633 1783139385.538299 mapread 0x02dbd4 thru 0x032433 (0x4860 bytes) 000634 1783139385.540502 trunc from 0x02653d to 0x0316c2 (0xb186 bytes) 000635 1783139385.557196 trunc from 0x038e95 to 0x03aa4a (0x1bb6 bytes) 000636 1783139385.563288 fallocate 0x0249ab thru 0x02f146 (0xa79c bytes) 000637 1783139385.577836 trunc from 0x010118 to 0x01c093 (0xbf7c bytes) 000638 1783139385.592701 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3aa4b): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa4b): Mirror 14: 0xc7b5f5df (stale) Mirror 16: 0xc7b5f5df (stale) Mirror 18: 0xc7b5f5df (stale) Mirror 19: 0xc7b5f5df (stale) Mirror 21: 0xc7b5f5df (stale) Mirror 22: 0xc7b5f5df (stale) Mirror 23: 0xc7b5f5df (stale) Mirror 24: 0xc7b5f5df (stale) Mirror 25: 0xc7b5f5df (stale) Mirror 26: 0xc7b5f5df (stale) Mirror 28: 0xc7b5f5df (stale) Mirror 29: 0xc7b5f5df (stale) Mirror 30: 0xc7b5f5df (stale) Mirror 31: 0xc7b5f5df (stale) CRC-32 checksum value for chunk [0, 0x3aa4b): Mirror 1: 0x47d0efca Mirror 32: 0x47d0efca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa4b: skipped 000639 1783139387.381286 mapread 0x03106b thru 0x039c72 (0x8c08 bytes) 000640 1783139387.448318 read 0x01e94c thru 0x022d7d (0x4432 bytes) 000641 1783139387.463395 read 0x001455 thru 0x0029aa (0x1556 bytes) 000642 1783139387.515531 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aa4b): Mirror 14: 0xc7b5f5df (stale) Mirror 16: 0xc7b5f5df (stale) Mirror 18: 0xc7b5f5df (stale) Mirror 19: 0xc7b5f5df (stale) Mirror 21: 0xc7b5f5df (stale) Mirror 22: 0xc7b5f5df (stale) Mirror 23: 0xc7b5f5df (stale) Mirror 25: 0xc7b5f5df (stale) Mirror 26: 0xc7b5f5df (stale) Mirror 28: 0xc7b5f5df (stale) Mirror 29: 0xc7b5f5df (stale) Mirror 30: 0xc7b5f5df (stale) Mirror 31: 0xc7b5f5df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa4b): Mirror 14: 0xc7b5f5df (stale) Mirror 16: 0xc7b5f5df (stale) Mirror 18: 0xc7b5f5df (stale) Mirror 19: 0xc7b5f5df (stale) Mirror 21: 0xc7b5f5df (stale) Mirror 22: 0xc7b5f5df (stale) Mirror 23: 0xc7b5f5df (stale) Mirror 25: 0xc7b5f5df (stale) Mirror 26: 0xc7b5f5df (stale) Mirror 28: 0xc7b5f5df (stale) Mirror 29: 0xc7b5f5df (stale) Mirror 30: 0xc7b5f5df (stale) Mirror 31: 0xc7b5f5df (stale) CRC-32 checksum value for chunk [0, 0x3aa4b): Mirror 1: 0x47d0efca Mirror 32: 0x47d0efca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa4b: skipped 000643 1783139389.060736 punch from 0x038f57 to 0x03aa4a (0x1af4 bytes) 000644 1783139389.233378 fallocate 0x011851 thru 0x021075 (0xf825 bytes) 000645 1783139389.250454 punch from 0x000f70 to 0x0086a6 (0x7737 bytes) 000646 1783139389.262604 write 0x015900 thru 0x01df13 (0x8614 bytes) 000647 1783139389.283564 mapwrite 0x01d311 thru 0x02d180 (0xfe70 bytes) 000648 1783139389.376165 read 0x02cf8d thru 0x03656c (0x95e0 bytes) 000649 1783139389.397649 read 0x00e774 thru 0x01e6ee (0xff7b bytes) 000650 1783139389.415400 read 0x022540 thru 0x02d4a4 (0xaf65 bytes) 000651 1783139389.416150 write 0x00903f thru 0x00a201 (0x11c3 bytes) 000652 1783139389.417158 fallocate 0x032759 thru 0x0355df (0x2e87 bytes) 000653 1783139389.425752 read 0x015770 thru 0x01754e (0x1ddf bytes) 000654 1783139389.429323 mapwrite 0x033d63 thru 0x03dfc5 (0xa263 bytes) 000655 1783139389.498490 trunc from 0x03dfc6 to 0x02878e (0x15837 bytes) 000656 1783139389.521143 fallocate 0x019805 thru 0x02879f (0xef9b bytes) 000657 1783139389.527589 mapread 0x01279c thru 0x01ff51 (0xd7b6 bytes) 000658 1783139389.529726 mapread 0x027db0 thru 0x02878e (0x9df bytes) 000659 1783139389.531476 punch from 0x006cc5 to 0x00f357 (0x8693 bytes) 000660 1783139389.537637 punch from 0x014b7b to 0x020cc7 (0xc14d bytes) 000661 1783139389.548582 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2878f): Mirror 14: 0xff20d37d (stale) Mirror 16: 0xff20d37d (stale) Mirror 18: 0xff20d37d (stale) Mirror 19: 0xff20d37d (stale) Mirror 21: 0xff20d37d (stale) Mirror 22: 0xff20d37d (stale) Mirror 23: 0xff20d37d (stale) Mirror 25: 0xff20d37d (stale) Mirror 26: 0xff20d37d (stale) Mirror 29: 0xff20d37d (stale) Mirror 30: 0xff20d37d (stale) Mirror 31: 0xff20d37d (stale) Mirror 32: 0x9c76a41b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2878f): Mirror 14: 0xff20d37d (stale) Mirror 16: 0xff20d37d (stale) Mirror 18: 0xff20d37d (stale) Mirror 19: 0xff20d37d (stale) Mirror 21: 0xff20d37d (stale) Mirror 22: 0xff20d37d (stale) Mirror 23: 0xff20d37d (stale) Mirror 25: 0xff20d37d (stale) Mirror 26: 0xff20d37d (stale) Mirror 29: 0xff20d37d (stale) Mirror 30: 0xff20d37d (stale) Mirror 31: 0xff20d37d (stale) Mirror 32: 0x9c76a41b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2878f: skipped 000662 1783139390.394390 write 0x0339fb thru 0x03b163 (0x7769 bytes) 000663 1783139390.405441 mapread 0x023aaf thru 0x025e64 (0x23b6 bytes) 000664 1783139390.426382 fallocate 0x007658 thru 0x00b289 (0x3c32 bytes) 000665 1783139390.432498 punch from 0x018296 to 0x01ff43 (0x7cae bytes) 000666 1783139390.442084 fallocate 0x01b0da thru 0x01b1d5 (0xfc bytes) 000668 1783139390.448815 write 0x0091bc thru 0x015056 (0xbe9b bytes) 000669 1783139390.474460 trunc from 0x03b164 to 0x001dee (0x39375 bytes) 000670 1783139390.496694 punch from 0x001720 to 0x001dee (0x6cf bytes) 000671 1783139390.506553 punch from 0x0019b8 to 0x001dee (0x437 bytes) 000672 1783139390.512364 trunc from 0x000820 to 0x001dee (0x15cf bytes) 000673 1783139390.518033 trunc from 0x001367 to 0x001dee (0xa88 bytes) 000674 1783139390.527136 punch from 0x00014b to 0x001dee (0x1ca4 bytes) 000675 1783139390.534619 trunc from 0x00037e to 0x001dee (0x1a71 bytes) 000676 1783139390.541629 read 0x001ba2 thru 0x001dee (0x24d bytes) 000677 1783139390.547312 punch from 0x001541 to 0x001dee (0x8ae bytes) 000678 1783139390.553740 mapread 0x00197d thru 0x001dee (0x472 bytes) 000679 1783139390.560660 trunc from 0x000112 to 0x001dee (0x1cdd bytes) 000680 1783139390.570305 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1def): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1def): Mirror 14: 0x7a6ba098 (stale) Mirror 16: 0x7a6ba098 (stale) Mirror 18: 0x7a6ba098 (stale) Mirror 19: 0x7a6ba098 (stale) Mirror 21: 0x7a6ba098 (stale) Mirror 22: 0x7a6ba098 (stale) Mirror 23: 0x7a6ba098 (stale) Mirror 25: 0x7a6ba098 (stale) Mirror 26: 0x7a6ba098 (stale) Mirror 29: 0x7a6ba098 (stale) Mirror 30: 0x7a6ba098 (stale) Mirror 31: 0x7a6ba098 (stale) Mirror 32: 0x7a6ba098 (stale) CRC-32 checksum value for chunk [0, 0x1def): Mirror 1: 0x7a6ba098 Mirror 33: 0x7a6ba098 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1def: skipped 000681 1783139391.816666 trunc from 0x000ee5 to 0x0010a4 (0x1c0 bytes) 000682 1783139391.869399 trunc from 0x001def to 0x020a96 (0x1eca8 bytes) 000683 1783139391.899742 trunc from 0x00a06f to 0x0168df (0xc871 bytes) 000684 1783139391.906366 trunc from 0x007bf5 to 0x0101c2 (0x85ce bytes) 000686 1783139391.913432 mapwrite 0x01b3fc thru 0x01f119 (0x3d1e bytes) 000687 1783139391.973568 punch from 0x01e638 to 0x020a96 (0x245f bytes) 000688 1783139391.982188 fallocate 0x0193e1 thru 0x02397b (0xa59b bytes) 000689 1783139391.991147 fallocate 0x027bbb thru 0x033e6d (0xc2b3 bytes) 000690 1783139391.997423 trunc from 0x026de1 to 0x02aa34 (0x3c54 bytes) 000691 1783139392.003390 trunc from 0x033e6e to 0x0131e1 (0x20c8c bytes) 000693 1783139392.026676 mapwrite 0x0140f1 thru 0x0168e3 (0x27f3 bytes) 000694 1783139392.093136 write 0x0337af thru 0x03580e (0x2060 bytes) 000695 1783139392.094381 mapwrite 0x017a70 thru 0x02073f (0x8cd0 bytes) 000696 1783139392.138591 trunc from 0x03580f to 0x00733f (0x2e4cf bytes) 000697 1783139392.159337 mapread 0x006c69 thru 0x00733f (0x6d7 bytes) 000698 1783139392.167898 trunc from 0x007340 to 0x032fd1 (0x2bc92 bytes) 000699 1783139392.190616 mapwrite 0x0003fc thru 0x002771 (0x2376 bytes) 000700 1783139392.211799 punch from 0x025108 to 0x032fd1 (0xdeca bytes) 000701 1783139392.218077 trunc from 0x02f7c8 to 0x030232 (0xa6b bytes) 000702 1783139392.223392 trunc from 0x032fd2 to 0x01603d (0x1cf94 bytes) 000703 1783139392.242694 fallocate 0x0330cd thru 0x038b85 (0x5ab9 bytes) 000705 1783139392.252541 fallocate 0x00d953 thru 0x01b742 (0xddf0 bytes) 000706 1783139392.258688 trunc from 0x00aa97 to 0x01603d (0xb5a7 bytes) 000707 1783139392.264581 read 0x012ed1 thru 0x014148 (0x1278 bytes) 000708 1783139392.271483 read 0x00c373 thru 0x00ccf9 (0x987 bytes) 000709 1783139392.282254 trunc from 0x0076ff to 0x01603d (0xe93f bytes) 000710 1783139392.292721 MIRROR_RESYNC 15 mirrors 000711 1783139394.169022 trunc from 0x01603e to 0x03c903 (0x268c6 bytes) 000712 1783139394.214434 write 0x02317f thru 0x02f574 (0xc3f6 bytes) 000714 1783139394.226348 mapread 0x010b29 thru 0x01567b (0x4b53 bytes) 000715 1783139394.256825 write 0x024bcc thru 0x03268d (0xdac2 bytes) 000716 1783139394.265444 punch from 0x037e12 to 0x03c903 (0x4af2 bytes) 000717 1783139394.271133 trunc from 0x03c904 to 0x01ce87 (0x1fa7c bytes) 000718 1783139394.294758 read 0x0142c6 thru 0x01ce87 (0x8bc2 bytes) 000719 1783139394.303267 write 0x01a6f5 thru 0x01f2d7 (0x4be3 bytes) 000720 1783139394.305418 write 0x0178b1 thru 0x0203fd (0x8b4d bytes) 000722 1783139394.307618 trunc from 0x0203fe to 0x014db7 (0xb646 bytes) 000723 1783139394.323637 read 0x0000c6 thru 0x005c9e (0x5bd9 bytes) 000724 1783139394.335696 mapread 0x0022ae thru 0x00c85a (0xa5ad bytes) 000725 1783139394.337157 fallocate 0x015de0 thru 0x018a63 (0x2c84 bytes) 000726 1783139394.343696 punch from 0x00604b to 0x013260 (0xd216 bytes) 000727 1783139394.350757 trunc from 0x018a64 to 0x00e623 (0xa440 bytes) 000728 1783139394.382806 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0xe624): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe624): Mirror 14: 0x797f72b (stale) Mirror 16: 0x797f72b (stale) Mirror 18: 0x797f72b (stale) Mirror 19: 0x797f72b (stale) Mirror 21: 0x797f72b (stale) Mirror 22: 0x797f72b (stale) Mirror 23: 0x797f72b (stale) Mirror 25: 0x797f72b (stale) Mirror 26: 0x797f72b (stale) Mirror 29: 0x797f72b (stale) Mirror 30: 0x797f72b (stale) Mirror 31: 0x797f72b (stale) Mirror 32: 0x797f72b (stale) Mirror 33: 0x797f72b (stale) CRC-32 checksum value for chunk [0, 0xe624): Mirror 1: 0x797f72b Mirror 34: 0x797f72b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe624: skipped 000729 1783139395.366421 fallocate 0x022190 thru 0x027e04 (0x5c75 bytes) 000730 1783139395.417497 read 0x025737 thru 0x027e04 (0x26ce bytes) 000731 1783139395.428108 read 0x01e3c8 thru 0x027e04 (0x9a3d bytes) 000733 1783139395.453780 mapwrite 0x014ef8 thru 0x0218d5 (0xc9de bytes) 000734 1783139395.482577 mapread 0x0056ef thru 0x0116a5 (0xbfb7 bytes) 000735 1783139395.484700 punch from 0x00c8c7 to 0x015757 (0x8e91 bytes) 000736 1783139395.492083 write 0x039293 thru 0x03ffff (0x6d6d bytes) 000737 1783139395.498744 fallocate 0x000c09 thru 0x009d62 (0x915a bytes) 000738 1783139395.506084 write 0x0205ac thru 0x02c081 (0xbad6 bytes) 000739 1783139395.513425 mapwrite 0x00f4bc thru 0x018ca4 (0x97e9 bytes) 000741 1783139395.603348 trunc from 0x040000 to 0x016866 (0x29799 bytes) 000742 1783139395.675508 write 0x0056b6 thru 0x0132c4 (0xdc0f bytes) 000743 1783139395.677837 trunc from 0x016867 to 0x011952 (0x4f14 bytes) 000744 1783139395.707467 trunc from 0x011953 to 0x012f18 (0x15c6 bytes) 000745 1783139395.723364 punch from 0x0022d9 to 0x0098b5 (0x75dd bytes) 000746 1783139395.757139 mapwrite 0x02975e thru 0x02bd00 (0x25a3 bytes) 000747 1783139395.796598 write 0x0057a9 thru 0x010d64 (0xb5bc bytes) 000748 1783139395.803594 trunc from 0x02bd01 to 0x02e59c (0x289c bytes) 000749 1783139395.823220 read 0x020694 thru 0x02e59c (0xdf09 bytes) 000750 1783139395.836068 trunc from 0x00c343 to 0x0127f4 (0x64b2 bytes) 000751 1783139395.856605 trunc from 0x02b256 to 0x02e59c (0x3347 bytes) 000752 1783139395.866676 read 0x001d63 thru 0x010be2 (0xee80 bytes) 000754 1783139395.877332 read 0x02254b thru 0x02a430 (0x7ee6 bytes) 000755 1783139395.878021 read 0x01e265 thru 0x02783b (0x95d7 bytes) 000756 1783139395.878627 fallocate 0x016f6f thru 0x0234c7 (0xc559 bytes) 000757 1783139395.884347 read 0x0025b0 thru 0x009bdd (0x762e bytes) 000758 1783139395.885267 punch from 0x018de9 to 0x02690c (0xdb24 bytes) 000759 1783139395.891218 punch from 0x00643c to 0x008d38 (0x28fd bytes) 000760 1783139395.896788 read 0x000591 thru 0x000ea8 (0x918 bytes) 000761 1783139395.898432 fallocate 0x001413 thru 0x00a554 (0x9142 bytes) 000762 1783139395.908555 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e59d): Mirror 14: 0x95d88513 (stale) Mirror 16: 0x95d88513 (stale) Mirror 18: 0x95d88513 (stale) Mirror 19: 0x95d88513 (stale) Mirror 21: 0x95d88513 (stale) Mirror 22: 0x95d88513 (stale) Mirror 23: 0x95d88513 (stale) Mirror 25: 0x95d88513 (stale) Mirror 29: 0x95d88513 (stale) Mirror 30: 0x95d88513 (stale) Mirror 31: 0x95d88513 (stale) Mirror 32: 0x95d88513 (stale) Mirror 33: 0x95d88513 (stale) Mirror 34: 0x95d88513 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e59d): Mirror 14: 0x95d88513 (stale) Mirror 16: 0x95d88513 (stale) Mirror 18: 0x95d88513 (stale) Mirror 19: 0x95d88513 (stale) Mirror 21: 0x95d88513 (stale) Mirror 22: 0x95d88513 (stale) Mirror 23: 0x95d88513 (stale) Mirror 25: 0x95d88513 (stale) Mirror 29: 0x95d88513 (stale) Mirror 30: 0x95d88513 (stale) Mirror 31: 0x95d88513 (stale) Mirror 32: 0x95d88513 (stale) Mirror 33: 0x95d88513 (stale) Mirror 34: 0x95d88513 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e59d: skipped 000763 1783139396.707096 trunc from 0x02e59d to 0x013667 (0x1af35 bytes) 000764 1783139396.732313 trunc from 0x013668 to 0x02bebe (0x18857 bytes) 000765 1783139396.741509 trunc from 0x00b1f5 to 0x019b90 (0xe99c bytes) 000766 1783139396.745423 read 0x011c22 thru 0x017772 (0x5b51 bytes) 000767 1783139396.754169 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bebf): Mirror 14: 0x872fe870 (stale) Mirror 16: 0x872fe870 (stale) Mirror 18: 0x872fe870 (stale) Mirror 19: 0x872fe870 (stale) Mirror 21: 0x872fe870 (stale) Mirror 23: 0x872fe870 (stale) Mirror 25: 0x872fe870 (stale) Mirror 29: 0x872fe870 (stale) Mirror 30: 0x872fe870 (stale) Mirror 31: 0x872fe870 (stale) Mirror 32: 0x872fe870 (stale) Mirror 33: 0x872fe870 (stale) Mirror 34: 0x872fe870 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bebf): Mirror 14: 0x872fe870 (stale) Mirror 16: 0x872fe870 (stale) Mirror 18: 0x872fe870 (stale) Mirror 19: 0x872fe870 (stale) Mirror 21: 0x872fe870 (stale) Mirror 23: 0x872fe870 (stale) Mirror 25: 0x872fe870 (stale) Mirror 29: 0x872fe870 (stale) Mirror 30: 0x872fe870 (stale) Mirror 31: 0x872fe870 (stale) Mirror 32: 0x872fe870 (stale) Mirror 33: 0x872fe870 (stale) Mirror 34: 0x872fe870 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bebf: skipped 000768 1783139397.406681 punch from 0x00b90c to 0x00e400 (0x2af5 bytes) 000769 1783139397.420261 trunc from 0x00a3fd to 0x0150d7 (0xacdb bytes) 000770 1783139397.427598 write 0x01aa68 thru 0x0212eb (0x6884 bytes) 000771 1783139397.452119 mapwrite 0x0247d9 thru 0x02d581 (0x8da9 bytes) 000772 1783139397.574841 trunc from 0x019fc5 to 0x01a349 (0x385 bytes) 000773 1783139397.581686 fallocate 0x02f1a4 thru 0x032e62 (0x3cbf bytes) 000774 1783139397.589838 mapread 0x019446 thru 0x02078f (0x734a bytes) 000775 1783139397.601103 trunc from 0x032e63 to 0x02e9ef (0x4473 bytes) 000776 1783139397.622545 trunc from 0x00a3c5 to 0x00fee5 (0x5b21 bytes) 000777 1783139397.629218 write 0x03d542 thru 0x03ffff (0x2abe bytes) 000778 1783139397.630707 punch from 0x02b2e3 to 0x03989a (0xe5b8 bytes) 000779 1783139397.638200 punch from 0x016336 to 0x01b80b (0x54d6 bytes) 000780 1783139397.646615 read 0x0151e1 thru 0x01f89c (0xa6bc bytes) 000781 1783139397.663727 mapread 0x030d7a thru 0x032707 (0x198e bytes) 000782 1783139397.688147 read 0x02b9cc thru 0x0384eb (0xcb20 bytes) 000783 1783139397.689177 write 0x00c874 thru 0x01934f (0xcadc bytes) 000784 1783139397.722990 punch from 0x031ffd to 0x03a843 (0x8847 bytes) 000785 1783139397.742808 mapread 0x0028f1 thru 0x00c659 (0x9d69 bytes) 000786 1783139397.744178 trunc from 0x040000 to 0x00e97a (0x31685 bytes) 000787 1783139397.800161 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0xe97b): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe97b): Mirror 14: 0xd07747e7 (stale) Mirror 16: 0xd07747e7 (stale) Mirror 18: 0xd07747e7 (stale) Mirror 19: 0xd07747e7 (stale) Mirror 21: 0xd07747e7 (stale) Mirror 23: 0xd07747e7 (stale) Mirror 25: 0xd07747e7 (stale) Mirror 29: 0xd07747e7 (stale) Mirror 30: 0xd07747e7 (stale) Mirror 31: 0xd07747e7 (stale) Mirror 32: 0xd07747e7 (stale) Mirror 33: 0xd07747e7 (stale) Mirror 34: 0xd07747e7 (stale) CRC-32 checksum value for chunk [0, 0xe97b): Mirror 1: 0x92b6ca01 Mirror 35: 0x92b6ca01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe97b: skipped 000788 1783139399.030514 MIRROR_RESYNC 15 mirrors 000789 1783139401.920438 punch from 0x00061c to 0x00cdba (0xc79f bytes) 000790 1783139401.978514 fallocate 0x035234 thru 0x03d15b (0x7f28 bytes) 000791 1783139401.985730 mapread 0x00d1f8 thru 0x00e97a (0x1783 bytes) 000792 1783139402.000615 fallocate 0x02d21e thru 0x03d14c (0xff2f bytes) 000793 1783139402.006478 mapread 0x01838d thru 0x01ab13 (0x2787 bytes) 000794 1783139402.028317 read 0x0286f9 thru 0x0303c8 (0x7cd0 bytes) 000796 1783139402.056386 fallocate 0x01617c thru 0x0183f1 (0x2276 bytes) 000797 1783139402.070313 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d14d): Mirror 14: 0x14eb5802 (stale) Mirror 16: 0x14eb5802 (stale) Mirror 18: 0x14eb5802 (stale) Mirror 19: 0x14eb5802 (stale) Mirror 21: 0x14eb5802 (stale) Mirror 25: 0x14eb5802 (stale) Mirror 29: 0x14eb5802 (stale) Mirror 30: 0x14eb5802 (stale) Mirror 31: 0x14eb5802 (stale) Mirror 32: 0x14eb5802 (stale) Mirror 33: 0x14eb5802 (stale) Mirror 34: 0x14eb5802 (stale) Mirror 35: 0x14eb5802 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d14d): Mirror 14: 0x14eb5802 (stale) Mirror 16: 0x14eb5802 (stale) Mirror 18: 0x14eb5802 (stale) Mirror 19: 0x14eb5802 (stale) Mirror 21: 0x14eb5802 (stale) Mirror 25: 0x14eb5802 (stale) Mirror 29: 0x14eb5802 (stale) Mirror 30: 0x14eb5802 (stale) Mirror 31: 0x14eb5802 (stale) Mirror 32: 0x14eb5802 (stale) Mirror 33: 0x14eb5802 (stale) Mirror 34: 0x14eb5802 (stale) Mirror 35: 0x14eb5802 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d14d: skipped 000798 1783139403.086709 fallocate 0x005d2d thru 0x00a5f1 (0x48c5 bytes) 000799 1783139403.091366 mapread 0x010330 thru 0x010e57 (0xb28 bytes) 000800 1783139403.098840 trunc from 0x008c59 to 0x016144 (0xd4ec bytes) 000802 1783139403.104605 trunc from 0x03d14d to 0x00969e (0x33aae bytes) 000803 1783139403.119059 read 0x0042fc thru 0x00969e (0x53a3 bytes) 000804 1783139403.126422 mapwrite 0x00d354 thru 0x010351 (0x2ffe bytes) 000805 1783139403.194079 fallocate 0x00fe2f thru 0x01f843 (0xfa15 bytes) 000806 1783139403.212089 punch from 0x00f36f to 0x010351 (0xfe3 bytes) 000807 1783139403.233161 trunc from 0x00d709 to 0x010351 (0x2c49 bytes) 000808 1783139403.263620 trunc from 0x00e1b7 to 0x010351 (0x219b bytes) 000809 1783139403.275995 trunc from 0x00710c to 0x010351 (0x9246 bytes) 000810 1783139403.297978 mapwrite 0x004f83 thru 0x00deb3 (0x8f31 bytes) 000811 1783139403.398689 write 0x01f20d thru 0x027a9b (0x888f bytes) 000812 1783139403.408603 punch from 0x0038ac to 0x011ecf (0xe624 bytes) 000813 1783139403.418598 trunc from 0x027a9c to 0x0324a6 (0xaa0b bytes) 000814 1783139403.463109 read 0x007205 thru 0x00ad38 (0x3b34 bytes) 000815 1783139403.492799 trunc from 0x000943 to 0x00bc1a (0xb2d8 bytes) 000816 1783139403.514811 punch from 0x01bdbb to 0x0225da (0x6820 bytes) 000817 1783139403.553913 MIRROR_RESYNC 14 mirrors 000818 1783139406.456456 write 0x0238ec thru 0x02da11 (0xa126 bytes) 000819 1783139406.508245 write 0x000133 thru 0x00790c (0x77da bytes) 000820 1783139406.520841 mapread 0x0114a5 thru 0x01ae0a (0x9966 bytes) 000822 1783139406.544438 trunc from 0x012df3 to 0x012e89 (0x97 bytes) 000823 1783139406.549629 trunc from 0x0324a7 to 0x00dcf1 (0x247b5 bytes) 000824 1783139406.577992 write 0x03c0af thru 0x03ffff (0x3f51 bytes) 000825 1783139406.589048 MIRROR_RESYNC 14 mirrors 000827 1783139408.839126 punch from 0x037713 to 0x03acf0 (0x35de bytes) 000829 1783139408.877382 trunc from 0x037d18 to 0x038fcc (0x12b5 bytes) 000830 1783139408.882166 fallocate 0x03a49b thru 0x03a52e (0x94 bytes) 000831 1783139408.887249 punch from 0x018758 to 0x021e1e (0x96c7 bytes) 000833 1783139408.897843 MIRROR_RESYNC 14 mirrors 000834 1783139410.829632 read 0x026fe5 thru 0x036072 (0xf08e bytes) 000835 1783139410.849917 mapread 0x021816 thru 0x0267bd (0x4fa8 bytes) 000836 1783139410.872810 fallocate 0x031722 thru 0x035d4c (0x462b bytes) 000837 1783139410.924229 MIRROR_RESYNC 14 mirrors 000838 1783139413.051677 mapwrite 0x00bf3a thru 0x00f0ef (0x31b6 bytes) 000839 1783139413.182470 MIRROR_RESYNC 14 mirrors 000840 1783139415.898087 trunc from 0x00da3a to 0x0169e6 (0x8fad bytes) 000841 1783139415.941266 mapwrite 0x023209 thru 0x02e2dd (0xb0d5 bytes) 000843 1783139415.997069 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3091fb6f (stale) Mirror 16: 0x3091fb6f (stale) Mirror 19: 0x3091fb6f (stale) Mirror 21: 0x3091fb6f (stale) Mirror 25: 0x3091fb6f (stale) Mirror 29: 0x3091fb6f (stale) Mirror 30: 0x3091fb6f (stale) Mirror 31: 0x3091fb6f (stale) Mirror 32: 0x3091fb6f (stale) Mirror 33: 0x3091fb6f (stale) Mirror 34: 0x3091fb6f (stale) Mirror 35: 0x3091fb6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3091fb6f (stale) Mirror 16: 0x3091fb6f (stale) Mirror 19: 0x3091fb6f (stale) Mirror 21: 0x3091fb6f (stale) Mirror 25: 0x3091fb6f (stale) Mirror 29: 0x3091fb6f (stale) Mirror 30: 0x3091fb6f (stale) Mirror 31: 0x3091fb6f (stale) Mirror 32: 0x3091fb6f (stale) Mirror 33: 0x3091fb6f (stale) Mirror 34: 0x3091fb6f (stale) Mirror 35: 0x3091fb6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000844 1783139416.760135 mapread 0x003e1a thru 0x009168 (0x534f bytes) 000845 1783139416.782489 trunc from 0x03a3b2 to 0x03ffff (0x5c4e bytes) 000846 1783139416.788377 mapwrite 0x020305 thru 0x023241 (0x2f3d bytes) 000848 1783139416.818364 punch from 0x03cd2d to 0x03ffff (0x32d3 bytes) 000849 1783139416.824434 read 0x006542 thru 0x0149ea (0xe4a9 bytes) 000850 1783139416.831512 read 0x00ec4c thru 0x01e0c3 (0xf478 bytes) 000851 1783139416.865906 trunc from 0x040000 to 0x01480e (0x2b7f1 bytes) 000852 1783139416.904223 punch from 0x006ab1 to 0x01480e (0xdd5e bytes) 000853 1783139416.913535 mapread 0x011ef6 thru 0x01480e (0x2919 bytes) 000854 1783139416.942303 read 0x0096fa thru 0x00e60c (0x4f13 bytes) 000855 1783139416.963384 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1480f): Mirror 16: 0xeb64b5d3 (stale) Mirror 19: 0xeb64b5d3 (stale) Mirror 21: 0xeb64b5d3 (stale) Mirror 25: 0xeb64b5d3 (stale) Mirror 29: 0xeb64b5d3 (stale) Mirror 30: 0xeb64b5d3 (stale) Mirror 31: 0xeb64b5d3 (stale) Mirror 32: 0xeb64b5d3 (stale) Mirror 33: 0xeb64b5d3 (stale) Mirror 34: 0xeb64b5d3 (stale) Mirror 35: 0xeb64b5d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1480f): Mirror 16: 0xeb64b5d3 (stale) Mirror 19: 0xeb64b5d3 (stale) Mirror 21: 0xeb64b5d3 (stale) Mirror 25: 0xeb64b5d3 (stale) Mirror 29: 0xeb64b5d3 (stale) Mirror 30: 0xeb64b5d3 (stale) Mirror 31: 0xeb64b5d3 (stale) Mirror 32: 0xeb64b5d3 (stale) Mirror 33: 0xeb64b5d3 (stale) Mirror 34: 0xeb64b5d3 (stale) Mirror 35: 0xeb64b5d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1480f: skipped 000859 1783139417.634577 read 0x009c2e thru 0x00d612 (0x39e5 bytes) 000860 1783139417.665293 fallocate 0x000266 thru 0x009f69 (0x9d04 bytes) 000861 1783139417.677996 trunc from 0x00775f to 0x01480e (0xd0b0 bytes) 000862 1783139417.696876 write 0x02b25f thru 0x033be0 (0x8982 bytes) 000863 1783139417.717767 punch from 0x00d3c7 to 0x019b6d (0xc7a7 bytes) 000864 1783139417.736457 trunc from 0x0296b6 to 0x0300aa (0x69f5 bytes) 000866 1783139417.769153 mapwrite 0x03e007 thru 0x03ec1a (0xc14 bytes) 000867 1783139417.867912 mapwrite 0x0352be thru 0x03887a (0x35bd bytes) 000868 1783139417.911269 trunc from 0x0329ae to 0x033c4d (0x12a0 bytes) 000869 1783139417.926614 mapwrite 0x0115da thru 0x012ae2 (0x1509 bytes) 000870 1783139417.954160 read 0x0003ae thru 0x0086bd (0x8310 bytes) 000871 1783139417.965057 read 0x00de4a thru 0x019f12 (0xc0c9 bytes) 000872 1783139417.965758 mapwrite 0x02627a thru 0x035657 (0xf3de bytes) 000873 1783139417.997602 read 0x01b271 thru 0x025e6e (0xabfe bytes) 000874 1783139417.999113 mapread 0x00b91e thru 0x018add (0xd1c0 bytes) 000875 1783139418.008567 mapwrite 0x039f9c thru 0x03c542 (0x25a7 bytes) 000876 1783139418.029323 trunc from 0x00f5dd to 0x01459b (0x4fbf bytes) 000878 1783139418.041499 write 0x01aac9 thru 0x02871e (0xdc56 bytes) 000880 1783139418.050307 punch from 0x03e013 to 0x03ec1a (0xc08 bytes) 000881 1783139418.064154 mapread 0x00d1a1 thru 0x01978d (0xc5ed bytes) 000882 1783139418.083930 punch from 0x030ef5 to 0x03a8e2 (0x99ee bytes) 000884 1783139418.091732 punch from 0x031cf8 to 0x03ec1a (0xcf23 bytes) 000885 1783139418.100306 trunc from 0x03ec1b to 0x00eeae (0x2fd6c bytes) 000886 1783139418.141015 mapread 0x0082f8 thru 0x00eeae (0x6bb7 bytes) 000887 1783139418.142317 trunc from 0x00eeaf to 0x018f51 (0xa0a3 bytes) 000888 1783139418.166283 fallocate 0x01fb57 thru 0x026fe7 (0x7491 bytes) 000890 1783139418.173449 trunc from 0x00e485 to 0x01ac65 (0xc7e1 bytes) 000891 1783139418.181556 fallocate 0x021399 thru 0x021be3 (0x84b bytes) 000892 1783139418.194169 trunc from 0x026fe8 to 0x0070e1 (0x1ff06 bytes) 000893 1783139418.239157 trunc from 0x0070e2 to 0x008561 (0x1480 bytes) 000894 1783139418.266297 write 0x03c9ec thru 0x03ffff (0x3614 bytes) 000895 1783139418.278572 MIRROR_RESYNC 12 mirrors 000896 1783139420.384515 trunc from 0x02166e to 0x028912 (0x72a5 bytes) 000897 1783139420.422125 write 0x0317e1 thru 0x03e63f (0xce5f bytes) 000898 1783139420.441221 mapread 0x0202b0 thru 0x0235ae (0x32ff bytes) 000899 1783139420.464258 mapwrite 0x033755 thru 0x0368ef (0x319b bytes) 000900 1783139420.486866 mapwrite 0x03fca9 thru 0x03ffff (0x357 bytes) 000901 1783139420.508817 write 0x0168e6 thru 0x024b7a (0xe295 bytes) 000903 1783139420.545062 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xad0a1717 (stale) Mirror 19: 0xad0a1717 (stale) Mirror 21: 0xad0a1717 (stale) Mirror 25: 0xad0a1717 (stale) Mirror 29: 0xad0a1717 (stale) Mirror 30: 0xad0a1717 (stale) Mirror 31: 0xad0a1717 (stale) Mirror 32: 0xad0a1717 (stale) Mirror 33: 0xad0a1717 (stale) Mirror 34: 0xad0a1717 (stale) Mirror 35: 0xad0a1717 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a19dce2 Mirror 36: 0x1a19dce2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000904 1783139421.700685 trunc from 0x02e5d0 to 0x03cc7f (0xe6b0 bytes) 000905 1783139421.757348 mapwrite 0x03d183 thru 0x03fb52 (0x29d0 bytes) 000906 1783139421.797804 trunc from 0x022b34 to 0x0303f6 (0xd8c3 bytes) 000907 1783139421.805815 mapwrite 0x00751e thru 0x01141b (0x9efe bytes) 000908 1783139421.848484 punch from 0x00b457 to 0x00e51d (0x30c7 bytes) 000909 1783139421.856984 punch from 0x03d170 to 0x03ffff (0x2e90 bytes) 000911 1783139421.870400 MIRROR_RESYNC 13 mirrors 000912 1783139425.645430 write 0x033939 thru 0x03a819 (0x6ee1 bytes) 000913 1783139425.703559 fallocate 0x004d8a thru 0x00d3f6 (0x866d bytes) 000915 1783139425.728393 punch from 0x03e8f5 to 0x03ffff (0x170b bytes) 000916 1783139425.742983 fallocate 0x015934 thru 0x01b6b0 (0x5d7d bytes) 000917 1783139425.755001 fallocate 0x01e9d1 thru 0x01f14b (0x77b bytes) 000918 1783139425.763235 trunc from 0x040000 to 0x02a28d (0x15d72 bytes) 000919 1783139425.788490 write 0x012372 thru 0x019960 (0x75ef bytes) 000920 1783139425.800073 punch from 0x015966 to 0x02181f (0xbeba bytes) 000922 1783139425.814369 mapread 0x01eaa8 thru 0x029dc5 (0xb31e bytes) 000923 1783139425.836310 punch from 0x006dd3 to 0x00db3b (0x6d69 bytes) 000924 1783139425.841799 fallocate 0x00b958 thru 0x0160e8 (0xa791 bytes) 000925 1783139425.847974 write 0x0252f7 thru 0x0284f2 (0x31fc bytes) 000926 1783139425.850225 fallocate 0x029b5b thru 0x02eab0 (0x4f56 bytes) 000927 1783139425.856365 trunc from 0x02eab1 to 0x0285ea (0x64c6 bytes) 000928 1783139425.888020 punch from 0x0032fb to 0x0103ad (0xd0b3 bytes) 000929 1783139425.901129 mapwrite 0x01e8d0 thru 0x02d389 (0xeaba bytes) 000930 1783139425.951970 mapread 0x0023ab thru 0x00992a (0x7580 bytes) 000932 1783139425.962663 punch from 0x0008cf to 0x00539b (0x4acd bytes) 000933 1783139425.968415 trunc from 0x02d38a to 0x027e85 (0x5504 bytes) 000934 1783139425.985052 fallocate 0x0211a8 thru 0x021e15 (0xc6e bytes) 000935 1783139425.990480 read 0x0235dd thru 0x027e85 (0x48a9 bytes) 000936 1783139425.991122 mapwrite 0x0326a7 thru 0x032c82 (0x5dc bytes) 000937 1783139426.035441 write 0x037d76 thru 0x039ff6 (0x2281 bytes) 000938 1783139426.036620 trunc from 0x00ab05 to 0x00d360 (0x285c bytes) 000939 1783139426.043476 read 0x039f1a thru 0x039ff6 (0xdd bytes) 000941 1783139426.045178 fallocate 0x019633 thru 0x01dbb9 (0x4587 bytes) 000942 1783139426.050690 trunc from 0x02fecd to 0x030a4a (0xb7e bytes) 000943 1783139426.060653 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x39ff7): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ff7): Mirror 16: 0x56a92a9b (stale) Mirror 19: 0x56a92a9b (stale) Mirror 21: 0x56a92a9b (stale) Mirror 25: 0x56a92a9b (stale) Mirror 29: 0x56a92a9b (stale) Mirror 30: 0x56a92a9b (stale) Mirror 31: 0x56a92a9b (stale) Mirror 32: 0x56a92a9b (stale) Mirror 33: 0x56a92a9b (stale) Mirror 34: 0x56a92a9b (stale) Mirror 35: 0x56a92a9b (stale) Mirror 36: 0x56a92a9b (stale) CRC-32 checksum value for chunk [0, 0x39ff7): Mirror 1: 0xb8c7a845 Mirror 37: 0xb8c7a845 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ff7: skipped 000944 1783139427.074449 mapwrite 0x000768 thru 0x009b20 (0x93b9 bytes) 000945 1783139427.200772 write 0x005759 thru 0x0146cc (0xef74 bytes) 000947 1783139427.213535 trunc from 0x039ff7 to 0x01ad8b (0x1f26b bytes) 000948 1783139427.299383 trunc from 0x0094a6 to 0x014d3a (0xb895 bytes) 000949 1783139427.317322 trunc from 0x0173e0 to 0x0186bc (0x12dd bytes) 000950 1783139427.322406 trunc from 0x01ad8c to 0x0386c5 (0x1d93a bytes) 000951 1783139427.338502 fallocate 0x00b0db thru 0x019d42 (0xec68 bytes) 000952 1783139427.343890 read 0x017788 thru 0x0220a4 (0xa91d bytes) 000953 1783139427.353534 trunc from 0x0386c6 to 0x016a3c (0x21c89 bytes) 000954 1783139427.379331 mapread 0x009211 thru 0x012327 (0x9117 bytes) 000955 1783139427.400819 mapread 0x001b91 thru 0x00d1b8 (0xb628 bytes) 000957 1783139427.402558 fallocate 0x0321e6 thru 0x03fe42 (0xdc5d bytes) 000958 1783139427.408732 write 0x010d14 thru 0x01d243 (0xc530 bytes) 000959 1783139427.411902 punch from 0x00003c to 0x009ee6 (0x9eab bytes) 000960 1783139427.418047 punch from 0x00ca28 to 0x012a31 (0x600a bytes) 000961 1783139427.432591 read 0x014ca2 thru 0x01bcc3 (0x7022 bytes) 000962 1783139427.433347 read 0x016637 thru 0x01d243 (0x6c0d bytes) 000964 1783139427.434625 fallocate 0x013039 thru 0x018e89 (0x5e51 bytes) 000965 1783139427.441535 punch from 0x00a294 to 0x010d61 (0x6ace bytes) 000966 1783139427.446876 write 0x0159c6 thru 0x022ca8 (0xd2e3 bytes) 000967 1783139427.450140 mapwrite 0x001b94 thru 0x0055de (0x3a4b bytes) 000968 1783139427.480243 mapread 0x022b73 thru 0x022ca8 (0x136 bytes) 000969 1783139427.486678 MIRROR_RESYNC 14 mirrors 000970 1783139430.870919 punch from 0x010bc8 to 0x01eec9 (0xe302 bytes) 000971 1783139430.955722 write 0x02ac67 thru 0x02b0e5 (0x47f bytes) 000972 1783139430.962842 MIRROR_RESYNC 14 mirrors 000973 1783139434.455644 fallocate 0x0378e2 thru 0x03ffff (0x871e bytes) 000974 1783139434.499381 write 0x038f72 thru 0x03ffff (0x708e bytes) 000975 1783139434.500761 trunc from 0x032e5d to 0x03ffff (0xd1a3 bytes) 000976 1783139434.516996 trunc from 0x02b407 to 0x02f588 (0x4182 bytes) 000977 1783139434.523967 mapread 0x009414 thru 0x00a04e (0xc3b bytes) 000978 1783139434.535695 trunc from 0x02b986 to 0x02d923 (0x1f9e bytes) 000979 1783139434.541144 write 0x007fae thru 0x00800e (0x61 bytes) 000980 1783139434.551147 mapwrite 0x00ac5b thru 0x018694 (0xda3a bytes) 000981 1783139434.592372 mapread 0x03f286 thru 0x03ffff (0xd7a bytes) 000982 1783139434.601290 trunc from 0x02fddf to 0x03c630 (0xc852 bytes) 000983 1783139434.606903 mapwrite 0x00eeb3 thru 0x01692f (0x7a7d bytes) 000984 1783139434.626500 mapwrite 0x0249a7 thru 0x02d53d (0x8b97 bytes) 000985 1783139434.671003 trunc from 0x040000 to 0x013f86 (0x2c079 bytes) 000986 1783139434.700261 punch from 0x00001f to 0x000719 (0x6fb bytes) 000987 1783139434.709358 mapread 0x007f2f thru 0x00ac15 (0x2ce7 bytes) 000988 1783139434.710396 mapread 0x002319 thru 0x004760 (0x2448 bytes) 000989 1783139434.717892 mapread 0x00f108 thru 0x011d94 (0x2c8d bytes) 000991 1783139434.721034 trunc from 0x013f87 to 0x00e9a7 (0x55df bytes) 000992 1783139434.759901 punch from 0x0069a1 to 0x00bb0b (0x516b bytes) 000993 1783139434.770729 trunc from 0x00e9a8 to 0x00e6fc (0x2ab bytes) 000994 1783139434.797111 trunc from 0x005bb2 to 0x00b98e (0x5ddd bytes) 000995 1783139434.803550 write 0x01a629 thru 0x01ba27 (0x13ff bytes) 000996 1783139434.807557 fallocate 0x033f17 thru 0x03beb6 (0x7fa0 bytes) 000997 1783139434.814652 read 0x018ad5 thru 0x01f15d (0x6689 bytes) 000998 1783139434.828683 trunc from 0x03beb7 to 0x029a46 (0x12470 bytes) 000999 1783139434.867840 punch from 0x017178 to 0x021161 (0x9fea bytes) 001000 1783139434.886287 mapwrite 0x0361fb thru 0x03a0a8 (0x3eae bytes) All operations completed A-OK! PASS 70a (191s) == sanity-flr test 71: check mirror extend parent fid ==== 00:30:40 (1783139440) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x2c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 00:30:52 (1783139452) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174478 s, 12.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697802 s, 15.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 00:31:00 (1783139460) Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre2 Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3595392 .. Extending file size to 730304 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 3210048 .. Extending file size to 5255648 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 4950304 .. Extending file size to 6632704 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7294720 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8281344 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7279616 .. Extending file size to 9034912 .. Extending file size to 8965120 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8992768 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9170976 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61238 61239 61242 61244 61245 done 192.168.202.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg206-client.virtnet /mnt/lustre2 (opts:) 192.168.202.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg206-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:32:24 (1783139544) Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre2 Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5604096 .. Extending file size to 3735680 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3863936 .. Extending file size to 7578688 .. Extending file size to 6536896 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7253408 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7946944 .. Extending file size to 8116768 .. Extending file size to 8490624 .. Extending file size to 8139424 .. Extending file size to 8143936 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8164864 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8176064 .. Extending file size to 9041376 .. Extending file size to 9124960 .. Extending file size to 8535936 .. Extending file size to 8805824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62574 62576 62578 62579 62580 192.168.202.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg206-client.virtnet /mnt/lustre2 (opts:) 192.168.202.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg206-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:33:41 (1783139621) Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2432 1268872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24540 3582480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18832 3588188 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43372 7170668 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2432 1268872 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24540 3582480 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18832 3588188 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43372 7170668 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0780323 s, 26.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247085 s, 8.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6f2a8c53 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg206-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:34:00 (1783139640) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0856373 s, 24.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 00:34:09 (1783139649) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:34:18 (1783139658) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104148 s, 20.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:34:28 (1783139668) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13043 s, 16.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:34:36 (1783139676) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138095 s, 15.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22e:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:35:10 (1783139710) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg206-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960167 s, 10.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x230:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x230:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22c:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:36:07 (1783139767) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg206-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0743473 s, 28.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:36:54 (1783139814) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg206-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg206-server: Pool lustre.test_204f created Adding targets to pool oleg206-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg206-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143315 s, 14.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg206-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg206-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg206-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:38:11 (1783139891) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:38:21 (1783139901) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:38:32 (1783139912) oleg206-server: Pool lustre.test_206 created Adding targets to pool oleg206-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg206-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg206-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg206-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg206-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:39:01 (1783139941) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923984 s, 11.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419109 s, 25.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd622bff0 Mirror 3: 0xd622bff0 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:39:13 (1783139953) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:39:17 (1783139957) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:39:21 (1783139961) Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.074485 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1170000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1670000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1670000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1670000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2500000 Read this (4 bytes): BBBB 192.168.202.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg206-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:40:04 (1783140004) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0172214 s, 476 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:40:12 (1783140012) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730778 s, 14.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5c lmm_fid: [0x200000405:0x5c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5c lmm_fid: [0x200000405:0x5c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611228 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-04 00:40:14.000000000 -0400 Modify: 2026-07-04 00:40:14.000000000 -0400 Change: 2026-07-04 00:40:16.000000000 -0400 Birth: 2026-07-04 00:40:13.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:40:21 (1783140021) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391448 s, 26.8 MB/s oleg206-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg206-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg206-server: O/280000401/d20/134: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:40:43 (1783140043) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.196757 s, 208 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0443922 s, 92.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 4 00:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 4 00:40 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:40:51 (1783140051) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:41:00 (1783140060) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2016 sec =========== 00:41:08 (1783140068) === sanity-flr: start cleanup 00:41:10 (1783140070) === === sanity-flr: finish cleanup 00:41:19 (1783140079) ===