-----============= acceptance-small: sanity-flr ============----- Sun Jun 7 13:10:35 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg415-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:10:59 (1780852259) === oleg415-client.virtnet: executing check_config_client /mnt/lustre oleg415-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg415-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93d003da7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93d003da7800.idle_timeout=debug disable quota as required oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:11:43 (1780852303) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8203] == sanity-flr test 0a: lfs mirror create with -N option == 13:11:45 (1780852305) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:12:05 (1780852325) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:12:09 (1780852329) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:12:12 (1780852332) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:12:35 (1780852355) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:12:38 (1780852358) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:12:41 (1780852361) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93d003da7800.stats=clear osc.lustre-OST0001-osc-ffff93d003da7800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:12:50 (1780852370) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:13:02 (1780852382) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:13:12 (1780852392) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:13:17 (1780852397) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:13:25 (1780852405) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 13:13:35 (1780852415) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:13:42 (1780852422) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:13:49 (1780852429) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:13:56 (1780852436) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:14:07 (1780852447) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:14:14 (1780852454) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.703541 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10456 s, 10.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:14:24 (1780852464) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.317562 s, 26.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:14:32 (1780852472) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:14:42 (1780852482) Stopping /mnt/lustre-ost1 (opts:) on oleg415-server oleg415-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg415-server: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg415-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg415-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg415-server oleg415-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg415-server: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg415-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (136s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:16:58 (1780852618) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.436782 s, 21.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg415-server oleg415-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg415-server: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg415-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:17:43 (1780852663) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg415-server oleg415-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg415-server: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg415-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg415-server oleg415-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg415-server: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg415-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:19:01 (1780852741) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.09258 s, 17.0 MB/s -rw-r--r-- 1 root root 52428800 Jun 7 13:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.457s user 0m0.000s sys 0m0.193s Stopping /mnt/lustre-ost1 (opts:) on oleg415-server oleg415-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d003da7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.137s user 0m0.000s sys 0m0.205s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg415-server: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg415-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg415-server oleg415-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.546s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg415-server: oleg415-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg415-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93d003da7800.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:20:36 (1780852836) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:20:39 (1780852839) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:20:42 (1780852842) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 13:20:45 (1780852845) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 13:20:51 (1780852851) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.577286 s, 21.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:21:02 (1780852862) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374123 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236008 s, 44.4 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:21:08 (1780852868) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:21:15 (1780852875) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00296451 s, 345 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00223921 s, 457 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00221159 s, 463 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125908 s, 813 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 13:21:25 (1780852885) create test files with size 15739503 .. 37749016702e2b03c39a774eccc9e7ed - eb1e9d03da388bb7e3d46ff3e63d2355 - f9676a28166e8291fd22cd6885fcc80c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg415-client.virtnet /mnt/lustre (opts:) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 13:21:47 (1780852907) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:21:59 (1780852919) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:22:06 (1780852926) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724918 s, 14.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492676 s, 21.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 13:22:15 (1780852935) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139178 s, 15.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.334203 s, 12.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140345 s, 14.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 13:22:29 (1780852949) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 13:22:32 (1780852952) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:22:35 (1780852955) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348967 s, 30.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 13:22:44 (1780852964) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.319456 s, 9.8 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:22:54 (1780852974) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:23:04 (1780852984) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.356777 s, 29.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg415-client.virtnet /mnt/lustre (opts:) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:23:16 (1780852996) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff93d0032e3000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.285289 s, 36.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg415-client.virtnet /mnt/lustre (opts:) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff93d01063e000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:23:33 (1780853013) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:24:10 (1780853050) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:24:53 (1780853093) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 13:24:56 (1780853096) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.375826 s, 8.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.365355 s, 8.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x743aa1b1 (nosync,stale) Mirror 2: 0x743aa1b1 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d341b51eecade54f5b30eff9928fadd5 - mirror 1 checksum: adf02a11aa91ef59e4d73b6830e02ebe - mirror 2 checksum: d341b51eecade54f5b30eff9928fadd5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jun 7 13:25:00 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jun 7 13:25:00 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d341b51eecade54f5b30eff9928fadd5 - mirror 1 checksum: adf02a11aa91ef59e4d73b6830e02ebe - mirror 2 checksum: adf02a11aa91ef59e4d73b6830e02ebe - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jun 7 13:25:00 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jun 7 13:25:00 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:25:09 (1780853109) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323299 s, 32.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:25:17 (1780853117) Failing mds1 on oleg415-server Stopping /mnt/lustre-mds1 (opts:) on oleg415-server 13:25:29 (1780853129) shut down facet: mds1 facet_host: oleg415-server facet_failover_host: oleg415-server Failover mds1 to oleg415-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 13:26:01 (1780853161) targets are mounted 13:26:01 (1780853161) facet_failover done oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265901 s, 39.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:26:17 (1780853177) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0495054 s, 414 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0184888 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:26:27 (1780853187) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0475328 s, 431 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0145845 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:26:39 (1780853199) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0120188 s, 1.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:26:48 (1780853208) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0725396 s, 282 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (11s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:26:59 (1780853219) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00551807 s, 742 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0389865 s, 105 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:27:06 (1780853226) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:27:31 (1780853251) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:27:59 (1780853279) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:28:22 (1780853302) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:28:46 (1780853326) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 13:28:54 (1780853334) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57162 mirror split pid 57163 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7730: 57162 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6904: 57163 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6904: 57163 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57163 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 13:30:02 (1780853402) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 3404 skipping zero size read skipping zero size read skipping zero size read 000004 1780853405.105398 write 0x003182 thru 0x007f2c (0x4dab bytes) 000005 1780853405.108731 mapread 0x0002c7 thru 0x007f2c (0x7c66 bytes) 000007 1780853405.130978 mapwrite 0x01f9a2 thru 0x021f19 (0x2578 bytes) truncating to largest ever: 0xd5cd 000008 1780853405.220632 trunc from 0x021f1a to 0x00d5cc (0x1494d bytes) 000010 1780853405.296413 punch from 0x007280 to 0x00d5cc (0x634d bytes) 000011 1780853405.313689 read 0x00bac9 thru 0x00d5cc (0x1b04 bytes) 000012 1780853405.325010 mapwrite 0x03a8fb thru 0x03ffff (0x5705 bytes) 000013 1780853405.468381 read 0x0251ac thru 0x0260bd (0xf12 bytes) 000014 1780853405.493291 mapread 0x01a1aa thru 0x01ba85 (0x18dc bytes) 000015 1780853405.496277 mapwrite 0x01870c thru 0x02575b (0xd050 bytes) 000016 1780853405.535281 write 0x009435 thru 0x014fd6 (0xbba2 bytes) 000018 1780853405.543566 write 0x015d34 thru 0x01c871 (0x6b3e bytes) 000019 1780853405.548593 read 0x03a437 thru 0x03ffff (0x5bc9 bytes) 000020 1780853405.550210 trunc from 0x040000 to 0x005800 (0x3a7ff bytes) 000021 1780853405.602655 read 0x005612 thru 0x005800 (0x1ef bytes) truncating to largest ever: 0x15650 000022 1780853405.603564 trunc from 0x005801 to 0x01564f (0xfe4f bytes) 000023 1780853405.651376 punch from 0x003e01 to 0x008a9e (0x4c9e bytes) 000024 1780853405.658987 mapwrite 0x01f86e thru 0x02eee3 (0xf676 bytes) 000025 1780853405.732842 trunc from 0x02a6fe to 0x02d157 (0x2a5a bytes) 000026 1780853405.738132 write 0x015578 thru 0x016b7a (0x1603 bytes) 000028 1780853405.742067 trunc from 0x02eee4 to 0x00dff9 (0x20eea bytes) 000029 1780853405.773779 punch from 0x00dcaf to 0x00dff9 (0x34b bytes) 000030 1780853405.781023 punch from 0x008abc to 0x00dff9 (0x553e bytes) 000031 1780853405.786664 write 0x03f6e6 thru 0x03ffff (0x91a bytes) 000032 1780853405.788282 write 0x00ff15 thru 0x01377b (0x3867 bytes) fallocating to largest ever: 0x40000 000033 1780853405.805983 fallocate 0x03f9a7 thru 0x03ffff (0x659 bytes) 000034 1780853405.819637 mapwrite 0x01e866 thru 0x02dbc8 (0xf363 bytes) 000035 1780853405.966847 trunc from 0x040000 to 0x016d7d (0x29282 bytes) 000036 1780853405.988346 punch from 0x016852 to 0x016d7d (0x52c bytes) 000038 1780853405.995122 fallocate 0x0023ff thru 0x010a69 (0xe66b bytes) 000039 1780853406.002702 trunc from 0x016d7e to 0x0144c8 (0x28b5 bytes) 000040 1780853406.022432 write 0x01992a thru 0x021f29 (0x8600 bytes) 000041 1780853406.026389 trunc from 0x021f2a to 0x005ecd (0x1c05c bytes) 000042 1780853406.050345 punch from 0x0027ff to 0x005ecd (0x36cf bytes) 000043 1780853406.059034 mapwrite 0x00687e thru 0x00b197 (0x491a bytes) 000044 1780853406.140338 read 0x0031cf thru 0x008c89 (0x5abb bytes) 000045 1780853406.151379 punch from 0x003e80 to 0x00b197 (0x7318 bytes) 000046 1780853406.164834 mapread 0x007bf5 thru 0x009c64 (0x2070 bytes) 000047 1780853406.185679 fallocate 0x00951f thru 0x00ede9 (0x58cb bytes) 000048 1780853406.192507 trunc from 0x0065b4 to 0x00ede9 (0x8836 bytes) 000049 1780853406.198876 punch from 0x00b146 to 0x00ede9 (0x3ca4 bytes) 000050 1780853406.204431 mapread 0x00c63d thru 0x00ede9 (0x27ad bytes) 000052 1780853406.221668 fallocate 0x01ce2c thru 0x023fab (0x7180 bytes) 000053 1780853406.229251 read 0x00c562 thru 0x00ede9 (0x2888 bytes) 000054 1780853406.230836 fallocate 0x00383d thru 0x010aaa (0xd26e bytes) 000055 1780853406.237434 read 0x000682 thru 0x00aa57 (0xa3d6 bytes) 000056 1780853406.255261 punch from 0x00e492 to 0x010aaa (0x2619 bytes) 000057 1780853406.263339 punch from 0x00ab77 to 0x00e089 (0x3513 bytes) 000058 1780853406.269919 punch from 0x0105dc to 0x010aaa (0x4cf bytes) 000059 1780853406.277112 write 0x01884f thru 0x027c0c (0xf3be bytes) 000060 1780853406.282680 mapread 0x003e22 thru 0x00a119 (0x62f8 bytes) 000061 1780853406.299695 fallocate 0x02b8b7 thru 0x02cb7e (0x12c8 bytes) 000062 1780853406.307432 fallocate 0x031b9e thru 0x035da2 (0x4205 bytes) 000063 1780853406.328026 MIRROR_EXTEND to 2 mirrors 000064 1780853407.152361 mapread 0x010f77 thru 0x0127b8 (0x1842 bytes) 000066 1780853407.180864 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) 000067 1780853407.387107 trunc from 0x02cb7f to 0x006110 (0x26a6e bytes) 000068 1780853407.446389 mapwrite 0x02bbda thru 0x032a0f (0x6e36 bytes) 000069 1780853407.506602 read 0x017b9c thru 0x022fae (0xb413 bytes) 000071 1780853407.519750 trunc from 0x032a10 to 0x022674 (0x1039b bytes) 000072 1780853407.537544 mapread 0x00bf3a thru 0x00e00b (0x20d2 bytes) 000073 1780853407.552278 trunc from 0x022675 to 0x009e56 (0x1881e bytes) 000074 1780853407.588466 trunc from 0x009e57 to 0x03e8b6 (0x34a60 bytes) 000075 1780853407.614117 mapwrite 0x02dc31 thru 0x033316 (0x56e6 bytes) 000076 1780853407.677047 fallocate 0x034c15 thru 0x03ffff (0xb3eb bytes) 000078 1780853407.684813 mapread 0x02f4df thru 0x030f46 (0x1a68 bytes) 000079 1780853407.691031 MIRROR_EXTEND to 2 mirrors 000080 1780853408.543389 fallocate 0x03950a thru 0x03ffff (0x6af6 bytes) 000081 1780853408.583339 punch from 0x007638 to 0x014bc6 (0xd58f bytes) 000082 1780853408.589306 fallocate 0x03b8a9 thru 0x03ffff (0x4757 bytes) 000083 1780853408.604488 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x257f6ac3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x257f6ac3 Mirror 3: 0x257f6ac3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000084 1780853409.613878 mapread 0x03be3c thru 0x03ffff (0x41c4 bytes) 000085 1780853409.651720 mapread 0x01b878 thru 0x0290d6 (0xd85f bytes) 000086 1780853409.693360 trunc from 0x040000 to 0x02da90 (0x1256f bytes) 000087 1780853409.773030 mapwrite 0x00afb2 thru 0x019498 (0xe4e7 bytes) 000088 1780853409.840916 mapread 0x0035a4 thru 0x006e05 (0x3862 bytes) 000089 1780853409.848020 trunc from 0x0096de to 0x00da8d (0x43b0 bytes) 000090 1780853409.873518 write 0x01cf57 thru 0x022d66 (0x5e10 bytes) 000091 1780853409.901373 read 0x01796b thru 0x01ba16 (0x40ac bytes) 000092 1780853409.902242 read 0x01bb55 thru 0x0290e3 (0xd58f bytes) 000093 1780853409.963481 mapwrite 0x018a26 thru 0x01c90d (0x3ee8 bytes) 000094 1780853410.007503 read 0x01e520 thru 0x0224c2 (0x3fa3 bytes) 000095 1780853410.008383 trunc from 0x02da91 to 0x03a552 (0xcac2 bytes) 000096 1780853410.041468 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a553): Mirror 3: 0xbffcf98a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a553): Mirror 3: 0xbffcf98a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a553: skipped 000097 1780853410.379375 read 0x016140 thru 0x018633 (0x24f4 bytes) 000098 1780853410.387008 trunc from 0x03a553 to 0x0203e1 (0x1a171 bytes) 000099 1780853410.421963 trunc from 0x0203e2 to 0x011b8b (0xe856 bytes) 000100 1780853410.447556 fallocate 0x03ecd9 thru 0x03ffff (0x1327 bytes) 000101 1780853410.457194 fallocate 0x010733 thru 0x01ef25 (0xe7f3 bytes) 000102 1780853410.463624 punch from 0x0142be to 0x019093 (0x4dd6 bytes) 000103 1780853410.470254 fallocate 0x027093 thru 0x030aa2 (0x9a10 bytes) 000104 1780853410.477474 mapwrite 0x005db6 thru 0x008511 (0x275c bytes) 000105 1780853410.527667 punch from 0x02f1fc to 0x03ab53 (0xb958 bytes) 000106 1780853410.539622 trunc from 0x040000 to 0x01a50a (0x25af5 bytes) 000107 1780853410.563951 read 0x019062 thru 0x01a50a (0x14a9 bytes) 000108 1780853410.580730 trunc from 0x01a50b to 0x0094bb (0x1104f bytes) 000109 1780853410.618678 MIRROR_RESYNC 2 mirrors 000110 1780853411.246965 mapwrite 0x0273ec thru 0x02fcef (0x8904 bytes) 000111 1780853411.401043 mapwrite 0x01602d thru 0x01a7e5 (0x47b9 bytes) 000113 1780853411.452639 trunc from 0x010e54 to 0x01cec8 (0xc075 bytes) 000114 1780853411.475508 mapwrite 0x00806a thru 0x00b095 (0x302c bytes) 000116 1780853411.555075 trunc from 0x02fcf0 to 0x038e7c (0x918d bytes) 000117 1780853411.607179 trunc from 0x038e7d to 0x000216 (0x38c66 bytes) 000119 1780853411.627233 write 0x038e2f thru 0x03fb48 (0x6d1a bytes) 000120 1780853411.629672 trunc from 0x026597 to 0x027cb8 (0x1722 bytes) 000121 1780853411.634876 trunc from 0x03fb49 to 0x02aff1 (0x14b57 bytes) 000122 1780853411.653886 write 0x028285 thru 0x030681 (0x83fd bytes) 000123 1780853411.668107 read 0x0234b0 thru 0x02531c (0x1e6d bytes) 000125 1780853411.687973 write 0x02eeac thru 0x037d3a (0x8e8f bytes) 000126 1780853411.691778 read 0x01fb5e thru 0x01fe64 (0x307 bytes) 000127 1780853411.705016 write 0x03abd1 thru 0x03c5c2 (0x19f2 bytes) 000128 1780853411.710966 MIRROR_RESYNC 2 mirrors 000129 1780853412.476200 write 0x00db54 thru 0x01860a (0xaab7 bytes) 000130 1780853412.558038 read 0x014780 thru 0x015f90 (0x1811 bytes) 000131 1780853412.560044 trunc from 0x002be4 to 0x012237 (0xf654 bytes) 000132 1780853412.585910 read 0x00066b thru 0x00a36d (0x9d03 bytes) 000133 1780853412.603352 trunc from 0x01d5a5 to 0x01df0d (0x969 bytes) 000134 1780853412.613189 trunc from 0x03885d to 0x03c5c2 (0x3d66 bytes) 000135 1780853412.631564 mapwrite 0x019a1e thru 0x01f7bb (0x5d9e bytes) 000136 1780853412.702672 trunc from 0x006423 to 0x012c86 (0xc864 bytes) 000137 1780853412.710093 fallocate 0x0273d0 thru 0x02a684 (0x32b5 bytes) 000138 1780853412.717931 trunc from 0x03a5a3 to 0x03c5c2 (0x2020 bytes) 000139 1780853412.735098 punch from 0x016bc7 to 0x017835 (0xc6f bytes) 000141 1780853412.756555 mapwrite 0x0086b0 thru 0x011c9f (0x95f0 bytes) 000142 1780853412.905073 mapread 0x0067ed thru 0x00be08 (0x561c bytes) 000143 1780853412.918723 trunc from 0x03c5c3 to 0x02e6a5 (0xdf1d bytes) 000144 1780853412.990871 trunc from 0x02e6a6 to 0x011bbc (0x1cae9 bytes) 000146 1780853413.044641 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) 000147 1780853413.238215 fallocate 0x0133b8 thru 0x01574b (0x2394 bytes) 000148 1780853413.248379 read 0x00fb48 thru 0x011bbc (0x2075 bytes) 000149 1780853413.253749 fallocate 0x03308e thru 0x0330ed (0x60 bytes) 000150 1780853413.263587 write 0x021060 thru 0x02de7f (0xce20 bytes) 000152 1780853413.298979 fallocate 0x003749 thru 0x00e4d3 (0xad8b bytes) 000153 1780853413.308357 punch from 0x001771 to 0x009209 (0x7a99 bytes) 000154 1780853413.326873 mapread 0x022e4d thru 0x02fb0a (0xccbe bytes) 000155 1780853413.349744 punch from 0x013fb8 to 0x014fb7 (0x1000 bytes) 000156 1780853413.359206 trunc from 0x0197ba to 0x027771 (0xdfb8 bytes) 000158 1780853413.394521 mapread 0x01a904 thru 0x024fa0 (0xa69d bytes) 000160 1780853413.428494 MIRROR_EXTEND to 2 mirrors 000162 1780853414.503266 trunc from 0x01e9e5 to 0x02ba94 (0xd0b0 bytes) 000163 1780853414.578239 mapread 0x01194c thru 0x018385 (0x6a3a bytes) 000164 1780853414.599425 write 0x00b114 thru 0x0151d7 (0xa0c4 bytes) 000165 1780853414.618868 mapread 0x00c975 thru 0x0185f1 (0xbc7d bytes) 000166 1780853414.620171 trunc from 0x01e85e to 0x024856 (0x5ff9 bytes) 000167 1780853414.626403 trunc from 0x0330ee to 0x00739f (0x2bd4e bytes) 000168 1780853414.660874 trunc from 0x0073a0 to 0x00ede1 (0x7a42 bytes) 000169 1780853414.706922 mapread 0x00ce02 thru 0x00ede1 (0x1fe0 bytes) 000170 1780853414.733272 read 0x004b43 thru 0x00c1b7 (0x7675 bytes) 000171 1780853414.734440 trunc from 0x00ede2 to 0x014fd5 (0x61f4 bytes) 000172 1780853414.763653 punch from 0x002da0 to 0x00bdce (0x902f bytes) 000173 1780853414.773845 mapwrite 0x00379b thru 0x00aace (0x7334 bytes) 000174 1780853414.868557 MIRROR_RESYNC 2 mirrors 000175 1780853415.598942 trunc from 0x014fd6 to 0x00c324 (0x8cb1 bytes) 000176 1780853415.658611 trunc from 0x0047c2 to 0x00c324 (0x7b63 bytes) 000177 1780853415.665872 mapwrite 0x01ce9c thru 0x02497c (0x7ae1 bytes) 000178 1780853415.758118 trunc from 0x01369d to 0x0234e0 (0xfe44 bytes) 000179 1780853415.765691 trunc from 0x001dc0 to 0x002b4f (0xd90 bytes) 000180 1780853415.776329 fallocate 0x03545b thru 0x03ffff (0xaba5 bytes) 000181 1780853415.789015 write 0x02f459 thru 0x036e4e (0x79f6 bytes) 000182 1780853415.800375 punch from 0x011a95 to 0x01ff1f (0xe48b bytes) 000183 1780853415.806810 trunc from 0x02d9e3 to 0x02e682 (0xca0 bytes) 000184 1780853415.814448 write 0x00b59f thru 0x0105aa (0x500c bytes) 000188 1780853415.843090 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) 000189 1780853416.044967 punch from 0x0230ac to 0x0284e6 (0x543b bytes) 000190 1780853416.052059 trunc from 0x040000 to 0x001b55 (0x3e4aa bytes) 000191 1780853416.093494 punch from 0x00186e to 0x001b55 (0x2e8 bytes) 000192 1780853416.103183 mapwrite 0x037330 thru 0x03ffff (0x8cd0 bytes) 000194 1780853416.172927 trunc from 0x040000 to 0x000517 (0x3fae8 bytes) 000195 1780853416.189746 mapread 0x000385 thru 0x000517 (0x193 bytes) 000196 1780853416.195921 mapwrite 0x02e219 thru 0x034ac6 (0x68ae bytes) 000197 1780853416.298878 mapread 0x02db58 thru 0x034ac6 (0x6f6f bytes) 000199 1780853416.302288 fallocate 0x0166f7 thru 0x0218ef (0xb1f9 bytes) 000200 1780853416.316622 read 0x0022fc thru 0x00be15 (0x9b1a bytes) 000201 1780853416.335125 write 0x008447 thru 0x00d339 (0x4ef3 bytes) 000203 1780853416.352035 mapwrite 0x00d76c thru 0x011cc2 (0x4557 bytes) 000204 1780853416.432182 trunc from 0x02c140 to 0x031f77 (0x5e38 bytes) 000205 1780853416.442863 write 0x033aa9 thru 0x03d4a1 (0x99f9 bytes) 000206 1780853416.444834 fallocate 0x009ca4 thru 0x00cb2f (0x2e8c bytes) 000207 1780853416.467050 punch from 0x023aa2 to 0x02ce4c (0x93ab bytes) 000208 1780853416.489818 trunc from 0x03d4a2 to 0x02bd56 (0x1174b bytes) 000209 1780853416.520327 mapwrite 0x031e0b thru 0x03ffff (0xe1f5 bytes) 000210 1780853416.608839 MIRROR_EXTEND to 2 mirrors 000211 1780853417.657011 trunc from 0x02f695 to 0x034442 (0x4dae bytes) 000214 1780853417.726297 MIRROR_RESYNC 2 mirrors 000215 1780853418.500443 MIRROR_EXTEND to 3 mirrors 000216 1780853419.237321 trunc from 0x004ee7 to 0x01213b (0xd255 bytes) 000217 1780853419.269830 trunc from 0x008e73 to 0x00bfe5 (0x3173 bytes) 000218 1780853419.281251 mapread 0x03425a thru 0x036365 (0x210c bytes) 000219 1780853419.322332 mapwrite 0x02f023 thru 0x033955 (0x4933 bytes) 000220 1780853419.363031 mapread 0x025e76 thru 0x02940d (0x3598 bytes) 000221 1780853419.394773 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x613ac244 (stale) Mirror 3: 0x613ac244 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5698a53c Mirror 4: 0x5698a53c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000222 1780853420.437536 trunc from 0x0337e3 to 0x03ffff (0xc81d bytes) 000223 1780853420.504317 punch from 0x039581 to 0x03d40a (0x3e8a bytes) 000224 1780853420.518255 read 0x00b00d thru 0x012ba2 (0x7b96 bytes) 000225 1780853420.549541 fallocate 0x030dcd thru 0x03a691 (0x98c5 bytes) 000226 1780853420.555874 punch from 0x03b26b to 0x03ff8e (0x4d24 bytes) 000227 1780853420.560622 trunc from 0x01f6f2 to 0x02dcc1 (0xe5d0 bytes) 000228 1780853420.566717 mapwrite 0x00819e thru 0x0144c2 (0xc325 bytes) 000229 1780853420.623785 fallocate 0x019614 thru 0x0242fd (0xacea bytes) 000230 1780853420.636024 punch from 0x00ad48 to 0x01607c (0xb335 bytes) 000231 1780853420.642004 mapread 0x0386fc thru 0x03987e (0x1183 bytes) 000232 1780853420.660397 trunc from 0x040000 to 0x037554 (0x8aab bytes) 000233 1780853420.738851 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37555): Mirror 2: 0x3180858c (stale) Mirror 4: 0x5ebb1c5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37555): Mirror 2: 0x3180858c (stale) Mirror 4: 0x5ebb1c5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37555: skipped 000234 1780853421.147309 mapread 0x017eec thru 0x022f41 (0xb056 bytes) 000235 1780853421.204198 trunc from 0x022cfa to 0x032310 (0xf617 bytes) 000236 1780853421.212589 mapwrite 0x0042db thru 0x004fe9 (0xd0f bytes) 000237 1780853421.247410 fallocate 0x01e8aa thru 0x02d48d (0xebe4 bytes) 000239 1780853421.255757 read 0x01fccd thru 0x029e92 (0xa1c6 bytes) 000240 1780853421.266323 mapwrite 0x019620 thru 0x01ece9 (0x56ca bytes) 000241 1780853421.291657 fallocate 0x00f2f3 thru 0x015ca6 (0x69b4 bytes) 000242 1780853421.305187 fallocate 0x02ae86 thru 0x034c97 (0x9e12 bytes) 000243 1780853421.318471 write 0x03e91a thru 0x03ffff (0x16e6 bytes) 000244 1780853421.327560 trunc from 0x0249d4 to 0x02a1d9 (0x5806 bytes) 000245 1780853421.339393 punch from 0x02b9c4 to 0x032a5f (0x709c bytes) 000246 1780853421.351524 read 0x0324e8 thru 0x03ffff (0xdb18 bytes) 000247 1780853421.369727 trunc from 0x040000 to 0x0109c1 (0x2f63e bytes) 000248 1780853421.405619 mapread 0x006567 thru 0x00688b (0x325 bytes) 000249 1780853421.406861 write 0x016963 thru 0x0227ef (0xbe8d bytes) 000250 1780853421.410234 fallocate 0x033070 thru 0x038f0a (0x5e9b bytes) 000251 1780853421.419606 trunc from 0x02c4e7 to 0x02ea42 (0x255c bytes) 000252 1780853421.430166 mapread 0x00c17b thru 0x01ae98 (0xed1e bytes) 000253 1780853421.490035 punch from 0x000d92 to 0x0034d1 (0x2740 bytes) 000254 1780853421.497016 write 0x03144f thru 0x03d473 (0xc025 bytes) 000255 1780853421.508729 trunc from 0x03d474 to 0x01c587 (0x20eec bytes) 000256 1780853421.531896 punch from 0x00415c to 0x00df9d (0x9e42 bytes) 000257 1780853421.540290 fallocate 0x006281 thru 0x012be9 (0xc969 bytes) 000258 1780853421.545908 mapwrite 0x03e8a6 thru 0x03ffff (0x175a bytes) 000259 1780853421.590797 fallocate 0x022f92 thru 0x0234d3 (0x542 bytes) 000260 1780853421.597132 write 0x009e96 thru 0x00a65b (0x7c6 bytes) 000261 1780853421.605455 trunc from 0x0219e4 to 0x0249a3 (0x2fc0 bytes) 000263 1780853421.624842 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5698a53c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5698a53c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000264 1780853422.007138 write 0x02e581 thru 0x03ba7d (0xd4fd bytes) 000265 1780853422.024391 trunc from 0x00ae32 to 0x014c42 (0x9e11 bytes) 000266 1780853422.029930 mapread 0x01c501 thru 0x027c6d (0xb76d bytes) 000267 1780853422.071194 MIRROR_RESYNC 2 mirrors 000269 1780853423.276894 mapread 0x0009a2 thru 0x004e9f (0x44fe bytes) 000270 1780853423.344946 mapread 0x013f7a thru 0x0153c0 (0x1447 bytes) 000271 1780853423.371570 read 0x021ba4 thru 0x021d14 (0x171 bytes) 000272 1780853423.385928 trunc from 0x006287 to 0x0082a1 (0x201b bytes) 000273 1780853423.437365 read 0x00c7aa thru 0x00eeed (0x2744 bytes) 000274 1780853423.456895 mapwrite 0x0190e1 thru 0x024d86 (0xbca6 bytes) 000275 1780853423.478015 trunc from 0x03e41e to 0x03f388 (0xf6b bytes) 000276 1780853423.482665 mapwrite 0x02ca20 thru 0x031660 (0x4c41 bytes) 000277 1780853423.501477 mapread 0x003e19 thru 0x009229 (0x5411 bytes) 000278 1780853423.503123 mapread 0x013b29 thru 0x019aa3 (0x5f7b bytes) 000279 1780853423.508952 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) 000280 1780853423.734018 fallocate 0x0297db thru 0x02e848 (0x506e bytes) 000281 1780853423.743137 trunc from 0x040000 to 0x024620 (0x1b9df bytes) 000282 1780853423.833914 fallocate 0x032408 thru 0x03fbec (0xd7e5 bytes) 000283 1780853423.842855 mapwrite 0x0215db thru 0x02c7b4 (0xb1da bytes) 000284 1780853423.972981 trunc from 0x03fbed to 0x01e502 (0x216ea bytes) 000285 1780853424.031916 punch from 0x01438c to 0x01e502 (0xa177 bytes) 000287 1780853424.049198 mapread 0x0173c6 thru 0x01e502 (0x713d bytes) 000288 1780853424.078596 mapread 0x00680b thru 0x00812b (0x1921 bytes) 000289 1780853424.098082 punch from 0x01a5d6 to 0x01e502 (0x3f2d bytes) 000290 1780853424.106058 fallocate 0x0118b4 thru 0x016251 (0x499e bytes) 000291 1780853424.113993 fallocate 0x00c62d thru 0x00e951 (0x2325 bytes) 000293 1780853424.127010 punch from 0x00fecf to 0x01955a (0x968c bytes) 000294 1780853424.136668 mapwrite 0x02ec4a thru 0x037de9 (0x91a0 bytes) 000295 1780853424.219829 MIRROR_EXTEND to 2 mirrors 000296 1780853424.990499 read 0x02a2e6 thru 0x032c15 (0x8930 bytes) 000297 1780853425.001071 write 0x01167c thru 0x01f937 (0xe2bc bytes) 000298 1780853425.083788 mapwrite 0x0066f6 thru 0x012bf4 (0xc4ff bytes) 000300 1780853425.133526 mapread 0x023618 thru 0x02e2cd (0xacb6 bytes) 000301 1780853425.169454 punch from 0x019063 to 0x0274d2 (0xe470 bytes) 000302 1780853425.176039 fallocate 0x03ab37 thru 0x03ffff (0x54c9 bytes) 000303 1780853425.182662 trunc from 0x009ab5 to 0x019a3e (0xff8a bytes) 000304 1780853425.190969 fallocate 0x02100b thru 0x02c652 (0xb648 bytes) 000305 1780853425.208812 trunc from 0x037dea to 0x00452c (0x338bd bytes) 000306 1780853425.300962 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) 000308 1780853425.519202 read 0x003c9a thru 0x00452c (0x893 bytes) 000310 1780853425.528308 fallocate 0x0055ea thru 0x00edc6 (0x97dd bytes) 000311 1780853425.537510 mapread 0x002031 thru 0x00452c (0x24fc bytes) 000313 1780853425.551467 mapwrite 0x0166ce thru 0x01ac00 (0x4533 bytes) 000315 1780853425.687903 read 0x0009c6 thru 0x007f59 (0x7594 bytes) 000318 1780853425.710089 fallocate 0x022741 thru 0x02bd9e (0x965e bytes) 000319 1780853425.719744 mapread 0x02bd5d thru 0x02bd9e (0x42 bytes) 000321 1780853425.762896 punch from 0x01a92d to 0x0295e9 (0xecbd bytes) 000322 1780853425.773062 trunc from 0x02bd9f to 0x004f8f (0x26e0f bytes) 000323 1780853425.809828 trunc from 0x002c61 to 0x004f8f (0x232f bytes) 000324 1780853425.819844 mapwrite 0x0263dc thru 0x02ead9 (0x86fe bytes) 000326 1780853425.940887 write 0x03dcb9 thru 0x03ffff (0x2347 bytes) 000327 1780853425.944833 read 0x0026fe thru 0x00841a (0x5d1d bytes) 000328 1780853425.972522 write 0x025645 thru 0x02db8d (0x8549 bytes) 000329 1780853425.973835 punch from 0x003e15 to 0x008a97 (0x4c83 bytes) 000330 1780853425.980551 write 0x0264c2 thru 0x02c969 (0x64a8 bytes) 000331 1780853425.986530 fallocate 0x025ba5 thru 0x0320c8 (0xc524 bytes) 000332 1780853425.992579 mapread 0x01dc71 thru 0x0208ac (0x2c3c bytes) 000333 1780853425.998387 mapwrite 0x02aaae thru 0x0349d7 (0x9f2a bytes) 000336 1780853426.050129 MIRROR_EXTEND to 2 mirrors 000337 1780853426.909322 fallocate 0x004116 thru 0x007872 (0x375d bytes) 000338 1780853426.978693 punch from 0x001ced to 0x011a9c (0xfdb0 bytes) 000339 1780853426.999014 mapwrite 0x01d163 thru 0x027234 (0xa0d2 bytes) 000340 1780853427.086219 fallocate 0x038ab4 thru 0x039f74 (0x14c1 bytes) 000341 1780853427.104221 punch from 0x01e682 to 0x0201c7 (0x1b46 bytes) 000342 1780853427.139883 punch from 0x00dc15 to 0x01560e (0x79fa bytes) 000343 1780853427.149173 write 0x03e526 thru 0x03e64c (0x127 bytes) 000344 1780853427.201434 trunc from 0x040000 to 0x02736b (0x18c94 bytes) 000345 1780853427.254986 trunc from 0x02736c to 0x036088 (0xed1d bytes) 000346 1780853427.296422 trunc from 0x00077f to 0x00975e (0x8fe0 bytes) 000347 1780853427.308367 write 0x03ef8c thru 0x03ffff (0x1074 bytes) 000348 1780853427.319345 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd56e6930 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x828a7e0c Mirror 3: 0x828a7e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000349 1780853428.695771 write 0x034daa thru 0x03d380 (0x85d7 bytes) 000350 1780853428.762551 mapread 0x033cc4 thru 0x034c30 (0xf6d bytes) 000351 1780853428.787322 trunc from 0x02a2b6 to 0x02a61b (0x366 bytes) 000352 1780853428.796632 fallocate 0x029eb5 thru 0x035c61 (0xbdad bytes) 000353 1780853428.801667 punch from 0x0367cc to 0x03ffff (0x9834 bytes) 000354 1780853428.833091 mapwrite 0x010f33 thru 0x0173f1 (0x64bf bytes) 000355 1780853428.891207 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x828a7e0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x828a7e0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000356 1780853429.290666 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x828a7e0c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3876fd8b Mirror 4: 0x3876fd8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000357 1780853430.401238 mapwrite 0x03265f thru 0x033e2b (0x17cd bytes) 000358 1780853430.568980 mapread 0x031c36 thru 0x03ffff (0xe3ca bytes) 000359 1780853430.587711 read 0x009a26 thru 0x0109c2 (0x6f9d bytes) 000360 1780853430.621808 MIRROR_RESYNC 3 mirrors 000361 1780853432.449496 write 0x01335b thru 0x018b62 (0x5808 bytes) 000362 1780853432.563113 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6d1cc9cb (stale) Mirror 4: 0x6d1cc9cb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe086497d Mirror 5: 0xe086497d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000363 1780853434.268310 punch from 0x008582 to 0x014b6a (0xc5e9 bytes) 000364 1780853434.403226 write 0x01ff81 thru 0x027377 (0x73f7 bytes) 000366 1780853434.438507 trunc from 0x02b01e to 0x03ad47 (0xfd2a bytes) 000367 1780853434.447652 trunc from 0x040000 to 0x011ef0 (0x2e10f bytes) 000368 1780853434.511554 trunc from 0x0097a4 to 0x011ef0 (0x874d bytes) 000369 1780853434.528613 punch from 0x007466 to 0x007f45 (0xae0 bytes) 000370 1780853434.551115 fallocate 0x037e34 thru 0x03a919 (0x2ae6 bytes) 000371 1780853434.561129 trunc from 0x03a91a to 0x01903a (0x218df bytes) 000372 1780853434.604771 mapread 0x018179 thru 0x01903a (0xec2 bytes) 000373 1780853434.635554 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1903b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1903b): Mirror 3: 0x5f5c35a8 (stale) Mirror 4: 0x5f5c35a8 (stale) Mirror 5: 0x8b73bf69 (stale) CRC-32 checksum value for chunk [0, 0x1903b): Mirror 1: 0x430a9d7c Mirror 6: 0x430a9d7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1903b: skipped 000374 1780853435.829615 mapwrite 0x012ec1 thru 0x020536 (0xd676 bytes) 000375 1780853436.056915 read 0x018854 thru 0x01efa5 (0x6752 bytes) 000376 1780853436.063069 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x20537): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20537): Mirror 3: 0xe7570484 (stale) Mirror 4: 0xe7570484 (stale) Mirror 5: 0x9aabc971 (stale) Mirror 6: 0xb075e84c (stale) CRC-32 checksum value for chunk [0, 0x20537): Mirror 1: 0x43199f12 Mirror 7: 0x43199f12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20537: skipped 000377 1780853437.248823 mapread 0x006b63 thru 0x00cdea (0x6288 bytes) 000378 1780853437.306138 write 0x0101e6 thru 0x0171f3 (0x700e bytes) 000379 1780853437.412285 fallocate 0x03086a thru 0x03d827 (0xcfbe bytes) 000380 1780853437.422587 mapwrite 0x01960c thru 0x02302d (0x9a22 bytes) 000381 1780853437.489279 mapwrite 0x02cd3e thru 0x02d22e (0x4f1 bytes) 000382 1780853437.553831 mapread 0x031dec thru 0x03d827 (0xba3c bytes) 000383 1780853437.584181 punch from 0x0098c8 to 0x017fd4 (0xe70d bytes) 000384 1780853437.597281 fallocate 0x03ede3 thru 0x03ffff (0x121d bytes) 000386 1780853437.612355 trunc from 0x040000 to 0x030bb0 (0xf44f bytes) 000387 1780853437.707857 mapread 0x0304d3 thru 0x030bb0 (0x6de bytes) 000388 1780853437.710893 write 0x032c97 thru 0x03cd4a (0xa0b4 bytes) 000389 1780853437.713026 trunc from 0x02cb6d to 0x033c68 (0x70fc bytes) 000390 1780853437.734343 mapwrite 0x006e11 thru 0x01080f (0x99ff bytes) 000391 1780853437.776599 punch from 0x01b03d to 0x024ea4 (0x9e68 bytes) 000392 1780853437.784241 write 0x015ceb thru 0x017c1f (0x1f35 bytes) 000393 1780853437.807864 write 0x00c44a thru 0x017ef0 (0xbaa7 bytes) 000394 1780853437.811621 trunc from 0x03cd4b to 0x032b70 (0xa1da bytes) 000395 1780853437.852859 mapread 0x01902f thru 0x027ac1 (0xea93 bytes) 000396 1780853437.872096 trunc from 0x032b71 to 0x021fcb (0x10ba5 bytes) 000397 1780853437.891253 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21fcc): Mirror 3: 0xf6b47317 (stale) Mirror 4: 0xf6b47317 (stale) Mirror 5: 0x533178af (stale) Mirror 6: 0xa442fe4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21fcc): Mirror 3: 0xf6b47317 (stale) Mirror 4: 0xf6b47317 (stale) Mirror 5: 0x533178af (stale) Mirror 6: 0xa442fe4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21fcc: skipped 000399 1780853438.450733 mapwrite 0x02c9bf thru 0x034112 (0x7754 bytes) 000400 1780853438.521995 mapread 0x0169bd thru 0x02003e (0x9682 bytes) 000401 1780853438.542787 mapwrite 0x0039e4 thru 0x010113 (0xc730 bytes) 000402 1780853438.578619 mapwrite 0x003445 thru 0x004803 (0x13bf bytes) 000403 1780853438.594089 write 0x03e881 thru 0x03ffff (0x177f bytes) 000404 1780853438.603372 mapwrite 0x0077f9 thru 0x010431 (0x8c39 bytes) 000405 1780853438.647413 trunc from 0x040000 to 0x00b3de (0x34c21 bytes) 000406 1780853438.686503 trunc from 0x003671 to 0x00680f (0x319f bytes) 000407 1780853438.693205 read 0x009f24 thru 0x00b3de (0x14bb bytes) 000408 1780853438.693967 read 0x00069e thru 0x002bdb (0x253e bytes) 000409 1780853438.705063 trunc from 0x002024 to 0x00b3de (0x93bb bytes) 000410 1780853438.713896 mapwrite 0x02c23c thru 0x0335f9 (0x73be bytes) 000411 1780853438.767753 write 0x002905 thru 0x00d094 (0xa790 bytes) 000412 1780853438.777674 write 0x03756b thru 0x03b295 (0x3d2b bytes) 000413 1780853438.782138 trunc from 0x03b296 to 0x03ed77 (0x3ae2 bytes) 000414 1780853438.821640 write 0x00fa88 thru 0x01f9ed (0xff66 bytes) 000415 1780853438.823666 write 0x03d6e2 thru 0x03ffff (0x291e bytes) 000416 1780853438.837977 punch from 0x011848 to 0x01c539 (0xacf2 bytes) 000417 1780853438.866673 mapread 0x03b34d thru 0x03d88a (0x253e bytes) 000418 1780853438.867894 trunc from 0x040000 to 0x024140 (0x1bebf bytes) 000419 1780853438.913498 fallocate 0x027160 thru 0x027bad (0xa4e bytes) 000420 1780853438.926043 trunc from 0x027bae to 0x010d6b (0x16e42 bytes) 000421 1780853438.979076 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x10d6c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d6c): Mirror 3: 0x53a4d936 (stale) Mirror 4: 0x53a4d936 (stale) Mirror 5: 0x53a4d936 (stale) Mirror 6: 0x53a4d936 (stale) CRC-32 checksum value for chunk [0, 0x10d6c): Mirror 1: 0x7ce37522 Mirror 7: 0x7ce37522 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d6c: skipped 000422 1780853440.511685 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x10d6c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d6c): Mirror 3: 0x53a4d936 (stale) Mirror 4: 0x53a4d936 (stale) Mirror 5: 0x53a4d936 (stale) Mirror 6: 0x53a4d936 (stale) CRC-32 checksum value for chunk [0, 0x10d6c): Mirror 1: 0x7ce37522 Mirror 7: 0x7ce37522 Mirror 8: 0x7ce37522 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d6c: skipped 000423 1780853441.777275 trunc from 0x002b56 to 0x00ebe5 (0xc090 bytes) 000424 1780853441.864279 trunc from 0x010d6c to 0x0064cf (0xa89c bytes) 000425 1780853441.906883 trunc from 0x0064d0 to 0x01ca4e (0x1657f bytes) 000426 1780853441.931273 trunc from 0x014ec4 to 0x01ca4e (0x7b8b bytes) 000427 1780853441.937644 write 0x00e653 thru 0x01496a (0x6318 bytes) 000428 1780853441.960952 read 0x011db0 thru 0x019f27 (0x8178 bytes) 000429 1780853441.978665 trunc from 0x01ca4f to 0x01a04d (0x2a01 bytes) 000430 1780853442.018055 mapread 0x00064c thru 0x001fb9 (0x196e bytes) 000431 1780853442.040904 trunc from 0x00c439 to 0x0120e0 (0x5ca8 bytes) 000432 1780853442.063271 trunc from 0x01a04e to 0x03d2ce (0x23281 bytes) 000433 1780853442.095292 trunc from 0x03d2cf to 0x01dbe4 (0x1f6ea bytes) 000434 1780853442.126068 punch from 0x0094b7 to 0x017e31 (0xe97b bytes) 000435 1780853442.137029 fallocate 0x00ef55 thru 0x01166f (0x271b bytes) 000437 1780853442.152846 trunc from 0x012973 to 0x01dbe4 (0xb272 bytes) 000439 1780853442.163813 read 0x009805 thru 0x017560 (0xdd5c bytes) 000440 1780853442.189199 trunc from 0x01dbe5 to 0x035f0a (0x18326 bytes) 000441 1780853442.222815 trunc from 0x025cb7 to 0x02ee33 (0x917d bytes) 000443 1780853442.235261 fallocate 0x02ff38 thru 0x038c0a (0x8cd3 bytes) 000444 1780853442.246637 mapread 0x0008e8 thru 0x00e05d (0xd776 bytes) 000445 1780853442.254718 punch from 0x0240e0 to 0x029c17 (0x5b38 bytes) 000447 1780853442.272574 fallocate 0x023f29 thru 0x0294e9 (0x55c1 bytes) 000448 1780853442.280640 write 0x03d5d2 thru 0x03ffff (0x2a2e bytes) 000449 1780853442.291507 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6d1cc9cb (stale) Mirror 4: 0x6d1cc9cb (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x1bf83c57 (stale) Mirror 8: 0x1bf83c57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6d1cc9cb (stale) Mirror 4: 0x6d1cc9cb (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x1bf83c57 (stale) Mirror 8: 0x1bf83c57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000450 1780853442.805786 fallocate 0x01a4be thru 0x01e311 (0x3e54 bytes) 000451 1780853442.815044 fallocate 0x03ecb3 thru 0x03ffff (0x134d bytes) 000452 1780853442.821424 write 0x025c53 thru 0x02c5aa (0x6958 bytes) 000453 1780853442.848825 trunc from 0x040000 to 0x02ac02 (0x153fd bytes) 000454 1780853442.871912 trunc from 0x02ac03 to 0x0278f0 (0x3312 bytes) 000455 1780853442.899230 trunc from 0x0278f1 to 0x03590f (0xe01f bytes) 000456 1780853442.936889 fallocate 0x0321e3 thru 0x034a17 (0x2835 bytes) 000457 1780853442.948240 fallocate 0x0390a0 thru 0x03ffff (0x6f60 bytes) 000458 1780853442.960049 write 0x0111b1 thru 0x01dd52 (0xcba2 bytes) 000459 1780853443.004130 fallocate 0x03cfd0 thru 0x03ffff (0x3030 bytes) 000460 1780853443.024330 punch from 0x00d356 to 0x00f33e (0x1fe9 bytes) 000461 1780853443.039822 mapwrite 0x00f79b thru 0x0169ea (0x7250 bytes) 000462 1780853443.093612 mapwrite 0x00fa3f thru 0x019611 (0x9bd3 bytes) 000463 1780853443.144762 trunc from 0x040000 to 0x0059e2 (0x3a61d bytes) 000464 1780853443.168001 mapread 0x005868 thru 0x0059e2 (0x17b bytes) 000465 1780853443.182032 mapread 0x004337 thru 0x0059e2 (0x16ac bytes) 000466 1780853443.187465 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x59e3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x59e3): Mirror 3: 0x40c981c8 (stale) Mirror 4: 0x40c981c8 (stale) Mirror 6: 0x40c981c8 (stale) Mirror 7: 0x5e7990f5 (stale) Mirror 8: 0x5e7990f5 (stale) CRC-32 checksum value for chunk [0, 0x59e3): Mirror 1: 0x599aee6b Mirror 9: 0x599aee6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x59e3: skipped 000467 1780853444.118268 write 0x0387a2 thru 0x03ffff (0x785e bytes) 000469 1780853444.187280 read 0x02eca9 thru 0x0344da (0x5832 bytes) 000470 1780853444.200773 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6d1cc9cb (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x1bf83c57 (stale) Mirror 8: 0x1bf83c57 (stale) Mirror 9: 0x18ac0581 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6d1cc9cb (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x1bf83c57 (stale) Mirror 8: 0x1bf83c57 (stale) Mirror 9: 0x18ac0581 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000472 1780853444.707531 read 0x003b8f thru 0x009af6 (0x5f68 bytes) 000473 1780853444.720803 write 0x028ed3 thru 0x037339 (0xe467 bytes) 000474 1780853444.743503 write 0x02ff72 thru 0x03f6d9 (0xf768 bytes) 000475 1780853444.753168 trunc from 0x040000 to 0x035e1c (0xa1e3 bytes) 000476 1780853444.799455 trunc from 0x01fc85 to 0x028bf6 (0x8f72 bytes) 000477 1780853444.819962 trunc from 0x035e1d to 0x024361 (0x11abb bytes) 000478 1780853444.847576 read 0x0003f4 thru 0x001a69 (0x1676 bytes) 000479 1780853444.901207 MIRROR_RESYNC 6 mirrors 000480 1780853446.211203 trunc from 0x024362 to 0x0073c2 (0x1cf9f bytes) 000481 1780853446.298279 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x73c3): Mirror 3: 0x53c2af91 (stale) Mirror 7: 0x53c2af91 (stale) Mirror 8: 0x53c2af91 (stale) Mirror 9: 0x53c2af91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x73c3): Mirror 3: 0x53c2af91 (stale) Mirror 7: 0x53c2af91 (stale) Mirror 8: 0x53c2af91 (stale) Mirror 9: 0x53c2af91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x73c3: skipped 000483 1780853446.851643 mapread 0x000dd0 thru 0x0073c2 (0x65f3 bytes) 000485 1780853446.882073 mapread 0x00464d thru 0x0073c2 (0x2d76 bytes) 000487 1780853446.883630 fallocate 0x00e82f thru 0x014efe (0x66d0 bytes) 000489 1780853446.889784 mapwrite 0x000ac7 thru 0x0028fd (0x1e37 bytes) 000490 1780853446.908758 read 0x01098d thru 0x013f23 (0x3597 bytes) 000491 1780853446.915640 read 0x005619 thru 0x0144c4 (0xeeac bytes) 000492 1780853446.929247 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14eff): Mirror 3: 0xc7b56fa5 (stale) Mirror 7: 0xc7b56fa5 (stale) Mirror 9: 0xc7b56fa5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14eff): Mirror 3: 0xc7b56fa5 (stale) Mirror 7: 0xc7b56fa5 (stale) Mirror 9: 0xc7b56fa5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14eff: skipped 000493 1780853447.355938 read 0x0048f0 thru 0x013dac (0xf4bd bytes) 000494 1780853447.371480 punch from 0x013acb to 0x014efe (0x1434 bytes) 000496 1780853447.386100 fallocate 0x03ca13 thru 0x03ffff (0x35ed bytes) 000498 1780853447.402208 mapread 0x00ba52 thru 0x00e12c (0x26db bytes) 000499 1780853447.403234 mapread 0x014368 thru 0x014efe (0xb97 bytes) 000501 1780853447.408595 read 0x009b3d thru 0x00b953 (0x1e17 bytes) 000502 1780853447.409284 mapwrite 0x0374dd thru 0x03d2c6 (0x5dea bytes) 000503 1780853447.474042 fallocate 0x038fb1 thru 0x03ffff (0x704f bytes) 000505 1780853447.483003 mapwrite 0x037381 thru 0x03ecd0 (0x7950 bytes) 000506 1780853447.567787 punch from 0x002471 to 0x008a06 (0x6596 bytes) 000507 1780853447.580816 punch from 0x037d4d to 0x03ecd0 (0x6f84 bytes) 000508 1780853447.593653 fallocate 0x031d99 thru 0x03964b (0x78b3 bytes) 000509 1780853447.600557 write 0x02d441 thru 0x039233 (0xbdf3 bytes) 000510 1780853447.606988 fallocate 0x01fa6b thru 0x023ad9 (0x406f bytes) 000511 1780853447.611982 write 0x00e53a thru 0x01cee9 (0xe9b0 bytes) 000512 1780853447.613567 trunc from 0x030054 to 0x03b849 (0xb7f6 bytes) 000513 1780853447.630637 punch from 0x01ce18 to 0x020d08 (0x3ef1 bytes) 000514 1780853447.636126 write 0x01c115 thru 0x021865 (0x5751 bytes) 000515 1780853447.646955 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ecd1): Mirror 7: 0x6a8f5166 (stale) Mirror 9: 0x6a8f5166 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecd1): Mirror 7: 0x6a8f5166 (stale) Mirror 9: 0x6a8f5166 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecd1: skipped 000516 1780853448.064894 trunc from 0x03ecd1 to 0x0044c4 (0x3a80c bytes) 000518 1780853448.096790 trunc from 0x0021c6 to 0x0044c4 (0x22ff bytes) 000519 1780853448.104255 mapread 0x001873 thru 0x0044c4 (0x2c52 bytes) 000520 1780853448.119633 trunc from 0x0042d3 to 0x0044c4 (0x1f2 bytes) 000521 1780853448.126804 punch from 0x003ac8 to 0x0044c4 (0x9fd bytes) 000522 1780853448.131600 read 0x002b50 thru 0x0044c4 (0x1975 bytes) 000523 1780853448.136838 trunc from 0x000b7e to 0x0044c4 (0x3947 bytes) 000524 1780853448.143339 mapwrite 0x011651 thru 0x016182 (0x4b32 bytes) 000525 1780853448.192963 punch from 0x004410 to 0x007c38 (0x3829 bytes) 000526 1780853448.201665 read 0x006847 thru 0x006cfe (0x4b8 bytes) 000527 1780853448.219181 write 0x018387 thru 0x01c592 (0x420c bytes) 000528 1780853448.220810 read 0x001dc4 thru 0x0119d6 (0xfc13 bytes) 000529 1780853448.223535 mapread 0x013e56 thru 0x01ac1a (0x6dc5 bytes) 000530 1780853448.229544 write 0x03c240 thru 0x03e477 (0x2238 bytes) 000531 1780853448.231393 punch from 0x03df52 to 0x03e477 (0x526 bytes) 000532 1780853448.244812 mapread 0x0196a9 thru 0x02196b (0x82c3 bytes) 000533 1780853448.255704 write 0x00b620 thru 0x00d09b (0x1a7c bytes) 000534 1780853448.261412 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e478): Mirror 7: 0xd390ca48 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e478): Mirror 7: 0xd390ca48 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e478: skipped 000535 1780853448.580074 fallocate 0x03e6e6 thru 0x03ffff (0x191a bytes) 000536 1780853448.589478 punch from 0x0210a9 to 0x021d58 (0xcb0 bytes) 000537 1780853448.594767 mapwrite 0x02c5bf thru 0x02ecff (0x2741 bytes) 000538 1780853448.634735 trunc from 0x040000 to 0x02e674 (0x1198b bytes) 000541 1780853448.662355 mapwrite 0x01d1de thru 0x0225a3 (0x53c6 bytes) 000542 1780853448.728238 fallocate 0x03fa40 thru 0x03ffff (0x5c0 bytes) 000543 1780853448.736784 read 0x02ddb3 thru 0x02e674 (0x8c2 bytes) 000544 1780853448.738004 trunc from 0x02e675 to 0x02aa2e (0x3c46 bytes) 000545 1780853448.772959 trunc from 0x01f2d3 to 0x02367b (0x43a9 bytes) 000546 1780853448.800820 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2aa2f): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa2f): Mirror 7: 0x361fc7cf (stale) CRC-32 checksum value for chunk [0, 0x2aa2f): Mirror 1: 0x124c237b Mirror 8: 0x124c237b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa2f: skipped 000547 1780853450.323932 MIRROR_RESYNC 3 mirrors 000548 1780853451.440761 mapwrite 0x010921 thru 0x0161f4 (0x58d4 bytes) 000549 1780853451.615468 trunc from 0x015048 to 0x016aac (0x1a65 bytes) 000550 1780853451.635782 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2aa2f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa2f): Mirror 7: 0x124c237b (stale) Mirror 8: 0x124c237b (stale) CRC-32 checksum value for chunk [0, 0x2aa2f): Mirror 1: 0x7c786a46 Mirror 9: 0x7c786a46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa2f: skipped 000551 1780853452.987354 trunc from 0x014be6 to 0x0222ff (0xd71a bytes) 000552 1780853453.033309 trunc from 0x01b38b to 0x022008 (0x6c7e bytes) 000553 1780853453.037788 trunc from 0x02aa2f to 0x01bd62 (0xeccc bytes) 000554 1780853453.077668 trunc from 0x01bd63 to 0x038030 (0x1c2ce bytes) 000555 1780853453.103360 write 0x022a9e thru 0x0242dd (0x1840 bytes) 000556 1780853453.119774 mapwrite 0x024c83 thru 0x034b9b (0xff19 bytes) 000557 1780853453.188468 mapwrite 0x016047 thru 0x024cc2 (0xec7c bytes) 000558 1780853453.250747 mapread 0x01dd35 thru 0x02aea0 (0xd16c bytes) 000559 1780853453.253356 trunc from 0x038031 to 0x008132 (0x2fefe bytes) 000560 1780853453.283802 trunc from 0x008133 to 0x024581 (0x1c44f bytes) 000561 1780853453.307932 read 0x0112f9 thru 0x012216 (0xf1e bytes) 000562 1780853453.322229 mapwrite 0x02f1b1 thru 0x0333c7 (0x4217 bytes) 000563 1780853453.431944 fallocate 0x00de65 thru 0x01c809 (0xe9a5 bytes) 000564 1780853453.474693 MIRROR_RESYNC 4 mirrors 000565 1780853454.758424 punch from 0x026134 to 0x0303c6 (0xa293 bytes) 000566 1780853454.816638 mapwrite 0x03c206 thru 0x03ffff (0x3dfa bytes) 000567 1780853454.923030 trunc from 0x040000 to 0x0344ab (0xbb54 bytes) 000568 1780853454.962093 trunc from 0x0070f9 to 0x0169e8 (0xf8f0 bytes) 000569 1780853454.977473 mapwrite 0x02ec9f thru 0x034fff (0x6361 bytes) 000570 1780853455.044009 write 0x03144a thru 0x039072 (0x7c29 bytes) 000571 1780853455.047687 fallocate 0x01548d thru 0x01d6b9 (0x822d bytes) 000572 1780853455.054221 mapread 0x021b35 thru 0x02d33b (0xb807 bytes) 000573 1780853455.074457 read 0x00a31b thru 0x00b49c (0x1182 bytes) 000574 1780853455.084007 mapwrite 0x003799 thru 0x006c58 (0x34c0 bytes) 000575 1780853455.144308 trunc from 0x039073 to 0x003d87 (0x352eb bytes) 000576 1780853455.184667 read 0x003336 thru 0x003d87 (0xa52 bytes) 000577 1780853455.203018 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d88): Mirror 8: 0x9a2dbaac (stale) Mirror 9: 0x9a2dbaac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d88): Mirror 8: 0x9a2dbaac (stale) Mirror 9: 0x9a2dbaac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d88: skipped 000578 1780853455.742903 fallocate 0x000666 thru 0x0022b0 (0x1c4b bytes) 000579 1780853455.750426 punch from 0x00048d to 0x003d87 (0x38fb bytes) 000580 1780853455.762903 read 0x000471 thru 0x002787 (0x2317 bytes) 000581 1780853455.772258 mapread 0x003556 thru 0x003d87 (0x832 bytes) 000582 1780853455.783103 mapread 0x00364b thru 0x003d87 (0x73d bytes) 000583 1780853455.786475 trunc from 0x003d88 to 0x01d05c (0x192d5 bytes) 000584 1780853455.816599 fallocate 0x03166a thru 0x03b7d3 (0xa16a bytes) 000585 1780853455.830184 write 0x0188bc thru 0x019ab1 (0x11f6 bytes) 000586 1780853455.863844 mapwrite 0x0243a3 thru 0x033c1e (0xf87c bytes) 000587 1780853455.997848 read 0x003418 thru 0x01242d (0xf016 bytes) 000588 1780853456.034506 mapread 0x02b262 thru 0x02ba23 (0x7c2 bytes) 000589 1780853456.037797 read 0x032ae6 thru 0x033c1e (0x1139 bytes) 000590 1780853456.043001 punch from 0x004422 to 0x00dd13 (0x98f2 bytes) 000591 1780853456.070376 punch from 0x01c02b to 0x024507 (0x84dd bytes) 000592 1780853456.087104 punch from 0x01e69e to 0x024cec (0x664f bytes) 000593 1780853456.098843 trunc from 0x033c1f to 0x0224e6 (0x11738 bytes) 000594 1780853456.135639 mapwrite 0x023226 thru 0x025451 (0x222c bytes) 000595 1780853456.208071 fallocate 0x014da5 thru 0x021a4d (0xcca9 bytes) 000597 1780853456.217705 write 0x03c310 thru 0x03ffff (0x3cf0 bytes) 000598 1780853456.222619 mapwrite 0x02d552 thru 0x030fed (0x3a9c bytes) 000599 1780853456.260208 mapwrite 0x02144c thru 0x02b8e5 (0xa49a bytes) 000600 1780853456.314186 punch from 0x01507a to 0x01a200 (0x5187 bytes) 000601 1780853456.323535 read 0x032f83 thru 0x039880 (0x68fe bytes) 000603 1780853456.325245 punch from 0x00ea72 to 0x016722 (0x7cb1 bytes) 000604 1780853456.344988 mapread 0x03c00d thru 0x03e3c7 (0x23bb bytes) 000605 1780853456.349055 fallocate 0x022a06 thru 0x02d55c (0xab57 bytes) 000606 1780853456.358479 trunc from 0x03bfcc to 0x03dcd2 (0x1d07 bytes) 000607 1780853456.365860 mapread 0x02b461 thru 0x036a29 (0xb5c9 bytes) 000608 1780853456.368998 trunc from 0x040000 to 0x007885 (0x3877a bytes) 000609 1780853456.399444 read 0x00045a thru 0x005b4d (0x56f4 bytes) 000610 1780853456.400323 mapwrite 0x0303d7 thru 0x038ee5 (0x8b0f bytes) 000611 1780853456.490726 trunc from 0x038ee6 to 0x03b7ec (0x2907 bytes) 000612 1780853456.508629 mapwrite 0x00874f thru 0x0134b1 (0xad63 bytes) 000613 1780853456.542220 mapread 0x035222 thru 0x03b7ec (0x65cb bytes) 000614 1780853456.563468 mapread 0x00ad7f thru 0x0144e3 (0x9765 bytes) 000615 1780853456.569498 read 0x032554 thru 0x036d24 (0x47d1 bytes) 000618 1780853456.571727 write 0x03ab7e thru 0x03ffff (0x5482 bytes) 000619 1780853456.581982 mapread 0x00e151 thru 0x017aa2 (0x9952 bytes) 000620 1780853456.585103 mapwrite 0x02d624 thru 0x033e36 (0x6813 bytes) 000621 1780853456.625698 punch from 0x0257eb to 0x0307a1 (0xafb7 bytes) 000622 1780853456.635436 MIRROR_RESYNC 3 mirrors 000623 1780853458.331952 write 0x02e345 thru 0x037428 (0x90e4 bytes) 000625 1780853458.397894 read 0x0250a9 thru 0x02dc7b (0x8bd3 bytes) 000627 1780853458.420842 trunc from 0x040000 to 0x007ead (0x38152 bytes) 000628 1780853458.537702 trunc from 0x000b37 to 0x007ead (0x7377 bytes) 000629 1780853458.550904 trunc from 0x007eae to 0x03b858 (0x339ab bytes) 000630 1780853458.592647 write 0x00ef0b thru 0x019cfc (0xadf2 bytes) 000631 1780853458.625042 read 0x013606 thru 0x01999d (0x6398 bytes) 000632 1780853458.626651 read 0x027bee thru 0x02eb2e (0x6f41 bytes) 000633 1780853458.655944 mapread 0x00bc44 thru 0x00d07f (0x143c bytes) 000634 1780853458.657239 trunc from 0x03b859 to 0x02e7c5 (0xd093 bytes) 000635 1780853458.680760 MIRROR_RESYNC 3 mirrors 000636 1780853459.803640 fallocate 0x001f35 thru 0x005060 (0x312c bytes) 000637 1780853459.848585 fallocate 0x025d73 thru 0x034a07 (0xec95 bytes) 000638 1780853459.861345 MIRROR_RESYNC 3 mirrors 000639 1780853461.210317 fallocate 0x00c665 thru 0x0188db (0xc277 bytes) 000640 1780853461.288639 mapwrite 0x02a9f2 thru 0x02c491 (0x1aa0 bytes) 000641 1780853461.360261 write 0x0345ae thru 0x03fb6d (0xb5c0 bytes) 000642 1780853461.366951 mapwrite 0x00a392 thru 0x00d0ea (0x2d59 bytes) 000643 1780853461.446053 write 0x01f88d thru 0x029432 (0x9ba6 bytes) 000644 1780853461.476135 trunc from 0x03fb6e to 0x03c97e (0x31ef bytes) 000645 1780853461.527461 mapwrite 0x034587 thru 0x03aed5 (0x694f bytes) 000646 1780853461.577697 fallocate 0x0128ea thru 0x014072 (0x1789 bytes) 000648 1780853461.603769 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c97f): Mirror 9: 0xcd333f53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c97f): Mirror 9: 0xcd333f53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c97f: skipped 000649 1780853462.198128 mapread 0x00b4fe thru 0x01b06c (0xfb6f bytes) 000650 1780853462.248459 mapread 0x03c096 thru 0x03c97e (0x8e9 bytes) 000651 1780853462.267641 trunc from 0x03c97f to 0x02151c (0x1b462 bytes) 000652 1780853462.292739 trunc from 0x02151d to 0x03ee85 (0x1d969 bytes) 000653 1780853462.342618 trunc from 0x02ff48 to 0x03ee85 (0xef3e bytes) 000654 1780853462.349945 mapread 0x03d109 thru 0x03ee85 (0x1d7d bytes) 000657 1780853462.401439 mapread 0x00021b thru 0x00e138 (0xdf1e bytes) 000658 1780853462.430129 write 0x03b2b2 thru 0x03ffff (0x4d4e bytes) 000659 1780853462.431883 mapread 0x009afc thru 0x0144a4 (0xa9a9 bytes) 000660 1780853462.447838 write 0x022c14 thru 0x031ab4 (0xeea1 bytes) 000661 1780853462.480823 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xac430452 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x89db36ff Mirror 10: 0x89db36ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000662 1780853463.781905 mapwrite 0x02f89f thru 0x03ab8a (0xb2ec bytes) 000663 1780853463.952799 trunc from 0x0179d6 to 0x01817a (0x7a5 bytes) 000664 1780853463.961851 fallocate 0x016b9c thru 0x0192d0 (0x2735 bytes) 000665 1780853463.984574 MIRROR_RESYNC 3 mirrors 000666 1780853465.221099 mapwrite 0x017e1a thru 0x018cbe (0xea5 bytes) 000667 1780853465.349972 read 0x02d0ea thru 0x0310e9 (0x4000 bytes) 000668 1780853465.365967 trunc from 0x03d7c5 to 0x03ffff (0x283b bytes) 000669 1780853465.372129 mapread 0x032876 thru 0x03c316 (0x9aa1 bytes) 000670 1780853465.396478 fallocate 0x027c44 thru 0x029f3b (0x22f8 bytes) 000671 1780853465.406116 write 0x01c061 thru 0x01dc12 (0x1bb2 bytes) 000672 1780853465.407292 read 0x0326f9 thru 0x036e15 (0x471d bytes) 000673 1780853465.423053 MIRROR_RESYNC 3 mirrors 000674 1780853466.881798 MIRROR_EXTEND to 4 mirrors 000675 1780853468.057796 trunc from 0x040000 to 0x01d419 (0x22be6 bytes) 000676 1780853468.148512 read 0x010006 thru 0x013dbb (0x3db6 bytes) 000677 1780853468.158056 trunc from 0x018667 to 0x01d419 (0x4db3 bytes) 000678 1780853468.164944 read 0x016054 thru 0x0189ef (0x299c bytes) 000680 1780853468.186103 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d41a): Mirror 10: 0xca12974a (stale) Mirror 11: 0xca12974a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d41a): Mirror 10: 0xca12974a (stale) Mirror 11: 0xca12974a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d41a: skipped 000681 1780853468.723390 read 0x00b07b thru 0x0160c0 (0xb046 bytes) 000682 1780853468.757885 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d41a): Mirror 11: 0xca12974a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d41a): Mirror 11: 0xca12974a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d41a: skipped 000683 1780853469.141517 write 0x024911 thru 0x02e4a3 (0x9b93 bytes) 000684 1780853469.153544 trunc from 0x028977 to 0x02e4a3 (0x5b2d bytes) 000685 1780853469.184806 read 0x01b537 thru 0x027201 (0xbccb bytes) 000686 1780853469.218082 MIRROR_RESYNC 2 mirrors 000687 1780853470.303411 write 0x01d24e thru 0x021243 (0x3ff6 bytes) 000688 1780853470.397491 trunc from 0x01a54f to 0x01c4b2 (0x1f64 bytes) 000689 1780853470.407130 mapread 0x00360b thru 0x011be8 (0xe5de bytes) 000690 1780853470.453480 mapread 0x0169a3 thru 0x01cbad (0x620b bytes) 000692 1780853470.472335 trunc from 0x0107e4 to 0x01d96f (0xd18c bytes) 000693 1780853470.500524 mapwrite 0x034831 thru 0x03ffff (0xb7cf bytes) 000694 1780853470.598571 mapwrite 0x00f6e1 thru 0x012cc3 (0x35e3 bytes) 000695 1780853470.625556 mapread 0x032300 thru 0x03cca9 (0xa9aa bytes) 000696 1780853470.627089 punch from 0x02a1b7 to 0x038886 (0xe6d0 bytes) 000699 1780853470.633875 mapwrite 0x0048c8 thru 0x0057df (0xf18 bytes) 000701 1780853470.653302 punch from 0x01cbe7 to 0x020485 (0x389f bytes) 000702 1780853470.676667 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xaf339bfd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeda33cba Mirror 12: 0xeda33cba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000703 1780853472.616811 trunc from 0x040000 to 0x03eb72 (0x148d bytes) 000704 1780853472.739512 write 0x005e45 thru 0x00dace (0x7c8a bytes) 000705 1780853472.779226 mapread 0x0061d1 thru 0x00deea (0x7d1a bytes) 000706 1780853472.782254 fallocate 0x01b875 thru 0x02ac8a (0xf416 bytes) 000707 1780853472.790603 write 0x02c730 thru 0x036bf4 (0xa4c5 bytes) 000708 1780853472.864362 trunc from 0x03eb73 to 0x0303a8 (0xe7ca bytes) 000709 1780853472.907727 punch from 0x01ce10 to 0x02cdcb (0xffbc bytes) 000710 1780853472.937005 read 0x02fc81 thru 0x0303a8 (0x728 bytes) 000711 1780853472.937720 punch from 0x02677e to 0x02b3f7 (0x4c7a bytes) 000712 1780853472.953512 read 0x006373 thru 0x012169 (0xbdf7 bytes) 000713 1780853472.981280 read 0x02adf7 thru 0x02d86b (0x2a75 bytes) 000714 1780853472.984215 fallocate 0x001f28 thru 0x006403 (0x44dc bytes) 000715 1780853473.004411 mapwrite 0x011e9d thru 0x0160e5 (0x4249 bytes) 000716 1780853473.059601 mapwrite 0x0304de thru 0x039a7b (0x959e bytes) 000717 1780853473.165515 fallocate 0x03d50b thru 0x03f41b (0x1f11 bytes) 000718 1780853473.188218 fallocate 0x0301bd thru 0x031ae3 (0x1927 bytes) 000719 1780853473.211209 read 0x029cd3 thru 0x02c531 (0x285f bytes) 000720 1780853473.211819 mapwrite 0x00c8a7 thru 0x00e86d (0x1fc7 bytes) 000721 1780853473.254422 fallocate 0x03fbab thru 0x03ffff (0x455 bytes) 000722 1780853473.287460 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 11: 0x16293a00 (stale) Mirror 12: 0x8111ebd5 (stale) CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 1: 0xbf920baa Mirror 13: 0xbf920baa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f41c: skipped 000723 1780853474.965689 write 0x0051e6 thru 0x01217b (0xcf96 bytes) 000724 1780853475.050639 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 11: 0x16293a00 (stale) Mirror 12: 0x8111ebd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 11: 0x16293a00 (stale) Mirror 12: 0x8111ebd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f41c: skipped 000725 1780853475.713116 read 0x028f8f thru 0x034816 (0xb888 bytes) 000726 1780853475.741049 trunc from 0x0374c5 to 0x03f41b (0x7f57 bytes) 000727 1780853475.748423 trunc from 0x01d0e5 to 0x023876 (0x6792 bytes) 000728 1780853475.754388 punch from 0x02241b to 0x031d9e (0xf984 bytes) 000729 1780853475.762650 mapread 0x0226ee thru 0x027a8d (0x53a0 bytes) 000730 1780853475.820429 mapread 0x010c6f thru 0x013bed (0x2f7f bytes) 000731 1780853475.849723 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 12: 0x8111ebd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f41c): Mirror 12: 0x8111ebd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f41c: skipped 000732 1780853476.368765 mapread 0x029edc thru 0x03900b (0xf130 bytes) 000733 1780853476.442822 trunc from 0x0158cb to 0x01c5b7 (0x6ced bytes) 000735 1780853476.448357 trunc from 0x03f41c to 0x011a73 (0x2d9a8 bytes) 000736 1780853476.490688 trunc from 0x011a74 to 0x0156ce (0x3c5b bytes) 000737 1780853476.516442 fallocate 0x01f5ca thru 0x02ad68 (0xb79f bytes) 000738 1780853476.522748 mapwrite 0x00f50f thru 0x01b9fb (0xc4ed bytes) 000739 1780853476.589484 fallocate 0x012c70 thru 0x01d117 (0xa4a8 bytes) 000740 1780853476.609549 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) 000741 1780853476.895283 fallocate 0x01c12f thru 0x020f97 (0x4e69 bytes) 000742 1780853476.911385 fallocate 0x025712 thru 0x026f75 (0x1864 bytes) 000743 1780853476.922842 trunc from 0x02ad69 to 0x00dcc4 (0x1d0a4 bytes) 000744 1780853476.967593 punch from 0x003b0d to 0x00910b (0x55ff bytes) 000745 1780853476.985997 trunc from 0x00dcc5 to 0x01d449 (0xf785 bytes) 000746 1780853477.014710 read 0x00b965 thru 0x00c4a7 (0xb43 bytes) 000747 1780853477.025967 punch from 0x01a5c3 to 0x01d449 (0x2e87 bytes) 000749 1780853477.035445 punch from 0x00dd4d to 0x01a5ed (0xc8a1 bytes) 000750 1780853477.042582 fallocate 0x029ed3 thru 0x0319bf (0x7aed bytes) 000751 1780853477.051086 write 0x031e48 thru 0x03fb55 (0xdd0e bytes) 000752 1780853477.053355 read 0x00e9c8 thru 0x015a9a (0x70d3 bytes) 000753 1780853477.109407 punch from 0x0002dc to 0x00bf4d (0xbc72 bytes) 000755 1780853477.123920 punch from 0x0077aa to 0x010848 (0x909f bytes) 000756 1780853477.144402 MIRROR_EXTEND to 2 mirrors 000757 1780853478.192602 write 0x033eb2 thru 0x03444e (0x59d bytes) 000758 1780853478.323674 write 0x00545c thru 0x007e1a (0x29bf bytes) 000759 1780853478.346274 trunc from 0x025398 to 0x032ad7 (0xd740 bytes) 000760 1780853478.360034 write 0x01d098 thru 0x0289e7 (0xb950 bytes) 000761 1780853478.398686 write 0x005f22 thru 0x00bda7 (0x5e86 bytes) 000762 1780853478.399892 trunc from 0x032855 to 0x035b61 (0x330d bytes) 000763 1780853478.436239 fallocate 0x032e57 thru 0x03cc44 (0x9dee bytes) 000764 1780853478.445924 punch from 0x01c5b2 to 0x020b99 (0x45e8 bytes) 000765 1780853478.458997 read 0x01c8fb thru 0x02b091 (0xe797 bytes) 000766 1780853478.476672 write 0x02714b thru 0x036cf1 (0xfba7 bytes) 000767 1780853478.482640 fallocate 0x009b9a thru 0x00f67c (0x5ae3 bytes) 000769 1780853478.501359 trunc from 0x01865f to 0x0200bb (0x7a5d bytes) 000770 1780853478.528338 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) 000771 1780853478.844835 trunc from 0x03fb56 to 0x02e686 (0x114cf bytes) 000772 1780853478.932375 trunc from 0x004fe6 to 0x013a8f (0xeaaa bytes) 000774 1780853478.946036 punch from 0x00b54a to 0x019d78 (0xe82f bytes) 000775 1780853478.984279 mapwrite 0x03c8b5 thru 0x03ffff (0x374b bytes) 000776 1780853479.120013 write 0x02b1e4 thru 0x037984 (0xc7a1 bytes) 000777 1780853479.140736 mapwrite 0x01a9c7 thru 0x020848 (0x5e82 bytes) 000779 1780853479.199266 trunc from 0x014326 to 0x01a51c (0x61f7 bytes) 000780 1780853479.207941 write 0x0269dc thru 0x02d632 (0x6c57 bytes) 000781 1780853479.209336 trunc from 0x036004 to 0x03ae40 (0x4e3d bytes) 000782 1780853479.221776 fallocate 0x019232 thru 0x01b212 (0x1fe1 bytes) 000784 1780853479.237056 trunc from 0x00b12b to 0x00e11e (0x2ff4 bytes) 000785 1780853479.255640 punch from 0x0086db to 0x015632 (0xcf58 bytes) 000786 1780853479.262796 write 0x018b0d thru 0x018e75 (0x369 bytes) 000788 1780853479.279095 mapread 0x01212a thru 0x01dc05 (0xbadc bytes) 000789 1780853479.280573 mapwrite 0x027502 thru 0x036627 (0xf126 bytes) 000790 1780853479.317037 trunc from 0x033686 to 0x03428c (0xc07 bytes) 000791 1780853479.350565 MIRROR_EXTEND to 2 mirrors 000792 1780853480.714745 punch from 0x024c35 to 0x02fbd2 (0xaf9e bytes) 000793 1780853480.827366 mapwrite 0x01c245 thru 0x01cf12 (0xcce bytes) 000794 1780853480.883547 fallocate 0x024017 thru 0x028200 (0x41ea bytes) 000795 1780853480.891658 trunc from 0x040000 to 0x024cbf (0x1b340 bytes) 000796 1780853480.947949 punch from 0x009b8a to 0x00a72c (0xba3 bytes) 000797 1780853480.959104 trunc from 0x00944f to 0x00e55f (0x5111 bytes) 000799 1780853480.987684 mapwrite 0x007c13 thru 0x013dce (0xc1bc bytes) 000800 1780853481.093797 mapread 0x0010ae thru 0x00bd73 (0xacc6 bytes) 000801 1780853481.099063 trunc from 0x024cc0 to 0x039cc9 (0x1500a bytes) 000802 1780853481.170478 MIRROR_RESYNC 2 mirrors 000803 1780853482.507288 mapread 0x00efd3 thru 0x0103b9 (0x13e7 bytes) 000804 1780853482.581683 read 0x00fba0 thru 0x01462b (0x4a8c bytes) 000805 1780853482.582185 mapwrite 0x034ae0 thru 0x039a92 (0x4fb3 bytes) 000806 1780853482.778015 trunc from 0x028c5c to 0x02e08f (0x5434 bytes) 000807 1780853482.784531 mapread 0x02b6f4 thru 0x0321d3 (0x6ae0 bytes) 000808 1780853482.793982 read 0x036a7c thru 0x039cc9 (0x324e bytes) 000809 1780853482.794740 write 0x03849c thru 0x03ffff (0x7b64 bytes) 000810 1780853482.797072 mapwrite 0x006186 thru 0x00de33 (0x7cae bytes) 000811 1780853482.849582 fallocate 0x006693 thru 0x0137cf (0xd13d bytes) 000812 1780853482.856801 mapwrite 0x02f48d thru 0x0390b6 (0x9c2a bytes) 000813 1780853482.879863 mapwrite 0x03e73f thru 0x03ffff (0x18c1 bytes) 000815 1780853482.900920 write 0x02644b thru 0x033c8b (0xd841 bytes) 000816 1780853482.902977 trunc from 0x024ba2 to 0x027323 (0x2782 bytes) 000817 1780853482.924921 write 0x032e67 thru 0x03a1c3 (0x735d bytes) 000818 1780853482.926348 read 0x03c80d thru 0x03d0a0 (0x894 bytes) 000819 1780853482.926953 mapread 0x00d258 thru 0x00d757 (0x500 bytes) 000820 1780853482.932822 read 0x00eb91 thru 0x019c6c (0xb0dc bytes) 000821 1780853482.933772 trunc from 0x040000 to 0x028835 (0x177ca bytes) 000822 1780853482.982628 punch from 0x00d58a to 0x00f83c (0x22b3 bytes) 000823 1780853482.991795 write 0x025ee2 thru 0x02a31e (0x443d bytes) 000824 1780853483.001464 trunc from 0x023eec to 0x02a31e (0x6433 bytes) 000825 1780853483.019205 trunc from 0x02a31f to 0x00e4b0 (0x1be6e bytes) 000826 1780853483.036525 trunc from 0x006ef7 to 0x00e4b0 (0x75ba bytes) 000827 1780853483.043305 trunc from 0x00e4b1 to 0x013dc7 (0x5917 bytes) 000829 1780853483.059632 fallocate 0x038615 thru 0x03af89 (0x2975 bytes) 000831 1780853483.071020 punch from 0x0003e8 to 0x000b48 (0x761 bytes) 000832 1780853483.076974 mapwrite 0x033fb3 thru 0x03d814 (0x9862 bytes) 000833 1780853483.128687 fallocate 0x03876e thru 0x03fbc7 (0x745a bytes) 000835 1780853483.137124 trunc from 0x03d815 to 0x02265e (0x1b1b6 bytes) 000836 1780853483.155393 fallocate 0x0058b9 thru 0x012383 (0xcacb bytes) 000837 1780853483.164071 read 0x00fa3c thru 0x01151c (0x1ae1 bytes) 000838 1780853483.175713 trunc from 0x02265f to 0x031acd (0xf46f bytes) 000839 1780853483.197618 trunc from 0x001de0 to 0x009dd9 (0x7ffa bytes) 000840 1780853483.206190 trunc from 0x031ace to 0x00be78 (0x25c55 bytes) 000842 1780853483.234060 mapwrite 0x02439a thru 0x02553e (0x11a5 bytes) 000843 1780853483.342993 read 0x01c789 thru 0x01efae (0x2826 bytes) 000845 1780853483.372045 punch from 0x013ddb to 0x01c1fb (0x8421 bytes) 000846 1780853483.382744 punch from 0x013d35 to 0x01d7d8 (0x9aa4 bytes) 000847 1780853483.415035 MIRROR_RESYNC 2 mirrors 000848 1780853484.107321 fallocate 0x003141 thru 0x008284 (0x5144 bytes) 000849 1780853484.177656 write 0x02572a thru 0x030c21 (0xb4f8 bytes) 000851 1780853484.208009 trunc from 0x020e15 to 0x02630c (0x54f8 bytes) 000852 1780853484.244845 trunc from 0x030c22 to 0x00ea54 (0x221cd bytes) 000853 1780853484.341656 fallocate 0x02249d thru 0x02fa15 (0xd579 bytes) 000855 1780853484.357044 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xea55): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea55): Mirror 2: 0xc9975ab0 (stale) CRC-32 checksum value for chunk [0, 0xea55): Mirror 1: 0xc9975ab0 Mirror 3: 0xc9975ab0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea55: skipped 000856 1780853485.689897 trunc from 0x00ea55 to 0x039b1c (0x2b0c8 bytes) 000857 1780853485.764258 fallocate 0x01b565 thru 0x01f555 (0x3ff1 bytes) 000858 1780853485.772853 punch from 0x00cbe9 to 0x01186a (0x4c82 bytes) 000859 1780853485.784460 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x158:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b1d): Mirror 3: 0xf460e125 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b1d): Mirror 3: 0xf460e125 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b1d: skipped 000860 1780853486.113385 trunc from 0x018757 to 0x025d2b (0xd5d5 bytes) 000861 1780853486.119695 punch from 0x013ca9 to 0x017f7a (0x42d2 bytes) 000862 1780853486.124660 trunc from 0x039b1d to 0x009d84 (0x2fd98 bytes) 000863 1780853486.147979 trunc from 0x006756 to 0x006bf6 (0x4a1 bytes) 000864 1780853486.156107 mapwrite 0x020a97 thru 0x027b18 (0x7082 bytes) 000865 1780853486.231118 write 0x00efdc thru 0x015077 (0x609c bytes) 000866 1780853486.251461 mapread 0x017afa thru 0x022bca (0xb0d1 bytes) 000867 1780853486.275910 read 0x027289 thru 0x027b18 (0x890 bytes) 000868 1780853486.280060 mapwrite 0x020d1c thru 0x023a02 (0x2ce7 bytes) 000869 1780853486.324264 trunc from 0x018418 to 0x026925 (0xe50e bytes) 000870 1780853486.334979 write 0x03f28a thru 0x03ff04 (0xc7b bytes) 000871 1780853486.342712 punch from 0x015ae5 to 0x01c9e1 (0x6efd bytes) 000872 1780853486.352955 mapwrite 0x011bc1 thru 0x0183b8 (0x67f8 bytes) 000873 1780853486.443659 write 0x01d494 thru 0x024c25 (0x7792 bytes) 000874 1780853486.461207 trunc from 0x0178d5 to 0x01fe70 (0x859c bytes) 000875 1780853486.507955 trunc from 0x03ff05 to 0x00b200 (0x34d04 bytes) 000876 1780853486.557886 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xb201): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb201): Mirror 3: 0xcd6f9ddb (stale) CRC-32 checksum value for chunk [0, 0xb201): Mirror 1: 0xcd6f9ddb Mirror 4: 0xcd6f9ddb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb201: skipped 000877 1780853487.936075 mapwrite 0x03e518 thru 0x03ffff (0x1ae8 bytes) 000878 1780853488.045547 fallocate 0x031a52 thru 0x03ae47 (0x93f6 bytes) 000879 1780853488.055462 fallocate 0x03f45b thru 0x03ffff (0xba5 bytes) 000882 1780853488.068417 trunc from 0x025fc4 to 0x02743c (0x1479 bytes) 000883 1780853488.075890 fallocate 0x03ee7d thru 0x03ffff (0x1183 bytes) 000884 1780853488.086054 mapwrite 0x002543 thru 0x008936 (0x63f4 bytes) 000885 1780853488.142021 punch from 0x03ef30 to 0x03ffff (0x10d0 bytes) 000886 1780853488.187813 MIRROR_RESYNC 3 mirrors 000887 1780853489.694916 MIRROR_EXTEND to 4 mirrors 000888 1780853490.649508 read 0x03a23c thru 0x03ffff (0x5dc4 bytes) 000889 1780853490.664846 read 0x02268a thru 0x02f552 (0xcec9 bytes) 000891 1780853490.686621 fallocate 0x02a5d9 thru 0x02d698 (0x30c0 bytes) 000892 1780853490.764410 punch from 0x006c0e to 0x012f01 (0xc2f4 bytes) 000893 1780853490.771980 read 0x0144ad thru 0x018551 (0x40a5 bytes) 000894 1780853490.854077 fallocate 0x030430 thru 0x032135 (0x1d06 bytes) 000895 1780853490.860024 trunc from 0x040000 to 0x00d92e (0x326d1 bytes) 000896 1780853490.892974 punch from 0x0091c5 to 0x00d6a7 (0x44e3 bytes) 000897 1780853490.900878 write 0x00275e thru 0x00ff6b (0xd80e bytes) 000898 1780853490.903664 mapwrite 0x03a0a3 thru 0x03ffff (0x5f5d bytes) 000899 1780853490.987111 mapwrite 0x00e0ba thru 0x011948 (0x388f bytes) 000900 1780853491.018106 write 0x031f17 thru 0x03ffff (0xe0e9 bytes) 000901 1780853491.026892 punch from 0x03d83f to 0x03ffff (0x27c1 bytes) 000902 1780853491.053078 punch from 0x00705f to 0x0120d6 (0xb078 bytes) 000903 1780853491.061946 fallocate 0x02c434 thru 0x036144 (0x9d11 bytes) 000904 1780853491.069923 mapwrite 0x0166b5 thru 0x0213b5 (0xad01 bytes) 000906 1780853491.179366 MIRROR_RESYNC 4 mirrors 000907 1780853493.162753 MIRROR_EXTEND to 5 mirrors 000908 1780853494.533119 mapread 0x00c7e4 thru 0x0186af (0xbecc bytes) 000909 1780853494.597777 write 0x00a8e7 thru 0x00c37c (0x1a96 bytes) 000910 1780853494.696912 mapread 0x009e42 thru 0x015767 (0xb926 bytes) 000911 1780853494.713726 read 0x019746 thru 0x0238a1 (0xa15c bytes) 000912 1780853494.726253 read 0x02c20c thru 0x03aef2 (0xece7 bytes) 000913 1780853494.749800 read 0x036de7 thru 0x037b14 (0xd2e bytes) 000914 1780853494.754573 fallocate 0x005cfe thru 0x00c7bd (0x6ac0 bytes) 000915 1780853494.765531 mapread 0x02b62f thru 0x036b0e (0xb4e0 bytes) 000916 1780853494.770098 trunc from 0x012591 to 0x018fdd (0x6a4d bytes) 000917 1780853494.784641 mapread 0x0328e3 thru 0x03b0f2 (0x8810 bytes) 000918 1780853494.796481 read 0x0254d3 thru 0x033f41 (0xea6f bytes) 000919 1780853494.797773 fallocate 0x012926 thru 0x02021f (0xd8fa bytes) 000920 1780853494.811200 trunc from 0x040000 to 0x008998 (0x37667 bytes) 000921 1780853494.864465 write 0x021771 thru 0x02983a (0x80ca bytes) 000922 1780853494.865951 mapread 0x01c45c thru 0x01d379 (0xf1e bytes) 000923 1780853494.889837 mapread 0x01b2b5 thru 0x0214f0 (0x623c bytes) 000924 1780853494.891533 mapwrite 0x01b342 thru 0x028fed (0xdcac bytes) 000925 1780853494.939789 trunc from 0x02983b to 0x00c1ca (0x1d670 bytes) 000926 1780853494.982204 trunc from 0x00c1cb to 0x023674 (0x174aa bytes) 000927 1780853495.012201 mapwrite 0x019fb3 thru 0x01b4cf (0x151d bytes) 000928 1780853495.066897 MIRROR_RESYNC 5 mirrors 000929 1780853497.153659 fallocate 0x01a823 thru 0x024d7a (0xa558 bytes) 000930 1780853497.222341 trunc from 0x00399f to 0x00fea7 (0xc509 bytes) 000932 1780853497.238630 fallocate 0x03ac9f thru 0x03ffff (0x5361 bytes) 000933 1780853497.243616 mapwrite 0x021945 thru 0x023d76 (0x2432 bytes) 000934 1780853497.315618 trunc from 0x024d7b to 0x034b08 (0xfd8e bytes) 000935 1780853497.353323 mapwrite 0x00371a thru 0x00bd10 (0x85f7 bytes) 000936 1780853497.400860 write 0x037fa8 thru 0x03ffff (0x8058 bytes) 000937 1780853497.406926 write 0x027b84 thru 0x0334b2 (0xb92f bytes) 000938 1780853497.440931 trunc from 0x034389 to 0x034426 (0x9e bytes) 000939 1780853497.451212 mapwrite 0x01aec4 thru 0x023621 (0x875e bytes) 000940 1780853497.509339 trunc from 0x002e25 to 0x00b570 (0x874c bytes) 000941 1780853497.515962 punch from 0x039a96 to 0x03ed5b (0x52c6 bytes) 000942 1780853497.522489 read 0x002300 thru 0x0028c2 (0x5c3 bytes) 000943 1780853497.528683 write 0x01ea91 thru 0x0217a8 (0x2d18 bytes) 000944 1780853497.530625 fallocate 0x00d770 thru 0x017e69 (0xa6fa bytes) 000945 1780853497.536340 mapread 0x0001d3 thru 0x00990b (0x9739 bytes) 000947 1780853497.572407 fallocate 0x03da4a thru 0x03ffff (0x25b6 bytes) 000949 1780853497.586541 fallocate 0x01b72d thru 0x01fbf2 (0x44c6 bytes) 000952 1780853497.599755 trunc from 0x02c894 to 0x03648c (0x9bf9 bytes) 000953 1780853497.607069 trunc from 0x040000 to 0x022af3 (0x1d50c bytes) 000955 1780853497.654934 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x22af4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22af4): Mirror 3: 0xd00f2b69 (stale) Mirror 4: 0xd00f2b69 (stale) Mirror 5: 0xd00f2b69 (stale) Mirror 6: 0xd00f2b69 (stale) CRC-32 checksum value for chunk [0, 0x22af4): Mirror 1: 0xc1c3aa96 Mirror 7: 0xc1c3aa96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22af4: skipped 000956 1780853499.084067 punch from 0x020add to 0x022af3 (0x2017 bytes) 000957 1780853499.158015 mapread 0x019f3e thru 0x022369 (0x842c bytes) 000958 1780853499.201300 fallocate 0x034bd6 thru 0x03ffff (0xb42a bytes) 000959 1780853499.216058 punch from 0x019fcc to 0x01ac52 (0xc87 bytes) 000960 1780853499.234976 write 0x0140e4 thru 0x018342 (0x425f bytes) 000961 1780853499.281015 write 0x0225d8 thru 0x02dbcf (0xb5f8 bytes) 000962 1780853499.298649 trunc from 0x018fb8 to 0x0264ec (0xd535 bytes) 000964 1780853499.342347 write 0x024e80 thru 0x034b3f (0xfcc0 bytes) 000965 1780853499.372999 trunc from 0x040000 to 0x02603e (0x19fc1 bytes) 000966 1780853499.422067 write 0x00cd2e thru 0x0140d7 (0x73aa bytes) 000967 1780853499.463429 trunc from 0x0175b9 to 0x022574 (0xafbc bytes) 000968 1780853499.474139 trunc from 0x012d63 to 0x017307 (0x45a5 bytes) 000969 1780853499.500388 punch from 0x013503 to 0x01f86e (0xc36c bytes) 000970 1780853499.510736 write 0x022415 thru 0x02c3a1 (0x9f8d bytes) 000972 1780853499.535420 punch from 0x012275 to 0x016a54 (0x47e0 bytes) 000973 1780853499.547077 MIRROR_RESYNC 6 mirrors 000974 1780853502.022966 MIRROR_RESYNC 6 mirrors 000975 1780853502.679114 write 0x00c99b thru 0x017e79 (0xb4df bytes) 000976 1780853502.759022 read 0x01e5a5 thru 0x02c3a1 (0xddfd bytes) 000977 1780853502.808238 punch from 0x02495d to 0x025afe (0x11a2 bytes) 000978 1780853502.819328 trunc from 0x012d99 to 0x01d7c5 (0xaa2d bytes) 000979 1780853502.843334 write 0x00c008 thru 0x00e4f2 (0x24eb bytes) 000980 1780853502.848973 fallocate 0x03d2a7 thru 0x03ffff (0x2d59 bytes) 000981 1780853502.861640 mapread 0x01aeea thru 0x01bcd0 (0xde7 bytes) 000982 1780853502.896515 mapwrite 0x01b269 thru 0x020f79 (0x5d11 bytes) 000983 1780853502.948424 MIRROR_RESYNC 6 mirrors 000984 1780853505.184498 punch from 0x02a160 to 0x02c3a1 (0x2242 bytes) 000985 1780853505.234781 trunc from 0x016fbb to 0x0223a1 (0xb3e7 bytes) 000986 1780853505.240839 mapread 0x0039df thru 0x004957 (0xf79 bytes) 000987 1780853505.261007 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c3a2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3a2): Mirror 3: 0xfd60f155 (stale) Mirror 4: 0xfd60f155 (stale) Mirror 5: 0xfd60f155 (stale) Mirror 6: 0xfd60f155 (stale) Mirror 7: 0xfd60f155 (stale) CRC-32 checksum value for chunk [0, 0x2c3a2): Mirror 1: 0xc22ab4e3 Mirror 8: 0xc22ab4e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3a2: skipped 000988 1780853506.602754 trunc from 0x015295 to 0x024497 (0xf203 bytes) 000989 1780853506.707554 trunc from 0x02b8bf to 0x02c3a1 (0xae3 bytes) 000990 1780853506.719826 fallocate 0x00a700 thru 0x017fef (0xd8f0 bytes) 000991 1780853506.732383 write 0x024dfb thru 0x03014e (0xb354 bytes) 000992 1780853506.755834 write 0x00e841 thru 0x01c5e4 (0xdda4 bytes) 000994 1780853506.782228 mapwrite 0x03c3c7 thru 0x03d344 (0xf7e bytes) 000995 1780853506.849284 mapread 0x000e70 thru 0x00917f (0x8310 bytes) 000997 1780853506.865672 trunc from 0x001724 to 0x00de68 (0xc745 bytes) 000998 1780853506.877415 write 0x004cfe thru 0x0054df (0x7e2 bytes) 000999 1780853506.902247 write 0x035c18 thru 0x03b9f9 (0x5de2 bytes) 001000 1780853506.903691 trunc from 0x03d345 to 0x01d810 (0x1fb34 bytes) All operations completed A-OK! PASS 70a (110s) == sanity-flr test 71: check mirror extend parent fid ==== 13:31:52 (1780853512) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x195:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 13:32:02 (1780853522) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.338719 s, 6.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177069 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 13:32:12 (1780853532) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5893024 .. Extending file size to 4620224 .. failed Extending file size to 4946304 .. Extending file size to 8364544 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6222848 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8170080 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8534368 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8786592 .. Extending file size to 8851264 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8525920 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60844 60845 60846 60848 60851 done 192.168.204.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre2 (opts:) 192.168.204.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:33:40 (1780853620) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3368960 .. Extending file size to 6253376 .. failed Extending file size to 6239040 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8100128 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6564768 .. Extending file size to 6636672 .. Extending file size to 8779424 .. Extending file size to 7331008 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8195008 .. Extending file size to 9285792 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8468288 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62348 62349 62350 62353 62355 failed 192.168.204.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre2 (opts:) 192.168.204.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:35:02 (1780853702) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2156 1269148 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24744 3582276 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18928 3588092 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43672 7170368 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2156 1269148 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24744 3582276 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18928 3588092 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43672 7170368 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129167 s, 16.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.330224 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3e6f41a6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:35:25 (1780853725) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12224 s, 17.2 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 13:35:34 (1780853734) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:35:42 (1780853742) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100597 s, 20.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:35:52 (1780853752) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0995083 s, 21.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:36:00 (1780853760) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156937 s, 13.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:36:35 (1780853795) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg415-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787411 s, 13.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:37:27 (1780853847) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg415-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0825012 s, 25.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:38:21 (1780853901) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg415-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg415-server: Pool lustre.test_204f created Adding targets to pool oleg415-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg415-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120301 s, 17.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg415-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg415-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg415-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:39:38 (1780853978) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:39:46 (1780853986) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:39:55 (1780853995) oleg415-server: Pool lustre.test_206 created Adding targets to pool oleg415-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg415-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg415-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg415-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg415-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:40:23 (1780854023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719502 s, 14.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485308 s, 21.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5dd01d47 Mirror 3: 0x5dd01d47 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:40:32 (1780854032) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:40:35 (1780854035) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:40:39 (1780854039) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0511468 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x690000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xdc0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xdc0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xdc0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ff0000 Read this (4 bytes): BBBB 192.168.204.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:41:16 (1780854076) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0170503 s, 480 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:41:22 (1780854082) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413493 s, 25.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x120:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611269 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 13:41:24.000000000 -0400 Modify: 2026-06-07 13:41:24.000000000 -0400 Change: 2026-06-07 13:41:26.000000000 -0400 Birth: 2026-06-07 13:41:25.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:41:32 (1780854092) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315572 s, 33.2 MB/s oleg415-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg415-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg415-server: O/280000401/d31/13f: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:41:57 (1780854117) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.223023 s, 184 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0352948 s, 116 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 7 13:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 7 13:42 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:42:05 (1780854125) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:42:12 (1780854132) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1905 sec =========== 13:42:22 (1780854142) === sanity-flr: start cleanup 13:42:23 (1780854143) === === sanity-flr: finish cleanup 13:42:32 (1780854152) ===