-----============= acceptance-small: sanity-flr ============----- Fri Aug 28 04:36:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg625-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:36:47 (1787906207) === oleg625-client.virtnet: executing check_config_client /mnt/lustre oleg625-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg625-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f99c5f7f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f99c5f7f800.idle_timeout=debug disable quota as required oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:37:29 (1787906249) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8252] == sanity-flr test 0a: lfs mirror create with -N option == 04:37:32 (1787906252) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:37:50 (1787906270) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:37:54 (1787906274) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:37:59 (1787906279) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:38:28 (1787906308) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:38:32 (1787906312) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 04:38:36 (1787906316) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f99c5f7f800.stats=clear osc.lustre-OST0001-osc-ffff8f99c5f7f800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:38:44 (1787906324) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:38:55 (1787906335) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:39:03 (1787906343) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:39:06 (1787906346) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:39:14 (1787906354) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 04:39:23 (1787906363) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:39:30 (1787906370) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:39:37 (1787906377) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:39:45 (1787906385) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:39:55 (1787906395) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:40:01 (1787906401) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.05808 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899114 s, 11.7 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:40:10 (1787906410) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.617794 s, 30.6 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:40:19 (1787906419) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:40:30 (1787906430) Stopping /mnt/lustre-ost1 (opts:) on oleg625-server oleg625-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg625-server: oleg625-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg625-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg625-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg625-server oleg625-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg625-server: oleg625-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg625-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (133s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:42:43 (1787906563) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.479033 s, 15.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg625-server oleg625-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg625-server: oleg625-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg625-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:43:27 (1787906607) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg625-server oleg625-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg625-server: oleg625-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg625-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg625-server oleg625-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg625-server: oleg625-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg625-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:44:50 (1787906690) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.26853 s, 16.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 28 04:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.397s user 0m0.001s sys 0m0.146s Stopping /mnt/lustre-ost1 (opts:) on oleg625-server oleg625-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f99c5f7f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.252s user 0m0.002s sys 0m0.149s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg625-server: oleg625-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg625-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg625-server oleg625-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.252s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg625-server: oleg625-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg625-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg625-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f99c5f7f800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:46:27 (1787906787) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:46:31 (1787906791) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:46:35 (1787906795) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 04:46:39 (1787906799) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 04:46:47 (1787906807) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.540441 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:47:01 (1787906821) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416622 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312405 s, 33.6 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:47:11 (1787906831) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:47:21 (1787906841) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00204958 s, 500 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118378 s, 865 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00156684 s, 654 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00186346 s, 550 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 04:47:31 (1787906851) create test files with size 15756007 .. 4533b4697fb3a96bc28bf89296a2517c - 5c266254f2c7ffc4e4a7344222ccfb86 - 461ac31a1c8e2eddcbb67f2d9aa84fd5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg625-client.virtnet /mnt/lustre (opts:) Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 04:47:55 (1787906875) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:48:07 (1787906887) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:48:14 (1787906894) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120612 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031296 s, 33.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 04:48:24 (1787906904) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0872937 s, 24.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.443953 s, 9.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135367 s, 15.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 04:48:42 (1787906922) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 04:48:46 (1787906926) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:48:50 (1787906930) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293489 s, 35.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 04:48:58 (1787906938) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.366494 s, 8.6 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:49:09 (1787906949) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:49:19 (1787906959) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.321251 s, 32.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg625-client.virtnet /mnt/lustre (opts:) Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:49:31 (1787906971) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8f99c8174000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.532594 s, 19.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg625-client.virtnet /mnt/lustre (opts:) Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8f99d1d0c800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:49:49 (1787906989) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:50:33 (1787907033) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:51:11 (1787907071) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:51:15 (1787907075) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.332791 s, 9.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.182955 s, 17.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6f49d2fc (nosync,stale) Mirror 2: 0x6f49d2fc (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3a314cdd781b5517a1ea9469f8ad8155 - mirror 1 checksum: 4f7d2430db987b9260a54d1bba4486b7 - mirror 2 checksum: 3a314cdd781b5517a1ea9469f8ad8155 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-28T04:51:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-28T04:51:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3a314cdd781b5517a1ea9469f8ad8155 - mirror 1 checksum: 4f7d2430db987b9260a54d1bba4486b7 - mirror 2 checksum: 4f7d2430db987b9260a54d1bba4486b7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-28T04:51:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-28T04:51:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:51:26 (1787907086) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387155 s, 27.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:51:34 (1787907094) Failing mds1 on oleg625-server Stopping /mnt/lustre-mds1 (opts:) on oleg625-server 04:51:47 (1787907107) shut down facet: mds1 facet_host: oleg625-server facet_failover_host: oleg625-server Failover mds1 to oleg625-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg625-server: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-MDT0000 04:52:12 (1787907132) targets are mounted 04:52:12 (1787907132) facet_failover done oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379447 s, 27.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } PASS 50A (55s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:52:30 (1787907150) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0485594 s, 422 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0111435 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:52:40 (1787907160) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.11285 s, 181 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0130857 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:52:50 (1787907170) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00681913 s, 2.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:52:58 (1787907178) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0700345 s, 292 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:53:07 (1787907187) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0115176 s, 356 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0233457 s, 175 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:53:15 (1787907195) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:53:40 (1787907220) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:54:07 (1787907247) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:54:31 (1787907271) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:54:55 (1787907295) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 04:55:04 (1787907304) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57616 mirror split pid 57617 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57616 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57617 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 04:56:11 (1787907371) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 7373 skipping zero length punch hole truncating to largest ever: 0x3b685 000002 1787907373.506295 trunc from 00000000 to 0x03b684 (0x3b685 bytes) 000003 1787907373.528606 trunc from 0x03b685 to 0x008984 (0x32d00 bytes) 000004 1787907373.545282 trunc from 0x008985 to 0x01ab03 (0x1217f bytes) 000006 1787907373.566773 trunc from 0x00415a to 0x00b4f9 (0x73a0 bytes) 000008 1787907373.572954 trunc from 0x008306 to 0x00ec60 (0x695b bytes) 000010 1787907373.579182 punch from 0x017e06 to 0x01ab03 (0x2cfe bytes) 000011 1787907373.584149 trunc from 0x00c6af to 0x018532 (0xbe84 bytes) 000012 1787907373.590583 read 0x00eff6 thru 0x01ab03 (0xbb0e bytes) 000013 1787907373.602371 fallocate 0x0172f4 thru 0x01cdee (0x5afb bytes) 000014 1787907373.613950 MIRROR_EXTEND to 2 mirrors 000015 1787907374.303594 punch from 0x01839c to 0x01cdee (0x4a53 bytes) 000016 1787907374.330987 read 0x00a910 thru 0x01326c (0x895d bytes) 000017 1787907374.339292 mapwrite 0x0085e7 thru 0x00882e (0x248 bytes) 000018 1787907374.352989 read 0x00e13d thru 0x01b005 (0xcec9 bytes) 000019 1787907374.366368 mapwrite 0x035f24 thru 0x03ffff (0xa0dc bytes) 000020 1787907374.416111 write 0x01bdbf thru 0x02bda2 (0xffe4 bytes) 000021 1787907374.422503 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe4793e80 Mirror 3: 0xe4793e80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000022 1787907375.425270 write 0x028754 thru 0x02ff39 (0x77e6 bytes) 000024 1787907375.489450 MIRROR_RESYNC 3 mirrors 000025 1787907376.580339 punch from 0x027a03 to 0x0368b4 (0xeeb2 bytes) 000026 1787907376.607476 write 0x0249d8 thru 0x027f50 (0x3579 bytes) 000027 1787907376.617725 read 0x0071a8 thru 0x00d1da (0x6033 bytes) 000028 1787907376.626003 punch from 0x0027b2 to 0x002e29 (0x678 bytes) 000030 1787907376.632246 punch from 0x028057 to 0x032f3d (0xaee7 bytes) 000031 1787907376.636829 punch from 0x02cd95 to 0x0339bf (0x6c2b bytes) 000032 1787907376.641858 mapread 0x026ec3 thru 0x028b97 (0x1cd5 bytes) 000034 1787907376.648552 mapread 0x011587 thru 0x018d52 (0x77cc bytes) 000035 1787907376.663267 write 0x01b265 thru 0x022639 (0x73d5 bytes) 000036 1787907376.671914 read 0x008848 thru 0x0139cf (0xb188 bytes) 000037 1787907376.672624 mapread 0x03f363 thru 0x03ffff (0xc9d bytes) 000038 1787907376.696270 MIRROR_RESYNC 3 mirrors 000039 1787907377.555405 trunc from 0x02c23d to 0x038969 (0xc72d bytes) 000040 1787907377.607182 mapread 0x005585 thru 0x00f761 (0xa1dd bytes) 000041 1787907377.640175 write 0x003007 thru 0x00f189 (0xc183 bytes) 000042 1787907377.643275 read 0x019bdc thru 0x01d163 (0x3588 bytes) 000043 1787907377.671039 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3936220c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3936220c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000044 1787907378.054860 mapread 0x017dd8 thru 0x025b42 (0xdd6b bytes) 000045 1787907378.102030 mapwrite 0x02879e thru 0x032826 (0xa089 bytes) 000046 1787907378.153528 write 0x01414d thru 0x021fd6 (0xde8a bytes) 000047 1787907378.180414 MIRROR_RESYNC 2 mirrors 000049 1787907379.015675 fallocate 0x02c8ef thru 0x03aa0c (0xe11e bytes) 000050 1787907379.053436 trunc from 0x0221de to 0x02cb51 (0xa974 bytes) 000051 1787907379.063856 mapwrite 0x02c601 thru 0x03821f (0xbc1f bytes) 000052 1787907379.128774 fallocate 0x000544 thru 0x006369 (0x5e26 bytes) 000053 1787907379.142156 punch from 0x030fdb to 0x03cd56 (0xbd7c bytes) 000054 1787907379.152280 read 0x0077fc thru 0x00fafc (0x8301 bytes) 000056 1787907379.179386 mapread 0x014e67 thru 0x017071 (0x220b bytes) 000057 1787907379.196933 write 0x028f9f thru 0x034d0d (0xbd6f bytes) 000058 1787907379.208119 write 0x02a13f thru 0x031e44 (0x7d06 bytes) 000059 1787907379.221855 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000060 1787907379.402235 read 0x01232b thru 0x01bb9b (0x9871 bytes) 000061 1787907379.417555 mapread 0x03dba1 thru 0x03e75f (0xbbf bytes) 000062 1787907379.434482 punch from 0x0284d5 to 0x0308f0 (0x841c bytes) 000063 1787907379.444901 fallocate 0x008e89 thru 0x00c0d3 (0x324b bytes) 000065 1787907379.468392 trunc from 0x040000 to 0x03a4dc (0x5b23 bytes) 000066 1787907379.527771 trunc from 0x03a4dd to 0x022e3b (0x176a1 bytes) 000067 1787907379.558097 MIRROR_EXTEND to 2 mirrors 000068 1787907380.525112 trunc from 0x022e3c to 0x02e9dc (0xbba1 bytes) 000069 1787907380.577939 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000070 1787907380.714395 write 0x002171 thru 0x00be4a (0x9cda bytes) 000071 1787907380.734486 trunc from 0x02884f to 0x02e9dc (0x618e bytes) 000072 1787907380.738929 trunc from 0x00565e to 0x00b445 (0x5de8 bytes) 000073 1787907380.755450 punch from 0x02b4ec to 0x02e9dc (0x34f1 bytes) 000074 1787907380.760002 write 0x034fd8 thru 0x03ffff (0xb028 bytes) 000075 1787907380.763149 write 0x008724 thru 0x014d89 (0xc666 bytes) 000076 1787907380.772479 write 0x019326 thru 0x01a5fa (0x12d5 bytes) 000077 1787907380.781188 fallocate 0x0283a9 thru 0x032fa1 (0xabf9 bytes) 000078 1787907380.786204 mapread 0x01a479 thru 0x025655 (0xb1dd bytes) 000079 1787907380.802832 read 0x021546 thru 0x0239d5 (0x2490 bytes) 000080 1787907380.807230 MIRROR_EXTEND to 2 mirrors 000081 1787907381.791233 MIRROR_RESYNC 2 mirrors 000082 1787907382.255816 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000083 1787907382.410990 trunc from 0x0098d2 to 0x00f18c (0x58bb bytes) 000085 1787907382.431512 write 0x018d67 thru 0x01de1f (0x50b9 bytes) 000086 1787907382.444996 fallocate 0x029e11 thru 0x02c192 (0x2382 bytes) 000087 1787907382.450183 mapwrite 0x0079f7 thru 0x011fc1 (0xa5cb bytes) fallocating to largest ever: 0x40000 000088 1787907382.489080 fallocate 0x03dc0a thru 0x03ffff (0x23f6 bytes) 000089 1787907382.494101 read 0x024da2 thru 0x034b83 (0xfde2 bytes) 000090 1787907382.504603 read 0x02ac34 thru 0x02d7bc (0x2b89 bytes) 000091 1787907382.505544 trunc from 0x040000 to 0x0373e9 (0x8c16 bytes) 000092 1787907382.524544 mapwrite 0x02cf87 thru 0x036b4c (0x9bc6 bytes) 000093 1787907382.602920 MIRROR_EXTEND to 2 mirrors 000094 1787907383.543554 mapwrite 0x013fae thru 0x01a74f (0x67a2 bytes) 000095 1787907383.671804 fallocate 0x00fab2 thru 0x01a7c0 (0xad0f bytes) 000096 1787907383.677347 punch from 0x015fbc to 0x01b1df (0x5224 bytes) 000097 1787907383.686036 fallocate 0x0138fd thru 0x016053 (0x2757 bytes) 000098 1787907383.693288 write 0x02f14b thru 0x03cd36 (0xdbec bytes) 000099 1787907383.703260 trunc from 0x03cd37 to 0x01aaff (0x22237 bytes) 000101 1787907383.731623 punch from 0x00a7c1 to 0x018cad (0xe4ed bytes) 000103 1787907383.739264 write 0x02f7ed thru 0x03b99f (0xc1b3 bytes) 000104 1787907383.745080 write 0x02f46b thru 0x03658f (0x7125 bytes) 000106 1787907383.748638 trunc from 0x03b9a0 to 0x022c7b (0x18d24 bytes) 000107 1787907383.777742 trunc from 0x00c3dd to 0x01911c (0xcd40 bytes) 000108 1787907383.784204 write 0x02671e thru 0x028a02 (0x22e5 bytes) 000109 1787907383.785360 mapread 0x015a90 thru 0x01ef78 (0x94e9 bytes) 000110 1787907383.840694 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000111 1787907384.050484 MIRROR_EXTEND to 2 mirrors 000115 1787907384.764708 punch from 0x0230fe to 0x028a02 (0x5905 bytes) 000116 1787907384.795525 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000117 1787907384.948124 fallocate 0x027e11 thru 0x02ac06 (0x2df6 bytes) 000118 1787907384.953524 read 0x016b0d thru 0x0213f0 (0xa8e4 bytes) 000119 1787907384.963622 mapwrite 0x018a54 thru 0x019677 (0xc24 bytes) 000120 1787907384.979719 trunc from 0x019644 to 0x024d16 (0xb6d3 bytes) 000121 1787907384.987086 mapwrite 0x00fb7c thru 0x017382 (0x7807 bytes) 000122 1787907385.024502 mapread 0x011448 thru 0x01ae3b (0x99f4 bytes) 000123 1787907385.041912 mapwrite 0x02b631 thru 0x0332d0 (0x7ca0 bytes) 000124 1787907385.118110 trunc from 0x02761b to 0x0332d0 (0xbcb6 bytes) 000126 1787907385.123415 mapread 0x00e99e thru 0x01ce20 (0xe483 bytes) 000128 1787907385.128199 write 0x022bf4 thru 0x02f535 (0xc942 bytes) 000130 1787907385.134415 write 0x003fbc thru 0x010745 (0xc78a bytes) 000131 1787907385.141541 fallocate 0x01f57f thru 0x02172f (0x21b1 bytes) 000132 1787907385.146565 read 0x02968b thru 0x032bde (0x9554 bytes) 000133 1787907385.168324 mapread 0x00e2e4 thru 0x010bd4 (0x28f1 bytes) 000134 1787907385.169345 trunc from 0x0332d1 to 0x021dac (0x11524 bytes) 000135 1787907385.190761 punch from 0x01cf50 to 0x020039 (0x30ea bytes) 000136 1787907385.195513 mapread 0x00b22d thru 0x013dd3 (0x8ba7 bytes) 000137 1787907385.196735 trunc from 0x021dad to 0x01e1e6 (0x3bc6 bytes) 000138 1787907385.210619 mapread 0x00197d thru 0x002742 (0xdc6 bytes) 000139 1787907385.212192 mapwrite 0x012759 thru 0x020bb8 (0xe460 bytes) 000140 1787907385.289826 trunc from 0x020bb9 to 0x006e9e (0x19d1a bytes) 000141 1787907385.313257 fallocate 0x03d16b thru 0x03ffff (0x2e95 bytes) 000142 1787907385.318500 fallocate 0x030ec2 thru 0x037dcc (0x6f0b bytes) 000143 1787907385.324551 fallocate 0x030ed4 thru 0x032370 (0x149d bytes) 000144 1787907385.333086 trunc from 0x037dcd to 0x01aaf5 (0x1d2d7 bytes) 000145 1787907385.365886 fallocate 0x032496 thru 0x03ffff (0xdb6a bytes) 000146 1787907385.379533 trunc from 0x01aaf6 to 0x02c85a (0x11d65 bytes) 000147 1787907385.404545 punch from 0x006320 to 0x00c7f3 (0x64d4 bytes) 000149 1787907385.415802 mapread 0x01d5e6 thru 0x026b50 (0x956b bytes) 000150 1787907385.496479 trunc from 0x01d3c4 to 0x02732b (0x9f68 bytes) 000151 1787907385.502490 write 0x00601d thru 0x00fb53 (0x9b37 bytes) 000152 1787907385.527749 punch from 0x012053 to 0x02041d (0xe3cb bytes) 000153 1787907385.537011 trunc from 0x02c85b to 0x01d089 (0xf7d1 bytes) 000154 1787907385.569292 read 0x00bac1 thru 0x00e179 (0x26b9 bytes) 000155 1787907385.570022 punch from 0x0192f3 to 0x01d089 (0x3d97 bytes) 000156 1787907385.582256 trunc from 0x01d08a to 0x02ae87 (0xddfe bytes) 000157 1787907385.605585 trunc from 0x02ae88 to 0x030efc (0x6075 bytes) 000158 1787907385.633697 trunc from 0x017987 to 0x01badc (0x4156 bytes) 000159 1787907385.647510 write 0x01f70b thru 0x0210ff (0x19f5 bytes) 000160 1787907385.687161 read 0x02249a thru 0x0252ba (0x2e21 bytes) 000161 1787907385.688111 trunc from 0x017e47 to 0x01c6c3 (0x487d bytes) 000162 1787907385.694083 read 0x00293a thru 0x00d32d (0xa9f4 bytes) 000163 1787907385.695173 mapwrite 0x011a9c thru 0x020a0b (0xef70 bytes) 000164 1787907385.759257 mapread 0x0006e5 thru 0x00e9eb (0xe307 bytes) 000165 1787907385.761439 trunc from 0x030efd to 0x011195 (0x1fd67 bytes) 000166 1787907385.779518 mapread 0x00802c thru 0x01113e (0x9113 bytes) 000168 1787907385.784086 mapwrite 0x009302 thru 0x0117e3 (0x84e2 bytes) 000169 1787907385.834072 mapread 0x00ef34 thru 0x0117e3 (0x28b0 bytes) 000170 1787907385.840610 MIRROR_EXTEND to 2 mirrors 000171 1787907386.623640 read 0x009a74 thru 0x00b649 (0x1bd6 bytes) 000172 1787907386.636465 punch from 0x010027 to 0x0117e3 (0x17bd bytes) 000174 1787907386.670764 write 0x02ff33 thru 0x036e57 (0x6f25 bytes) 000175 1787907386.677625 mapread 0x02219d thru 0x02f102 (0xcf66 bytes) 000176 1787907386.697831 punch from 0x001875 to 0x01036a (0xeaf6 bytes) 000177 1787907386.702815 fallocate 0x031328 thru 0x03ffff (0xecd8 bytes) 000178 1787907386.707575 trunc from 0x02b374 to 0x02df3e (0x2bcb bytes) 000179 1787907386.712216 fallocate 0x039875 thru 0x03ffff (0x678b bytes) 000180 1787907386.717143 trunc from 0x008cb6 to 0x00dfa1 (0x52ec bytes) 000181 1787907386.726426 trunc from 0x037302 to 0x03ffff (0x8cfe bytes) 000183 1787907386.736449 mapread 0x031462 thru 0x03d5cd (0xc16c bytes) 000184 1787907386.747730 mapwrite 0x00bac9 thru 0x011ea4 (0x63dc bytes) 000185 1787907386.778336 read 0x01de47 thru 0x02bce4 (0xde9e bytes) 000186 1787907386.795816 punch from 0x020377 to 0x028cc7 (0x8951 bytes) 000187 1787907386.801787 mapwrite 0x0274dd thru 0x0362e5 (0xee09 bytes) 000188 1787907386.847225 punch from 0x03c4e2 to 0x03ffff (0x3b1e bytes) 000189 1787907386.855903 fallocate 0x038a6e thru 0x03ffff (0x7592 bytes) 000190 1787907386.867935 MIRROR_RESYNC 2 mirrors 000191 1787907387.736391 write 0x028e44 thru 0x035650 (0xc80d bytes) 000192 1787907387.777823 trunc from 0x040000 to 0x01b051 (0x24fae bytes) 000193 1787907387.806015 read 0x00004c thru 0x00d073 (0xd028 bytes) 000195 1787907387.817671 fallocate 0x03d8e3 thru 0x03ffff (0x271d bytes) 000196 1787907387.826764 punch from 0x0281ef to 0x0286f4 (0x506 bytes) 000197 1787907387.836600 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x73166a5d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd1ab35af Mirror 3: 0xd1ab35af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1787907388.749203 trunc from 0x040000 to 0x00930a (0x36cf5 bytes) 000199 1787907388.816866 MIRROR_RESYNC 3 mirrors 000200 1787907389.696186 read 0x00728e thru 0x00930a (0x207d bytes) 000201 1787907389.705252 mapwrite 0x017c73 thru 0x026a2f (0xedbd bytes) 000202 1787907389.916738 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26a30): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a30): Mirror 2: 0x817afa49 (stale) Mirror 3: 0x817afa49 (stale) CRC-32 checksum value for chunk [0, 0x26a30): Mirror 1: 0x41e6e2bd Mirror 4: 0x41e6e2bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a30: skipped 000203 1787907390.808424 punch from 0x021510 to 0x026a2f (0x5520 bytes) 000204 1787907390.877880 mapwrite 0x00828b thru 0x00c27b (0x3ff1 bytes) 000205 1787907390.955086 trunc from 0x00a542 to 0x014602 (0xa0c1 bytes) 000206 1787907390.963118 fallocate 0x024516 thru 0x025572 (0x105d bytes) 000207 1787907390.972167 fallocate 0x01a982 thru 0x0215db (0x6c5a bytes) 000208 1787907390.983190 write 0x03dd7f thru 0x03ffff (0x2281 bytes) 000209 1787907390.996058 mapread 0x00a532 thru 0x018ee3 (0xe9b2 bytes) 000210 1787907391.035425 trunc from 0x01a3a1 to 0x02a23a (0xfe9a bytes) 000211 1787907391.041236 fallocate 0x02d41a thru 0x0314d0 (0x40b7 bytes) 000212 1787907391.047413 trunc from 0x040000 to 0x0045fe (0x3ba01 bytes) 000213 1787907391.082540 fallocate 0x004ad7 thru 0x012445 (0xd96f bytes) 000214 1787907391.094959 mapwrite 0x02e700 thru 0x039302 (0xac03 bytes) 000215 1787907391.173179 mapwrite 0x0391c2 thru 0x03bf87 (0x2dc6 bytes) 000216 1787907391.228557 punch from 0x02ca1d to 0x02fbf5 (0x31d9 bytes) 000217 1787907391.233601 trunc from 0x016c52 to 0x02188c (0xac3b bytes) 000219 1787907391.241244 fallocate 0x03c534 thru 0x03ffff (0x3acc bytes) 000220 1787907391.253061 trunc from 0x03bf88 to 0x015ba5 (0x263e2 bytes) 000222 1787907391.275104 mapwrite 0x03a39c thru 0x03ffff (0x5c64 bytes) 000223 1787907391.333150 MIRROR_RESYNC 4 mirrors 000224 1787907392.310609 write 0x02264b thru 0x025661 (0x3017 bytes) 000225 1787907392.358916 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91ce975 (stale) Mirror 3: 0x91ce975 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91ce975 (stale) Mirror 3: 0x91ce975 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000226 1787907392.733043 punch from 0x03ff47 to 0x03ffff (0xb9 bytes) 000227 1787907392.743576 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91ce975 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91ce975 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000228 1787907393.030956 trunc from 0x03a348 to 0x03b12f (0xde8 bytes) 000229 1787907393.052976 read 0x00b8de thru 0x014013 (0x8736 bytes) 000230 1787907393.078523 mapread 0x026863 thru 0x02953f (0x2cdd bytes) 000231 1787907393.112471 trunc from 0x033510 to 0x033a5e (0x54f bytes) 000233 1787907393.129860 trunc from 0x040000 to 0x02f68a (0x10975 bytes) 000234 1787907393.183041 write 0x02e822 thru 0x030798 (0x1f77 bytes) 000235 1787907393.184478 mapread 0x002cbc thru 0x004336 (0x167b bytes) 000236 1787907393.205148 MIRROR_RESYNC 2 mirrors 000237 1787907393.901107 mapwrite 0x0050b2 thru 0x008b82 (0x3ad1 bytes) 000238 1787907394.119156 punch from 0x02ebf6 to 0x030798 (0x1ba3 bytes) 000239 1787907394.127118 read 0x012602 thru 0x01c671 (0xa070 bytes) 000240 1787907394.150020 punch from 0x01d337 to 0x0280f8 (0xadc2 bytes) 000241 1787907394.157481 punch from 0x00c7a6 to 0x00cc40 (0x49b bytes) 000242 1787907394.179201 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000243 1787907394.336043 mapwrite 0x00ca39 thru 0x01ae37 (0xe3ff bytes) 000244 1787907394.443920 punch from 0x00c1e7 to 0x00e949 (0x2763 bytes) 000245 1787907394.448831 trunc from 0x030799 to 0x02b4ff (0x5299 bytes) 000246 1787907394.495536 mapwrite 0x006bb6 thru 0x009625 (0x2a70 bytes) 000250 1787907394.528806 trunc from 0x02b500 to 0x030c67 (0x5768 bytes) 000251 1787907394.557594 read 0x01d38b thru 0x0252d1 (0x7f47 bytes) 000252 1787907394.567386 read 0x0084d3 thru 0x01480a (0xc338 bytes) 000253 1787907394.568877 trunc from 0x0233ce to 0x030c67 (0xd89a bytes) 000255 1787907394.575849 trunc from 0x030c68 to 0x012699 (0x1e5ce bytes) 000256 1787907394.598393 punch from 0x011aec to 0x012699 (0xbae bytes) 000257 1787907394.603049 read 0x004805 thru 0x005c00 (0x13fc bytes) 000258 1787907394.604227 read 0x0008bb thru 0x000f77 (0x6bd bytes) 000259 1787907394.613330 read 0x000b34 thru 0x00995f (0x8e2c bytes) 000261 1787907394.615276 mapwrite 0x01754a thru 0x018b26 (0x15dd bytes) 000262 1787907394.669830 trunc from 0x018b27 to 0x02beb5 (0x1338f bytes) 000263 1787907394.701229 mapread 0x01438e thru 0x019fed (0x5c60 bytes) 000264 1787907394.713513 punch from 0x0231df to 0x02beb5 (0x8cd7 bytes) 000265 1787907394.718723 mapwrite 0x00c027 thru 0x0102b5 (0x428f bytes) 000266 1787907394.742406 punch from 0x008e78 to 0x00eb3c (0x5cc5 bytes) 000267 1787907394.748227 fallocate 0x01b9f7 thru 0x026e9f (0xb4a9 bytes) 000268 1787907394.754922 read 0x029700 thru 0x02beb5 (0x27b6 bytes) 000269 1787907394.773491 punch from 0x02201a to 0x022845 (0x82c bytes) 000270 1787907394.784133 fallocate 0x0399d0 thru 0x03ffff (0x6630 bytes) 000271 1787907394.796620 trunc from 0x0240ca to 0x02beb5 (0x7dec bytes) 000272 1787907394.809545 trunc from 0x01bef1 to 0x02618d (0xa29d bytes) 000273 1787907394.829523 mapread 0x0234fd thru 0x02beb5 (0x89b9 bytes) 000274 1787907394.928872 read 0x00cfa1 thru 0x016105 (0x9165 bytes) 000275 1787907394.931240 mapwrite 0x004e1b thru 0x00a3c2 (0x55a8 bytes) 000276 1787907394.973321 write 0x01fd72 thru 0x021391 (0x1620 bytes) 000277 1787907395.007838 MIRROR_EXTEND to 2 mirrors 000278 1787907395.991920 MIRROR_RESYNC 2 mirrors 000279 1787907396.379481 MIRROR_EXTEND to 3 mirrors 000280 1787907397.297761 mapread 0x009f82 thru 0x018cc6 (0xed45 bytes) 000281 1787907397.349010 fallocate 0x0168fe thru 0x01b51c (0x4c1f bytes) 000283 1787907397.417161 trunc from 0x02beb6 to 0x021fe7 (0x9ece bytes) 000284 1787907397.455266 trunc from 0x021fe8 to 0x02ff5d (0xdf76 bytes) 000285 1787907397.487099 mapwrite 0x02ed3b thru 0x030aa9 (0x1d6f bytes) 000286 1787907397.547226 MIRROR_RESYNC 3 mirrors 000287 1787907398.739525 MIRROR_RESYNC 3 mirrors 000289 1787907399.169441 mapwrite 0x02d19c thru 0x02d795 (0x5fa bytes) 000290 1787907399.308868 trunc from 0x01c4ea to 0x0237f1 (0x7308 bytes) 000292 1787907399.316906 punch from 0x00e0e0 to 0x014731 (0x6652 bytes) 000293 1787907399.337067 trunc from 0x009666 to 0x013efb (0xa896 bytes) 000295 1787907399.352786 punch from 0x0210e4 to 0x023488 (0x23a5 bytes) 000296 1787907399.368104 trunc from 0x030aaa to 0x02902a (0x7a7f bytes) 000297 1787907399.415300 punch from 0x0069e5 to 0x01491d (0xdf39 bytes) 000298 1787907399.425580 write 0x02bbb7 thru 0x02d2ae (0x16f8 bytes) 000299 1787907399.434969 write 0x033504 thru 0x038eb1 (0x59ae bytes) 000300 1787907399.436414 write 0x037641 thru 0x03a1a7 (0x2b67 bytes) 000301 1787907399.438849 write 0x01389c thru 0x01ba36 (0x819b bytes) 000302 1787907399.469005 punch from 0x00e88f to 0x0162da (0x7a4c bytes) 000303 1787907399.495768 mapwrite 0x015988 thru 0x01c3fd (0x6a76 bytes) 000304 1787907399.530778 trunc from 0x03a1a8 to 0x028be6 (0x115c1 bytes) 000305 1787907399.549147 punch from 0x01a74e to 0x01aa12 (0x2c5 bytes) 000306 1787907399.571313 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28be7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28be7): Mirror 2: 0x57f01514 (stale) Mirror 3: 0x57f01514 (stale) CRC-32 checksum value for chunk [0, 0x28be7): Mirror 1: 0x2ebcffe3 Mirror 4: 0x2ebcffe3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28be7: skipped 000307 1787907400.622871 trunc from 0x028be7 to 0x009fd9 (0x1ec0d bytes) 000308 1787907400.696539 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9fda): Mirror 3: 0x2e56aa97 (stale) Mirror 4: 0xef7beafc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9fda): Mirror 3: 0x2e56aa97 (stale) Mirror 4: 0xef7beafc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9fda: skipped 000309 1787907401.100866 fallocate 0x03f035 thru 0x03ffff (0xfcb bytes) 000310 1787907401.107548 mapread 0x00880a thru 0x009fd9 (0x17d0 bytes) 000311 1787907401.123840 read 0x008ca5 thru 0x009fd9 (0x1335 bytes) 000312 1787907401.129552 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9fda): Mirror 4: 0xef7beafc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9fda): Mirror 4: 0xef7beafc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9fda: skipped 000313 1787907401.404368 fallocate 0x03fa75 thru 0x03ffff (0x58b bytes) 000315 1787907401.409098 write 0x00fafc thru 0x01de33 (0xe338 bytes) 000316 1787907401.427380 fallocate 0x03009a thru 0x036141 (0x60a8 bytes) 000317 1787907401.434416 trunc from 0x02d012 to 0x03971e (0xc70d bytes) 000318 1787907401.444188 read 0x00dc57 thru 0x0199e9 (0xbd93 bytes) 000319 1787907401.454222 write 0x0064b7 thru 0x00f05b (0x8ba5 bytes) 000320 1787907401.466913 mapwrite 0x033bc7 thru 0x038fa5 (0x53df bytes) 000321 1787907401.554959 MIRROR_RESYNC 2 mirrors 000322 1787907402.293042 write 0x01446b thru 0x01ff37 (0xbacd bytes) 000323 1787907402.319959 read 0x019a0f thru 0x019ec8 (0x4ba bytes) 000324 1787907402.320688 read 0x000acf thru 0x004c8d (0x41bf bytes) 000325 1787907402.336700 mapwrite 0x03a64c thru 0x03ffff (0x59b4 bytes) 000326 1787907402.357432 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000327 1787907402.501032 write 0x03fca0 thru 0x03ffff (0x360 bytes) 000329 1787907402.517642 MIRROR_EXTEND to 2 mirrors 000330 1787907403.382326 trunc from 0x015806 to 0x01bc76 (0x6471 bytes) 000331 1787907403.458466 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000332 1787907403.633172 mapwrite 0x007447 thru 0x00cced (0x58a7 bytes) 000333 1787907403.676854 read 0x00143a thru 0x00ccf4 (0xb8bb bytes) 000334 1787907403.687437 punch from 0x025c93 to 0x02dd77 (0x80e5 bytes) 000335 1787907403.697140 punch from 0x01fdfb to 0x024c93 (0x4e99 bytes) 000336 1787907403.706513 trunc from 0x040000 to 0x01af8b (0x25074 bytes) 000338 1787907403.731381 trunc from 0x00365d to 0x00533d (0x1ce1 bytes) 000340 1787907403.738847 punch from 0x001825 to 0x00c740 (0xaf1c bytes) 000341 1787907403.750318 mapread 0x00a9ba thru 0x0198af (0xeef6 bytes) 000342 1787907403.788616 read 0x01a21e thru 0x01af8b (0xd6e bytes) 000343 1787907403.789392 mapread 0x015edb thru 0x01af8b (0x50b1 bytes) 000344 1787907403.795075 MIRROR_EXTEND to 2 mirrors 000345 1787907404.455194 MIRROR_RESYNC 2 mirrors 000346 1787907404.890853 trunc from 0x016a9d to 0x01af8b (0x44ef bytes) 000348 1787907404.922406 fallocate 0x01f17a thru 0x01fab2 (0x939 bytes) 000349 1787907404.931980 mapwrite 0x0268a1 thru 0x03210d (0xb86d bytes) 000350 1787907405.056040 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3210e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3210e): Mirror 2: 0x30f726ca (stale) CRC-32 checksum value for chunk [0, 0x3210e): Mirror 1: 0x8b6dfbdb Mirror 3: 0x8b6dfbdb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3210e: skipped 000351 1787907406.245294 mapwrite 0x01a66a thru 0x0259a6 (0xb33d bytes) 000352 1787907406.502912 MIRROR_RESYNC 3 mirrors 000353 1787907407.624787 MIRROR_RESYNC 3 mirrors 000355 1787907408.014489 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000356 1787907408.396462 mapread 0x02c91c thru 0x03210d (0x57f2 bytes) 000357 1787907408.418060 punch from 0x00b342 to 0x019376 (0xe035 bytes) 000358 1787907408.457568 read 0x00e87f thru 0x01c0cd (0xd84f bytes) 000359 1787907408.468643 trunc from 0x030e01 to 0x03210d (0x130d bytes) 000360 1787907408.477745 write 0x00bca9 thru 0x01b86e (0xfbc6 bytes) 000361 1787907408.492580 read 0x02d261 thru 0x02e14d (0xeed bytes) 000362 1787907408.503546 read 0x011463 thru 0x01b48a (0xa028 bytes) 000363 1787907408.505792 write 0x02e962 thru 0x03640a (0x7aa9 bytes) 000364 1787907408.510705 read 0x0331c8 thru 0x03640a (0x3243 bytes) 000365 1787907408.511890 write 0x00871f thru 0x00ff1c (0x77fe bytes) 000366 1787907408.516312 write 0x031ac8 thru 0x038887 (0x6dc0 bytes) 000367 1787907408.518629 trunc from 0x0184ed to 0x0267d6 (0xe2ea bytes) 000368 1787907408.554491 trunc from 0x02522a to 0x0349fd (0xf7d4 bytes) 000369 1787907408.568816 fallocate 0x02bfe5 thru 0x037ac9 (0xbae5 bytes) 000370 1787907408.582556 read 0x00709d thru 0x008aad (0x1a11 bytes) 000371 1787907408.587805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38888): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38888): Mirror 3: 0x7a699bcc (stale) CRC-32 checksum value for chunk [0, 0x38888): Mirror 1: 0xc16fbab7 Mirror 4: 0xc16fbab7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38888: skipped 000372 1787907409.776526 mapwrite 0x000bcf thru 0x0065ec (0x5a1e bytes) 000373 1787907410.002030 mapwrite 0x02e31a thru 0x0395b1 (0xb298 bytes) 000375 1787907410.145322 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x395b2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395b2): Mirror 3: 0x611bd00b (stale) Mirror 4: 0x345a2e18 (stale) CRC-32 checksum value for chunk [0, 0x395b2): Mirror 1: 0xb826e3d4 Mirror 5: 0xb826e3d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395b2: skipped 000376 1787907411.396889 fallocate 0x036052 thru 0x03be14 (0x5dc3 bytes) 000377 1787907411.430587 read 0x02e5de thru 0x030519 (0x1f3c bytes) 000378 1787907411.445175 MIRROR_RESYNC 4 mirrors 000379 1787907412.662109 mapwrite 0x0048ef thru 0x00cff3 (0x8705 bytes) 000380 1787907412.760163 trunc from 0x010b4d to 0x014c76 (0x412a bytes) 000381 1787907412.768080 trunc from 0x02ed2e to 0x030d55 (0x2028 bytes) 000383 1787907412.774632 punch from 0x0041db to 0x008d65 (0x4b8b bytes) 000384 1787907412.779566 mapwrite 0x02dc86 thru 0x03387c (0x5bf7 bytes) 000385 1787907412.804086 trunc from 0x027dc8 to 0x029df0 (0x2029 bytes) 000387 1787907412.808701 trunc from 0x0395b2 to 0x0168f4 (0x22cbd bytes) 000388 1787907412.833605 trunc from 0x0168f5 to 0x03e892 (0x27f9e bytes) 000390 1787907412.854468 punch from 0x031023 to 0x03ae01 (0x9ddf bytes) 000391 1787907412.863945 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e893): Mirror 4: 0x6eef2168 (stale) Mirror 5: 0x6eef2168 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e893): Mirror 4: 0x6eef2168 (stale) Mirror 5: 0x6eef2168 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e893: skipped 000392 1787907413.174474 mapread 0x039341 thru 0x03dddd (0x4a9d bytes) 000393 1787907413.188438 punch from 0x0105ce to 0x014365 (0x3d98 bytes) 000394 1787907413.196089 mapwrite 0x03805b thru 0x03ffff (0x7fa5 bytes) 000395 1787907413.237644 mapread 0x037713 thru 0x03ffff (0x88ed bytes) 000396 1787907413.238976 read 0x02f851 thru 0x039c91 (0xa441 bytes) 000397 1787907413.244749 fallocate 0x00b30f thru 0x01577f (0xa471 bytes) 000398 1787907413.251762 trunc from 0x03dbe2 to 0x03ffff (0x241e bytes) 000399 1787907413.258623 fallocate 0x0305d4 thru 0x0389e5 (0x8412 bytes) 000400 1787907413.265017 read 0x015cc8 thru 0x019ded (0x4126 bytes) 000401 1787907413.284826 mapwrite 0x02ee5d thru 0x033c94 (0x4e38 bytes) 000402 1787907413.308368 mapread 0x03be48 thru 0x03da3f (0x1bf8 bytes) 000403 1787907413.314113 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6df2388c (stale) Mirror 5: 0x6df2388c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfc9edf99 Mirror 6: 0xfc9edf99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000404 1787907414.752472 trunc from 0x005b5d to 0x00eb90 (0x9034 bytes) 000405 1787907414.802269 read 0x038a7c thru 0x03ffff (0x7584 bytes) 000406 1787907414.822259 mapwrite 0x016bc5 thru 0x024bbb (0xdff7 bytes) 000407 1787907414.890246 mapwrite 0x01927e thru 0x023415 (0xa198 bytes) 000408 1787907414.915172 trunc from 0x040000 to 0x01c425 (0x23bda bytes) 000409 1787907414.946815 trunc from 0x01c426 to 0x00caca (0xf95b bytes) 000410 1787907414.975813 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xcacb): Mirror 4: 0xf139459c (stale) Mirror 6: 0x7b8428f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcacb): Mirror 4: 0xf139459c (stale) Mirror 6: 0x7b8428f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcacb: skipped 000412 1787907415.441320 punch from 0x003e5a to 0x009663 (0x580a bytes) 000413 1787907415.446523 mapwrite 0x026212 thru 0x034b0c (0xe8fb bytes) 000414 1787907415.506940 punch from 0x018809 to 0x0191c7 (0x9bf bytes) 000415 1787907415.512428 punch from 0x01f86a to 0x023619 (0x3db0 bytes) 000416 1787907415.517183 punch from 0x02e64c to 0x034b0c (0x64c1 bytes) 000417 1787907415.528518 read 0x008c5e thru 0x00a7b3 (0x1b56 bytes) 000418 1787907415.541807 MIRROR_RESYNC 3 mirrors 000420 1787907416.542327 mapread 0x00c526 thru 0x016ef5 (0xa9d0 bytes) 000421 1787907416.573021 fallocate 0x00485f thru 0x00b3d3 (0x6b75 bytes) 000422 1787907416.612338 read 0x027827 thru 0x034b0c (0xd2e6 bytes) 000423 1787907416.640636 mapread 0x027c09 thru 0x0285c8 (0x9c0 bytes) 000424 1787907416.644488 mapread 0x00558c thru 0x011587 (0xbffc bytes) 000425 1787907416.680452 trunc from 0x034b0d to 0x033f65 (0xba7 bytes) 000426 1787907416.731437 trunc from 0x033f66 to 0x02669f (0xd8c6 bytes) 000427 1787907416.753947 trunc from 0x0266a0 to 0x03cf2d (0x1688e bytes) 000428 1787907416.775393 trunc from 0x00c722 to 0x018b3c (0xc41b bytes) 000429 1787907416.790159 mapread 0x018683 thru 0x01fce0 (0x765e bytes) 000430 1787907416.804710 write 0x02e8bf thru 0x03c79e (0xdee0 bytes) 000431 1787907416.823033 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cf2e): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf2e): Mirror 4: 0xdf0837a2 (stale) Mirror 6: 0xdf0837a2 (stale) CRC-32 checksum value for chunk [0, 0x3cf2e): Mirror 1: 0x942e3ab4 Mirror 7: 0x942e3ab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf2e: skipped 000432 1787907418.081275 trunc from 0x03cf2e to 0x008d8c (0x341a1 bytes) 000433 1787907418.228231 fallocate 0x02d02b thru 0x03cc5c (0xfc32 bytes) 000434 1787907418.238557 trunc from 0x008d8d to 0x02af6a (0x221de bytes) 000435 1787907418.291062 trunc from 0x028224 to 0x02af6a (0x2d47 bytes) 000436 1787907418.331000 fallocate 0x017989 thru 0x019dd6 (0x244e bytes) 000437 1787907418.347454 fallocate 0x03f769 thru 0x03ffff (0x897 bytes) 000438 1787907418.368033 punch from 0x00b9ae to 0x00fc15 (0x4268 bytes) 000439 1787907418.379941 read 0x01e19c thru 0x02af6a (0xcdcf bytes) 000440 1787907418.414765 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2af6b): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af6b): Mirror 4: 0x398e4956 (stale) Mirror 6: 0x398e4956 (stale) Mirror 7: 0x34b26a89 (stale) CRC-32 checksum value for chunk [0, 0x2af6b): Mirror 1: 0x67592f8b Mirror 8: 0x67592f8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af6b: skipped 000441 1787907419.468213 read 0x011984 thru 0x01baa9 (0xa126 bytes) 000442 1787907419.489378 fallocate 0x03fec2 thru 0x03ffff (0x13e bytes) 000443 1787907419.547954 punch from 0x01b5e8 to 0x01e819 (0x3232 bytes) 000444 1787907419.562533 mapwrite 0x015144 thru 0x01c58e (0x744b bytes) 000445 1787907419.616034 read 0x01a374 thru 0x022a50 (0x86dd bytes) 000447 1787907419.631815 read 0x027fc0 thru 0x02af6a (0x2fab bytes) 000448 1787907419.632639 read 0x005b88 thru 0x012e13 (0xd28c bytes) 000449 1787907419.633944 trunc from 0x02af6b to 0x0375c2 (0xc658 bytes) 000450 1787907419.674082 mapwrite 0x00981c thru 0x009d01 (0x4e6 bytes) 000451 1787907419.684936 trunc from 0x00de73 to 0x014ab9 (0x6c47 bytes) 000452 1787907419.692388 mapread 0x020c81 thru 0x02ffab (0xf32b bytes) 000453 1787907419.703565 fallocate 0x033fce thru 0x03b81d (0x7850 bytes) 000454 1787907419.709260 mapwrite 0x0002cd thru 0x005842 (0x5576 bytes) 000455 1787907419.726725 trunc from 0x033db0 to 0x034133 (0x384 bytes) 000457 1787907419.731861 trunc from 0x0330a4 to 0x03b81d (0x877a bytes) 000458 1787907419.737348 mapwrite 0x0101c1 thru 0x01f007 (0xee47 bytes) 000459 1787907419.754891 punch from 0x02d5e9 to 0x02f829 (0x2241 bytes) 000460 1787907419.760172 mapread 0x012c0d thru 0x01bc71 (0x9065 bytes) 000461 1787907419.765333 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b81e): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b81e): Mirror 4: 0x70bba4d (stale) Mirror 6: 0x70bba4d (stale) Mirror 7: 0xfb87c6b6 (stale) Mirror 8: 0x62dac228 (stale) CRC-32 checksum value for chunk [0, 0x3b81e): Mirror 1: 0x456c4d75 Mirror 9: 0x456c4d75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b81e: skipped 000463 1787907420.999600 trunc from 0x033535 to 0x03b81d (0x82e9 bytes) 000464 1787907421.044335 mapread 0x030b5e thru 0x035d1f (0x51c2 bytes) 000465 1787907421.075642 read 0x02a39a thru 0x032adf (0x8746 bytes) 000466 1787907421.103478 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b81e): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b81e): Mirror 4: 0x70bba4d (stale) Mirror 6: 0x70bba4d (stale) Mirror 7: 0xfb87c6b6 (stale) Mirror 8: 0x62dac228 (stale) Mirror 9: 0x456c4d75 (stale) CRC-32 checksum value for chunk [0, 0x3b81e): Mirror 1: 0x456c4d75 Mirror 10: 0x456c4d75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b81e: skipped 000467 1787907422.740557 punch from 0x0277ee to 0x02d2e0 (0x5af3 bytes) 000468 1787907422.789311 MIRROR_RESYNC 7 mirrors 000469 1787907424.920436 read 0x02d39d thru 0x03b81d (0xe481 bytes) 000470 1787907424.961422 write 0x0355d2 thru 0x03ffff (0xaa2e bytes) 000471 1787907425.017855 trunc from 0x02dd78 to 0x02e8ef (0xb78 bytes) 000472 1787907425.027591 trunc from 0x040000 to 0x027b42 (0x184bd bytes) 000473 1787907425.087502 mapwrite 0x015d39 thru 0x0176b2 (0x197a bytes) 000474 1787907425.161396 write 0x01e3c0 thru 0x01fa20 (0x1661 bytes) 000475 1787907425.163124 mapwrite 0x038545 thru 0x03ee0b (0x68c7 bytes) 000476 1787907425.256029 trunc from 0x01e243 to 0x01e799 (0x557 bytes) 000477 1787907425.289100 trunc from 0x03ee0c to 0x007172 (0x37c99 bytes) 000478 1787907425.341372 trunc from 0x006ae2 to 0x007172 (0x691 bytes) 000479 1787907425.368982 MIRROR_RESYNC 7 mirrors 000480 1787907426.670970 write 0x023731 thru 0x03040a (0xccda bytes) 000481 1787907426.757971 MIRROR_RESYNC 7 mirrors 000482 1787907428.348428 write 0x00bd85 thru 0x016d8b (0xb007 bytes) 000483 1787907428.392247 mapwrite 0x01a057 thru 0x01b821 (0x17cb bytes) 000484 1787907428.447274 read 0x02561a thru 0x02a23b (0x4c22 bytes) 000485 1787907428.468463 mapwrite 0x03f5d9 thru 0x03ffff (0xa27 bytes) 000486 1787907428.584143 fallocate 0x01d791 thru 0x025dca (0x863a bytes) 000487 1787907428.597967 write 0x01d00d thru 0x01dd88 (0xd7c bytes) 000488 1787907428.615452 mapwrite 0x0093df thru 0x0121fc (0x8e1e bytes) 000489 1787907428.708243 trunc from 0x040000 to 0x0328fa (0xd705 bytes) 000490 1787907428.748565 read 0x00e099 thru 0x00f9e6 (0x194e bytes) 000491 1787907428.749611 trunc from 0x023c2d to 0x0261a5 (0x2579 bytes) 000492 1787907428.758762 trunc from 0x0328fb to 0x03acf2 (0x83f8 bytes) 000493 1787907428.798285 trunc from 0x0009a4 to 0x008440 (0x7a9d bytes) 000494 1787907428.823917 mapread 0x01d93c thru 0x01f0f7 (0x17bc bytes) 000495 1787907428.824932 mapread 0x027816 thru 0x02c834 (0x501f bytes) 000496 1787907428.848558 read 0x00ca38 thru 0x018a85 (0xc04e bytes) 000497 1787907428.849239 fallocate 0x00235b thru 0x00c10b (0x9db1 bytes) 000498 1787907428.861212 punch from 0x038e9c to 0x03acf2 (0x1e57 bytes) 000500 1787907428.869373 trunc from 0x030dfc to 0x03acf2 (0x9ef7 bytes) 000501 1787907428.888592 read 0x02da1c thru 0x0383f4 (0xa9d9 bytes) 000502 1787907428.910752 trunc from 0x00595d to 0x00b824 (0x5ec8 bytes) 000503 1787907428.924581 trunc from 0x037ab2 to 0x03acf2 (0x3241 bytes) 000504 1787907428.931463 read 0x033713 thru 0x038df6 (0x56e4 bytes) 000506 1787907428.943261 punch from 0x00cee3 to 0x0166f5 (0x9813 bytes) 000507 1787907428.949489 trunc from 0x03acf3 to 0x028725 (0x125cd bytes) 000508 1787907428.966450 fallocate 0x0386c5 thru 0x03f7bb (0x70f7 bytes) 000509 1787907428.973423 mapwrite 0x031de3 thru 0x0393bf (0x75dd bytes) 000510 1787907429.109891 trunc from 0x0393c0 to 0x01ee28 (0x1a597 bytes) 000511 1787907429.146028 mapwrite 0x026326 thru 0x0295c3 (0x329e bytes) 000512 1787907429.229629 punch from 0x005196 to 0x00a8b9 (0x5724 bytes) 000513 1787907429.239609 punch from 0x005253 to 0x012167 (0xcf15 bytes) 000514 1787907429.249577 mapwrite 0x0244ac thru 0x032e63 (0xe9b8 bytes) 000515 1787907429.439384 write 0x0114f9 thru 0x01aed7 (0x99df bytes) 000516 1787907429.454123 trunc from 0x032e64 to 0x032cbe (0x1a5 bytes) 000517 1787907429.471986 fallocate 0x01fe55 thru 0x027c59 (0x7e05 bytes) 000518 1787907429.495042 MIRROR_RESYNC 7 mirrors 000519 1787907431.922847 mapread 0x01effb thru 0x024081 (0x5087 bytes) 000520 1787907431.943550 mapwrite 0x0000ac thru 0x008790 (0x86e5 bytes) 000521 1787907432.044778 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32cbf): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32cbf): Mirror 4: 0xf45fae00 (stale) Mirror 6: 0xf45fae00 (stale) Mirror 7: 0xf45fae00 (stale) Mirror 8: 0xf45fae00 (stale) Mirror 9: 0xf45fae00 (stale) Mirror 10: 0xf45fae00 (stale) CRC-32 checksum value for chunk [0, 0x32cbf): Mirror 1: 0x6f416b5a Mirror 11: 0x6f416b5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32cbf: skipped 000522 1787907433.593981 read 0x028bb2 thru 0x032cbe (0xa10d bytes) 000523 1787907433.628478 mapwrite 0x030960 thru 0x036d4f (0x63f0 bytes) 000524 1787907433.832210 MIRROR_RESYNC 8 mirrors 000525 1787907436.282168 write 0x01d7ef thru 0x0285ac (0xadbe bytes) 000526 1787907436.347000 read 0x01f0da thru 0x027cbb (0x8be2 bytes) 000527 1787907436.347791 write 0x01e0bb thru 0x024f3d (0x6e83 bytes) 000528 1787907436.354799 punch from 0x02eb53 to 0x0313c4 (0x2872 bytes) 000529 1787907436.361622 write 0x0069c6 thru 0x0077db (0xe16 bytes) 000530 1787907436.384851 write 0x00be9a thru 0x016cdf (0xae46 bytes) 000531 1787907436.399524 punch from 0x024e0c to 0x02fac2 (0xacb7 bytes) 000532 1787907436.424288 MIRROR_RESYNC 8 mirrors 000533 1787907438.681497 write 0x01041d thru 0x012733 (0x2317 bytes) 000534 1787907438.741116 trunc from 0x036d50 to 0x03486d (0x24e2 bytes) 000535 1787907438.801185 fallocate 0x009818 thru 0x010dff (0x75e8 bytes) 000536 1787907438.819473 MIRROR_RESYNC 8 mirrors 000537 1787907441.597605 mapread 0x02a15e thru 0x02ed1d (0x4bc0 bytes) 000538 1787907441.684449 mapread 0x014d12 thru 0x01c3e3 (0x76d2 bytes) 000539 1787907441.706848 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000541 1787907442.306737 write 0x037a7f thru 0x03c52c (0x4aae bytes) 000543 1787907442.373430 mapread 0x026a19 thru 0x0356e2 (0xecca bytes) 000545 1787907442.433587 fallocate 0x034b5f thru 0x035877 (0xd19 bytes) 000547 1787907442.441099 mapwrite 0x02c6b2 thru 0x02f9c7 (0x3316 bytes) 000548 1787907442.470596 punch from 0x02e79a to 0x033089 (0x48f0 bytes) 000549 1787907442.478484 punch from 0x0063b1 to 0x014945 (0xe595 bytes) 000550 1787907442.496556 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c52d): Mirror 6: 0x6490944f (stale) Mirror 7: 0x6490944f (stale) Mirror 8: 0x6490944f (stale) Mirror 9: 0x6490944f (stale) Mirror 11: 0x6490944f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c52d): Mirror 6: 0x6490944f (stale) Mirror 7: 0x6490944f (stale) Mirror 8: 0x6490944f (stale) Mirror 9: 0x6490944f (stale) Mirror 11: 0x6490944f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c52d: skipped 000551 1787907443.369609 trunc from 0x03c52d to 0x0151f2 (0x2733a bytes) 000552 1787907443.449538 trunc from 0x001310 to 0x008e16 (0x7b07 bytes) 000553 1787907443.457098 read 0x00204f thru 0x004d1f (0x2cd1 bytes) 000554 1787907443.464130 write 0x005f9b thru 0x009600 (0x3666 bytes) 000555 1787907443.506333 fallocate 0x01f042 thru 0x02d586 (0xe545 bytes) 000556 1787907443.512566 trunc from 0x02d587 to 0x017f3d (0x15649 bytes) 000557 1787907443.550339 read 0x00d8c8 thru 0x017f3d (0xa676 bytes) 000558 1787907443.561747 punch from 0x01226c to 0x012a76 (0x80b bytes) 000559 1787907443.572326 trunc from 0x017f3e to 0x0194e4 (0x15a7 bytes) 000560 1787907443.632893 mapwrite 0x022e91 thru 0x029599 (0x6709 bytes) 000562 1787907443.736257 write 0x035663 thru 0x0358ca (0x268 bytes) 000563 1787907443.740383 mapread 0x01be39 thru 0x01bff8 (0x1c0 bytes) 000564 1787907443.747277 mapread 0x00a256 thru 0x01962e (0xf3d9 bytes) 000565 1787907443.765406 trunc from 0x009edc to 0x01830b (0xe430 bytes) 000566 1787907443.782361 fallocate 0x00bb7d thru 0x014d47 (0x91cb bytes) 000567 1787907443.815206 trunc from 0x03051e to 0x0358ca (0x53ad bytes) 000568 1787907443.836876 mapwrite 0x02f663 thru 0x03874b (0x90e9 bytes) 000569 1787907443.980331 punch from 0x0219e6 to 0x02db6a (0xc185 bytes) 000570 1787907443.992185 punch from 0x00b03e to 0x0164e3 (0xb4a6 bytes) 000571 1787907444.008246 mapread 0x01a91d thru 0x0236f6 (0x8dda bytes) 000572 1787907444.042494 mapread 0x006542 thru 0x010c63 (0xa722 bytes) 000573 1787907444.149832 trunc from 0x02ec70 to 0x03874b (0x9adc bytes) 000574 1787907444.163548 trunc from 0x02a35e to 0x02fe13 (0x5ab6 bytes) 000575 1787907444.179151 trunc from 0x03874c to 0x00fc84 (0x28ac7 bytes) 000576 1787907444.209996 trunc from 0x00cc83 to 0x00fc84 (0x3002 bytes) 000577 1787907444.215739 punch from 0x0091c0 to 0x00fc84 (0x6ac5 bytes) 000578 1787907444.233964 punch from 0x00d385 to 0x00fc84 (0x2900 bytes) 000579 1787907444.250360 punch from 0x001667 to 0x00f955 (0xe2ef bytes) 000580 1787907444.261625 write 0x00c63c thru 0x01c5a1 (0xff66 bytes) 000581 1787907444.286319 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c5a2): Mirror 6: 0x1721e2e0 (stale) Mirror 7: 0x1721e2e0 (stale) Mirror 8: 0x1721e2e0 (stale) Mirror 9: 0x1721e2e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c5a2): Mirror 6: 0x1721e2e0 (stale) Mirror 7: 0x1721e2e0 (stale) Mirror 8: 0x1721e2e0 (stale) Mirror 9: 0x1721e2e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c5a2: skipped 000582 1787907444.848586 fallocate 0x01e674 thru 0x02af03 (0xc890 bytes) 000583 1787907444.859100 mapwrite 0x0232d7 thru 0x03175e (0xe488 bytes) 000584 1787907444.919856 write 0x02a66a thru 0x02da00 (0x3397 bytes) 000585 1787907444.921112 trunc from 0x03175f to 0x024d03 (0xca5b bytes) 000586 1787907444.941075 mapwrite 0x02f336 thru 0x039666 (0xa331 bytes) 000587 1787907445.003396 write 0x036a73 thru 0x03ffff (0x958d bytes) 000589 1787907445.009942 trunc from 0x040000 to 0x02780b (0x187f4 bytes) 000590 1787907445.034269 mapwrite 0x0075db thru 0x015d00 (0xe726 bytes) 000591 1787907445.087955 punch from 0x022d44 to 0x02780b (0x4ac8 bytes) 000592 1787907445.093584 mapwrite 0x011ac7 thru 0x0128bf (0xdf9 bytes) 000593 1787907445.111937 MIRROR_RESYNC 5 mirrors 000594 1787907446.770469 trunc from 0x020c10 to 0x02780b (0x6bfc bytes) 000595 1787907446.859502 read 0x0109e0 thru 0x015654 (0x4c75 bytes) 000596 1787907446.884223 read 0x01cef2 thru 0x02780b (0xa91a bytes) 000597 1787907446.916134 fallocate 0x018780 thru 0x01c8a3 (0x4124 bytes) 000598 1787907446.936625 trunc from 0x027576 to 0x02780b (0x296 bytes) 000599 1787907446.957531 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2780c): Mirror 6: 0xc96853c6 (stale) Mirror 8: 0xc96853c6 (stale) Mirror 9: 0xc96853c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2780c): Mirror 6: 0xc96853c6 (stale) Mirror 8: 0xc96853c6 (stale) Mirror 9: 0xc96853c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2780c: skipped 000600 1787907447.639997 MIRROR_RESYNC 4 mirrors 000601 1787907448.809713 mapread 0x01975b thru 0x01e148 (0x49ee bytes) 000602 1787907448.860447 MIRROR_EXTEND to 5 mirrors 000603 1787907449.793831 read 0x0163fe thru 0x01e350 (0x7f53 bytes) 000604 1787907449.813842 MIRROR_EXTEND to 6 mirrors 000605 1787907450.950613 fallocate 0x02bc43 thru 0x03711c (0xb4da bytes) 000606 1787907451.012980 read 0x0164d4 thru 0x0223b3 (0xbee0 bytes) 000607 1787907451.038108 trunc from 0x02780c to 0x0102c2 (0x17549 bytes) 000608 1787907451.073535 punch from 0x00db9b to 0x00e6db (0xb41 bytes) 000610 1787907451.079466 punch from 0x000b23 to 0x007c80 (0x715e bytes) 000611 1787907451.086122 mapread 0x005736 thru 0x0102c2 (0xab8d bytes) 000612 1787907451.116095 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x102c3): Mirror 8: 0x5dabd48d (stale) Mirror 9: 0x5dabd48d (stale) Mirror 10: 0x5dabd48d (stale) Mirror 11: 0x5dabd48d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x102c3): Mirror 8: 0x5dabd48d (stale) Mirror 9: 0x5dabd48d (stale) Mirror 10: 0x5dabd48d (stale) Mirror 11: 0x5dabd48d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x102c3: skipped 000613 1787907451.696157 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x102c3): Mirror 9: 0x5dabd48d (stale) Mirror 10: 0x5dabd48d (stale) Mirror 11: 0x5dabd48d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x102c3): Mirror 9: 0x5dabd48d (stale) Mirror 10: 0x5dabd48d (stale) Mirror 11: 0x5dabd48d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x102c3: skipped 000614 1787907452.060437 fallocate 0x02926a thru 0x038c3e (0xf9d5 bytes) 000615 1787907452.070920 mapwrite 0x015e44 thru 0x019d72 (0x3f2f bytes) 000616 1787907452.191704 punch from 0x007f42 to 0x016b10 (0xebcf bytes) 000617 1787907452.202446 punch from 0x0147fc to 0x019d72 (0x5577 bytes) 000618 1787907452.214515 mapwrite 0x005257 thru 0x0087f9 (0x35a3 bytes) 000619 1787907452.265836 write 0x036046 thru 0x03baa2 (0x5a5d bytes) 000620 1787907452.268389 read 0x01f758 thru 0x024b16 (0x53bf bytes) 000621 1787907452.282158 trunc from 0x03baa3 to 0x03cf41 (0x149f bytes) 000622 1787907452.300917 trunc from 0x03cf42 to 0x0101da (0x2cd67 bytes) 000623 1787907452.324199 read 0x00a107 thru 0x0101da (0x60d4 bytes) 000624 1787907452.325449 read 0x000757 thru 0x002640 (0x1eea bytes) 000626 1787907452.332429 read 0x00775a thru 0x009719 (0x1fc0 bytes) 000627 1787907452.334112 punch from 0x00c481 to 0x0101da (0x3d5a bytes) 000628 1787907452.342163 write 0x02a8da thru 0x03470e (0x9e35 bytes) 000630 1787907452.346310 write 0x01be4b thru 0x023972 (0x7b28 bytes) 000631 1787907452.387393 MIRROR_RESYNC 4 mirrors 000632 1787907453.628960 trunc from 0x0106bb to 0x013141 (0x2a87 bytes) 000633 1787907453.669367 fallocate 0x0131c2 thru 0x017d77 (0x4bb6 bytes) 000634 1787907453.674675 mapwrite 0x02fb11 thru 0x03dabe (0xdfae bytes) 000635 1787907453.755343 read 0x018b9f thru 0x021323 (0x8785 bytes) 000636 1787907453.781574 mapread 0x019a8d thru 0x025e14 (0xc388 bytes) 000637 1787907453.801870 mapread 0x02751f thru 0x03573f (0xe221 bytes) 000638 1787907453.803453 write 0x02c3e2 thru 0x03971d (0xd33c bytes) 000639 1787907453.811807 fallocate 0x03e99d thru 0x03ffff (0x1663 bytes) 000640 1787907453.827305 punch from 0x02ec3d to 0x034fdf (0x63a3 bytes) 000641 1787907453.850158 trunc from 0x040000 to 0x03a5d9 (0x5a26 bytes) 000642 1787907453.875321 fallocate 0x0310c6 thru 0x033daa (0x2ce5 bytes) 000643 1787907453.881326 mapread 0x006dd2 thru 0x014a57 (0xdc86 bytes) 000644 1787907453.905372 trunc from 0x03a5da to 0x00aaba (0x2fb1f bytes) 000645 1787907453.954330 fallocate 0x026dd9 thru 0x029fb8 (0x31e0 bytes) 000646 1787907453.970285 mapread 0x00d5ee thru 0x00df51 (0x964 bytes) 000647 1787907453.980324 punch from 0x00827a to 0x016adf (0xe866 bytes) 000648 1787907453.992525 fallocate 0x03c8e4 thru 0x03ffff (0x371c bytes) 000650 1787907454.002423 mapread 0x037f52 thru 0x03ffff (0x80ae bytes) 000651 1787907454.050313 fallocate 0x031071 thru 0x033c63 (0x2bf3 bytes) 000652 1787907454.057767 mapread 0x0134dc thru 0x01d8b4 (0xa3d9 bytes) 000653 1787907454.103766 fallocate 0x0344be thru 0x03457c (0xbf bytes) 000654 1787907454.122761 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x45bd25d1 (stale) Mirror 11: 0x45bd25d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x45bd25d1 (stale) Mirror 11: 0x45bd25d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000655 1787907454.440144 trunc from 0x040000 to 0x02a9f2 (0x1560d bytes) 000656 1787907454.475910 trunc from 0x02a9f3 to 0x017a49 (0x12fa9 bytes) 000657 1787907454.504255 trunc from 0x00d075 to 0x00e2bc (0x1248 bytes) 000658 1787907454.512388 trunc from 0x017a4a to 0x011ddd (0x5c6c bytes) 000659 1787907454.531691 trunc from 0x00042f to 0x00fe3d (0xfa0f bytes) 000660 1787907454.540552 fallocate 0x010d91 thru 0x01cd56 (0xbfc6 bytes) 000661 1787907454.548166 write 0x036b1e thru 0x03ffff (0x94e2 bytes) 000662 1787907454.549766 fallocate 0x0277a2 thru 0x0317b1 (0xa010 bytes) 000664 1787907454.558774 read 0x03c25a thru 0x03ffff (0x3da6 bytes) 000665 1787907454.563504 read 0x03094c thru 0x0372df (0x6994 bytes) 000666 1787907454.584480 trunc from 0x032433 to 0x03552f (0x30fd bytes) 000667 1787907454.590728 punch from 0x001f8e to 0x00a604 (0x8677 bytes) 000669 1787907454.599868 mapwrite 0x00679d thru 0x00e079 (0x78dd bytes) 000670 1787907454.676032 punch from 0x00783b to 0x00f908 (0x80ce bytes) 000672 1787907454.683304 mapwrite 0x00b1f9 thru 0x00f7c4 (0x45cc bytes) 000673 1787907454.723081 punch from 0x0303f2 to 0x035746 (0x5355 bytes) 000674 1787907454.731303 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x45bd25d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x45bd25d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000675 1787907455.069386 write 0x036d94 thru 0x03e58f (0x77fc bytes) 000676 1787907455.101294 mapread 0x028870 thru 0x02d6cd (0x4e5e bytes) 000677 1787907455.127109 read 0x00bf40 thru 0x019c5c (0xdd1d bytes) 000678 1787907455.135536 read 0x00790b thru 0x00a5df (0x2cd5 bytes) 000679 1787907455.158805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x45bd25d1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x92198ce2 Mirror 10: 0x92198ce2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000681 1787907456.158494 write 0x0230b6 thru 0x029845 (0x6790 bytes) 000682 1787907456.207998 read 0x01f7dc thru 0x029cd9 (0xa4fe bytes) 000683 1787907456.228777 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x92198ce2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x92198ce2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000684 1787907456.572248 read 0x006d82 thru 0x0132e0 (0xc55f bytes) 000685 1787907456.591853 punch from 0x030bab to 0x03550b (0x4961 bytes) 000686 1787907456.598582 trunc from 0x040000 to 0x019f58 (0x260a7 bytes) 000687 1787907456.647471 write 0x028f43 thru 0x032e17 (0x9ed5 bytes) 000688 1787907456.650556 write 0x0068bd thru 0x011ec2 (0xb606 bytes) 000689 1787907456.653488 trunc from 0x012c74 to 0x019ee5 (0x7272 bytes) 000690 1787907456.658601 read 0x02a4f8 thru 0x0318d4 (0x73dd bytes) 000691 1787907456.659315 punch from 0x00f6c9 to 0x016016 (0x694e bytes) 000692 1787907456.678833 fallocate 0x02fdcc thru 0x03f8db (0xfb10 bytes) 000693 1787907456.712122 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f8dc): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8dc): Mirror 10: 0x3d8a1b65 (stale) CRC-32 checksum value for chunk [0, 0x3f8dc): Mirror 1: 0x852130f4 Mirror 11: 0x852130f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8dc: skipped 000694 1787907458.146517 read 0x017fe0 thru 0x01dd90 (0x5db1 bytes) 000695 1787907458.162376 write 0x00a2d1 thru 0x0157b7 (0xb4e7 bytes) 000696 1787907458.230741 read 0x02cccd thru 0x0375ee (0xa922 bytes) 000697 1787907458.255137 read 0x03b7c9 thru 0x03f8db (0x4113 bytes) 000698 1787907458.255876 trunc from 0x01dc68 to 0x021fd6 (0x436f bytes) 000699 1787907458.265134 trunc from 0x03f8dc to 0x035bd0 (0x9d0b bytes) 000700 1787907458.296025 punch from 0x01b344 to 0x01fe7f (0x4b3c bytes) 000701 1787907458.303137 mapread 0x007b58 thru 0x012146 (0xa5ef bytes) 000702 1787907458.310834 write 0x02690f thru 0x02dfbf (0x76b1 bytes) 000703 1787907458.312916 trunc from 0x028880 to 0x02d1b0 (0x4931 bytes) 000704 1787907458.348951 write 0x00048a thru 0x00c690 (0xc207 bytes) 000705 1787907458.352613 read 0x018aa8 thru 0x024b66 (0xc0bf bytes) 000706 1787907458.367265 punch from 0x009771 to 0x013c79 (0xa509 bytes) 000707 1787907458.398378 fallocate 0x0068e4 thru 0x00cdb4 (0x64d1 bytes) 000708 1787907458.412553 trunc from 0x035bd1 to 0x03a5d5 (0x4a05 bytes) 000709 1787907458.448772 read 0x017297 thru 0x023e67 (0xcbd1 bytes) 000710 1787907458.449515 mapwrite 0x035c66 thru 0x03ffff (0xa39a bytes) 000711 1787907458.615881 punch from 0x003bdc to 0x00726c (0x3691 bytes) 000712 1787907458.625830 mapread 0x02f305 thru 0x03351b (0x4217 bytes) 000713 1787907458.627884 punch from 0x01ff10 to 0x02f774 (0xf865 bytes) 000714 1787907458.635221 mapwrite 0x023c9e thru 0x029cce (0x6031 bytes) 000715 1787907458.699536 fallocate 0x0372cc thru 0x03ffff (0x8d34 bytes) 000717 1787907458.705573 fallocate 0x027b9d thru 0x029f71 (0x23d5 bytes) 000718 1787907458.714601 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1764b47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1764b47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000719 1787907459.016507 trunc from 0x00ff53 to 0x013de9 (0x3e97 bytes) 000720 1787907459.029186 trunc from 0x00f557 to 0x01d742 (0xe1ec bytes) 000721 1787907459.045139 trunc from 0x040000 to 0x02db8b (0x12474 bytes) 000722 1787907459.106524 read 0x00b3e6 thru 0x00d504 (0x211f bytes) 000723 1787907459.122256 trunc from 0x02db8c to 0x015f94 (0x17bf7 bytes) 000724 1787907459.158093 fallocate 0x03639f thru 0x03afc6 (0x4c28 bytes) 000725 1787907459.164484 fallocate 0x003aeb thru 0x007b68 (0x407e bytes) 000726 1787907459.169443 trunc from 0x015f95 to 0x006df5 (0xf19f bytes) 000728 1787907459.191604 mapwrite 0x01832e thru 0x01c6d2 (0x43a5 bytes) 000729 1787907459.269933 fallocate 0x02b655 thru 0x031a9d (0x6449 bytes) 000730 1787907459.276795 punch from 0x016084 to 0x01c6d2 (0x664f bytes) 000731 1787907459.282948 mapread 0x00d6fe thru 0x01c6d2 (0xefd5 bytes) 000732 1787907459.326414 MIRROR_RESYNC 2 mirrors 000733 1787907460.081728 trunc from 0x01c6d3 to 0x002830 (0x19ea2 bytes) 000735 1787907460.187283 mapread 0x00107a thru 0x002830 (0x17b7 bytes) 000737 1787907460.200801 mapwrite 0x03bb14 thru 0x03e2f3 (0x27e0 bytes) 000738 1787907460.287710 mapwrite 0x005410 thru 0x00cfda (0x7bcb bytes) 000739 1787907460.352516 mapwrite 0x017a0f thru 0x01d38f (0x5981 bytes) 000740 1787907460.429788 punch from 0x01f16b to 0x02e391 (0xf227 bytes) 000741 1787907460.456067 mapwrite 0x03adb1 thru 0x03ffff (0x524f bytes) 000742 1787907460.537693 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x6c6dbfaf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2effb7b7 Mirror 12: 0x2effb7b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000743 1787907461.654039 punch from 0x01e227 to 0x023f20 (0x5cfa bytes) 000744 1787907461.735006 read 0x002485 thru 0x00390f (0x148b bytes) 000745 1787907461.745972 trunc from 0x0023fd to 0x011982 (0xf586 bytes) 000746 1787907461.752931 trunc from 0x03ec33 to 0x03fefd (0x12cb bytes) 000747 1787907461.773764 mapwrite 0x01db23 thru 0x023745 (0x5c23 bytes) 000748 1787907461.858422 mapwrite 0x028a19 thru 0x0356b8 (0xcca0 bytes) 000749 1787907461.943239 fallocate 0x02f9a9 thru 0x037164 (0x77bc bytes) 000751 1787907461.958380 mapwrite 0x007c24 thru 0x014ed1 (0xd2ae bytes) 000753 1787907462.028375 read 0x001481 thru 0x0069ee (0x556e bytes) 000754 1787907462.029945 mapwrite 0x03c9f3 thru 0x03ffff (0x360d bytes) 000756 1787907462.081838 mapread 0x03bacd thru 0x03ffff (0x4533 bytes) 000757 1787907462.082890 trunc from 0x023dba to 0x029c73 (0x5eba bytes) 000758 1787907462.098897 write 0x0140b2 thru 0x01de84 (0x9dd3 bytes) 000760 1787907462.119889 read 0x01baa1 thru 0x02b61e (0xfb7e bytes) 000761 1787907462.144393 fallocate 0x033e9e thru 0x03cb06 (0x8c69 bytes) 000762 1787907462.181031 MIRROR_RESYNC 3 mirrors 000763 1787907463.207539 trunc from 0x040000 to 0x036e43 (0x91bc bytes) 000764 1787907463.300599 mapread 0x008144 thru 0x013efa (0xbdb7 bytes) 000765 1787907463.341508 mapread 0x010fa0 thru 0x019585 (0x85e6 bytes) 000766 1787907463.344873 mapread 0x01c9e8 thru 0x0215ca (0x4be3 bytes) 000767 1787907463.363524 write 0x03fb05 thru 0x03ffff (0x4fb bytes) 000768 1787907463.369297 write 0x030ce0 thru 0x039613 (0x8934 bytes) 000769 1787907463.402274 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf8656c03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf8656c03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000770 1787907463.892059 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf8656c03 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeabd7689 Mirror 13: 0xeabd7689 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1787907465.195621 trunc from 0x02d544 to 0x031f65 (0x4a22 bytes) 000774 1787907465.262918 mapread 0x033dac thru 0x03e52f (0xa784 bytes) 000775 1787907465.307306 read 0x006f37 thru 0x01135e (0xa428 bytes) 000776 1787907465.316567 write 0x0239e9 thru 0x02a9c7 (0x6fdf bytes) 000777 1787907465.338426 fallocate 0x03d856 thru 0x03ffff (0x27aa bytes) 000778 1787907465.347952 mapread 0x03200f thru 0x039316 (0x7308 bytes) 000779 1787907465.348910 fallocate 0x0114e4 thru 0x0129aa (0x14c7 bytes) 000780 1787907465.365188 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xeabd7689 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xeabd7689 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1787907465.793099 trunc from 0x029d72 to 0x02c461 (0x26f0 bytes) 000782 1787907465.798040 trunc from 0x040000 to 0x000b86 (0x3f479 bytes) 000783 1787907465.839579 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) 000784 1787907466.127524 write 0x03c06a thru 0x03f777 (0x370e bytes) 000786 1787907466.141169 trunc from 0x038150 to 0x03f777 (0x7628 bytes) 000787 1787907466.163304 write 0x031d57 thru 0x03db67 (0xbe11 bytes) 000790 1787907466.181698 punch from 0x02a037 to 0x0303f4 (0x63be bytes) 000791 1787907466.196977 mapwrite 0x00c63e thru 0x00e40b (0x1dce bytes) 000792 1787907466.259722 mapwrite 0x015d5b thru 0x0187f7 (0x2a9d bytes) 000793 1787907466.299423 read 0x00a973 thru 0x019361 (0xe9ef bytes) 000794 1787907466.302068 trunc from 0x036064 to 0x03b88c (0x5829 bytes) 000795 1787907466.311491 trunc from 0x03f778 to 0x00d34b (0x3242c bytes) 000796 1787907466.337185 fallocate 0x00ce0b thru 0x016cea (0x9ee0 bytes) 000797 1787907466.355288 punch from 0x007d53 to 0x00d34b (0x55f9 bytes) 000798 1787907466.370310 mapwrite 0x03af17 thru 0x03ffff (0x50e9 bytes) 000800 1787907466.443041 trunc from 0x040000 to 0x029154 (0x16eab bytes) 000801 1787907466.465350 trunc from 0x00badf to 0x01b69f (0xfbc1 bytes) 000802 1787907466.472331 fallocate 0x0122e3 thru 0x01a35f (0x807d bytes) 000803 1787907466.482334 punch from 0x018ad9 to 0x020dd1 (0x82f9 bytes) 000805 1787907466.491031 read 0x01ac42 thru 0x01f1f9 (0x45b8 bytes) 000806 1787907466.535190 read 0x00f44c thru 0x0146e2 (0x5297 bytes) 000807 1787907466.537208 punch from 0x01e783 to 0x021016 (0x2894 bytes) 000808 1787907466.551625 punch from 0x01ce5e to 0x02214e (0x52f1 bytes) 000809 1787907466.563435 MIRROR_EXTEND to 2 mirrors 000810 1787907467.325888 MIRROR_RESYNC 2 mirrors 000811 1787907468.008875 mapwrite 0x0295f9 thru 0x031aa1 (0x84a9 bytes) 000812 1787907468.149224 fallocate 0x033b95 thru 0x03ed80 (0xb1ec bytes) 000813 1787907468.157439 fallocate 0x012f0e thru 0x012f92 (0x85 bytes) 000814 1787907468.170620 trunc from 0x031aa2 to 0x02e586 (0x351b bytes) 000815 1787907468.199829 write 0x01c129 thru 0x01cd1b (0xbf3 bytes) 000816 1787907468.227452 MIRROR_RESYNC 2 mirrors 000817 1787907469.256756 MIRROR_RESYNC 2 mirrors 000818 1787907469.659203 trunc from 0x02e587 to 0x028ac0 (0x5ac6 bytes) 000819 1787907469.753608 trunc from 0x006f55 to 0x013259 (0xc305 bytes) 000820 1787907469.762883 mapread 0x002f8c thru 0x0076e6 (0x475b bytes) 000822 1787907469.792370 trunc from 0x004fa7 to 0x00c12a (0x7184 bytes) 000823 1787907469.800999 fallocate 0x037296 thru 0x038513 (0x127e bytes) 000824 1787907469.807352 write 0x03e533 thru 0x03ffff (0x1acd bytes) 000825 1787907469.811733 trunc from 0x040000 to 0x03f477 (0xb88 bytes) 000826 1787907469.839637 read 0x02d998 thru 0x036c78 (0x92e1 bytes) 000829 1787907469.871699 mapread 0x02e050 thru 0x032a42 (0x49f3 bytes) 000830 1787907469.873285 write 0x03c703 thru 0x03df5e (0x185c bytes) 000831 1787907469.891257 punch from 0x02b272 to 0x03547b (0xa20a bytes) 000832 1787907469.907358 MIRROR_RESYNC 2 mirrors 000833 1787907470.795098 trunc from 0x03f478 to 0x009b65 (0x35912 bytes) 000834 1787907470.851395 mapread 0x001f0d thru 0x0035fe (0x16f2 bytes) 000835 1787907470.892927 write 0x010385 thru 0x01bb67 (0xb7e3 bytes) 000837 1787907470.896466 mapread 0x002b62 thru 0x003522 (0x9c1 bytes) 000838 1787907470.898132 trunc from 0x0078bd to 0x0102e8 (0x8a2c bytes) 000839 1787907470.911115 mapread 0x00073f thru 0x000cdb (0x59d bytes) 000840 1787907470.923175 punch from 0x010251 to 0x015d33 (0x5ae3 bytes) 000841 1787907470.928902 write 0x01e02a thru 0x0226fd (0x46d4 bytes) 000842 1787907470.931250 fallocate 0x02dce0 thru 0x03bc74 (0xdf95 bytes) 000844 1787907470.937067 trunc from 0x02006d to 0x02f731 (0xf6c5 bytes) 000845 1787907470.950460 trunc from 0x03bc75 to 0x011a14 (0x2a260 bytes) 000846 1787907470.974500 MIRROR_RESYNC 2 mirrors 000847 1787907471.622935 trunc from 0x011a15 to 0x028245 (0x16831 bytes) 000848 1787907471.670948 punch from 0x024a09 to 0x028245 (0x383d bytes) 000849 1787907471.683782 mapread 0x01d24c thru 0x028245 (0xaffa bytes) 000850 1787907471.726095 mapwrite 0x01d9a0 thru 0x025b6b (0x81cc bytes) 000851 1787907471.752718 mapread 0x024921 thru 0x028245 (0x3925 bytes) 000852 1787907471.753761 trunc from 0x028246 to 0x031038 (0x8df3 bytes) 000854 1787907471.792346 fallocate 0x0360db thru 0x03ffff (0x9f25 bytes) 000855 1787907471.805409 trunc from 0x024dc8 to 0x031038 (0xc271 bytes) 000856 1787907471.814372 mapwrite 0x022426 thru 0x028201 (0x5ddc bytes) 000857 1787907471.877344 trunc from 0x031039 to 0x0197a5 (0x17893 bytes) 000858 1787907471.921609 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x197a6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x197a6): Mirror 2: 0x4fd51a93 (stale) CRC-32 checksum value for chunk [0, 0x197a6): Mirror 1: 0x4fd51a93 Mirror 3: 0x4fd51a93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x197a6: skipped 000860 1787907473.206331 mapread 0x018d16 thru 0x0197a5 (0xa90 bytes) 000861 1787907473.242166 trunc from 0x00226e to 0x0079ea (0x577d bytes) 000862 1787907473.288436 mapwrite 0x00b3cc thru 0x018b34 (0xd769 bytes) 000863 1787907473.334290 mapwrite 0x013973 thru 0x015599 (0x1c27 bytes) 000864 1787907473.346964 mapwrite 0x000a31 thru 0x0083b9 (0x7989 bytes) 000865 1787907473.388640 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x197a6): Mirror 3: 0x4fd51a93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x197a6): Mirror 3: 0x4fd51a93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x197a6: skipped 000866 1787907473.738917 fallocate 0x01de6d thru 0x0287f7 (0xa98b bytes) 000867 1787907473.744108 trunc from 0x0287f8 to 0x00d2ef (0x1b508 bytes) 000868 1787907473.775139 MIRROR_RESYNC 2 mirrors 000870 1787907474.528331 MIRROR_RESYNC 2 mirrors 000871 1787907474.957278 read 0x002aae thru 0x00d2ef (0xa842 bytes) 000873 1787907474.976375 trunc from 0x000d26 to 0x00a1df (0x94ba bytes) 000874 1787907475.030197 MIRROR_RESYNC 2 mirrors 000875 1787907475.645199 write 0x03339f thru 0x03ffff (0xcc61 bytes) 000878 1787907475.696343 read 0x01b0ad thru 0x022a68 (0x79bc bytes) 000879 1787907475.705897 read 0x027123 thru 0x033c08 (0xcae6 bytes) 000880 1787907475.719767 read 0x03c991 thru 0x03ffff (0x366f bytes) 000881 1787907475.720900 read 0x03a9cb thru 0x03ffff (0x5635 bytes) 000882 1787907475.721949 punch from 0x00d458 to 0x0144ec (0x7095 bytes) 000883 1787907475.727364 mapwrite 0x013476 thru 0x01d920 (0xa4ab bytes) 000884 1787907475.785284 punch from 0x01d91a to 0x028cad (0xb394 bytes) 000885 1787907475.797589 write 0x009edd thru 0x00c0c3 (0x21e7 bytes) 000886 1787907475.802796 read 0x034bb8 thru 0x035b6e (0xfb7 bytes) 000887 1787907475.803972 mapread 0x01097e thru 0x01b319 (0xa99c bytes) 000888 1787907475.808165 mapwrite 0x018b79 thru 0x01d75e (0x4be6 bytes) 000889 1787907475.867825 read 0x036054 thru 0x036e2d (0xdda bytes) 000890 1787907475.868509 trunc from 0x00ba35 to 0x01390a (0x7ed6 bytes) 000891 1787907475.875273 fallocate 0x015a45 thru 0x017b5f (0x211b bytes) 000892 1787907475.896041 trunc from 0x040000 to 0x0334d0 (0xcb2f bytes) 000893 1787907475.935060 trunc from 0x022c2f to 0x031705 (0xead7 bytes) 000895 1787907475.948100 fallocate 0x038421 thru 0x03ffff (0x7bdf bytes) 000897 1787907475.954751 read 0x01823a thru 0x02655e (0xe325 bytes) 000898 1787907475.974518 MIRROR_RESYNC 2 mirrors 000899 1787907476.755023 mapread 0x02f997 thru 0x03650c (0x6b76 bytes) 000900 1787907476.816003 write 0x03975e thru 0x03c445 (0x2ce8 bytes) 000901 1787907476.862899 MIRROR_RESYNC 2 mirrors 000902 1787907477.959328 MIRROR_RESYNC 2 mirrors 000903 1787907478.411335 read 0x01166b thru 0x01ab05 (0x949b bytes) 000904 1787907478.423848 punch from 0x00878c to 0x012ffc (0xa871 bytes) 000905 1787907478.450326 fallocate 0x015753 thru 0x0249aa (0xf258 bytes) 000906 1787907478.455641 mapread 0x02aed3 thru 0x02cd84 (0x1eb2 bytes) 000907 1787907478.475632 trunc from 0x026297 to 0x02e31d (0x8087 bytes) 000908 1787907478.480416 read 0x012bc8 thru 0x013269 (0x6a2 bytes) 000910 1787907478.492407 fallocate 0x02d738 thru 0x032f5f (0x5828 bytes) 000911 1787907478.497178 trunc from 0x001096 to 0x008055 (0x6fc0 bytes) 000912 1787907478.502202 mapwrite 0x036871 thru 0x03fa56 (0x91e6 bytes) 000913 1787907478.521009 write 0x019622 thru 0x023200 (0x9bdf bytes) 000914 1787907478.523171 trunc from 0x02027c to 0x028e5e (0x8be3 bytes) 000915 1787907478.544802 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6f8cd274 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x60fb7f64 Mirror 4: 0x60fb7f64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000916 1787907479.658635 read 0x034c03 thru 0x03ffff (0xb3fd bytes) 000917 1787907479.684901 trunc from 0x040000 to 0x01f6b8 (0x20947 bytes) 000919 1787907479.771933 punch from 0x015b08 to 0x01c97c (0x6e75 bytes) 000920 1787907479.784020 punch from 0x00239a to 0x0104ad (0xe114 bytes) 000921 1787907479.795361 fallocate 0x03e8cb thru 0x03ee4f (0x585 bytes) 000922 1787907479.803077 write 0x02499e thru 0x029eb9 (0x551c bytes) 000923 1787907479.812881 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29eba): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29eba): Mirror 3: 0x84c8402d (stale) Mirror 4: 0xcd85923d (stale) CRC-32 checksum value for chunk [0, 0x29eba): Mirror 1: 0x160c690b Mirror 5: 0x160c690b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29eba: skipped 000924 1787907480.900510 mapwrite 0x02b793 thru 0x02c0d7 (0x945 bytes) 000925 1787907480.979253 trunc from 0x01c77e to 0x023a0a (0x728d bytes) 000926 1787907480.984709 trunc from 0x02c0d8 to 0x00a8c6 (0x21811 bytes) 000927 1787907481.012288 trunc from 0x004758 to 0x008d8e (0x4637 bytes) 000928 1787907481.021419 write 0x01782f thru 0x017ebc (0x68e bytes) 000929 1787907481.022763 trunc from 0x007d6f to 0x0112ce (0x9560 bytes) 000930 1787907481.029063 trunc from 0x017ebd to 0x000c95 (0x17227 bytes) 000932 1787907481.052074 write 0x0394e9 thru 0x03ffff (0x6b17 bytes) 000933 1787907481.054595 mapwrite 0x009316 thru 0x016294 (0xcf7f bytes) 000934 1787907481.125559 mapwrite 0x02fc9d thru 0x03efa3 (0xf307 bytes) 000935 1787907481.201353 read 0x03d939 thru 0x03e0c7 (0x78f bytes) 000936 1787907481.202134 read 0x03d5e4 thru 0x03ffff (0x2a1c bytes) 000937 1787907481.207348 mapwrite 0x012c22 thru 0x020e30 (0xe20f bytes) 000938 1787907481.293470 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x60fb7f64 (stale) Mirror 5: 0x7e38468a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x60fb7f64 (stale) Mirror 5: 0x7e38468a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000939 1787907481.665220 mapread 0x03add6 thru 0x03ffff (0x522a bytes) 000940 1787907481.683369 punch from 0x024c4c to 0x0286a5 (0x3a5a bytes) 000941 1787907481.688077 trunc from 0x040000 to 0x02ed3d (0x112c2 bytes) 000942 1787907481.712390 mapwrite 0x03fed0 thru 0x03ffff (0x130 bytes) 000944 1787907481.791626 read 0x0295fe thru 0x02e4b5 (0x4eb8 bytes) 000945 1787907481.817589 read 0x02307c thru 0x030739 (0xd6be bytes) 000946 1787907481.836135 mapwrite 0x030328 thru 0x0335ef (0x32c8 bytes) 000947 1787907481.862973 punch from 0x029a81 to 0x035b29 (0xc0a9 bytes) 000948 1787907481.868774 fallocate 0x00f11c thru 0x0109b6 (0x189b bytes) 000949 1787907481.884416 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x60fb7f64 (stale) Mirror 5: 0x7e38468a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6319cee4 Mirror 6: 0x6319cee4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000950 1787907483.105217 punch from 0x01fe34 to 0x02766a (0x7837 bytes) 000952 1787907483.185713 fallocate 0x0030c1 thru 0x00c8df (0x981f bytes) 000953 1787907483.198683 mapread 0x033563 thru 0x03a4d1 (0x6f6f bytes) 000955 1787907483.218295 read 0x01eec4 thru 0x028444 (0x9581 bytes) 000956 1787907483.233430 MIRROR_RESYNC 4 mirrors 000957 1787907484.490948 mapread 0x00ae88 thru 0x015336 (0xa4af bytes) 000960 1787907484.563041 fallocate 0x00b921 thru 0x010ce7 (0x53c7 bytes) 000961 1787907484.618412 fallocate 0x01ed32 thru 0x025c92 (0x6f61 bytes) 000962 1787907484.625436 fallocate 0x016297 thru 0x01e068 (0x7dd2 bytes) 000964 1787907484.634000 mapread 0x00a6ea thru 0x013b96 (0x94ad bytes) 000966 1787907484.674081 mapwrite 0x02a598 thru 0x035730 (0xb199 bytes) 000967 1787907484.754050 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x36910875 (stale) Mirror 5: 0x36910875 (stale) Mirror 6: 0x36910875 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa372f2c2 Mirror 7: 0xa372f2c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000968 1787907485.857364 mapread 0x003d1d thru 0x006ba7 (0x2e8b bytes) 000969 1787907485.932024 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x36910875 (stale) Mirror 5: 0x36910875 (stale) Mirror 6: 0x36910875 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa372f2c2 Mirror 7: 0xa372f2c2 Mirror 8: 0xa372f2c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000970 1787907486.942407 trunc from 0x040000 to 0x011041 (0x2efbe bytes) 000971 1787907487.015275 mapwrite 0x02defb thru 0x03a111 (0xc217 bytes) 000972 1787907487.082811 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a112): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a112): Mirror 4: 0x38612636 (stale) Mirror 5: 0x38612636 (stale) Mirror 6: 0x38612636 (stale) Mirror 7: 0x62b1b07d (stale) Mirror 8: 0x62b1b07d (stale) CRC-32 checksum value for chunk [0, 0x3a112): Mirror 1: 0x658c7b96 Mirror 9: 0x658c7b96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a112: skipped 000973 1787907488.356225 mapread 0x035010 thru 0x03a111 (0x5102 bytes) 000974 1787907488.434805 write 0x006fac thru 0x0097c5 (0x281a bytes) 000975 1787907488.500347 trunc from 0x014364 to 0x020ba8 (0xc845 bytes) 000977 1787907488.507355 trunc from 0x01e563 to 0x021ae5 (0x3583 bytes) 000978 1787907488.514901 mapread 0x0182e6 thru 0x018739 (0x454 bytes) 000979 1787907488.535348 MIRROR_RESYNC 7 mirrors 000980 1787907490.450944 fallocate 0x028a4c thru 0x03009b (0x7650 bytes) 000981 1787907490.534578 mapwrite 0x035ee9 thru 0x035f2b (0x43 bytes) 000982 1787907490.554496 mapwrite 0x039a75 thru 0x03ffff (0x658b bytes) 000984 1787907490.627772 write 0x022fe7 thru 0x02f841 (0xc85b bytes) 000985 1787907490.650318 trunc from 0x040000 to 0x0296f2 (0x1690d bytes) 000986 1787907490.681318 write 0x03965b thru 0x03ffff (0x69a5 bytes) 000987 1787907490.682943 mapwrite 0x0146fa thru 0x01f473 (0xad7a bytes) 000988 1787907490.742348 trunc from 0x040000 to 0x0209b8 (0x1f647 bytes) 000989 1787907490.775221 mapread 0x007049 thru 0x012268 (0xb220 bytes) 000990 1787907490.811923 punch from 0x00580b to 0x010d6c (0xb562 bytes) 000991 1787907490.822212 mapread 0x01b0a9 thru 0x01fcc6 (0x4c1e bytes) 000992 1787907490.824197 fallocate 0x0228d7 thru 0x02dd93 (0xb4bd bytes) 000993 1787907490.829618 trunc from 0x01b052 to 0x0209b8 (0x5967 bytes) 000994 1787907490.845814 MIRROR_RESYNC 7 mirrors 000996 1787907493.237685 mapwrite 0x015050 thru 0x01f180 (0xa131 bytes) 000997 1787907493.426527 write 0x02b480 thru 0x032560 (0x70e1 bytes) 000998 1787907493.438195 read 0x012cd3 thru 0x013245 (0x573 bytes) 000999 1787907493.439096 mapwrite 0x028bb5 thru 0x02e203 (0x564f bytes) 001000 1787907493.512119 trunc from 0x032561 to 0x02a53a (0x8026 bytes) All operations completed A-OK! PASS 70a (127s) == sanity-flr test 71: check mirror extend parent fid ==== 04:58:19 (1787907499) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x2e:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 04:58:29 (1787907509) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177608 s, 11.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14743 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 04:58:37 (1787907517) Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre2 Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6337664 .. Extending file size to 4844032 .. Extending file size to 7553344 .. Extending file size to 8256640 .. Extending file size to 8192192 .. Extending file size to 8372320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8686112 .. Extending file size to 8887328 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9240768 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9064160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61524 61525 61528 61530 61531 done 192.168.206.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg625-client.virtnet /mnt/lustre2 (opts:) 192.168.206.125@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg625-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:00:02 (1787907602) Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre2 Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6646880 .. Extending file size to 4944480 .. Extending file size to 6467392 .. Extending file size to 8083616 .. Extending file size to 6745152 .. Extending file size to 7555840 .. done Extending file size to 8986464 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9242688 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63141 63142 63144 63146 63148 done 192.168.206.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg625-client.virtnet /mnt/lustre2 (opts:) 192.168.206.125@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg625-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:01:20 (1787907680) Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2252 1269052 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24856 3582164 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18984 3588036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43840 7170200 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2252 1269052 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24856 3582164 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18984 3588036 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43840 7170200 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123078 s, 17.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216907 s, 9.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe016b458 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg625-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:01:40 (1787907700) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115973 s, 18.1 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 05:01:48 (1787907708) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:01:56 (1787907716) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0887775 s, 23.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x275:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x276:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:02:05 (1787907725) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123346 s, 17.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:02:16 (1787907736) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121496 s, 17.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:02:52 (1787907772) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg625-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949494 s, 11.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x278:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x277:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x278:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x277:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x275:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:03:47 (1787907827) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg625-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0933284 s, 22.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:04:35 (1787907875) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg625-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg625-server: Pool lustre.test_204f created Adding targets to pool oleg625-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg625-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151592 s, 13.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg625-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg625-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg625-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:05:43 (1787907943) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:05:51 (1787907951) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:05:58 (1787907958) oleg625-server: Pool lustre.test_206 created Adding targets to pool oleg625-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg625-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg625-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg625-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg625-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:06:26 (1787907986) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804812 s, 13.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323351 s, 32.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5e0d20fa Mirror 3: 0x5e0d20fa PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:06:37 (1787907997) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:06:41 (1787908001) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:06:44 (1787908004) Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0402755 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xad0000 Read this (4 bytes): BBBB 192.168.206.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg625-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:07:20 (1787908040) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0120118 s, 682 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:07:28 (1787908048) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308684 s, 34.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5c lmm_fid: [0x200000405:0x5c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5c lmm_fid: [0x200000405:0x5c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611228 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-28 05:07:29.000000000 -0400 Modify: 2026-08-28 05:07:29.000000000 -0400 Change: 2026-08-28 05:07:31.000000000 -0400 Birth: 2026-08-28 05:07:29.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:07:36 (1787908056) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557819 s, 18.8 MB/s oleg625-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg625-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg625-server: O/280000401/d20/134: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:07:56 (1787908076) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.241958 s, 169 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0186003 s, 220 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 28 05:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 28 05:07 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:08:05 (1787908085) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:08:11 (1787908091) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1910 sec =========== 05:08:20 (1787908100) === sanity-flr: start cleanup 05:08:22 (1787908102) === === sanity-flr: finish cleanup 05:08:30 (1787908110) ===