-----============= acceptance-small: sanity-flr ============----- Fri Jun 26 22:59:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg201-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:59:33 (1782529173) === oleg201-client.virtnet: executing check_config_client /mnt/lustre oleg201-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg201-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9afc10d17800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9afc10d17800.idle_timeout=debug disable quota as required oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:00:08 (1782529208) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8235] == sanity-flr test 0a: lfs mirror create with -N option == 23:00:10 (1782529210) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:00:25 (1782529225) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:00:28 (1782529228) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:00:31 (1782529231) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:00:52 (1782529252) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:00:55 (1782529255) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:00:59 (1782529259) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9afc10d17800.stats=clear osc.lustre-OST0001-osc-ffff9afc10d17800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:01:06 (1782529266) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:01:16 (1782529276) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:01:23 (1782529283) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:01:26 (1782529286) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:01:33 (1782529293) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 23:01:42 (1782529302) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:01:49 (1782529309) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:01:56 (1782529316) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:02:04 (1782529324) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:02:14 (1782529334) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:02:21 (1782529341) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.743643 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544994 s, 19.2 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:02:28 (1782529348) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0768812 s, 54.6 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:02:35 (1782529355) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:02:44 (1782529364) Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg201-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (131s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:04:54 (1782529494) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.193256 s, 16.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:05:36 (1782529536) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:06:53 (1782529613) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.40935 s, 21.8 MB/s -rw-r--r-- 1 root root 52428800 Jun 26 23:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.850s user 0m0.001s sys 0m0.157s Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9afc10d17800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.940s user 0m0.000s sys 0m0.186s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.165s user 0m0.001s sys 0m0.182s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg201-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9afc10d17800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:08:23 (1782529703) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:08:27 (1782529707) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:08:30 (1782529710) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:08:33 (1782529713) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 23:08:39 (1782529719) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.494973 s, 25.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:08:52 (1782529732) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267887 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241695 s, 43.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:08:59 (1782529739) PASS 36c (11s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:09:11 (1782529751) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000835995 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113224 s, 904 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.012435 s, 82.3 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163418 s, 627 kB/s flv=10 olv=10 PASS 36d (15s) == sanity-flr test 37: mirror I/O API verification ======= 23:09:26 (1782529766) create test files with size 15755262 .. 0eeec05b01679e26e4b7e88fd69b2c3b - 74b55b871b2cc83f6ce65dbbe9578288 - 8255509b9552247a5b4656e29f28731e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 23:09:49 (1782529789) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:10:00 (1782529800) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:10:07 (1782529807) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865114 s, 12.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208604 s, 50.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 23:10:17 (1782529817) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0661567 s, 31.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.222145 s, 18.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138753 s, 15.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 23:10:34 (1782529834) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 23:10:37 (1782529837) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:10:41 (1782529841) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261503 s, 40.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 23:10:49 (1782529849) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.299368 s, 10.5 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:10:58 (1782529858) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:11:08 (1782529868) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.375829 s, 27.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:11:20 (1782529880) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9afc2f8df800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.471451 s, 22.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9afc1346f800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:11:39 (1782529899) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:12:21 (1782529941) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:13:04 (1782529984) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 23:13:08 (1782529988) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.299952 s, 10.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.23376 s, 13.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x874ad68a (nosync,stale) Mirror 2: 0x874ad68a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6330b99db218cb0c71df876923b76422 - mirror 1 checksum: 97649ff4c0bd54b255dc07fbe4c873c9 - mirror 2 checksum: 6330b99db218cb0c71df876923b76422 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T23:13:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T23:13:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6330b99db218cb0c71df876923b76422 - mirror 1 checksum: 97649ff4c0bd54b255dc07fbe4c873c9 - mirror 2 checksum: 97649ff4c0bd54b255dc07fbe4c873c9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T23:13:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T23:13:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:13:18 (1782529998) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684853 s, 15.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:13:26 (1782530006) Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server 23:13:38 (1782530018) shut down facet: mds1 facet_host: oleg201-server facet_failover_host: oleg201-server Failover mds1 to oleg201-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 23:14:03 (1782530043) targets are mounted 23:14:03 (1782530043) facet_failover done oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595003 s, 17.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:14:22 (1782530062) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0263306 s, 778 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0139713 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:14:35 (1782530075) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0782394 s, 262 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00846076 s, 2.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:14:45 (1782530085) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00552077 s, 3.0 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:14:54 (1782530094) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0693801 s, 295 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:15:03 (1782530103) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0118181 s, 347 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0292247 s, 140 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:15:12 (1782530112) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:15:33 (1782530133) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:16:02 (1782530162) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:16:25 (1782530185) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:16:48 (1782530208) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 23:16:56 (1782530216) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57396 mirror split pid 57397 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57396 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57397 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 23:18:03 (1782530283) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 286 fallocating to largest ever: 0x3d261 000001 1782530286.326775 fallocate 0x035396 thru 0x03d260 (0x7ecb bytes) fallocating to largest ever: 0x3d7c9 000002 1782530286.336177 fallocate 0x03c09d thru 0x03d7c8 (0x172c bytes) 000003 1782530286.342315 mapread 0x0174eb thru 0x01a42e (0x2f44 bytes) 000004 1782530286.365820 read 0x0122f8 thru 0x0124cd (0x1d6 bytes) 000005 1782530286.366725 write 0x0174a9 thru 0x0247a9 (0xd301 bytes) 000006 1782530286.373533 mapread 0x017f78 thru 0x01f9ec (0x7a75 bytes) 000007 1782530286.378478 MIRROR_EXTEND to 2 mirrors 000008 1782530286.872015 write 0x0123a4 thru 0x01b487 (0x90e4 bytes) 000009 1782530286.923531 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d7c9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7c9): Mirror 2: 0xa5a0879d (stale) CRC-32 checksum value for chunk [0, 0x3d7c9): Mirror 1: 0x9719c928 Mirror 3: 0x9719c928 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7c9: skipped 000010 1782530288.116754 read 0x02e65f thru 0x03c968 (0xe30a bytes) 000011 1782530288.185179 trunc from 0x03d7c9 to 0x00a5d2 (0x331f6 bytes) 000012 1782530288.299602 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xa5d3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa5d3): Mirror 2: 0x499b9b23 (stale) Mirror 3: 0x499b9b23 (stale) CRC-32 checksum value for chunk [0, 0xa5d3): Mirror 1: 0x499b9b23 Mirror 4: 0x499b9b23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa5d3: skipped 000014 1782530289.562594 mapread 0x004944 thru 0x009130 (0x47ed bytes) 000015 1782530289.643659 punch from 0x007795 to 0x00a5d2 (0x2e3e bytes) 000016 1782530289.696665 trunc from 0x0004c5 to 0x009621 (0x915d bytes) 000017 1782530289.701982 mapread 0x005451 thru 0x00a5d2 (0x5182 bytes) 000018 1782530289.733652 mapread 0x006166 thru 0x00a5d2 (0x446d bytes) 000019 1782530289.734864 read 0x00851d thru 0x00a5d2 (0x20b6 bytes) 000020 1782530289.746216 mapread 0x0094cb thru 0x00a5d2 (0x1108 bytes) 000021 1782530289.748612 read 0x000ba7 thru 0x005963 (0x4dbd bytes) 000022 1782530289.769245 punch from 0x0054d4 to 0x00a5d2 (0x50ff bytes) 000023 1782530289.782084 trunc from 0x00a5d3 to 0x030439 (0x25e67 bytes) 000024 1782530289.854431 MIRROR_RESYNC 4 mirrors 000025 1782530290.473683 mapwrite 0x003b0a thru 0x00a3ca (0x68c1 bytes) 000026 1782530290.579992 trunc from 0x000c33 to 0x010300 (0xf6ce bytes) 000027 1782530290.585610 fallocate 0x008ebf thru 0x00b4fd (0x263f bytes) 000030 1782530290.592089 trunc from 0x03043a to 0x005e32 (0x2a607 bytes) 000031 1782530290.618023 mapread 0x004ffc thru 0x005e32 (0xe37 bytes) 000032 1782530290.631517 read 0x002e24 thru 0x005e32 (0x300f bytes) 000033 1782530290.633012 mapread 0x0058af thru 0x005e32 (0x584 bytes) 000034 1782530290.638325 MIRROR_RESYNC 4 mirrors 000037 1782530291.393281 punch from 0x004204 to 0x005e32 (0x1c2f bytes) 000038 1782530291.431131 trunc from 0x005e33 to 0x01e4d8 (0x186a6 bytes) 000039 1782530291.458393 fallocate 0x01c36e thru 0x0295fe (0xd291 bytes) 000040 1782530291.465126 fallocate 0x01f9a9 thru 0x025f71 (0x65c9 bytes) 000041 1782530291.475293 punch from 0x01ba9d to 0x025f71 (0xa4d5 bytes) 000042 1782530291.488418 mapwrite 0x01ddae thru 0x029511 (0xb764 bytes) 000043 1782530291.570053 trunc from 0x029512 to 0x01c680 (0xce91 bytes) 000044 1782530291.599372 MIRROR_RESYNC 4 mirrors 000045 1782530292.323053 write 0x03f4d2 thru 0x03ffff (0xb2e bytes) 000046 1782530292.363768 punch from 0x022a95 to 0x032746 (0xfcb2 bytes) 000047 1782530292.372821 mapread 0x01bf79 thru 0x02a92e (0xe9b6 bytes) 000048 1782530292.389283 mapread 0x02dc7c thru 0x03b046 (0xd3cb bytes) 000050 1782530292.406361 fallocate 0x01485e thru 0x0186fb (0x3e9e bytes) 000051 1782530292.411866 mapread 0x03bcae thru 0x03ffff (0x4352 bytes) fallocating to largest ever: 0x40000 000053 1782530292.414638 fallocate 0x037d97 thru 0x03ffff (0x8269 bytes) 000054 1782530292.424275 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb396ede9 Mirror 5: 0xb396ede9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000055 1782530293.029814 MIRROR_RESYNC 5 mirrors 000056 1782530293.697669 punch from 0x0252dd to 0x02bb5c (0x6880 bytes) 000057 1782530293.729419 fallocate 0x019bed thru 0x01b088 (0x149c bytes) 000058 1782530293.739600 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb396ede9 (stale) Mirror 3: 0xb396ede9 (stale) Mirror 4: 0xb396ede9 (stale) Mirror 5: 0xb396ede9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb396ede9 Mirror 6: 0xb396ede9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000060 1782530294.396233 fallocate 0x03ea92 thru 0x03ffff (0x156e bytes) 000061 1782530294.430351 mapread 0x026fc2 thru 0x035a86 (0xeac5 bytes) 000062 1782530294.454339 fallocate 0x030f1e thru 0x03651a (0x55fd bytes) 000063 1782530294.459035 trunc from 0x011110 to 0x01bc89 (0xab7a bytes) 000064 1782530294.463837 fallocate 0x018453 thru 0x01cd75 (0x4923 bytes) 000066 1782530294.468987 mapread 0x024a31 thru 0x02d771 (0x8d41 bytes) 000067 1782530294.470233 mapwrite 0x0166ff thru 0x023d3d (0xd63f bytes) 000068 1782530294.501214 trunc from 0x03f479 to 0x03ffff (0xb87 bytes) 000069 1782530294.505548 fallocate 0x037304 thru 0x03d6d1 (0x63ce bytes) 000070 1782530294.510612 punch from 0x020f15 to 0x023493 (0x257f bytes) 000071 1782530294.516272 punch from 0x0133a9 to 0x016309 (0x2f61 bytes) 000072 1782530294.523562 punch from 0x03144e to 0x031c34 (0x7e7 bytes) 000073 1782530294.528474 write 0x030d1d thru 0x0317f2 (0xad6 bytes) 000074 1782530294.537317 mapwrite 0x035867 thru 0x03bffa (0x6794 bytes) 000075 1782530294.564072 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb396ede9 (stale) Mirror 3: 0xb396ede9 (stale) Mirror 4: 0xb396ede9 (stale) Mirror 6: 0xb396ede9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb396ede9 (stale) Mirror 3: 0xb396ede9 (stale) Mirror 4: 0xb396ede9 (stale) Mirror 6: 0xb396ede9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000076 1782530294.901371 mapwrite 0x027aaa thru 0x02f52a (0x7a81 bytes) 000078 1782530294.935662 mapread 0x02dcff thru 0x032b3d (0x4e3f bytes) 000079 1782530294.944594 mapread 0x02725e thru 0x032245 (0xafe8 bytes) 000080 1782530294.946154 punch from 0x0354c7 to 0x03ffff (0xab39 bytes) 000081 1782530294.951121 fallocate 0x00dff0 thru 0x012a58 (0x4a69 bytes) 000082 1782530294.955377 write 0x022cbb thru 0x0270ae (0x43f4 bytes) 000083 1782530294.959797 trunc from 0x040000 to 0x0364a2 (0x9b5d bytes) 000084 1782530294.980455 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364a3): Mirror 2: 0xe63a4dd4 (stale) Mirror 3: 0xe63a4dd4 (stale) Mirror 6: 0xe63a4dd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364a3): Mirror 2: 0xe63a4dd4 (stale) Mirror 3: 0xe63a4dd4 (stale) Mirror 6: 0xe63a4dd4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364a3: skipped 000085 1782530295.290697 mapwrite 0x02b1a2 thru 0x035287 (0xa0e6 bytes) 000086 1782530295.333735 punch from 0x021dfc to 0x02b60c (0x9811 bytes) 000087 1782530295.339166 fallocate 0x01fc4e thru 0x02c6d8 (0xca8b bytes) 000088 1782530295.349948 write 0x033b22 thru 0x03546b (0x194a bytes) 000089 1782530295.357180 fallocate 0x00258f thru 0x00c2c4 (0x9d36 bytes) 000090 1782530295.363736 trunc from 0x0364a3 to 0x011f34 (0x2456e bytes) 000091 1782530295.374566 read 0x002f7a thru 0x003ebf (0xf46 bytes) 000092 1782530295.379854 mapwrite 0x006869 thru 0x007920 (0x10b8 bytes) 000093 1782530295.401128 fallocate 0x00bce2 thru 0x018d01 (0xd020 bytes) 000095 1782530295.414710 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18d02): Mirror 2: 0x83d493d9 (stale) Mirror 6: 0x83d493d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18d02): Mirror 2: 0x83d493d9 (stale) Mirror 6: 0x83d493d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18d02: skipped 000096 1782530295.742356 trunc from 0x018d02 to 0x0042aa (0x14a57 bytes) 000097 1782530295.757845 mapread 0x0002e2 thru 0x0042aa (0x3fc9 bytes) 000098 1782530295.788206 punch from 0x003ac0 to 0x0042aa (0x7eb bytes) 000099 1782530295.797819 fallocate 0x0380bd thru 0x03ffff (0x7f43 bytes) 000100 1782530295.819253 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb396ede9 (stale) Mirror 6: 0xb396ede9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 7: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000101 1782530296.623191 mapread 0x021c0f thru 0x0289e8 (0x6dda bytes) 000102 1782530296.655199 trunc from 0x040000 to 0x03aec0 (0x513f bytes) 000103 1782530296.747870 fallocate 0x027905 thru 0x031b8b (0xa287 bytes) 000104 1782530296.753823 read 0x000c9e thru 0x00ebc4 (0xdf27 bytes) 000105 1782530296.765475 mapwrite 0x0238fa thru 0x02edf7 (0xb4fe bytes) 000106 1782530296.797717 trunc from 0x03aec1 to 0x01847d (0x22a43 bytes) 000107 1782530296.815513 trunc from 0x01847e to 0x003290 (0x151ed bytes) 000108 1782530296.833458 punch from 0x00131a to 0x003290 (0x1f77 bytes) 000109 1782530296.842577 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3291): Mirror 6: 0x82237829 (stale) Mirror 7: 0x82237829 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3291): Mirror 6: 0x82237829 (stale) Mirror 7: 0x82237829 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3291: skipped 000110 1782530297.071542 trunc from 0x003291 to 0x0017de (0x1ab2 bytes) 000111 1782530297.093383 fallocate 0x0036d8 thru 0x0123be (0xece7 bytes) 000112 1782530297.098426 trunc from 0x0008d8 to 0x0017de (0xf07 bytes) 000113 1782530297.102560 trunc from 0x0017b5 to 0x0017de (0x2a bytes) 000114 1782530297.106251 write 0x03db74 thru 0x03ffff (0x248c bytes) 000115 1782530297.108725 fallocate 0x01be7c thru 0x01f90d (0x3a92 bytes) 000117 1782530297.126056 trunc from 0x0241e1 to 0x02a5ea (0x640a bytes) 000118 1782530297.136670 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000119 1782530297.356420 punch from 0x01a4ab to 0x024c4e (0xa7a4 bytes) 000121 1782530297.365887 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x35c30a7f Mirror 8: 0x35c30a7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000122 1782530298.048360 mapread 0x016634 thru 0x019ac2 (0x348f bytes) 000123 1782530298.081306 read 0x03c76d thru 0x03cee2 (0x776 bytes) 000124 1782530298.096404 write 0x00d05e thru 0x01b407 (0xe3aa bytes) 000125 1782530298.168160 fallocate 0x007c61 thru 0x00f1c8 (0x7568 bytes) 000126 1782530298.174126 read 0x011dba thru 0x018d7b (0x6fc2 bytes) 000127 1782530298.175180 read 0x004993 thru 0x0071bd (0x282b bytes) 000128 1782530298.189280 write 0x01066e thru 0x010aa5 (0x438 bytes) 000129 1782530298.190615 trunc from 0x040000 to 0x0077a3 (0x3885c bytes) 000130 1782530298.213848 trunc from 0x001a42 to 0x0077a3 (0x5d62 bytes) 000131 1782530298.221274 mapread 0x003467 thru 0x0071f3 (0x3d8d bytes) 000132 1782530298.255369 punch from 0x004740 to 0x0077a3 (0x3064 bytes) 000133 1782530298.262040 write 0x02293d thru 0x02f2ea (0xc9ae bytes) 000134 1782530298.265134 write 0x02bd60 thru 0x0301ac (0x444d bytes) 000136 1782530298.268591 write 0x021ddf thru 0x029bca (0x7dec bytes) 000137 1782530298.274622 trunc from 0x02dab6 to 0x0301ac (0x26f7 bytes) 000138 1782530298.291878 trunc from 0x0301ad to 0x00ad4e (0x2545e bytes) 000139 1782530298.313527 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xad4f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xad4f): Mirror 7: 0xe46cf93c (stale) Mirror 8: 0xe46cf93c (stale) CRC-32 checksum value for chunk [0, 0xad4f): Mirror 1: 0xe46cf93c Mirror 9: 0xe46cf93c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xad4f: skipped 000140 1782530298.983076 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xad4f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xad4f): Mirror 7: 0xe46cf93c (stale) Mirror 8: 0xe46cf93c (stale) CRC-32 checksum value for chunk [0, 0xad4f): Mirror 1: 0xe46cf93c Mirror 9: 0xe46cf93c Mirror 10: 0xe46cf93c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xad4f: skipped 000141 1782530299.686903 write 0x0131e3 thru 0x016f04 (0x3d22 bytes) 000142 1782530299.762276 mapread 0x011445 thru 0x016f04 (0x5ac0 bytes) 000143 1782530299.771497 trunc from 0x016f05 to 0x014d6d (0x2197 bytes) 000144 1782530299.835154 mapwrite 0x039d3c thru 0x03e16d (0x4432 bytes) 000145 1782530299.884052 trunc from 0x03a249 to 0x03dee3 (0x3c9b bytes) 000146 1782530299.890963 trunc from 0x03b253 to 0x03e16d (0x2f1b bytes) 000147 1782530299.900300 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e16e): Mirror 8: 0xb073c10a (stale) Mirror 9: 0xc27633ab (stale) Mirror 10: 0xc27633ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e16e): Mirror 8: 0xb073c10a (stale) Mirror 9: 0xc27633ab (stale) Mirror 10: 0xc27633ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e16e: skipped 000148 1782530300.175146 fallocate 0x03eeb2 thru 0x03ffff (0x114e bytes) 000149 1782530300.180303 mapwrite 0x0100ae thru 0x01395a (0x38ad bytes) 000150 1782530300.211952 read 0x03569c thru 0x03b6c1 (0x6026 bytes) 000151 1782530300.219385 fallocate 0x02e940 thru 0x02f868 (0xf29 bytes) 000152 1782530300.224392 fallocate 0x034928 thru 0x03ffff (0xb6d8 bytes) 000153 1782530300.229563 trunc from 0x0143c6 to 0x02169b (0xd2d6 bytes) 000154 1782530300.234523 punch from 0x03cb80 to 0x03ffff (0x3480 bytes) 000155 1782530300.238846 mapwrite 0x01eb0a thru 0x022791 (0x3c88 bytes) 000156 1782530300.264059 trunc from 0x026bfc to 0x02f9f1 (0x8df6 bytes) 000157 1782530300.268353 fallocate 0x00a08d thru 0x00a63f (0x5b3 bytes) 000158 1782530300.273441 punch from 0x01b569 to 0x024f13 (0x99ab bytes) 000159 1782530300.277701 punch from 0x01e37d to 0x0210fd (0x2d81 bytes) 000160 1782530300.281559 trunc from 0x040000 to 0x010be7 (0x2f418 bytes) 000161 1782530300.298542 read 0x0049c3 thru 0x00bbe7 (0x7225 bytes) 000162 1782530300.311249 trunc from 0x010be8 to 0x01295e (0x1d77 bytes) 000163 1782530300.323990 punch from 0x0108dd to 0x01295e (0x2082 bytes) 000164 1782530300.329225 trunc from 0x01295f to 0x03ac8a (0x2832c bytes) 000165 1782530300.341401 read 0x02470f thru 0x02ecb0 (0xa5a2 bytes) 000166 1782530300.351111 mapwrite 0x039ea9 thru 0x03ffff (0x6157 bytes) 000167 1782530300.393300 fallocate 0x01c010 thru 0x02464b (0x863c bytes) 000168 1782530300.398603 write 0x033dc3 thru 0x034b37 (0xd75 bytes) 000169 1782530300.400224 read 0x0200d5 thru 0x026998 (0x68c4 bytes) 000170 1782530300.401458 punch from 0x006cf9 to 0x00e0a6 (0x73ae bytes) 000172 1782530300.406954 write 0x01ce20 thru 0x020cb5 (0x3e96 bytes) 000173 1782530300.412407 MIRROR_RESYNC 4 mirrors 000175 1782530301.365160 mapwrite 0x0281a5 thru 0x02c149 (0x3fa5 bytes) 000176 1782530301.447421 punch from 0x0372eb to 0x03ffff (0x8d15 bytes) 000178 1782530301.453067 fallocate 0x00ec97 thru 0x017500 (0x886a bytes) 000179 1782530301.457893 mapread 0x011650 thru 0x020e83 (0xf834 bytes) 000180 1782530301.474588 write 0x00b66b thru 0x01504a (0x99e0 bytes) 000181 1782530301.476504 punch from 0x021f30 to 0x028d94 (0x6e65 bytes) 000182 1782530301.485362 MIRROR_RESYNC 4 mirrors 000183 1782530302.479984 fallocate 0x03bcdb thru 0x03c75f (0xa85 bytes) 000184 1782530302.511227 write 0x0054fc thru 0x01455d (0xf062 bytes) 000185 1782530302.524288 punch from 0x03e4fa to 0x03f471 (0xf78 bytes) 000186 1782530302.528477 mapread 0x03e1c3 thru 0x03ffff (0x1e3d bytes) 000187 1782530302.539803 trunc from 0x02be88 to 0x033618 (0x7791 bytes) 000188 1782530302.544525 trunc from 0x040000 to 0x039911 (0x66ee bytes) 000189 1782530302.559996 read 0x0024e5 thru 0x0030e8 (0xc04 bytes) 000190 1782530302.565967 trunc from 0x034755 to 0x039911 (0x51bd bytes) 000192 1782530302.576447 read 0x036b2b thru 0x038db3 (0x2289 bytes) 000194 1782530302.586637 punch from 0x017023 to 0x01d8b7 (0x6895 bytes) 000195 1782530302.591127 read 0x0037a3 thru 0x0090cc (0x592a bytes) 000196 1782530302.596277 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39912): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39912): Mirror 8: 0xcc4db9a7 (stale) Mirror 9: 0xcc4db9a7 (stale) Mirror 10: 0xcc4db9a7 (stale) CRC-32 checksum value for chunk [0, 0x39912): Mirror 1: 0x69678a9d Mirror 11: 0x69678a9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39912: skipped 000197 1782530303.298507 punch from 0x01b3db to 0x01c55e (0x1184 bytes) 000198 1782530303.388987 trunc from 0x039912 to 0x0044e5 (0x3542c bytes) 000200 1782530303.432769 punch from 0x000651 to 0x0044e5 (0x3e95 bytes) 000201 1782530303.441547 mapread 0x000325 thru 0x0044e5 (0x41c1 bytes) 000202 1782530303.465204 punch from 0x000b6d to 0x00188e (0xd22 bytes) 000203 1782530303.471424 trunc from 0x0044e6 to 0x03a1e9 (0x35d04 bytes) 000206 1782530303.493411 mapwrite 0x029507 thru 0x035458 (0xbf52 bytes) 000207 1782530303.537305 trunc from 0x03a1ea to 0x013aa5 (0x26744 bytes) 000208 1782530303.563782 mapread 0x009fe4 thru 0x013aa5 (0x9ac2 bytes) 000209 1782530303.589104 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x13aa6): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13aa6): Mirror 8: 0x23ad3af3 (stale) Mirror 9: 0x23ad3af3 (stale) Mirror 10: 0x23ad3af3 (stale) Mirror 11: 0x77e02726 (stale) CRC-32 checksum value for chunk [0, 0x13aa6): Mirror 1: 0xd673849d Mirror 12: 0xd673849d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13aa6: skipped 000210 1782530304.391449 mapread 0x00cba5 thru 0x013aa5 (0x6f01 bytes) 000211 1782530304.457390 MIRROR_RESYNC 6 mirrors 000213 1782530305.368787 fallocate 0x007a5b thru 0x008195 (0x73b bytes) 000216 1782530305.417476 punch from 0x001a0d to 0x00870f (0x6d03 bytes) 000217 1782530305.422971 punch from 0x008843 to 0x0105d2 (0x7d90 bytes) 000218 1782530305.432201 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x13aa6): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13aa6): Mirror 8: 0xd673849d (stale) Mirror 9: 0xd673849d (stale) Mirror 10: 0xd673849d (stale) Mirror 11: 0xd673849d (stale) Mirror 12: 0xd673849d (stale) CRC-32 checksum value for chunk [0, 0x13aa6): Mirror 1: 0xd673849d Mirror 13: 0xd673849d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13aa6: skipped 000219 1782530306.336892 fallocate 0x03b4af thru 0x03ffff (0x4b51 bytes) 000220 1782530306.386281 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000222 1782530306.770352 mapwrite 0x022097 thru 0x02a3ff (0x8369 bytes) 000223 1782530306.804396 trunc from 0x040000 to 0x005265 (0x3ad9a bytes) 000224 1782530306.827203 fallocate 0x02bdec thru 0x035c8e (0x9ea3 bytes) 000225 1782530306.832828 write 0x025730 thru 0x027a4a (0x231b bytes) 000226 1782530306.844531 write 0x029d35 thru 0x03267d (0x8949 bytes) 000227 1782530306.854746 trunc from 0x035c8f to 0x00b752 (0x2a53c bytes) 000228 1782530306.867866 mapread 0x00359c thru 0x009bc1 (0x6626 bytes) 000229 1782530306.880402 mapwrite 0x0039cb thru 0x0118f4 (0xdf2a bytes) 000230 1782530306.921278 trunc from 0x0118f5 to 0x03b61d (0x29d29 bytes) 000231 1782530306.934884 trunc from 0x03b61e to 0x03b230 (0x3ed bytes) 000232 1782530306.948263 mapwrite 0x03b8ac thru 0x03ffff (0x4754 bytes) 000233 1782530306.981838 write 0x000e39 thru 0x0094aa (0x8672 bytes) 000234 1782530306.989383 mapwrite 0x03dd50 thru 0x03ffff (0x22b0 bytes) 000235 1782530307.005451 write 0x021e38 thru 0x02e94e (0xcb17 bytes) 000236 1782530307.015304 mapread 0x004953 thru 0x0062bc (0x196a bytes) 000237 1782530307.016354 trunc from 0x040000 to 0x010e02 (0x2f1fd bytes) 000238 1782530307.041060 MIRROR_RESYNC 6 mirrors 000239 1782530307.948892 mapread 0x010913 thru 0x010e02 (0x4f0 bytes) 000240 1782530307.962167 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000241 1782530308.231670 mapwrite 0x03fc29 thru 0x03ffff (0x3d7 bytes) 000242 1782530308.301219 mapread 0x0155ed thru 0x020016 (0xaa2a bytes) 000243 1782530308.323458 mapwrite 0x032345 thru 0x03ffff (0xdcbb bytes) 000244 1782530308.360423 trunc from 0x040000 to 0x003d82 (0x3c27d bytes) 000245 1782530308.377571 fallocate 0x0318a1 thru 0x033516 (0x1c76 bytes) 000246 1782530308.400741 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d83): Mirror 9: 0xff10fca2 (stale) Mirror 11: 0xff10fca2 (stale) Mirror 12: 0xff10fca2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d83): Mirror 9: 0xff10fca2 (stale) Mirror 11: 0xff10fca2 (stale) Mirror 12: 0xff10fca2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d83: skipped 000247 1782530308.698812 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d83): Mirror 11: 0xff10fca2 (stale) Mirror 12: 0xff10fca2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d83): Mirror 11: 0xff10fca2 (stale) Mirror 12: 0xff10fca2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d83: skipped 000248 1782530309.002810 mapwrite 0x019723 thru 0x024d1d (0xb5fb bytes) 000249 1782530309.140257 mapread 0x01fd01 thru 0x024d1d (0x501d bytes) 000250 1782530309.144007 mapread 0x01162b thru 0x018134 (0x6b0a bytes) 000251 1782530309.167966 trunc from 0x024b62 to 0x024d1d (0x1bc bytes) 000252 1782530309.194458 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24d1e): Mirror 12: 0x6633346b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d1e): Mirror 12: 0x6633346b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d1e: skipped 000253 1782530309.561857 write 0x004be4 thru 0x008197 (0x35b4 bytes) 000254 1782530309.577367 trunc from 0x024d1e to 0x0033e6 (0x21937 bytes) 000255 1782530309.593343 mapwrite 0x01fc0a thru 0x02ba2a (0xbe21 bytes) 000256 1782530309.657860 write 0x000b49 thru 0x00472f (0x3be7 bytes) 000257 1782530309.672715 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ba2b): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba2b): Mirror 12: 0x42e79ef1 (stale) CRC-32 checksum value for chunk [0, 0x2ba2b): Mirror 1: 0x4a2c8b57 Mirror 13: 0x4a2c8b57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba2b: skipped 000258 1782530310.417457 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000260 1782530310.790336 punch from 0x023a9f to 0x025832 (0x1d94 bytes) 000261 1782530310.859006 read 0x00326c thru 0x005130 (0x1ec5 bytes) 000263 1782530310.871008 punch from 0x0196c8 to 0x01e6e0 (0x5019 bytes) 000265 1782530310.880812 write 0x03515f thru 0x03dd35 (0x8bd7 bytes) 000266 1782530310.889271 mapwrite 0x019472 thru 0x01a252 (0xde1 bytes) 000267 1782530310.946821 MIRROR_RESYNC 2 mirrors 000268 1782530311.620880 trunc from 0x03dd36 to 0x0184e9 (0x2584c bytes) 000269 1782530311.655184 fallocate 0x008a74 thru 0x018105 (0xf692 bytes) 000271 1782530311.661494 fallocate 0x03b9ff thru 0x03ffff (0x4601 bytes) 000272 1782530311.667176 write 0x01613b thru 0x02143a (0xb300 bytes) 000273 1782530311.678022 fallocate 0x0021c9 thru 0x00f7ff (0xd637 bytes) 000274 1782530311.682399 read 0x008997 thru 0x011147 (0x87b1 bytes) 000277 1782530311.695461 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000278 1782530311.814351 fallocate 0x02306e thru 0x02a8c1 (0x7854 bytes) 000279 1782530311.820462 fallocate 0x021f57 thru 0x02fbc2 (0xdc6c bytes) 000280 1782530311.825981 trunc from 0x007af9 to 0x012170 (0xa678 bytes) 000281 1782530311.834673 MIRROR_EXTEND to 2 mirrors 000283 1782530312.434581 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000284 1782530312.555020 read 0x030df1 thru 0x03bf01 (0xb111 bytes) 000285 1782530312.565252 trunc from 0x03d0b8 to 0x03ffff (0x2f48 bytes) 000287 1782530312.580729 trunc from 0x040000 to 0x004ccb (0x3b334 bytes) 000288 1782530312.605279 punch from 0x00037d to 0x004ccb (0x494f bytes) 000289 1782530312.612950 read 0x000ea9 thru 0x004ccb (0x3e23 bytes) 000290 1782530312.623051 mapread 0x003f8b thru 0x004ccb (0xd41 bytes) 000291 1782530312.624565 write 0x01a477 thru 0x01e2a7 (0x3e31 bytes) 000292 1782530312.627354 trunc from 0x01c6ac to 0x01e2a7 (0x1bfc bytes) 000293 1782530312.650967 mapwrite 0x034cb5 thru 0x03ffff (0xb34b bytes) 000295 1782530312.775026 write 0x00f77f thru 0x015055 (0x58d7 bytes) 000297 1782530312.787084 fallocate 0x004054 thru 0x007761 (0x370e bytes) 000298 1782530312.798776 read 0x017f60 thru 0x0259ab (0xda4c bytes) 000299 1782530312.800376 fallocate 0x033e5b thru 0x03f92f (0xbad5 bytes) 000301 1782530312.814024 trunc from 0x016cbd to 0x01dbce (0x6f12 bytes) 000302 1782530312.819422 mapwrite 0x005134 thru 0x00e4ae (0x937b bytes) 000303 1782530312.865318 trunc from 0x03acc2 to 0x03ffff (0x533e bytes) 000304 1782530312.871822 trunc from 0x00fc4c to 0x01b3af (0xb764 bytes) 000305 1782530312.877419 trunc from 0x040000 to 0x02e758 (0x118a7 bytes) 000306 1782530312.899500 punch from 0x0163fe to 0x0198d7 (0x34da bytes) 000310 1782530312.922421 mapwrite 0x031c61 thru 0x03570e (0x3aae bytes) 000311 1782530313.020294 trunc from 0x03570f to 0x019b70 (0x1bb9e bytes) 000312 1782530313.054491 mapread 0x005ce0 thru 0x00ca7c (0x6d9d bytes) 000313 1782530313.055807 read 0x00460a thru 0x005723 (0x111a bytes) 000314 1782530313.056444 read 0x00c55c thru 0x00ef47 (0x29ec bytes) 000315 1782530313.057082 mapread 0x012724 thru 0x019b70 (0x744d bytes) 000316 1782530313.062063 punch from 0x008e42 to 0x013d21 (0xaee0 bytes) 000317 1782530313.070977 mapread 0x014d78 thru 0x019b70 (0x4df9 bytes) 000318 1782530313.084077 trunc from 0x00fa4d to 0x019b70 (0xa124 bytes) 000319 1782530313.095107 fallocate 0x00de46 thru 0x0110fd (0x32b8 bytes) 000320 1782530313.105481 mapread 0x001dff thru 0x00cd6e (0xaf70 bytes) 000321 1782530313.128466 punch from 0x005e57 to 0x0063f9 (0x5a3 bytes) 000322 1782530313.146442 write 0x01cd1c thru 0x020704 (0x39e9 bytes) 000323 1782530313.147789 mapread 0x013c39 thru 0x020704 (0xcacc bytes) 000324 1782530313.181041 punch from 0x01b083 to 0x01e730 (0x36ae bytes) 000326 1782530313.211640 fallocate 0x024128 thru 0x028838 (0x4711 bytes) 000327 1782530313.227310 punch from 0x0181db to 0x01eb07 (0x692d bytes) 000328 1782530313.237371 trunc from 0x028839 to 0x0108bf (0x17f79 bytes) 000329 1782530313.275655 trunc from 0x00dd0e to 0x0108bf (0x2bb2 bytes) 000330 1782530313.283394 fallocate 0x03e12f thru 0x03ffff (0x1ed1 bytes) 000331 1782530313.296489 trunc from 0x00fa22 to 0x0108bf (0xe9e bytes) 000332 1782530313.311086 trunc from 0x0108c0 to 0x004337 (0xc588 bytes) 000333 1782530313.345223 fallocate 0x03bcfc thru 0x03de04 (0x2109 bytes) 000334 1782530313.356806 write 0x00994f thru 0x01267d (0x8d2f bytes) 000335 1782530313.386263 read 0x025b10 thru 0x030f6f (0xb460 bytes) 000336 1782530313.387413 read 0x01c7f3 thru 0x02a7e0 (0xdfee bytes) 000337 1782530313.400457 read 0x0064cb thru 0x014f86 (0xeabc bytes) 000338 1782530313.401180 punch from 0x02e49a to 0x035af8 (0x765f bytes) 000339 1782530313.407452 trunc from 0x03de05 to 0x008d61 (0x350a3 bytes) 000340 1782530313.432486 fallocate 0x007ad7 thru 0x00e007 (0x6531 bytes) 000341 1782530313.456243 write 0x010435 thru 0x01eb73 (0xe73f bytes) 000342 1782530313.463555 write 0x00d8b0 thru 0x01a7ac (0xcefd bytes) 000343 1782530313.482507 punch from 0x0064b3 to 0x00940b (0x2f59 bytes) 000344 1782530313.506565 write 0x0195ef thru 0x019700 (0x112 bytes) 000345 1782530313.507406 mapwrite 0x02c578 thru 0x02fe68 (0x38f1 bytes) 000346 1782530313.626633 read 0x014295 thru 0x015172 (0xede bytes) 000347 1782530313.627308 mapwrite 0x02d2eb thru 0x03ab8c (0xd8a2 bytes) 000348 1782530313.708236 fallocate 0x03b1a2 thru 0x03ffff (0x4e5e bytes) 000349 1782530313.721714 mapwrite 0x009f2a thru 0x015398 (0xb46f bytes) 000352 1782530313.748279 write 0x01da2a thru 0x021b66 (0x413d bytes) 000353 1782530313.749201 mapread 0x0198ae thru 0x02211c (0x886f bytes) 000354 1782530313.755333 fallocate 0x018d2c thru 0x01a58f (0x1864 bytes) 000355 1782530313.770872 mapread 0x0159f7 thru 0x01e460 (0x8a6a bytes) 000356 1782530313.773756 fallocate 0x01c922 thru 0x021b31 (0x5210 bytes) 000358 1782530313.784542 trunc from 0x01f73e to 0x026166 (0x6a29 bytes) 000359 1782530313.800089 read 0x02b801 thru 0x02c36e (0xb6e bytes) 000361 1782530313.802000 trunc from 0x007335 to 0x00c4ec (0x51b8 bytes) 000362 1782530313.806596 trunc from 0x03ab8d to 0x031998 (0x91f4 bytes) 000363 1782530313.827681 trunc from 0x031999 to 0x02c62c (0x536c bytes) 000364 1782530313.844754 read 0x010f3b thru 0x01cb9a (0xbc60 bytes) 000365 1782530313.846148 punch from 0x02586a to 0x02c62c (0x6dc3 bytes) 000366 1782530313.852273 punch from 0x023bd2 to 0x02ada7 (0x71d6 bytes) 000367 1782530313.860324 mapread 0x00fd7a thru 0x018a23 (0x8caa bytes) 000368 1782530313.868127 trunc from 0x02c62d to 0x01283e (0x19dee bytes) 000369 1782530313.891113 MIRROR_EXTEND to 2 mirrors 000370 1782530314.496481 fallocate 0x01d73d thru 0x027564 (0x9e28 bytes) 000371 1782530314.544469 trunc from 0x027565 to 0x0314f9 (0x9f95 bytes) 000372 1782530314.576068 mapwrite 0x011b97 thru 0x016ed4 (0x533e bytes) 000374 1782530314.624377 read 0x02a274 thru 0x0302c3 (0x6050 bytes) 000375 1782530314.635208 mapwrite 0x01436c thru 0x021c1d (0xd8b2 bytes) 000376 1782530314.677296 punch from 0x01de8c to 0x02d284 (0xf3f9 bytes) 000377 1782530314.690665 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000378 1782530314.837490 write 0x03e461 thru 0x03ffff (0x1b9f bytes) 000379 1782530314.843875 fallocate 0x01870b thru 0x01f9dc (0x72d2 bytes) 000380 1782530314.850309 punch from 0x00f472 to 0x01374c (0x42db bytes) 000381 1782530314.859222 trunc from 0x040000 to 0x007127 (0x38ed8 bytes) 000382 1782530314.877881 punch from 0x002e50 to 0x00356b (0x71c bytes) 000383 1782530314.883624 trunc from 0x007128 to 0x000fb0 (0x6177 bytes) 000384 1782530314.907474 write 0x031c80 thru 0x03aea7 (0x9228 bytes) 000385 1782530314.908996 mapread 0x035853 thru 0x03a931 (0x50df bytes) 000386 1782530314.911767 mapread 0x021b0e thru 0x027c9b (0x618e bytes) 000387 1782530314.928312 read 0x017840 thru 0x02607c (0xe83d bytes) 000388 1782530314.934821 punch from 0x0337b9 to 0x03aea7 (0x76ef bytes) 000389 1782530314.950165 trunc from 0x00002d to 0x006acd (0x6aa1 bytes) 000390 1782530314.956319 fallocate 0x03feee thru 0x03ffff (0x112 bytes) 000391 1782530314.963157 write 0x033677 thru 0x0364d3 (0x2e5d bytes) 000393 1782530314.975686 mapread 0x039047 thru 0x03ffff (0x6fb9 bytes) 000394 1782530314.986498 mapwrite 0x004bc2 thru 0x0103c3 (0xb802 bytes) 000395 1782530315.029588 write 0x00376f thru 0x005ee2 (0x2774 bytes) 000396 1782530315.030687 punch from 0x021fdb to 0x0287e0 (0x6806 bytes) 000397 1782530315.036241 mapread 0x02f8df thru 0x03b5a0 (0xbcc2 bytes) 000398 1782530315.048096 punch from 0x02a0bf to 0x02b4c4 (0x1406 bytes) 000399 1782530315.053312 read 0x026d74 thru 0x031011 (0xa29e bytes) 000400 1782530315.062610 punch from 0x00a4da to 0x00aa47 (0x56e bytes) 000401 1782530315.067741 mapwrite 0x016a8e thru 0x01e5bb (0x7b2e bytes) 000402 1782530315.090800 mapwrite 0x02a79d thru 0x034ac3 (0xa327 bytes) 000403 1782530315.116165 mapwrite 0x02bf86 thru 0x03a44e (0xe4c9 bytes) 000404 1782530315.138984 trunc from 0x032ef2 to 0x03366d (0x77c bytes) 000405 1782530315.145158 punch from 0x02f7dc to 0x034e27 (0x564c bytes) 000406 1782530315.151290 read 0x011a59 thru 0x016362 (0x490a bytes) 000409 1782530315.156232 trunc from 0x02e0bc to 0x032784 (0x46c9 bytes) 000410 1782530315.162090 mapwrite 0x02417e thru 0x02f6ae (0xb531 bytes) 000411 1782530315.192855 mapread 0x01ffad thru 0x0245f5 (0x4649 bytes) 000412 1782530315.194704 trunc from 0x01b5be to 0x028d69 (0xd7ac bytes) 000413 1782530315.200134 write 0x0095fe thru 0x00f100 (0x5b03 bytes) 000414 1782530315.201709 mapwrite 0x011506 thru 0x01cd46 (0xb841 bytes) 000415 1782530315.239048 fallocate 0x0170f5 thru 0x01a2e7 (0x31f3 bytes) 000416 1782530315.243472 trunc from 0x032cad to 0x03f14c (0xc4a0 bytes) 000417 1782530315.251234 trunc from 0x040000 to 0x027cf9 (0x18306 bytes) 000418 1782530315.264987 trunc from 0x027cfa to 0x007202 (0x20af7 bytes) 000419 1782530315.292024 mapwrite 0x01e3d5 thru 0x01fbf4 (0x1820 bytes) 000420 1782530315.333289 read 0x01490d thru 0x01e7d0 (0x9ec4 bytes) 000421 1782530315.343955 mapwrite 0x0145c1 thru 0x01e3a8 (0x9de8 bytes) 000422 1782530315.367081 trunc from 0x012629 to 0x01fbf4 (0xd5cc bytes) 000423 1782530315.373835 write 0x0299a5 thru 0x02d870 (0x3ecc bytes) 000424 1782530315.375851 fallocate 0x03c588 thru 0x03ffff (0x3a78 bytes) 000425 1782530315.382280 fallocate 0x02738c thru 0x03712d (0xfda2 bytes) 000428 1782530315.388447 mapwrite 0x025c3b thru 0x034d0d (0xf0d3 bytes) 000430 1782530315.438799 fallocate 0x01016f thru 0x0184f1 (0x8383 bytes) 000432 1782530315.448163 punch from 0x009206 to 0x015a0d (0xc808 bytes) 000433 1782530315.454165 mapread 0x029b0e thru 0x03555d (0xba50 bytes) 000434 1782530315.457095 mapread 0x013d16 thru 0x01b0c8 (0x73b3 bytes) 000435 1782530315.513049 read 0x010434 thru 0x016624 (0x61f1 bytes) 000436 1782530315.530168 punch from 0x027e4c to 0x02940c (0x15c1 bytes) 000437 1782530315.538470 punch from 0x032033 to 0x03712d (0x50fb bytes) 000438 1782530315.546232 punch from 0x000a75 to 0x00a6af (0x9c3b bytes) 000439 1782530315.553254 write 0x031e94 thru 0x03ef99 (0xd106 bytes) 000440 1782530315.556448 punch from 0x03a03f to 0x03ef99 (0x4f5b bytes) 000441 1782530315.572044 write 0x00da35 thru 0x017d28 (0xa2f4 bytes) 000442 1782530315.574291 read 0x023365 thru 0x026660 (0x32fc bytes) 000443 1782530315.576085 trunc from 0x03ef9a to 0x02ce4d (0x1214c bytes) 000444 1782530315.595778 write 0x038eaf thru 0x03ffff (0x7151 bytes) 000445 1782530315.598735 mapread 0x00c5ae thru 0x00d447 (0xe9a bytes) 000446 1782530315.600169 trunc from 0x01cd0e to 0x028733 (0xba26 bytes) 000447 1782530315.605383 write 0x019fba thru 0x0209e7 (0x6a2e bytes) 000448 1782530315.616005 write 0x006a7d thru 0x00aede (0x4462 bytes) 000449 1782530315.621322 read 0x025fed thru 0x02f5fd (0x9611 bytes) 000450 1782530315.631343 mapread 0x03a365 thru 0x03fe1f (0x5abb bytes) 000451 1782530315.632880 mapwrite 0x00adba thru 0x00efd8 (0x421f bytes) 000452 1782530315.651738 mapwrite 0x02224b thru 0x02e45b (0xc211 bytes) 000454 1782530315.672571 trunc from 0x040000 to 0x03b398 (0x4c67 bytes) 000456 1782530315.688007 trunc from 0x03b399 to 0x03f02a (0x3c92 bytes) 000458 1782530315.702597 mapread 0x02f139 thru 0x031223 (0x20eb bytes) 000459 1782530315.705262 mapread 0x003a98 thru 0x00f1b8 (0xb721 bytes) 000460 1782530315.708866 trunc from 0x00039f to 0x00abac (0xa80e bytes) 000461 1782530315.715953 mapread 0x033b5f thru 0x03cd60 (0x9202 bytes) 000462 1782530315.721861 mapwrite 0x005bd5 thru 0x009979 (0x3da5 bytes) 000463 1782530315.762864 trunc from 0x03f02b to 0x033b02 (0xb528 bytes) 000464 1782530315.776600 mapread 0x0119c1 thru 0x01dfa6 (0xc5e6 bytes) 000465 1782530315.779225 punch from 0x01eb5f to 0x02b41f (0xc8c1 bytes) 000466 1782530315.786151 punch from 0x00ac9b to 0x00e9f1 (0x3d57 bytes) 000467 1782530315.792297 punch from 0x02243d to 0x02d509 (0xb0cd bytes) 000468 1782530315.800475 mapwrite 0x01f928 thru 0x028f04 (0x95dd bytes) 000469 1782530315.864932 trunc from 0x033b03 to 0x00f950 (0x241b2 bytes) 000470 1782530315.882133 mapwrite 0x030d9e thru 0x038f9e (0x8201 bytes) 000471 1782530315.960103 mapwrite 0x006202 thru 0x008f29 (0x2d28 bytes) 000472 1782530315.972696 mapwrite 0x032416 thru 0x03ffff (0xdbea bytes) 000473 1782530316.038739 punch from 0x01fc3e to 0x02f96a (0xfd2d bytes) 000474 1782530316.043424 mapwrite 0x031194 thru 0x032eb9 (0x1d26 bytes) 000475 1782530316.059996 fallocate 0x01c6cf thru 0x01e1a4 (0x1ad6 bytes) 000476 1782530316.070082 fallocate 0x01ed66 thru 0x02ebd5 (0xfe70 bytes) 000477 1782530316.076197 read 0x0098cc thru 0x010d8b (0x74c0 bytes) 000478 1782530316.091216 trunc from 0x040000 to 0x03ee80 (0x117f bytes) 000480 1782530316.114089 MIRROR_EXTEND to 2 mirrors 000481 1782530316.714132 trunc from 0x012f20 to 0x0148a0 (0x1981 bytes) 000482 1782530316.763038 MIRROR_RESYNC 2 mirrors 000484 1782530317.323061 trunc from 0x03ee81 to 0x00ac7f (0x34201 bytes) 000485 1782530317.366276 mapread 0x0025ce thru 0x007859 (0x528c bytes) 000486 1782530317.388055 mapread 0x004fa3 thru 0x00ac7f (0x5cdd bytes) 000487 1782530317.389306 trunc from 0x003d5f to 0x00ac7f (0x6f21 bytes) 000488 1782530317.399282 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000489 1782530317.582253 MIRROR_EXTEND to 2 mirrors 000490 1782530318.377431 trunc from 0x0017cf to 0x0022e9 (0xb1b bytes) 000491 1782530318.416429 write 0x0315ff thru 0x036cf9 (0x56fb bytes) 000492 1782530318.427085 trunc from 0x021f0a to 0x030109 (0xe200 bytes) 000493 1782530318.433554 read 0x025e8a thru 0x02a8a6 (0x4a1d bytes) 000494 1782530318.445326 trunc from 0x036cfa to 0x029707 (0xd5f2 bytes) 000495 1782530318.466463 trunc from 0x029708 to 0x0323fa (0x8cf3 bytes) 000496 1782530318.486339 mapwrite 0x01ddbf thru 0x025f55 (0x8197 bytes) 000497 1782530318.544871 mapread 0x029742 thru 0x0323fa (0x8cb9 bytes) 000498 1782530318.577281 trunc from 0x0323fb to 0x023818 (0xebe2 bytes) 000499 1782530318.602557 mapwrite 0x00cec5 thru 0x00edf6 (0x1f32 bytes) 000500 1782530318.639572 read 0x023274 thru 0x023818 (0x5a5 bytes) 000501 1782530318.640308 trunc from 0x023819 to 0x006ef1 (0x1c927 bytes) 000502 1782530318.673295 read 0x006d33 thru 0x006ef1 (0x1bf bytes) 000503 1782530318.685621 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x6ef2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6ef2): Mirror 2: 0xbde2f949 (stale) CRC-32 checksum value for chunk [0, 0x6ef2): Mirror 1: 0xbde2f949 Mirror 3: 0xbde2f949 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6ef2: skipped 000504 1782530319.639303 mapread 0x005617 thru 0x006ef1 (0x18db bytes) 000505 1782530319.677599 fallocate 0x00b8eb thru 0x019191 (0xd8a7 bytes) 000506 1782530319.720068 mapread 0x0067bf thru 0x006ef1 (0x733 bytes) 000508 1782530319.731782 punch from 0x0068fa to 0x006ef1 (0x5f8 bytes) 000509 1782530319.741304 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x6ef2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6ef2): Mirror 2: 0xbde2f949 (stale) Mirror 3: 0xbde2f949 (stale) CRC-32 checksum value for chunk [0, 0x6ef2): Mirror 1: 0xbde2f949 Mirror 4: 0xbde2f949 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6ef2: skipped 000510 1782530320.671600 fallocate 0x03da4c thru 0x03ffff (0x25b4 bytes) 000511 1782530320.735985 mapread 0x0021a2 thru 0x006ef1 (0x4d50 bytes) 000512 1782530320.760625 write 0x031605 thru 0x03f131 (0xdb2d bytes) 000513 1782530320.767078 punch from 0x02a2f6 to 0x0387c2 (0xe4cd bytes) 000514 1782530320.783814 punch from 0x02e27c to 0x0389ba (0xa73f bytes) 000515 1782530320.793416 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f132): Mirror 3: 0xac2249a6 (stale) Mirror 4: 0xac2249a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f132): Mirror 3: 0xac2249a6 (stale) Mirror 4: 0xac2249a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f132: skipped 000517 1782530321.122407 read 0x03b5c9 thru 0x03ecfe (0x3736 bytes) 000518 1782530321.130044 fallocate 0x008e1c thru 0x012aef (0x9cd4 bytes) 000519 1782530321.141301 read 0x005b17 thru 0x00e57d (0x8a67 bytes) 000521 1782530321.156841 trunc from 0x03f132 to 0x027ce9 (0x17448 bytes) 000522 1782530321.203245 trunc from 0x001eb1 to 0x008b69 (0x6cb9 bytes) 000523 1782530321.216272 trunc from 0x027cea to 0x02827d (0x594 bytes) 000524 1782530321.245309 trunc from 0x01e001 to 0x021449 (0x3449 bytes) 000525 1782530321.265092 read 0x016b42 thru 0x01e5a8 (0x7a67 bytes) 000526 1782530321.287427 trunc from 0x02827e to 0x01710a (0x11173 bytes) 000527 1782530321.317308 trunc from 0x01710b to 0x02454a (0xd440 bytes) 000528 1782530321.338359 write 0x00a898 thru 0x016d82 (0xc4eb bytes) 000529 1782530321.341987 read 0x020ae1 thru 0x02454a (0x3a6a bytes) 000530 1782530321.357825 punch from 0x01f924 to 0x02454a (0x4c27 bytes) 000531 1782530321.366590 fallocate 0x0374fa thru 0x038e98 (0x199f bytes) 000532 1782530321.379826 fallocate 0x027847 thru 0x036f43 (0xf6fd bytes) 000533 1782530321.393874 fallocate 0x00524c thru 0x012d1d (0xdad2 bytes) 000534 1782530321.400222 fallocate 0x0073b8 thru 0x00ec67 (0x78b0 bytes) 000535 1782530321.427033 trunc from 0x02454b to 0x02fc7c (0xb732 bytes) 000537 1782530321.464276 fallocate 0x034a1b thru 0x03e9c4 (0x9faa bytes) 000538 1782530321.475168 write 0x0224ac thru 0x02b5b7 (0x910c bytes) 000539 1782530321.508094 trunc from 0x02fc7d to 0x00e75e (0x2151e bytes) 000540 1782530321.534520 mapwrite 0x007eaf thru 0x008518 (0x66a bytes) 000541 1782530321.553015 read 0x0031ee thru 0x00e75e (0xb571 bytes) 000542 1782530321.558389 MIRROR_RESYNC 3 mirrors 000543 1782530322.233323 trunc from 0x00e75f to 0x015b7c (0x741e bytes) 000544 1782530322.305741 trunc from 0x00e90a to 0x0108b9 (0x1fb0 bytes) 000545 1782530322.311021 write 0x01e4b4 thru 0x026b44 (0x8691 bytes) 000547 1782530322.313203 trunc from 0x026b45 to 0x01dc0e (0x8f36 bytes) 000548 1782530322.337432 trunc from 0x01dc0f to 0x00546c (0x187a2 bytes) 000549 1782530322.356868 mapread 0x00318f thru 0x00546c (0x22de bytes) 000550 1782530322.382152 punch from 0x003b05 to 0x00546c (0x1968 bytes) 000551 1782530322.387278 trunc from 0x0013e0 to 0x00546c (0x408d bytes) 000552 1782530322.399702 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x546d): Mirror 4: 0x8730b824 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x546d): Mirror 4: 0x8730b824 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x546d: skipped 000553 1782530322.689130 trunc from 0x004f6b to 0x00546c (0x502 bytes) 000554 1782530322.709048 trunc from 0x00319f to 0x00546c (0x22ce bytes) 000555 1782530322.714072 trunc from 0x001905 to 0x00546c (0x3b68 bytes) 000556 1782530322.734246 fallocate 0x016156 thru 0x01f45d (0x9308 bytes) 000557 1782530322.740006 read 0x00d39f thru 0x00d97d (0x5df bytes) 000558 1782530322.745501 mapread 0x00efda thru 0x01782a (0x8851 bytes) 000559 1782530322.779018 trunc from 0x003f0a to 0x010d5f (0xce56 bytes) 000561 1782530322.787180 mapwrite 0x008a78 thru 0x014a83 (0xc00c bytes) 000562 1782530322.832311 read 0x011eac thru 0x014742 (0x2897 bytes) 000563 1782530322.833198 mapread 0x019306 thru 0x01f45d (0x6158 bytes) 000564 1782530322.844549 trunc from 0x0174b2 to 0x01f45d (0x7fac bytes) 000565 1782530322.854047 mapread 0x001ae7 thru 0x006918 (0x4e32 bytes) 000567 1782530322.859324 fallocate 0x025328 thru 0x02be67 (0x6b40 bytes) 000568 1782530322.867858 write 0x017a82 thru 0x022795 (0xad14 bytes) 000569 1782530322.884986 punch from 0x018bd9 to 0x01affc (0x2424 bytes) 000570 1782530322.918485 mapread 0x011b84 thru 0x01a7a5 (0x8c22 bytes) 000571 1782530322.934637 fallocate 0x0283c8 thru 0x02c6c9 (0x4302 bytes) 000572 1782530322.946410 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000573 1782530323.116104 mapread 0x01527a thru 0x021fde (0xcd65 bytes) 000575 1782530323.169068 trunc from 0x022796 to 0x024b71 (0x23dc bytes) 000576 1782530323.212627 mapread 0x0014e7 thru 0x001861 (0x37b bytes) 000577 1782530323.235981 fallocate 0x00a730 thru 0x01121d (0x6aee bytes) 000579 1782530323.256734 trunc from 0x00d391 to 0x013ffa (0x6c6a bytes) 000580 1782530323.273247 trunc from 0x009a7d to 0x00ca19 (0x2f9d bytes) 000581 1782530323.287164 write 0x00c9a2 thru 0x014490 (0x7aef bytes) 000582 1782530323.295016 write 0x01d4f3 thru 0x02ce7b (0xf989 bytes) 000583 1782530323.304144 trunc from 0x0298db to 0x02b039 (0x175f bytes) 000584 1782530323.332839 read 0x003a87 thru 0x004b64 (0x10de bytes) 000585 1782530323.333681 write 0x03c76a thru 0x03ffff (0x3896 bytes) 000586 1782530323.348859 fallocate 0x00893c thru 0x013c5e (0xb323 bytes) 000587 1782530323.365902 read 0x0244dc thru 0x030229 (0xbd4e bytes) 000588 1782530323.387710 punch from 0x03aaeb to 0x03c74b (0x1c61 bytes) 000589 1782530323.425977 trunc from 0x040000 to 0x032432 (0xdbcd bytes) 000590 1782530323.454956 trunc from 0x032433 to 0x03bb8f (0x975d bytes) 000592 1782530323.491958 trunc from 0x03618d to 0x03bb8f (0x5a03 bytes) 000594 1782530323.507093 mapwrite 0x00e93c thru 0x01a8ef (0xbfb4 bytes) 000595 1782530323.605157 read 0x02ed53 thru 0x0398b9 (0xab67 bytes) 000596 1782530323.617396 trunc from 0x034c0c to 0x0357a4 (0xb99 bytes) 000599 1782530323.627869 trunc from 0x01f393 to 0x023e81 (0x4aef bytes) 000600 1782530323.633329 mapread 0x0286b7 thru 0x02e841 (0x618b bytes) 000602 1782530323.635758 write 0x03075c thru 0x03677f (0x6024 bytes) 000603 1782530323.639254 fallocate 0x00fe8f thru 0x010176 (0x2e8 bytes) 000604 1782530323.646761 read 0x004d83 thru 0x005e3a (0x10b8 bytes) 000605 1782530323.648087 read 0x00dfa6 thru 0x017be9 (0x9c44 bytes) 000606 1782530323.651019 fallocate 0x02fbe6 thru 0x039600 (0x9a1b bytes) 000608 1782530323.663176 write 0x015d4d thru 0x01895e (0x2c12 bytes) 000610 1782530323.666160 punch from 0x02d72c to 0x02f62d (0x1f02 bytes) 000611 1782530323.671348 fallocate 0x000fec thru 0x00b590 (0xa5a5 bytes) 000612 1782530323.678855 trunc from 0x03bb90 to 0x03518a (0x6a05 bytes) 000613 1782530323.704112 mapread 0x013365 thru 0x01cab8 (0x9754 bytes) 000614 1782530323.706206 write 0x028c99 thru 0x02c8ad (0x3c15 bytes) 000615 1782530323.709223 mapwrite 0x039537 thru 0x03ffff (0x6ac9 bytes) 000616 1782530323.770685 read 0x03a212 thru 0x03bbe5 (0x19d4 bytes) 000617 1782530323.771377 mapwrite 0x016a4d thru 0x0190fd (0x26b1 bytes) 000618 1782530323.789056 read 0x020159 thru 0x022c10 (0x2ab8 bytes) 000619 1782530323.794016 MIRROR_EXTEND to 2 mirrors 000620 1782530324.684852 MIRROR_RESYNC 2 mirrors 000621 1782530325.006950 MIRROR_EXTEND to 3 mirrors 000622 1782530325.696787 mapread 0x035220 thru 0x03813f (0x2f20 bytes) 000623 1782530325.711426 mapwrite 0x003692 thru 0x005bff (0x256e bytes) 000624 1782530325.796850 fallocate 0x00e0d1 thru 0x014458 (0x6388 bytes) 000625 1782530325.802783 mapread 0x01bb1c thru 0x01c223 (0x708 bytes) 000626 1782530325.816100 fallocate 0x01cc09 thru 0x022ac7 (0x5ebf bytes) 000627 1782530325.828412 punch from 0x026ad5 to 0x032030 (0xb55c bytes) 000628 1782530325.836258 read 0x00675e thru 0x008923 (0x21c6 bytes) 000629 1782530325.837467 trunc from 0x040000 to 0x034949 (0xb6b6 bytes) 000630 1782530325.864237 mapwrite 0x03fc04 thru 0x03ffff (0x3fc bytes) 000631 1782530325.903675 mapwrite 0x0241b4 thru 0x02707a (0x2ec7 bytes) 000632 1782530325.936536 punch from 0x01fda5 to 0x023f17 (0x4173 bytes) 000633 1782530325.940534 trunc from 0x040000 to 0x02d90a (0x126f5 bytes) 000634 1782530325.965103 mapwrite 0x022c50 thru 0x028cb0 (0x6061 bytes) 000635 1782530325.997181 trunc from 0x02d90b to 0x031fe0 (0x46d6 bytes) 000636 1782530326.011288 write 0x00c619 thru 0x014115 (0x7afd bytes) 000637 1782530326.012834 mapread 0x00315d thru 0x007706 (0x45aa bytes) 000639 1782530326.016767 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31fe1): Mirror 2: 0x3a25d1eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31fe1): Mirror 2: 0x3a25d1eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31fe1: skipped 000640 1782530326.421998 read 0x00e098 thru 0x018544 (0xa4ad bytes) 000641 1782530326.432865 mapread 0x017f48 thru 0x02026b (0x8324 bytes) 000642 1782530326.452064 trunc from 0x02355e to 0x02dbed (0xa690 bytes) 000643 1782530326.458745 punch from 0x0196fb to 0x01a6cb (0xfd1 bytes) 000644 1782530326.464740 mapread 0x01a7a6 thru 0x0268ef (0xc14a bytes) 000645 1782530326.486823 trunc from 0x0017ca to 0x00a0d3 (0x890a bytes) 000646 1782530326.492021 mapwrite 0x02709b thru 0x028543 (0x14a9 bytes) 000647 1782530326.510276 mapwrite 0x03192b thru 0x032002 (0x6d8 bytes) 000649 1782530326.617892 trunc from 0x032003 to 0x00918c (0x28e76 bytes) 000650 1782530326.651127 fallocate 0x006681 thru 0x00c2d1 (0x5c51 bytes) 000651 1782530326.658162 read 0x00714f thru 0x00918c (0x203e bytes) 000652 1782530326.671508 mapwrite 0x00eac5 thru 0x00fa59 (0xf95 bytes) 000653 1782530326.748405 trunc from 0x00fa5a to 0x027d26 (0x182cd bytes) 000655 1782530326.788019 trunc from 0x027d27 to 0x02ddba (0x6094 bytes) 000656 1782530326.828781 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000657 1782530327.060139 punch from 0x01e915 to 0x0250cf (0x67bb bytes) 000658 1782530327.074823 trunc from 0x02ddbb to 0x03202f (0x4275 bytes) 000659 1782530327.138308 trunc from 0x032030 to 0x02df76 (0x40b9 bytes) 000660 1782530327.192690 mapwrite 0x013e7d thru 0x01c2ed (0x8471 bytes) 000661 1782530327.278266 read 0x00b281 thru 0x016060 (0xade0 bytes) 000663 1782530327.300755 mapread 0x024134 thru 0x02df76 (0x9e43 bytes) 000664 1782530327.320767 punch from 0x014357 to 0x023a39 (0xf6e3 bytes) 000665 1782530327.327171 trunc from 0x00c09a to 0x00ee95 (0x2dfc bytes) 000666 1782530327.338000 mapwrite 0x038eb2 thru 0x03ffff (0x714e bytes) 000667 1782530327.419380 trunc from 0x040000 to 0x00685d (0x397a2 bytes) 000668 1782530327.446262 trunc from 0x00685e to 0x03a804 (0x33fa7 bytes) 000669 1782530327.473142 fallocate 0x002e20 thru 0x00efb0 (0xc191 bytes) 000670 1782530327.478777 fallocate 0x0008ff thru 0x00d25d (0xc95f bytes) 000671 1782530327.485745 mapread 0x0132e6 thru 0x022181 (0xee9c bytes) 000672 1782530327.516240 mapwrite 0x036661 thru 0x03d341 (0x6ce1 bytes) 000673 1782530327.606065 mapread 0x01195b thru 0x018416 (0x6abc bytes) 000675 1782530327.607859 read 0x01c224 thru 0x02b70c (0xf4e9 bytes) 000676 1782530327.624131 write 0x03dc71 thru 0x03ffff (0x238f bytes) 000677 1782530327.625638 trunc from 0x040000 to 0x03df6d (0x2092 bytes) 000678 1782530327.645525 mapwrite 0x00880a thru 0x01386c (0xb063 bytes) 000680 1782530327.678948 read 0x018e67 thru 0x01b2c7 (0x2461 bytes) 000681 1782530327.683211 trunc from 0x03df6e to 0x0135f8 (0x2a975 bytes) 000684 1782530327.747443 mapwrite 0x008ffc thru 0x0169ce (0xd9d3 bytes) 000685 1782530327.828036 write 0x023fd1 thru 0x030407 (0xc437 bytes) 000686 1782530327.833325 punch from 0x02d385 to 0x02dcb3 (0x92f bytes) 000688 1782530327.886048 read 0x001d2c thru 0x01084d (0xeb22 bytes) 000689 1782530327.886796 read 0x02f1c1 thru 0x030407 (0x1247 bytes) 000690 1782530327.898702 fallocate 0x029e91 thru 0x02aa2c (0xb9c bytes) 000691 1782530327.905097 mapwrite 0x01efb6 thru 0x021b26 (0x2b71 bytes) 000692 1782530327.934557 punch from 0x005783 to 0x006448 (0xcc6 bytes) 000693 1782530327.942436 mapread 0x0256e8 thru 0x0290ec (0x3a05 bytes) 000694 1782530327.943536 write 0x03dbc4 thru 0x03f79b (0x1bd8 bytes) 000695 1782530327.946360 trunc from 0x034939 to 0x036f51 (0x2619 bytes) 000696 1782530327.951947 write 0x0337f1 thru 0x03ffff (0xc80f bytes) 000697 1782530327.965288 trunc from 0x0147fd to 0x023f34 (0xf738 bytes) 000698 1782530327.980728 read 0x002bf9 thru 0x011624 (0xea2c bytes) 000699 1782530327.991464 mapread 0x01ea19 thru 0x01f251 (0x839 bytes) 000700 1782530327.992518 trunc from 0x040000 to 0x02a6f2 (0x1590d bytes) 000701 1782530328.009611 fallocate 0x033379 thru 0x034842 (0x14ca bytes) 000702 1782530328.014167 fallocate 0x016c93 thru 0x018c16 (0x1f84 bytes) 000703 1782530328.019473 trunc from 0x02a6f3 to 0x018c2c (0x11ac6 bytes) 000704 1782530328.036863 write 0x01a80d thru 0x024674 (0x9e68 bytes) 000705 1782530328.039189 trunc from 0x011a89 to 0x012383 (0x8fb bytes) 000706 1782530328.049277 punch from 0x01f461 to 0x01fcbd (0x85d bytes) 000707 1782530328.066338 write 0x025d05 thru 0x0319d4 (0xbcd0 bytes) 000708 1782530328.068004 trunc from 0x0319d5 to 0x00ff5e (0x21a76 bytes) 000709 1782530328.102755 MIRROR_EXTEND to 2 mirrors 000710 1782530328.956084 punch from 0x00e287 to 0x00ff5e (0x1cd8 bytes) 000711 1782530328.984670 trunc from 0x00682d to 0x00d2bd (0x6a91 bytes) 000712 1782530328.988776 read 0x005b3d thru 0x00ff5e (0xa422 bytes) 000713 1782530328.995068 write 0x00160c thru 0x006d55 (0x574a bytes) 000715 1782530329.002003 mapwrite 0x026294 thru 0x02a900 (0x466d bytes) 000716 1782530329.047602 MIRROR_RESYNC 2 mirrors 000717 1782530329.698826 mapwrite 0x026abc thru 0x02d172 (0x66b7 bytes) 000718 1782530329.891150 trunc from 0x02d173 to 0x03ff62 (0x12df0 bytes) 000719 1782530329.944067 fallocate 0x014ca2 thru 0x01cdf1 (0x8150 bytes) 000720 1782530329.961006 punch from 0x03525e to 0x037132 (0x1ed5 bytes) 000722 1782530329.979365 fallocate 0x032848 thru 0x03ffff (0xd7b8 bytes) 000723 1782530329.987944 trunc from 0x02cb97 to 0x03c8dd (0xfd47 bytes) 000724 1782530330.009842 mapwrite 0x01fa44 thru 0x022a11 (0x2fce bytes) 000725 1782530330.061033 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ff63): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff63): Mirror 2: 0xab71fe26 (stale) CRC-32 checksum value for chunk [0, 0x3ff63): Mirror 1: 0xc6a945b7 Mirror 3: 0xc6a945b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff63: skipped 000726 1782530331.190444 mapwrite 0x01b89e thru 0x01d868 (0x1fcb bytes) 000727 1782530331.283270 mapread 0x0306fa thru 0x03cae6 (0xc3ed bytes) 000728 1782530331.301309 mapread 0x02eb22 thru 0x0351c1 (0x66a0 bytes) 000729 1782530331.302604 write 0x0246ad thru 0x03254d (0xdea1 bytes) 000730 1782530331.309237 trunc from 0x03ff63 to 0x025f83 (0x19fdf bytes) 000731 1782530331.350810 write 0x01d8f9 thru 0x021c20 (0x4328 bytes) 000732 1782530331.352293 mapwrite 0x018977 thru 0x01c714 (0x3d9e bytes) 000733 1782530331.372185 write 0x0132b1 thru 0x02038b (0xd0db bytes) 000734 1782530331.378875 fallocate 0x0170c6 thru 0x01dbe5 (0x6b20 bytes) 000735 1782530331.385966 write 0x023416 thru 0x02849c (0x5087 bytes) 000736 1782530331.387312 mapwrite 0x00bfc0 thru 0x00c1f2 (0x233 bytes) 000737 1782530331.420041 read 0x009411 thru 0x00e163 (0x4d53 bytes) 000738 1782530331.422455 read 0x00f41e thru 0x015524 (0x6107 bytes) 000740 1782530331.424733 read 0x00ccae thru 0x0106f5 (0x3a48 bytes) 000742 1782530331.433538 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2849d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2849d): Mirror 2: 0x8f0c68f (stale) Mirror 3: 0x1a95e897 (stale) CRC-32 checksum value for chunk [0, 0x2849d): Mirror 1: 0x2dba68fc Mirror 4: 0x2dba68fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2849d: skipped 000745 1782530332.491739 read 0x012d39 thru 0x0214f3 (0xe7bb bytes) 000746 1782530332.534413 punch from 0x01496f to 0x01b4f6 (0x6b88 bytes) 000748 1782530332.635331 write 0x0118cf thru 0x017cdc (0x640e bytes) 000749 1782530332.662796 trunc from 0x023e8a to 0x02849c (0x4613 bytes) 000750 1782530332.670630 fallocate 0x01bbad thru 0x02766e (0xbac2 bytes) 000751 1782530332.686283 write 0x01a7ca thru 0x01f516 (0x4d4d bytes) 000752 1782530332.702237 write 0x028616 thru 0x029282 (0xc6d bytes) 000753 1782530332.712819 write 0x02de1e thru 0x0323ed (0x45d0 bytes) 000754 1782530332.716809 fallocate 0x03e39b thru 0x03f46b (0x10d1 bytes) 000755 1782530332.728236 trunc from 0x03f46c to 0x028402 (0x17069 bytes) 000756 1782530332.782314 write 0x00c895 thru 0x00f1c8 (0x2934 bytes) 000757 1782530332.802678 MIRROR_RESYNC 4 mirrors 000758 1782530334.334877 write 0x0223b3 thru 0x02f213 (0xce61 bytes) 000760 1782530334.374597 mapwrite 0x023376 thru 0x029d11 (0x699c bytes) 000761 1782530334.404879 mapread 0x02b302 thru 0x02d5e4 (0x22e3 bytes) 000762 1782530334.406249 trunc from 0x01f27a to 0x01f41f (0x1a6 bytes) 000763 1782530334.416103 read 0x00c5ff thru 0x0165c4 (0x9fc6 bytes) 000764 1782530334.426307 write 0x0069ee thru 0x009089 (0x269c bytes) 000765 1782530334.439104 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f214): Mirror 2: 0x5368b49f (stale) Mirror 4: 0x5368b49f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f214): Mirror 2: 0x5368b49f (stale) Mirror 4: 0x5368b49f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f214: skipped 000766 1782530334.896257 fallocate 0x0025c5 thru 0x005a9f (0x34db bytes) 000767 1782530334.906751 MIRROR_RESYNC 3 mirrors 000768 1782530335.696205 MIRROR_RESYNC 3 mirrors 000769 1782530336.095113 mapwrite 0x0226d5 thru 0x027244 (0x4b70 bytes) 000770 1782530336.205840 mapread 0x001107 thru 0x0036f5 (0x25ef bytes) 000771 1782530336.222751 mapread 0x01d5ca thru 0x02bc2c (0xe663 bytes) 000772 1782530336.225838 write 0x00437e thru 0x007e69 (0x3aec bytes) 000773 1782530336.229206 read 0x01b7ed thru 0x02393a (0x814e bytes) 000774 1782530336.244192 fallocate 0x00f0ac thru 0x0169ff (0x7954 bytes) 000775 1782530336.253248 trunc from 0x007fb6 to 0x00a982 (0x29cd bytes) 000776 1782530336.281257 read 0x010b79 thru 0x01aa9f (0x9f27 bytes) 000777 1782530336.301954 read 0x00722c thru 0x00fbdb (0x89b0 bytes) 000778 1782530336.304555 mapread 0x02da1b thru 0x02f213 (0x17f9 bytes) 000779 1782530336.306786 mapread 0x024396 thru 0x02c04d (0x7cb8 bytes) 000780 1782530336.308101 write 0x03dafc thru 0x03ffff (0x2504 bytes) 000781 1782530336.311174 write 0x021977 thru 0x02585e (0x3ee8 bytes) 000782 1782530336.313807 mapread 0x03d4b5 thru 0x03ffff (0x2b4b bytes) 000783 1782530336.315847 trunc from 0x040000 to 0x0343ed (0xbc12 bytes) 000784 1782530336.345853 fallocate 0x00fe53 thru 0x01ba65 (0xbc13 bytes) 000785 1782530336.352257 trunc from 0x0343ee to 0x01a9e2 (0x19a0b bytes) 000786 1782530336.376730 trunc from 0x01a9e3 to 0x0042ec (0x166f6 bytes) 000788 1782530336.410624 trunc from 0x0042ed to 0x00165a (0x2c92 bytes) 000789 1782530336.441856 trunc from 0x00165b to 0x0248cb (0x23271 bytes) 000790 1782530336.473095 trunc from 0x0248cc to 0x00a4d4 (0x1a3f7 bytes) 000791 1782530336.526701 trunc from 0x00a4d5 to 0x01d102 (0x12c2e bytes) 000792 1782530336.573211 fallocate 0x0164ed thru 0x021769 (0xb27d bytes) 000793 1782530336.580875 trunc from 0x02176a to 0x0099b7 (0x17db2 bytes) 000794 1782530336.616948 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x99b8): Mirror 2: 0x1c807aff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x99b8): Mirror 2: 0x1c807aff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x99b8: skipped 000795 1782530336.981280 fallocate 0x00c310 thru 0x01b95e (0xf64f bytes) 000796 1782530336.992856 MIRROR_RESYNC 2 mirrors 000798 1782530337.508597 mapwrite 0x027d2b thru 0x037ce5 (0xffbb bytes) 000799 1782530337.625002 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000800 1782530337.903241 trunc from 0x02a080 to 0x02e4d6 (0x4457 bytes) 000801 1782530337.909518 write 0x00572b thru 0x00705c (0x1932 bytes) 000802 1782530337.926848 mapwrite 0x025a90 thru 0x02d405 (0x7976 bytes) 000803 1782530337.979026 mapread 0x027917 thru 0x035997 (0xe081 bytes) 000804 1782530337.998994 write 0x03415e thru 0x03e567 (0xa40a bytes) 000805 1782530338.023014 MIRROR_EXTEND to 2 mirrors 000806 1782530338.761877 trunc from 0x03e568 to 0x03dc1d (0x94a bytes) 000807 1782530338.807659 fallocate 0x00333f thru 0x01014a (0xce0c bytes) 000808 1782530338.816114 MIRROR_RESYNC 2 mirrors 000809 1782530339.403030 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000810 1782530339.550875 write 0x0112e9 thru 0x02000d (0xed25 bytes) 000811 1782530339.579726 mapwrite 0x008a60 thru 0x00afce (0x256f bytes) 000812 1782530339.614143 punch from 0x018d7d to 0x01d13e (0x43c2 bytes) 000813 1782530339.629058 write 0x026c8c thru 0x0303ea (0x975f bytes) 000814 1782530339.656897 punch from 0x032d01 to 0x03a417 (0x7717 bytes) 000815 1782530339.662830 trunc from 0x03dc1e to 0x032d13 (0xaf0a bytes) 000817 1782530339.684069 write 0x010211 thru 0x013509 (0x32f9 bytes) 000818 1782530339.685664 write 0x03297e thru 0x03625f (0x38e2 bytes) 000820 1782530339.696271 write 0x01e447 thru 0x023369 (0x4f23 bytes) 000821 1782530339.708553 trunc from 0x03385e to 0x03625f (0x2a02 bytes) 000822 1782530339.736798 write 0x005373 thru 0x008843 (0x34d1 bytes) 000823 1782530339.738720 mapwrite 0x03147b thru 0x03ffff (0xeb85 bytes) 000824 1782530339.805290 MIRROR_EXTEND to 2 mirrors 000826 1782530340.507617 trunc from 0x040000 to 0x007397 (0x38c68 bytes) 000827 1782530340.551160 punch from 0x003ce1 to 0x007397 (0x36b7 bytes) 000828 1782530340.557688 mapwrite 0x026d4d thru 0x0354ed (0xe7a1 bytes) 000829 1782530340.631218 trunc from 0x022412 to 0x02e5eb (0xc1da bytes) 000830 1782530340.638785 mapread 0x03180f thru 0x0354ed (0x3cdf bytes) 000831 1782530340.646588 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000832 1782530340.792122 punch from 0x020dfb to 0x02aaa8 (0x9cae bytes) 000833 1782530340.797428 mapread 0x00b958 thru 0x0152c9 (0x9972 bytes) 000834 1782530340.820104 write 0x02f09a thru 0x037659 (0x85c0 bytes) 000835 1782530340.831079 trunc from 0x022c3c to 0x0327bb (0xfb80 bytes) 000836 1782530340.847128 fallocate 0x037e4e thru 0x03e6ff (0x68b2 bytes) 000837 1782530340.858210 mapwrite 0x036bf3 thru 0x0392a5 (0x26b3 bytes) 000838 1782530340.896580 punch from 0x0077d7 to 0x016d3a (0xf564 bytes) 000839 1782530340.901743 mapwrite 0x005278 thru 0x007307 (0x2090 bytes) 000840 1782530340.943553 fallocate 0x0363fa thru 0x03989a (0x34a1 bytes) 000841 1782530340.965065 read 0x01f4cf thru 0x026fef (0x7b21 bytes) 000842 1782530340.983168 punch from 0x033b3f to 0x03da46 (0x9f08 bytes) 000843 1782530341.003304 write 0x02a9f7 thru 0x03715f (0xc769 bytes) 000844 1782530341.032127 trunc from 0x03e700 to 0x01295a (0x2bda5 bytes) 000845 1782530341.068211 trunc from 0x01295b to 0x03a534 (0x27bda bytes) 000846 1782530341.097540 trunc from 0x03a535 to 0x03f494 (0x4f60 bytes) 000847 1782530341.112773 read 0x019f21 thru 0x01fa94 (0x5b74 bytes) 000848 1782530341.143532 mapread 0x00cd4c thru 0x0148bb (0x7b70 bytes) 000849 1782530341.144797 trunc from 0x00b279 to 0x01846d (0xd1f5 bytes) 000851 1782530341.153613 trunc from 0x006ccf to 0x012196 (0xb4c8 bytes) 000852 1782530341.159155 trunc from 0x0055cf to 0x00b951 (0x6383 bytes) 000853 1782530341.170354 punch from 0x0070d3 to 0x0102bc (0x91ea bytes) 000854 1782530341.177356 read 0x0013b4 thru 0x00768b (0x62d8 bytes) 000855 1782530341.200300 punch from 0x0067fc to 0x00dee1 (0x76e6 bytes) 000856 1782530341.207189 mapread 0x030187 thru 0x0385b5 (0x842f bytes) 000858 1782530341.213121 punch from 0x012592 to 0x01d465 (0xaed4 bytes) 000859 1782530341.219630 trunc from 0x03f495 to 0x006d87 (0x3870d bytes) 000860 1782530341.241653 fallocate 0x02e5b9 thru 0x03c449 (0xde91 bytes) 000862 1782530341.250117 mapread 0x001cd8 thru 0x006d87 (0x50b0 bytes) 000863 1782530341.256431 trunc from 0x0002dd to 0x006d87 (0x6aab bytes) 000864 1782530341.262499 mapread 0x002880 thru 0x006d87 (0x4508 bytes) 000865 1782530341.294569 trunc from 0x006d88 to 0x016846 (0xfabf bytes) 000866 1782530341.318041 write 0x03476b thru 0x03cf74 (0x880a bytes) 000867 1782530341.319549 read 0x001b08 thru 0x0103b4 (0xe8ad bytes) 000868 1782530341.330838 fallocate 0x000b65 thru 0x000b89 (0x25 bytes) 000869 1782530341.338236 punch from 0x017bfd to 0x024cb0 (0xd0b4 bytes) 000870 1782530341.346163 MIRROR_EXTEND to 2 mirrors 000871 1782530341.859617 write 0x03900c thru 0x03ffff (0x6ff4 bytes) 000872 1782530341.906069 mapwrite 0x02d61f thru 0x036ba3 (0x9585 bytes) 000873 1782530341.949768 fallocate 0x027dab thru 0x02b040 (0x3296 bytes) 000874 1782530341.955182 trunc from 0x032611 to 0x03d89f (0xb28f bytes) 000875 1782530341.961472 mapread 0x029c34 thru 0x036deb (0xd1b8 bytes) 000876 1782530341.971493 punch from 0x02ab55 to 0x02b3eb (0x897 bytes) 000877 1782530341.975843 trunc from 0x040000 to 0x02bb93 (0x1446c bytes) 000878 1782530342.019124 mapread 0x00b977 thru 0x01a563 (0xebed bytes) 000879 1782530342.037000 mapwrite 0x021696 thru 0x030f11 (0xf87c bytes) 000880 1782530342.084330 fallocate 0x00551a thru 0x00dab2 (0x8599 bytes) 000881 1782530342.095632 read 0x018f3a thru 0x027431 (0xe4f8 bytes) 000882 1782530342.100755 fallocate 0x035992 thru 0x03ffff (0xa66e bytes) 000883 1782530342.109355 read 0x02412b thru 0x0273f0 (0x32c6 bytes) 000885 1782530342.110544 fallocate 0x01b01e thru 0x025a3c (0xaa1f bytes) 000887 1782530342.119591 mapwrite 0x03c1d0 thru 0x03e00f (0x1e40 bytes) 000888 1782530342.153139 MIRROR_RESYNC 2 mirrors 000889 1782530342.770003 trunc from 0x040000 to 0x00ecdd (0x31322 bytes) 000891 1782530342.853817 punch from 0x005ed9 to 0x00ecdd (0x8e05 bytes) 000892 1782530342.859468 write 0x005b1c thru 0x006e67 (0x134c bytes) 000893 1782530342.870549 read 0x009bfe thru 0x00ecdd (0x50e0 bytes) 000894 1782530342.880024 read 0x006f60 thru 0x00dd72 (0x6e13 bytes) 000895 1782530342.889986 trunc from 0x00797c to 0x00ecdd (0x7362 bytes) 000896 1782530342.895201 mapwrite 0x01a20a thru 0x01ef2f (0x4d26 bytes) 000897 1782530342.944497 fallocate 0x00d5aa thru 0x019dac (0xc803 bytes) 000898 1782530342.949641 read 0x019fa1 thru 0x01ef2f (0x4f8f bytes) 000899 1782530342.950954 write 0x00137f thru 0x00caff (0xb781 bytes) 000900 1782530342.959173 punch from 0x0162fc to 0x01dcc1 (0x79c6 bytes) 000901 1782530342.965123 trunc from 0x003c7d to 0x0084b3 (0x4837 bytes) 000902 1782530342.981048 trunc from 0x01ef30 to 0x02030d (0x13de bytes) 000903 1782530343.000632 mapread 0x004cf2 thru 0x00fdd8 (0xb0e7 bytes) 000904 1782530343.012102 read 0x01b27b thru 0x02030d (0x5093 bytes) 000905 1782530343.013884 punch from 0x008fc3 to 0x01641c (0xd45a bytes) 000906 1782530343.018953 trunc from 0x001cc9 to 0x010f38 (0xf270 bytes) 000907 1782530343.024401 trunc from 0x00a4ef to 0x00ece3 (0x47f5 bytes) 000908 1782530343.029433 trunc from 0x00ac51 to 0x010501 (0x58b1 bytes) 000909 1782530343.036069 mapwrite 0x02df5f thru 0x02f9f4 (0x1a96 bytes) 000910 1782530343.077164 read 0x005ddc thru 0x01433d (0xe562 bytes) 000911 1782530343.079072 read 0x012e7d thru 0x020863 (0xd9e7 bytes) 000912 1782530343.080903 trunc from 0x022888 to 0x02f9f4 (0xd16d bytes) 000913 1782530343.088630 mapwrite 0x03b019 thru 0x03ffff (0x4fe7 bytes) 000914 1782530343.134133 mapread 0x01c2a9 thru 0x01e0f7 (0x1e4f bytes) 000915 1782530343.136134 trunc from 0x007282 to 0x00f2e7 (0x8066 bytes) 000916 1782530343.143633 punch from 0x03392e to 0x03cabc (0x918f bytes) 000918 1782530343.150591 write 0x02cc01 thru 0x03025a (0x365a bytes) 000919 1782530343.152676 read 0x03991c thru 0x03ee89 (0x556e bytes) 000920 1782530343.175347 write 0x036783 thru 0x03aff6 (0x4874 bytes) 000921 1782530343.178340 fallocate 0x037efd thru 0x03ffff (0x8103 bytes) 000922 1782530343.184259 fallocate 0x008a2c thru 0x017e12 (0xf3e7 bytes) 000923 1782530343.191506 punch from 0x02b464 to 0x03b275 (0xfe12 bytes) 000924 1782530343.208223 punch from 0x01ce72 to 0x0249b5 (0x7b44 bytes) 000925 1782530343.215145 trunc from 0x040000 to 0x0319ef (0xe610 bytes) 000926 1782530343.247183 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) 000927 1782530343.423206 trunc from 0x0319f0 to 0x00254d (0x2f4a2 bytes) 000928 1782530343.457604 mapwrite 0x031710 thru 0x03c5dc (0xaecd bytes) 000929 1782530343.605234 punch from 0x01ef05 to 0x025aa7 (0x6ba3 bytes) 000930 1782530343.615085 read 0x032d93 thru 0x03c5dc (0x984a bytes) 000931 1782530343.618628 trunc from 0x03c5dd to 0x016784 (0x25e58 bytes) 000932 1782530343.652220 fallocate 0x0299ed thru 0x02c4af (0x2ac3 bytes) 000933 1782530343.661544 write 0x0356d5 thru 0x03a077 (0x49a3 bytes) 000934 1782530343.665213 read 0x02d3c4 thru 0x02d559 (0x196 bytes) 000935 1782530343.679136 mapread 0x0252ae thru 0x028296 (0x2fe9 bytes) 000937 1782530343.683933 write 0x03ffa3 thru 0x03ffff (0x5d bytes) 000939 1782530343.695333 MIRROR_EXTEND to 2 mirrors 000940 1782530344.521898 mapwrite 0x011507 thru 0x015ace (0x45c8 bytes) 000941 1782530344.698178 punch from 0x019504 to 0x02610b (0xcc08 bytes) 000942 1782530344.709553 trunc from 0x040000 to 0x0378be (0x8741 bytes) 000943 1782530344.761844 mapread 0x032928 thru 0x0344df (0x1bb8 bytes) 000946 1782530344.777068 write 0x023e1b thru 0x023ea9 (0x8f bytes) 000947 1782530344.784404 write 0x00e2d5 thru 0x010115 (0x1e41 bytes) 000948 1782530344.788158 trunc from 0x0378bf to 0x0241fd (0x136c1 bytes) 000949 1782530344.824451 trunc from 0x0241fe to 0x01eab2 (0x574b bytes) 000950 1782530344.862238 mapread 0x016f65 thru 0x01eab2 (0x7b4e bytes) 000951 1782530344.874066 read 0x0162a9 thru 0x01eab2 (0x880a bytes) 000952 1782530344.875521 punch from 0x00e34d to 0x01b895 (0xd549 bytes) 000953 1782530344.889273 mapread 0x00c92f thru 0x0110c5 (0x4797 bytes) 000954 1782530344.906163 mapread 0x01786c thru 0x017acf (0x264 bytes) 000955 1782530344.919467 fallocate 0x000426 thru 0x00eeae (0xea89 bytes) 000956 1782530344.932217 read 0x00eacd thru 0x01a10e (0xb642 bytes) 000957 1782530344.935840 mapread 0x01c8ef thru 0x01eab2 (0x21c4 bytes) 000958 1782530344.939551 mapread 0x01e9b9 thru 0x01eab2 (0xfa bytes) 000960 1782530344.942895 punch from 0x00ad11 to 0x00bf39 (0x1229 bytes) 000961 1782530344.948271 trunc from 0x00d834 to 0x01210d (0x48da bytes) 000962 1782530344.960243 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1eab3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eab3): Mirror 2: 0x7ab5ba9a (stale) CRC-32 checksum value for chunk [0, 0x1eab3): Mirror 1: 0x7ab5ba9a Mirror 3: 0x7ab5ba9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eab3: skipped 000963 1782530346.062155 trunc from 0x01eab3 to 0x03fe7e (0x213cc bytes) 000964 1782530346.137863 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fe7f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe7f): Mirror 2: 0x3f49b483 (stale) Mirror 3: 0xb5aac9e (stale) CRC-32 checksum value for chunk [0, 0x3fe7f): Mirror 1: 0xb5aac9e Mirror 4: 0xb5aac9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe7f: skipped 000965 1782530346.989524 fallocate 0x03e804 thru 0x03ffff (0x17fc bytes) 000966 1782530347.038036 punch from 0x02860e to 0x02ab99 (0x258c bytes) 000967 1782530347.043365 trunc from 0x040000 to 0x0267ec (0x19813 bytes) 000968 1782530347.067673 write 0x03ffad thru 0x03ffff (0x53 bytes) 000970 1782530347.069420 trunc from 0x040000 to 0x02bec8 (0x14137 bytes) 000971 1782530347.092067 mapread 0x00a5b2 thru 0x014313 (0x9d62 bytes) 000972 1782530347.113292 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bec9): Mirror 2: 0x7c4a7974 (stale) Mirror 3: 0x7c4a7974 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bec9): Mirror 2: 0x7c4a7974 (stale) Mirror 3: 0x7c4a7974 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bec9: skipped 000973 1782530347.399205 read 0x015b46 thru 0x0173b3 (0x186e bytes) 000974 1782530347.411599 MIRROR_RESYNC 3 mirrors 000975 1782530348.164344 mapwrite 0x010f4b thru 0x01bc6b (0xad21 bytes) 000976 1782530348.321304 read 0x002d03 thru 0x003422 (0x720 bytes) 000977 1782530348.329065 mapread 0x026dd9 thru 0x02a65d (0x3885 bytes) 000978 1782530348.359685 read 0x004839 thru 0x00f24e (0xaa16 bytes) 000980 1782530348.375774 mapread 0x027420 thru 0x02bec8 (0x4aa9 bytes) 000981 1782530348.388221 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2bec9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bec9): Mirror 2: 0x7c4a7974 (stale) Mirror 3: 0x7c4a7974 (stale) CRC-32 checksum value for chunk [0, 0x2bec9): Mirror 1: 0x69370ae9 Mirror 4: 0x69370ae9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bec9: skipped 000982 1782530349.141373 trunc from 0x02bec9 to 0x02e414 (0x254c bytes) 000983 1782530349.209202 punch from 0x00232e to 0x00549e (0x3171 bytes) 000984 1782530349.213372 trunc from 0x003bd4 to 0x012095 (0xe4c2 bytes) 000985 1782530349.217806 mapread 0x018718 thru 0x025bdd (0xd4c6 bytes) 000986 1782530349.250934 mapwrite 0x01c410 thru 0x0272b1 (0xaea2 bytes) 000987 1782530349.284301 trunc from 0x02e415 to 0x01d454 (0x10fc0 bytes) 000988 1782530349.320650 punch from 0x00a4d6 to 0x00e2d8 (0x3e03 bytes) 000990 1782530349.327161 read 0x0136db thru 0x01d454 (0x9d7a bytes) 000991 1782530349.327863 mapwrite 0x02ae8b thru 0x030a74 (0x5bea bytes) 000993 1782530349.376156 read 0x024a64 thru 0x02c237 (0x77d4 bytes) 000995 1782530349.377119 trunc from 0x030a75 to 0x012b5d (0x1df17 bytes) 000996 1782530349.394030 read 0x000e8e thru 0x00ea95 (0xdc08 bytes) 000997 1782530349.400752 trunc from 0x004ff7 to 0x00e645 (0x964f bytes) 000998 1782530349.408425 trunc from 0x00ca6f to 0x0129cf (0x5f61 bytes) 000999 1782530349.423722 punch from 0x0110c1 to 0x012291 (0x11d1 bytes) 001000 1782530349.427627 punch from 0x005abe to 0x0067b0 (0xcf3 bytes) All operations completed A-OK! PASS 70a (71s) == sanity-flr test 71: check mirror extend parent fid ==== 23:19:15 (1782530355) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1dd:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 23:19:25 (1782530365) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.301869 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128583 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 23:19:34 (1782530374) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5895584 .. Extending file size to 3230688 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7191200 .. Extending file size to 7018912 .. Extending file size to 7737632 .. Extending file size to 7418688 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8598592 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8542592 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8635808 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8957728 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9075296 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61221 61223 61224 61227 61228 done 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) 192.168.202.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:20:56 (1782530456) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 849088 .. Extending file size to 2054144 .. done Extending file size to 7595328 .. Extending file size to 7549760 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7751200 .. Extending file size to 9143904 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7942752 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8119072 .. Extending file size to 8502176 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8827296 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 62924 62925 62926 62929 62930 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) 192.168.202.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:22:17 (1782530537) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25652 3581368 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17856 3589164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43508 7170532 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25652 3581368 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17856 3589164 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43508 7170532 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131609 s, 15.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274629 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4f9e0d82 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:22:40 (1782530560) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0950111 s, 22.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:22:47 (1782530567) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:22:56 (1782530576) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0958007 s, 21.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:23:06 (1782530586) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114735 s, 18.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:23:17 (1782530597) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177845 s, 11.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24d:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:23:52 (1782530632) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099698 s, 10.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:24:48 (1782530688) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190159 s, 11.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:25:46 (1782530746) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg201-server: Pool lustre.test_204f created Adding targets to pool oleg201-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg201-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211974 s, 9.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x251:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x251:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg201-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg201-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg201-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:27:04 (1782530824) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:27:12 (1782530832) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:27:21 (1782530841) oleg201-server: Pool lustre.test_206 created Adding targets to pool oleg201-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg201-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg201-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg201-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg201-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:27:52 (1782530872) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437667 s, 24.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220668 s, 47.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x617ca64b Mirror 3: 0x617ca64b PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:28:04 (1782530884) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:28:09 (1782530889) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:28:13 (1782530893) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0649533 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x700000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x19c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x19c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x19c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ae0000 Read this (4 bytes): BBBB 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) PASS 209a (49s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:29:02 (1782530942) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0294424 s, 278 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:29:09 (1782530949) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759559 s, 13.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611259 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 23:29:11.000000000 -0400 Modify: 2026-06-26 23:29:11.000000000 -0400 Change: 2026-06-26 23:29:12.000000000 -0400 Birth: 2026-06-26 23:29:10.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:29:19 (1782530959) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713259 s, 14.7 MB/s oleg201-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg201-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg201-server: O/280000401/d22/136: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:29:45 (1782530985) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.379882 s, 108 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0692986 s, 59.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 26 23:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x80:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 26 23:29 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:29:56 (1782530996) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:30:05 (1782531005) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1858 sec =========== 23:30:15 (1782531015) === sanity-flr: start cleanup 23:30:16 (1782531016) === === sanity-flr: finish cleanup 23:30:26 (1782531026) ===