-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 14:57:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg308-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:57:41 (1787597861) === oleg308-client.virtnet: executing check_config_client /mnt/lustre oleg308-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg308-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8cd4150d0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cd4150d0800.idle_timeout=debug disable quota as required oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:58:23 (1787597903) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8214] == sanity-flr test 0a: lfs mirror create with -N option == 14:58:25 (1787597905) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:58:45 (1787597925) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:58:49 (1787597929) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:58:52 (1787597932) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:59:19 (1787597959) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:59:22 (1787597962) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:59:26 (1787597966) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8cd4150d0800.stats=clear osc.lustre-OST0001-osc-ffff8cd4150d0800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:59:35 (1787597975) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:59:47 (1787597987) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:59:55 (1787597995) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:59:59 (1787597999) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:00:07 (1787598007) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 15:00:16 (1787598016) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:00:25 (1787598025) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:00:33 (1787598033) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:00:42 (1787598042) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:00:54 (1787598054) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:01:01 (1787598061) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.579958 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126216 s, 8.3 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:01:10 (1787598070) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.364154 s, 43.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:01:19 (1787598079) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:01:29 (1787598089) Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:03:46 (1787598226) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0992473 s, 21.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:04:30 (1787598270) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:05:47 (1787598347) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.22812 s, 16.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 15:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.888s user 0m0.001s sys 0m0.201s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd4150d0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.779s user 0m0.001s sys 0m0.197s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.839s user 0m0.001s sys 0m0.188s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd4150d0800.ost_server_uuid in FULL state after 0 sec PASS 33b (80s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:07:08 (1787598428) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:07:10 (1787598430) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:07:13 (1787598433) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:07:16 (1787598436) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 15:07:23 (1787598443) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.507933 s, 24.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:07:35 (1787598455) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362371 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714447 s, 14.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:07:43 (1787598463) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:07:51 (1787598471) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128368 s, 798 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000831534 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00364366 s, 281 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102389 s, 1.0 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 15:08:01 (1787598481) create test files with size 15729205 .. 02996d011aba651ac105d8fe35475a9b - d403ae4b16500909c4b31723f1bc03f4 - 7330888344dd4b4e2ed97ce67b3bdcf0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 15:08:21 (1787598501) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:08:33 (1787598513) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:08:40 (1787598520) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100955 s, 10.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565036 s, 18.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 15:08:51 (1787598531) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146506 s, 14.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.36289 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199732 s, 10.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 15:09:09 (1787598549) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:09:12 (1787598552) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:09:16 (1787598556) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497878 s, 21.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:09:25 (1787598565) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.409732 s, 7.7 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:09:37 (1787598577) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:09:49 (1787598589) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.377487 s, 27.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:10:02 (1787598602) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8cd403a7b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.387517 s, 27.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8cd408abf000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:10:21 (1787598621) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:11:03 (1787598663) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:11:40 (1787598700) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:11:44 (1787598704) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.305731 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.238626 s, 13.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x988c9add (nosync,stale) Mirror 2: 0x988c9add (nosync,stale) ** verify mirror 2 doesn't change original checksum: 860284ecd88dee1e31c9350f08b8fb86 - mirror 1 checksum: b4bf38b5a287991403db8787937fcda8 - mirror 2 checksum: 860284ecd88dee1e31c9350f08b8fb86 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T15:11:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T15:11:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 860284ecd88dee1e31c9350f08b8fb86 - mirror 1 checksum: b4bf38b5a287991403db8787937fcda8 - mirror 2 checksum: b4bf38b5a287991403db8787937fcda8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T15:11:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T15:11:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:11:55 (1787598715) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463817 s, 22.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:12:03 (1787598723) Failing mds1 on oleg308-server Stopping /mnt/lustre-mds1 (opts:) on oleg308-server 15:12:18 (1787598738) shut down facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server Failover mds1 to oleg308-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 15:12:52 (1787598772) targets are mounted 15:12:52 (1787598772) facet_failover done oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02031 s, 51.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:13:09 (1787598789) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0523911 s, 391 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0237884 s, 861 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:13:21 (1787598801) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.101877 s, 201 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0255308 s, 802 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:13:33 (1787598813) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00742779 s, 2.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:13:42 (1787598822) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0716943 s, 286 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:13:50 (1787598830) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00998615 s, 410 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0295965 s, 138 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:13:57 (1787598837) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:14:22 (1787598862) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:14:52 (1787598892) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:15:16 (1787598916) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:15:43 (1787598943) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 15:15:52 (1787598952) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57584 mirror split pid 57586 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57584 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57586 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 15:17:00 (1787599020) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 9022 skipping zero size read skipping zero length zero range skipping zero length zero range skipping zero length punch hole 000007 1787599022.846152 mapwrite 0x014877 thru 0x022c0a (0xe394 bytes) 000008 1787599022.959274 read 0x0012be thru 0x00665d (0x53a0 bytes) 000009 1787599022.972785 mapwrite 0x006f0e thru 0x00eb74 (0x7c67 bytes) 000010 1787599023.008140 mapread 0x00fae4 thru 0x01f40a (0xf927 bytes) truncating to largest ever: 0x15ea 000012 1787599023.012908 trunc from 0x022c0b to 0x0015e9 (0x21621 bytes) 000013 1787599023.045812 mapread 0x0013e7 thru 0x0015e9 (0x203 bytes) 000014 1787599023.047183 mapwrite 0x025bb8 thru 0x031e5a (0xc2a3 bytes) zero_range to largest ever: 0xc03f 000015 1787599023.162498 trunc from 0x006f65 to 0x00c03e (0x50da bytes) 000016 1787599023.172787 MIRROR_EXTEND to 2 mirrors 000017 1787599023.945859 mapread 0x026d88 thru 0x031e5a (0xb0d3 bytes) 000018 1787599023.997722 trunc from 0x031e5b to 0x005fdf (0x2be7b bytes) truncating to largest ever: 0xe68f 000021 1787599024.180987 trunc from 0x005fe0 to 0x00e68e (0x86af bytes) 000022 1787599024.222908 mapread 0x004e80 thru 0x008eef (0x4070 bytes) 000023 1787599024.244035 mapread 0x009c3f thru 0x00e68e (0x4a50 bytes) 000025 1787599024.246808 mapwrite 0x032b8e thru 0x036f06 (0x4379 bytes) 000026 1787599024.321285 write 0x022656 thru 0x025e3b (0x37e6 bytes) 000027 1787599024.330877 write 0x026d47 thru 0x02995e (0x2c18 bytes) 000028 1787599024.345773 MIRROR_RESYNC 2 mirrors 000029 1787599025.034820 mapread 0x014750 thru 0x022100 (0xd9b1 bytes) zero_range to largest ever: 0x1951d 000030 1787599025.063237 trunc from 0x00f875 to 0x01951c (0x9ca8 bytes) 000031 1787599025.117368 fallocate 0x00366d thru 0x004ecb (0x185f bytes) 000032 1787599025.124489 trunc from 0x000482 to 0x00e3b6 (0xdf35 bytes) 000033 1787599025.132495 read 0x007268 thru 0x00fdeb (0x8b84 bytes) 000034 1787599025.142026 write 0x0397b7 thru 0x03ffff (0x6849 bytes) 000035 1787599025.145890 write 0x02be62 thru 0x02ed52 (0x2ef1 bytes) 000036 1787599025.159399 trunc from 0x030f3e to 0x038fe3 (0x80a6 bytes) 000037 1787599025.168888 write 0x0222f3 thru 0x024d7b (0x2a89 bytes) 000038 1787599025.197082 mapwrite 0x03c37e thru 0x03ffff (0x3c82 bytes) 000039 1787599025.225968 read 0x010499 thru 0x019851 (0x93b9 bytes) 000041 1787599025.246031 read 0x02b8b3 thru 0x02b8c2 (0x10 bytes) 000043 1787599025.248082 trunc from 0x006489 to 0x011376 (0xaeee bytes) 000044 1787599025.267868 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) 000045 1787599025.426645 read 0x00aeb5 thru 0x015abb (0xac07 bytes) 000046 1787599025.449944 mapread 0x006c51 thru 0x00c006 (0x53b6 bytes) 000047 1787599025.468743 read 0x01caa0 thru 0x0256b1 (0x8c12 bytes) 000048 1787599025.484977 trunc from 0x040000 to 0x0134f6 (0x2cb09 bytes) 000049 1787599025.532256 trunc from 0x00611b to 0x0070f2 (0xfd8 bytes) 000050 1787599025.537119 mapwrite 0x0293a6 thru 0x033f18 (0xab73 bytes) 000053 1787599025.682665 MIRROR_EXTEND to 2 mirrors 000054 1787599026.392951 trunc from 0x033f19 to 0x0026a4 (0x31874 bytes) truncating to largest ever: 0x1c4b4 000055 1787599026.461033 trunc from 0x0026a5 to 0x01c4b3 (0x19e0f bytes) 000056 1787599026.482568 mapread 0x00680a thru 0x01430d (0xdb04 bytes) 000057 1787599026.511185 mapread 0x0104f2 thru 0x018283 (0x7d92 bytes) 000058 1787599026.512226 write 0x023fb7 thru 0x02ec5c (0xaca6 bytes) 000059 1787599026.515356 punch from 0x012041 to 0x01293d (0x8fd bytes) 000060 1787599026.523257 read 0x00475b thru 0x0054bc (0xd62 bytes) 000061 1787599026.529823 MIRROR_RESYNC 2 mirrors 000062 1787599027.247841 read 0x027e42 thru 0x02ec5c (0x6e1b bytes) 000063 1787599027.263036 read 0x00eff9 thru 0x01ee7e (0xfe86 bytes) 000064 1787599027.296656 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) 000066 1787599027.451769 mapread 0x001a65 thru 0x01075a (0xecf6 bytes) 000067 1787599027.509174 read 0x02a1f9 thru 0x02ec5c (0x4a64 bytes) 000068 1787599027.545258 mapwrite 0x0360fe thru 0x03e2c0 (0x81c3 bytes) 000070 1787599027.711842 trunc from 0x02001b to 0x02d496 (0xd47c bytes) 000071 1787599027.727397 trunc from 0x039b8d to 0x03ae91 (0x1305 bytes) 000072 1787599027.743453 mapread 0x0382a6 thru 0x03e2c0 (0x601b bytes) fallocating to largest ever: 0x2bfca 000073 1787599027.759460 fallocate 0x01e281 thru 0x02bfc9 (0xdd49 bytes) 000074 1787599027.770628 mapread 0x032af6 thru 0x0360ed (0x35f8 bytes) 000075 1787599027.772033 trunc from 0x0155d8 to 0x0197ad (0x41d6 bytes) 000076 1787599027.788149 write 0x026517 thru 0x02f7f6 (0x92e0 bytes) 000077 1787599027.799003 read 0x028ef5 thru 0x02a694 (0x17a0 bytes) 000079 1787599027.807862 fallocate 0x015fd4 thru 0x01721f (0x124c bytes) 000080 1787599027.815708 write 0x031b91 thru 0x034c6f (0x30df bytes) 000081 1787599027.818722 fallocate 0x00e42d thru 0x01e30e (0xfee2 bytes) 000082 1787599027.826741 mapwrite 0x02029a thru 0x0207ae (0x515 bytes) 000083 1787599027.898389 trunc from 0x0247a4 to 0x0300a5 (0xb902 bytes) 000084 1787599027.904448 mapread 0x004df3 thru 0x00b3f4 (0x6602 bytes) 000085 1787599027.905453 punch from 0x0257c2 to 0x027bfe (0x243d bytes) fallocating to largest ever: 0x323cf 000087 1787599027.917152 fallocate 0x02e4b2 thru 0x0323ce (0x3f1d bytes) 000088 1787599027.926804 trunc from 0x026738 to 0x029157 (0x2a20 bytes) 000089 1787599027.936764 write 0x03fd80 thru 0x03ffff (0x280 bytes) 000090 1787599027.940798 punch from 0x007116 to 0x009417 (0x2302 bytes) 000091 1787599027.954741 read 0x026467 thru 0x02959c (0x3136 bytes) 000092 1787599027.979558 mapwrite 0x01a3c7 thru 0x01f7bd (0x53f7 bytes) 000093 1787599028.024440 write 0x022ded thru 0x0302af (0xd4c3 bytes) 000094 1787599028.026136 punch from 0x00fba1 to 0x012510 (0x2970 bytes) 000095 1787599028.040032 mapwrite 0x00e034 thru 0x00e47a (0x447 bytes) 000096 1787599028.110666 punch from 0x01b94a to 0x02847f (0xcb36 bytes) 000097 1787599028.124858 punch from 0x019f47 to 0x01f915 (0x59cf bytes) 000098 1787599028.135940 read 0x03cb04 thru 0x03ffff (0x34fc bytes) 000100 1787599028.137138 punch from 0x0320a9 to 0x03a5c4 (0x851c bytes) 000101 1787599028.176869 MIRROR_EXTEND to 2 mirrors 000102 1787599029.201284 read 0x010bdd thru 0x0179c4 (0x6de8 bytes) 000103 1787599029.212561 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) 000104 1787599029.354448 write 0x00e209 thru 0x015846 (0x763e bytes) 000105 1787599029.389364 mapwrite 0x02a258 thru 0x032ee7 (0x8c90 bytes) 000106 1787599029.482143 MIRROR_EXTEND to 2 mirrors 000107 1787599030.457616 write 0x02e568 thru 0x0311ba (0x2c53 bytes) 000108 1787599030.565630 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6fc4566 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3553c825 Mirror 3: 0x3553c825 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000109 1787599031.936455 mapwrite 0x0058db thru 0x00e61f (0x8d45 bytes) 000110 1787599032.079278 mapwrite 0x01d211 thru 0x02a08e (0xce7e bytes) 000111 1787599032.139705 write 0x017064 thru 0x025af4 (0xea91 bytes) 000112 1787599032.148846 punch from 0x0398f9 to 0x03ffff (0x6707 bytes) 000113 1787599032.158347 write 0x0372c5 thru 0x039616 (0x2352 bytes) 000115 1787599032.174286 write 0x00f48f thru 0x0134c1 (0x4033 bytes) 000116 1787599032.186296 trunc from 0x040000 to 0x0145c4 (0x2ba3b bytes) 000117 1787599032.210024 mapwrite 0x037bf9 thru 0x03ffff (0x8407 bytes) 000118 1787599032.289790 write 0x0319f9 thru 0x0322b2 (0x8ba bytes) 000119 1787599032.295389 trunc from 0x033de1 to 0x03ffff (0xc21f bytes) 000120 1787599032.324156 read 0x0068de thru 0x00c330 (0x5a53 bytes) 000121 1787599032.324910 read 0x019169 thru 0x01bee3 (0x2d7b bytes) 000122 1787599032.350750 fallocate 0x01dd11 thru 0x0224a1 (0x4791 bytes) fallocating to largest ever: 0x40000 000124 1787599032.379177 fallocate 0x03c2c0 thru 0x03ffff (0x3d40 bytes) 000125 1787599032.391024 trunc from 0x040000 to 0x026ecc (0x19133 bytes) 000126 1787599032.430450 fallocate 0x03fdaf thru 0x03ffff (0x251 bytes) 000127 1787599032.449099 mapwrite 0x022448 thru 0x02c4be (0xa077 bytes) 000128 1787599032.617246 mapwrite 0x033cf6 thru 0x03ffff (0xc30a bytes) 000130 1787599032.822434 mapread 0x017408 thru 0x01f712 (0x830b bytes) 000131 1787599032.824251 mapwrite 0x00abf9 thru 0x00ec65 (0x406d bytes) 000132 1787599032.844582 fallocate 0x00e0f6 thru 0x00f199 (0x10a4 bytes) 000133 1787599032.877012 MIRROR_RESYNC 3 mirrors 000134 1787599034.172427 fallocate 0x026721 thru 0x02bf8b (0x586b bytes) 000136 1787599034.244120 trunc from 0x040000 to 0x03fe0f (0x1f0 bytes) 000137 1787599034.275000 trunc from 0x03fe10 to 0x035ec9 (0x9f46 bytes) 000138 1787599034.294646 punch from 0x001664 to 0x0096ab (0x8048 bytes) 000140 1787599034.306190 write 0x02887b thru 0x035cd0 (0xd456 bytes) 000142 1787599034.339053 write 0x00e316 thru 0x010974 (0x265f bytes) 000143 1787599034.366414 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35eca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35eca): Mirror 2: 0xd33e5b09 (stale) Mirror 3: 0xd33e5b09 (stale) CRC-32 checksum value for chunk [0, 0x35eca): Mirror 1: 0xbd248819 Mirror 4: 0xbd248819 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35eca: skipped 000144 1787599035.528049 mapread 0x00a25f thru 0x01233d (0x80df bytes) 000146 1787599035.567613 mapread 0x019405 thru 0x027153 (0xdd4f bytes) 000148 1787599035.619252 MIRROR_RESYNC 4 mirrors 000149 1787599036.715929 punch from 0x03142e to 0x031b90 (0x763 bytes) 000150 1787599036.759696 mapread 0x007d0c thru 0x011a78 (0x9d6d bytes) 000151 1787599036.781759 trunc from 0x022638 to 0x031eff (0xf8c8 bytes) 000152 1787599036.789018 mapwrite 0x018d88 thru 0x023ac7 (0xad40 bytes) 000153 1787599036.886582 MIRROR_RESYNC 4 mirrors 000154 1787599038.277005 trunc from 0x02d349 to 0x0319e2 (0x469a bytes) 000155 1787599038.312144 write 0x003ed4 thru 0x012ef5 (0xf022 bytes) 000156 1787599038.330435 trunc from 0x0109a6 to 0x012bda (0x2235 bytes) 000157 1787599038.358091 write 0x02db1f thru 0x03058e (0x2a70 bytes) 000158 1787599038.370203 trunc from 0x0086e6 to 0x00f8bc (0x71d7 bytes) 000159 1787599038.377257 trunc from 0x005043 to 0x00e6cf (0x968d bytes) 000160 1787599038.384173 mapwrite 0x036bab thru 0x036bb3 (0x9 bytes) 000161 1787599038.456891 mapread 0x014ebc thru 0x022232 (0xd377 bytes) 000162 1787599038.494784 trunc from 0x01fc9f to 0x025029 (0x538b bytes) 000163 1787599038.507909 mapwrite 0x00ce4d thru 0x015a7a (0x8c2e bytes) 000164 1787599038.565304 read 0x029e40 thru 0x035031 (0xb1f2 bytes) 000165 1787599038.581725 trunc from 0x036bb4 to 0x01491a (0x22299 bytes) 000166 1787599038.624089 mapwrite 0x018eb6 thru 0x01da6a (0x4bb5 bytes) 000167 1787599038.705702 trunc from 0x00b4a8 to 0x01b33f (0xfe98 bytes) 000168 1787599038.712649 trunc from 0x01da6b to 0x03fcfb (0x22291 bytes) 000169 1787599038.731581 trunc from 0x0251bd to 0x02e918 (0x975c bytes) 000170 1787599038.737713 read 0x00b10f thru 0x017884 (0xc776 bytes) 000171 1787599038.748488 trunc from 0x00a6cb to 0x018c95 (0xe5cb bytes) 000172 1787599038.754253 mapwrite 0x01ca0a thru 0x023277 (0x686e bytes) 000173 1787599038.775726 punch from 0x008dc8 to 0x011fd3 (0x920c bytes) 000174 1787599038.782386 mapread 0x037270 thru 0x038ea6 (0x1c37 bytes) 000176 1787599038.790071 MIRROR_RESYNC 4 mirrors 000177 1787599040.013189 fallocate 0x007e4c thru 0x011b82 (0x9d37 bytes) 000178 1787599040.063164 write 0x00da5c thru 0x00e6de (0xc83 bytes) 000179 1787599040.075810 trunc from 0x02e10d to 0x03275c (0x4650 bytes) 000180 1787599040.088724 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fcfc): Mirror 2: 0x43515b8f (stale) Mirror 3: 0x43515b8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fcfc): Mirror 2: 0x43515b8f (stale) Mirror 3: 0x43515b8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fcfc: skipped 000182 1787599040.610084 fallocate 0x02fb25 thru 0x03ae1a (0xb2f6 bytes) 000184 1787599040.624234 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fcfc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fcfc): Mirror 2: 0x43515b8f (stale) Mirror 3: 0x43515b8f (stale) CRC-32 checksum value for chunk [0, 0x3fcfc): Mirror 1: 0xed97fed9 Mirror 4: 0xed97fed9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fcfc: skipped 000185 1787599042.019491 trunc from 0x03fcfc to 0x014aa6 (0x2b255 bytes) 000186 1787599042.107438 trunc from 0x014aa7 to 0x028902 (0x13e5c bytes) 000187 1787599042.139937 trunc from 0x01c8db to 0x023465 (0x6b8b bytes) 000188 1787599042.149642 write 0x030a99 thru 0x032b93 (0x20fb bytes) 000189 1787599042.151049 trunc from 0x00e6d2 to 0x0115ad (0x2edc bytes) 000191 1787599042.173986 mapread 0x025b1b thru 0x02a10f (0x45f5 bytes) 000192 1787599042.200732 write 0x00e1a7 thru 0x01c9c3 (0xe81d bytes) 000193 1787599042.223704 mapwrite 0x002bb7 thru 0x00d465 (0xa8af bytes) 000194 1787599042.275411 punch from 0x031180 to 0x032b93 (0x1a14 bytes) 000195 1787599042.282756 read 0x0000bc thru 0x00b4cf (0xb414 bytes) 000196 1787599042.297749 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32b94): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b94): Mirror 2: 0x4591901b (stale) Mirror 3: 0x4591901b (stale) Mirror 4: 0x22a18e9b (stale) CRC-32 checksum value for chunk [0, 0x32b94): Mirror 1: 0x199fa9d2 Mirror 5: 0x199fa9d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b94: skipped 000197 1787599043.410216 fallocate 0x023f2b thru 0x02cefe (0x8fd4 bytes) 000198 1787599043.468968 write 0x0212d2 thru 0x024016 (0x2d45 bytes) 000199 1787599043.494002 mapread 0x016b51 thru 0x01dc4f (0x70ff bytes) 000200 1787599043.588106 mapwrite 0x018662 thru 0x018984 (0x323 bytes) 000201 1787599043.634828 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32b94): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b94): Mirror 2: 0x4591901b (stale) Mirror 3: 0x4591901b (stale) Mirror 4: 0x22a18e9b (stale) Mirror 5: 0x199fa9d2 (stale) CRC-32 checksum value for chunk [0, 0x32b94): Mirror 1: 0x4d12164f Mirror 6: 0x4d12164f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b94: skipped 000203 1787599045.096216 mapread 0x0143d4 thru 0x01b706 (0x7333 bytes) 000205 1787599045.144458 fallocate 0x034b24 thru 0x03f77c (0xac59 bytes) 000206 1787599045.212122 write 0x0394e4 thru 0x03ba02 (0x251f bytes) 000207 1787599045.224183 write 0x026685 thru 0x029ea1 (0x381d bytes) 000208 1787599045.239270 mapread 0x036c92 thru 0x03ba02 (0x4d71 bytes) 000209 1787599045.252767 write 0x003e62 thru 0x008941 (0x4ae0 bytes) 000210 1787599045.267702 trunc from 0x004958 to 0x007674 (0x2d1d bytes) 000211 1787599045.291150 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba03): Mirror 3: 0xa1d1830e (stale) Mirror 4: 0xcf960a0e (stale) Mirror 5: 0xef5cbfeb (stale) Mirror 6: 0xf5e8e26d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba03): Mirror 3: 0xa1d1830e (stale) Mirror 4: 0xcf960a0e (stale) Mirror 5: 0xef5cbfeb (stale) Mirror 6: 0xf5e8e26d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba03: skipped 000212 1787599045.749643 mapwrite 0x038491 thru 0x03ffff (0x7b6f bytes) 000213 1787599045.850122 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe655bec3 (stale) Mirror 4: 0x846a9505 (stale) Mirror 5: 0x345fca40 (stale) Mirror 6: 0x3ab9703b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xccc41c2e Mirror 7: 0xccc41c2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000214 1787599047.291580 write 0x0233e8 thru 0x02980a (0x6423 bytes) 000215 1787599047.344453 mapread 0x024aee thru 0x024bd2 (0xe5 bytes) 000216 1787599047.345606 mapwrite 0x033e14 thru 0x035870 (0x1a5d bytes) 000217 1787599047.381668 mapread 0x00527c thru 0x011ab0 (0xc835 bytes) 000219 1787599047.405569 read 0x01b8c6 thru 0x01c049 (0x784 bytes) 000220 1787599047.410345 trunc from 0x01dbaa to 0x026557 (0x89ae bytes) 000221 1787599047.416013 fallocate 0x0023f3 thru 0x00ec6f (0xc87d bytes) 000222 1787599047.422684 trunc from 0x03fbd9 to 0x03ffff (0x427 bytes) 000224 1787599047.428687 mapread 0x02c0a8 thru 0x032ca6 (0x6bff bytes) 000225 1787599047.438427 fallocate 0x03574e thru 0x03bf94 (0x6847 bytes) 000226 1787599047.442799 trunc from 0x040000 to 0x0050f2 (0x3af0d bytes) 000227 1787599047.513594 punch from 0x002218 to 0x0050f2 (0x2edb bytes) 000228 1787599047.522763 mapwrite 0x011486 thru 0x0192cf (0x7e4a bytes) 000229 1787599047.632481 trunc from 0x00bd90 to 0x0192cf (0xd540 bytes) 000230 1787599047.646383 mapread 0x011244 thru 0x0152b8 (0x4075 bytes) 000231 1787599047.699882 fallocate 0x03f216 thru 0x03ffff (0xdea bytes) 000232 1787599047.712247 trunc from 0x0192d0 to 0x01c1e5 (0x2f16 bytes) 000233 1787599047.728994 fallocate 0x0165ec thru 0x019a22 (0x3437 bytes) 000234 1787599047.736389 fallocate 0x033079 thru 0x037da8 (0x4d30 bytes) 000235 1787599047.741680 mapread 0x00ffca thru 0x010559 (0x590 bytes) 000236 1787599047.744176 punch from 0x004eac to 0x0076c7 (0x281c bytes) 000237 1787599047.748603 mapread 0x035260 thru 0x037da8 (0x2b49 bytes) 000238 1787599047.767381 trunc from 0x00a117 to 0x016eec (0xcdd6 bytes) 000239 1787599047.777029 mapread 0x033889 thru 0x037da8 (0x4520 bytes) 000240 1787599047.785426 punch from 0x030a8b to 0x032b46 (0x20bc bytes) 000242 1787599047.800389 write 0x02e9aa thru 0x0373b2 (0x8a09 bytes) 000243 1787599047.815895 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37da9): Mirror 3: 0x96bb1e4c (stale) Mirror 5: 0xd68214bc (stale) Mirror 6: 0xc4034670 (stale) Mirror 7: 0xfe7e6f8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37da9): Mirror 3: 0x96bb1e4c (stale) Mirror 5: 0xd68214bc (stale) Mirror 6: 0xc4034670 (stale) Mirror 7: 0xfe7e6f8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37da9: skipped 000244 1787599048.387566 trunc from 0x00f9f7 to 0x019a00 (0xa00a bytes) 000245 1787599048.396926 trunc from 0x02371e to 0x02416d (0xa50 bytes) 000246 1787599048.403993 read 0x0041ae thru 0x004447 (0x29a bytes) 000250 1787599048.412205 fallocate 0x00ad2f thru 0x00f7e3 (0x4ab5 bytes) 000252 1787599048.422438 mapread 0x0120ce thru 0x01c74f (0xa682 bytes) 000253 1787599048.462483 punch from 0x00d3e4 to 0x00f77e (0x239b bytes) 000254 1787599048.473372 punch from 0x01ae73 to 0x01cfdf (0x216d bytes) 000255 1787599048.481015 fallocate 0x00a057 thru 0x0138e5 (0x988f bytes) 000256 1787599048.488612 trunc from 0x037da9 to 0x016646 (0x21762 bytes) 000257 1787599048.531763 trunc from 0x016647 to 0x023549 (0xcf03 bytes) 000258 1787599048.552609 fallocate 0x014631 thru 0x01d65e (0x902e bytes) 000259 1787599048.560824 read 0x022fde thru 0x023549 (0x56c bytes) 000260 1787599048.592032 trunc from 0x0209e7 to 0x023549 (0x2b63 bytes) 000261 1787599048.595957 read 0x01cb11 thru 0x0200f0 (0x35e0 bytes) 000262 1787599048.611380 mapwrite 0x02f162 thru 0x03b70a (0xc5a9 bytes) 000263 1787599048.730718 mapwrite 0x004002 thru 0x010c3a (0xcc39 bytes) 000264 1787599048.755044 trunc from 0x03b70b to 0x03df0c (0x2802 bytes) 000265 1787599048.770217 punch from 0x0164d7 to 0x01c81c (0x6346 bytes) 000266 1787599048.776888 read 0x03b874 thru 0x03cce9 (0x1476 bytes) 000267 1787599048.801550 mapread 0x00aedc thru 0x012294 (0x73b9 bytes) 000268 1787599048.802468 mapwrite 0x0209f4 thru 0x025fe3 (0x55f0 bytes) 000269 1787599048.854405 write 0x01c4a5 thru 0x021b40 (0x569c bytes) 000271 1787599048.856385 mapwrite 0x0040e4 thru 0x00dc65 (0x9b82 bytes) 000272 1787599048.907993 read 0x02761a thru 0x03350d (0xbef4 bytes) 000273 1787599048.909199 mapread 0x02edf9 thru 0x03a5d7 (0xb7df bytes) 000274 1787599048.911136 mapread 0x03c7ab thru 0x03df0c (0x1762 bytes) 000275 1787599048.914567 trunc from 0x03df0d to 0x03e440 (0x534 bytes) 000276 1787599048.959665 trunc from 0x03e441 to 0x01b513 (0x22f2d bytes) 000277 1787599048.988962 punch from 0x00a7e0 to 0x011547 (0x6d68 bytes) 000278 1787599048.999835 mapread 0x00a71a thru 0x01579e (0xb085 bytes) 000279 1787599049.010816 trunc from 0x01b514 to 0x029bff (0xe6ec bytes) 000280 1787599049.028893 trunc from 0x019d57 to 0x0239e2 (0x9c8c bytes) 000281 1787599049.040573 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29c00): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c00): Mirror 3: 0x147ae16a (stale) Mirror 5: 0xb4bfe95f (stale) Mirror 6: 0xf1bc18be (stale) Mirror 7: 0xbf73696e (stale) CRC-32 checksum value for chunk [0, 0x29c00): Mirror 1: 0x149ca57d Mirror 8: 0x149ca57d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c00: skipped 000282 1787599050.233557 write 0x0156af thru 0x023ff9 (0xe94b bytes) 000283 1787599050.291786 trunc from 0x029c00 to 0x02da8e (0x3e8f bytes) 000284 1787599050.331377 MIRROR_RESYNC 6 mirrors 000285 1787599052.527421 punch from 0x005e98 to 0x014327 (0xe490 bytes) 000286 1787599052.575849 trunc from 0x02da8f to 0x03a393 (0xc905 bytes) 000287 1787599052.602633 punch from 0x021b1f to 0x03083e (0xed20 bytes) 000289 1787599052.610631 trunc from 0x03a394 to 0x025894 (0x14aff bytes) 000290 1787599052.629634 trunc from 0x025895 to 0x024f5e (0x936 bytes) 000291 1787599052.647841 fallocate 0x022762 thru 0x02829a (0x5b39 bytes) 000292 1787599052.653094 read 0x015abc thru 0x02592d (0xfe72 bytes) 000293 1787599052.664827 mapread 0x008777 thru 0x00c215 (0x3a9f bytes) 000294 1787599052.685879 punch from 0x01578d to 0x01f5da (0x9e4e bytes) 000295 1787599052.691310 punch from 0x00c6fa to 0x01a35d (0xdc64 bytes) 000296 1787599052.696630 punch from 0x021c4a to 0x02829a (0x6651 bytes) 000297 1787599052.701998 mapread 0x01c32a thru 0x020288 (0x3f5f bytes) 000298 1787599052.713747 write 0x0052d0 thru 0x00aaac (0x57dd bytes) 000299 1787599052.714993 read 0x01fbdd thru 0x02829a (0x86be bytes) 000300 1787599052.725964 mapwrite 0x03e893 thru 0x03ffff (0x176d bytes) 000301 1787599052.777847 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x437c5e24 (stale) Mirror 5: 0x437c5e24 (stale) Mirror 6: 0x437c5e24 (stale) Mirror 7: 0x437c5e24 (stale) Mirror 8: 0x437c5e24 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1add953b Mirror 9: 0x1add953b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000302 1787599054.123711 MIRROR_RESYNC 7 mirrors 000303 1787599056.302698 mapread 0x02544c thru 0x02c54a (0x70ff bytes) 000304 1787599056.357064 fallocate 0x0242a4 thru 0x02a390 (0x60ed bytes) 000305 1787599056.432764 mapread 0x003319 thru 0x010636 (0xd31e bytes) 000306 1787599056.451894 MIRROR_RESYNC 7 mirrors 000307 1787599059.197417 punch from 0x001b08 to 0x00c92a (0xae23 bytes) 000308 1787599059.236309 fallocate 0x00092a thru 0x00656a (0x5c41 bytes) 000309 1787599059.241113 read 0x015561 thru 0x01c7a9 (0x7249 bytes) 000310 1787599059.254730 punch from 0x02e806 to 0x0369c9 (0x81c4 bytes) 000311 1787599059.260361 read 0x02b1ac thru 0x03556d (0xa3c2 bytes) 000312 1787599059.276395 fallocate 0x027a24 thru 0x032411 (0xa9ee bytes) 000313 1787599059.283157 fallocate 0x028ecf thru 0x02b638 (0x276a bytes) 000314 1787599059.288452 trunc from 0x040000 to 0x033382 (0xcc7d bytes) 000315 1787599059.367805 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33383): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33383): Mirror 3: 0xd20300d9 (stale) Mirror 5: 0xd20300d9 (stale) Mirror 6: 0xd20300d9 (stale) Mirror 7: 0xd20300d9 (stale) Mirror 8: 0xd20300d9 (stale) Mirror 9: 0xd20300d9 (stale) CRC-32 checksum value for chunk [0, 0x33383): Mirror 1: 0x3f7f47de Mirror 10: 0x3f7f47de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33383: skipped 000316 1787599060.486421 write 0x0326d1 thru 0x03ffff (0xd92f bytes) 000317 1787599060.591014 punch from 0x02bf5f to 0x033cc9 (0x7d6b bytes) 000318 1787599060.619688 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1add953b (stale) Mirror 5: 0x1add953b (stale) Mirror 6: 0x1add953b (stale) Mirror 7: 0x1add953b (stale) Mirror 9: 0x1add953b (stale) Mirror 10: 0x1312bebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1add953b (stale) Mirror 5: 0x1add953b (stale) Mirror 6: 0x1add953b (stale) Mirror 7: 0x1add953b (stale) Mirror 9: 0x1add953b (stale) Mirror 10: 0x1312bebf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000320 1787599061.129549 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1add953b (stale) Mirror 5: 0x1add953b (stale) Mirror 6: 0x1add953b (stale) Mirror 7: 0x1add953b (stale) Mirror 9: 0x1add953b (stale) Mirror 10: 0x1312bebf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb246252 Mirror 11: 0xb246252 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000321 1787599062.468460 mapread 0x032a0b thru 0x03abaf (0x81a5 bytes) 000322 1787599062.525546 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1add953b (stale) Mirror 6: 0x1add953b (stale) Mirror 7: 0x1add953b (stale) Mirror 9: 0x1add953b (stale) Mirror 10: 0x1312bebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1add953b (stale) Mirror 6: 0x1add953b (stale) Mirror 7: 0x1add953b (stale) Mirror 9: 0x1add953b (stale) Mirror 10: 0x1312bebf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb246252 Mirror 11: 0xb246252 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000323 1787599063.573791 trunc from 0x03bd75 to 0x03ffff (0x428b bytes) 000324 1787599063.667802 punch from 0x01cb16 to 0x024b2a (0x8015 bytes) 000325 1787599063.672634 mapread 0x006813 thru 0x00902b (0x2819 bytes) 000326 1787599063.706098 mapwrite 0x00a290 thru 0x013366 (0x90d7 bytes) 000327 1787599063.745139 mapwrite 0x0291fa thru 0x03901d (0xfe24 bytes) 000328 1787599063.822220 trunc from 0x040000 to 0x002f95 (0x3d06a bytes) 000330 1787599063.865447 trunc from 0x002538 to 0x002f95 (0xa5e bytes) 000331 1787599063.875576 trunc from 0x0001ec to 0x000d37 (0xb4c bytes) 000332 1787599063.881227 fallocate 0x010cac thru 0x0146b0 (0x3a05 bytes) 000333 1787599063.889326 fallocate 0x029041 thru 0x0385ae (0xf56e bytes) 000334 1787599063.895211 mapwrite 0x0136ac thru 0x01b083 (0x79d8 bytes) 000335 1787599063.942354 fallocate 0x005255 thru 0x00c67b (0x7427 bytes) 000336 1787599063.949327 trunc from 0x019374 to 0x01b083 (0x1d10 bytes) 000337 1787599063.960863 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b084): Mirror 5: 0x14c81d43 (stale) Mirror 6: 0x14c81d43 (stale) Mirror 7: 0x14c81d43 (stale) Mirror 9: 0x14c81d43 (stale) Mirror 10: 0x48cea954 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b084): Mirror 5: 0x14c81d43 (stale) Mirror 6: 0x14c81d43 (stale) Mirror 7: 0x14c81d43 (stale) Mirror 9: 0x14c81d43 (stale) Mirror 10: 0x48cea954 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b084: skipped 000338 1787599064.464022 read 0x0132c3 thru 0x01b083 (0x7dc1 bytes) 000339 1787599064.473804 mapread 0x000cc3 thru 0x00b51f (0xa85d bytes) 000340 1787599064.496459 punch from 0x003820 to 0x00e329 (0xab0a bytes) 000341 1787599064.501912 write 0x01173a thru 0x01db9c (0xc463 bytes) 000342 1787599064.510931 fallocate 0x032ee1 thru 0x03ffe0 (0xd100 bytes) 000343 1787599064.515973 trunc from 0x03ffe1 to 0x010888 (0x2f758 bytes) 000344 1787599064.533356 mapwrite 0x018d9b thru 0x025080 (0xc2e6 bytes) 000345 1787599064.597282 trunc from 0x01c09c to 0x024cc3 (0x8c28 bytes) 000346 1787599064.604056 read 0x000a4d thru 0x001901 (0xeb5 bytes) 000347 1787599064.605603 write 0x011884 thru 0x0197f5 (0x7f72 bytes) 000348 1787599064.613473 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25081): Mirror 6: 0xfacbae78 (stale) Mirror 7: 0xfacbae78 (stale) Mirror 9: 0xfacbae78 (stale) Mirror 10: 0x2c9ecfad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25081): Mirror 6: 0xfacbae78 (stale) Mirror 7: 0xfacbae78 (stale) Mirror 9: 0xfacbae78 (stale) Mirror 10: 0x2c9ecfad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25081: skipped 000349 1787599065.020064 fallocate 0x02bf78 thru 0x0335a1 (0x762a bytes) 000350 1787599065.026070 mapwrite 0x03bb20 thru 0x03dd9f (0x2280 bytes) 000351 1787599065.138959 read 0x01d4b8 thru 0x024430 (0x6f79 bytes) 000352 1787599065.153419 mapwrite 0x03e57b thru 0x03f46f (0xef5 bytes) 000353 1787599065.219386 trunc from 0x011b1d to 0x0142d6 (0x27ba bytes) 000354 1787599065.230212 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f470): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f470): Mirror 6: 0xd98310b5 (stale) Mirror 7: 0xd98310b5 (stale) Mirror 9: 0xd98310b5 (stale) Mirror 10: 0xd3d75e53 (stale) CRC-32 checksum value for chunk [0, 0x3f470): Mirror 1: 0x383193e0 Mirror 11: 0x383193e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f470: skipped 000355 1787599066.495474 trunc from 0x03f470 to 0x021cf5 (0x1d77a bytes) 000356 1787599066.629808 trunc from 0x021cf6 to 0x03c5a7 (0x1a8b2 bytes) 000357 1787599066.674565 trunc from 0x00f03d to 0x01c4ad (0xd471 bytes) 000360 1787599066.696264 MIRROR_RESYNC 6 mirrors 000361 1787599067.845324 write 0x0021a4 thru 0x00e33b (0xc198 bytes) 000362 1787599067.879533 trunc from 0x0380ae to 0x03c5a7 (0x44fa bytes) 000363 1787599067.885119 read 0x022b39 thru 0x028cd2 (0x619a bytes) 000364 1787599067.894380 fallocate 0x00df7b thru 0x0164c2 (0x8548 bytes) 000365 1787599067.899099 read 0x00cbb9 thru 0x01a67b (0xdac3 bytes) 000366 1787599067.908145 mapread 0x00a59c thru 0x016a5a (0xc4bf bytes) 000367 1787599067.909215 write 0x00da0b thru 0x015ec2 (0x84b8 bytes) 000368 1787599067.910465 fallocate 0x03c80e thru 0x03ffff (0x37f2 bytes) 000369 1787599067.915346 fallocate 0x01b8ed thru 0x02b51d (0xfc31 bytes) 000370 1787599067.920025 fallocate 0x03d57b thru 0x03ffff (0x2a85 bytes) 000371 1787599067.924240 fallocate 0x02eb18 thru 0x031549 (0x2a32 bytes) 000372 1787599067.928433 write 0x01fada thru 0x028322 (0x8849 bytes) 000373 1787599067.930349 mapwrite 0x0159f3 thru 0x016091 (0x69f bytes) 000374 1787599067.953456 read 0x020b22 thru 0x026824 (0x5d03 bytes) 000375 1787599067.954152 trunc from 0x031769 to 0x03b336 (0x9bce bytes) 000376 1787599067.959174 fallocate 0x01aea5 thru 0x02520e (0xa36a bytes) 000377 1787599067.964479 mapread 0x01e75e thru 0x01f646 (0xee9 bytes) 000378 1787599067.966093 punch from 0x035a45 to 0x037f9d (0x2559 bytes) 000379 1787599067.971193 read 0x0246ab thru 0x033b91 (0xf4e7 bytes) 000380 1787599067.982005 read 0x0179f7 thru 0x026102 (0xe70c bytes) 000381 1787599067.983029 mapread 0x0358d6 thru 0x0387b7 (0x2ee2 bytes) 000382 1787599067.987033 read 0x020161 thru 0x02236d (0x220d bytes) 000383 1787599067.987931 trunc from 0x040000 to 0x0272f5 (0x18d0a bytes) 000384 1787599068.019316 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x272f6): Mirror 6: 0xd002508 (stale) Mirror 7: 0xd002508 (stale) Mirror 10: 0xd002508 (stale) Mirror 11: 0xd002508 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x272f6): Mirror 6: 0xd002508 (stale) Mirror 7: 0xd002508 (stale) Mirror 10: 0xd002508 (stale) Mirror 11: 0xd002508 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x272f6: skipped 000386 1787599068.613631 mapwrite 0x029e13 thru 0x036326 (0xc514 bytes) 000387 1787599068.733215 punch from 0x02f4a7 to 0x036326 (0x6e80 bytes) 000388 1787599068.744445 read 0x02d99a thru 0x036326 (0x898d bytes) 000389 1787599068.759978 mapwrite 0x03cc8c thru 0x03ffff (0x3374 bytes) 000390 1787599068.827946 trunc from 0x00e287 to 0x00f136 (0xeb0 bytes) 000391 1787599068.838194 trunc from 0x040000 to 0x001177 (0x3ee88 bytes) 000392 1787599068.928001 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1178): Mirror 6: 0x51f17238 (stale) Mirror 7: 0x51f17238 (stale) Mirror 10: 0x51f17238 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1178): Mirror 6: 0x51f17238 (stale) Mirror 7: 0x51f17238 (stale) Mirror 10: 0x51f17238 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1178: skipped 000393 1787599069.367136 trunc from 0x000dfc to 0x001177 (0x37c bytes) 000394 1787599069.372257 write 0x01b424 thru 0x020ff8 (0x5bd5 bytes) 000395 1787599069.377489 mapwrite 0x00d47f thru 0x0143e9 (0x6f6b bytes) 000396 1787599069.436152 read 0x011bc9 thru 0x01660f (0x4a47 bytes) 000397 1787599069.437059 mapwrite 0x013119 thru 0x019c61 (0x6b49 bytes) 000398 1787599069.461568 read 0x00a343 thru 0x00e73f (0x43fd bytes) 000399 1787599069.462241 punch from 0x00cb24 to 0x011a3b (0x4f18 bytes) 000400 1787599069.466320 fallocate 0x02a3a1 thru 0x030497 (0x60f7 bytes) 000401 1787599069.470419 mapread 0x023fc0 thru 0x02ecfb (0xad3c bytes) 000402 1787599069.483982 mapread 0x020845 thru 0x024ce9 (0x44a5 bytes) 000403 1787599069.498993 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30498): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30498): Mirror 6: 0x6c8aa71e (stale) Mirror 7: 0x6c8aa71e (stale) Mirror 10: 0x6c8aa71e (stale) CRC-32 checksum value for chunk [0, 0x30498): Mirror 1: 0x98295024 Mirror 11: 0x98295024 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30498: skipped 000404 1787599070.467433 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30498): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30498): Mirror 6: 0x6c8aa71e (stale) Mirror 7: 0x6c8aa71e (stale) Mirror 10: 0x6c8aa71e (stale) CRC-32 checksum value for chunk [0, 0x30498): Mirror 1: 0x98295024 Mirror 11: 0x98295024 Mirror 12: 0x98295024 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30498: skipped 000405 1787599071.319760 trunc from 0x030498 to 0x000e94 (0x2f603 bytes) 000406 1787599071.369263 mapread 0x000592 thru 0x000e94 (0x903 bytes) 000407 1787599071.378775 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe95): Mirror 7: 0x565f7c5 (stale) Mirror 10: 0x565f7c5 (stale) Mirror 11: 0x565f7c5 (stale) Mirror 12: 0x565f7c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe95): Mirror 7: 0x565f7c5 (stale) Mirror 10: 0x565f7c5 (stale) Mirror 11: 0x565f7c5 (stale) Mirror 12: 0x565f7c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe95: skipped 000408 1787599071.740104 mapread 0x0009e8 thru 0x000e94 (0x4ad bytes) 000409 1787599071.751002 mapread 0x000937 thru 0x000e94 (0x55e bytes) 000410 1787599071.753516 trunc from 0x000e95 to 0x00cb67 (0xbcd3 bytes) 000411 1787599071.780348 punch from 0x008d45 to 0x00cb67 (0x3e23 bytes) 000412 1787599071.784856 fallocate 0x0164b4 thru 0x01afa5 (0x4af2 bytes) 000413 1787599071.793852 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1afa6): Mirror 10: 0x1d136b90 (stale) Mirror 11: 0x256f3e04 (stale) Mirror 12: 0x256f3e04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1afa6): Mirror 10: 0x1d136b90 (stale) Mirror 11: 0x256f3e04 (stale) Mirror 12: 0x256f3e04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1afa6: skipped 000414 1787599072.172319 trunc from 0x01afa6 to 0x0300d3 (0x1512e bytes) 000415 1787599072.209610 trunc from 0x0300d4 to 0x03e778 (0xe6a5 bytes) 000416 1787599072.244640 mapwrite 0x028d2c thru 0x0357d5 (0xcaaa bytes) 000418 1787599072.332621 trunc from 0x03a44d to 0x03ac67 (0x81b bytes) 000419 1787599072.342895 write 0x009730 thru 0x018434 (0xed05 bytes) 000420 1787599072.382034 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e779): Mirror 11: 0x9ec3ca7d (stale) Mirror 12: 0x9ec3ca7d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e779): Mirror 11: 0x9ec3ca7d (stale) Mirror 12: 0x9ec3ca7d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e779: skipped 000421 1787599072.972442 read 0x004ab0 thru 0x0139da (0xef2b bytes) 000423 1787599072.988202 trunc from 0x02ea5f to 0x03bf69 (0xd50b bytes) 000424 1787599072.993023 fallocate 0x03eafe thru 0x03ffff (0x1502 bytes) 000425 1787599073.013384 MIRROR_RESYNC 3 mirrors 000426 1787599074.216048 mapwrite 0x007012 thru 0x00a3b8 (0x33a7 bytes) 000427 1787599074.390376 write 0x005b09 thru 0x00c8bd (0x6db5 bytes) 000428 1787599074.394885 read 0x00a4fc thru 0x00ea42 (0x4547 bytes) 000429 1787599074.396522 punch from 0x03657a to 0x03e778 (0x81ff bytes) 000430 1787599074.406673 trunc from 0x02c8eb to 0x02d278 (0x98e bytes) 000431 1787599074.414029 mapwrite 0x01a8ba thru 0x025c1e (0xb365 bytes) 000432 1787599074.510432 mapread 0x035fa5 thru 0x03e778 (0x87d4 bytes) 000433 1787599074.549646 mapwrite 0x031569 thru 0x032d33 (0x17cb bytes) 000434 1787599074.574968 mapread 0x01a729 thru 0x02711e (0xc9f6 bytes) 000435 1787599074.580890 read 0x021c2d thru 0x02f4ff (0xd8d3 bytes) 000436 1787599074.590239 mapread 0x019977 thru 0x024d20 (0xb3aa bytes) 000437 1787599074.591190 write 0x00822c thru 0x011211 (0x8fe6 bytes) 000438 1787599074.592367 trunc from 0x03e779 to 0x008e98 (0x358e0 bytes) 000439 1787599074.614374 punch from 0x007076 to 0x008e98 (0x1e23 bytes) 000440 1787599074.630892 fallocate 0x0331bc thru 0x03f801 (0xc646 bytes) 000441 1787599074.635986 mapread 0x02ba0a thru 0x02eee2 (0x34d9 bytes) 000442 1787599074.657323 mapread 0x02d891 thru 0x03398e (0x60fe bytes) 000443 1787599074.682794 write 0x03db5a thru 0x03ffff (0x24a6 bytes) 000444 1787599074.687725 fallocate 0x01b7f1 thru 0x022ab8 (0x72c8 bytes) 000445 1787599074.692462 trunc from 0x040000 to 0x0210af (0x1ef50 bytes) 000446 1787599074.718330 fallocate 0x008e0a thru 0x012b01 (0x9cf8 bytes) 000447 1787599074.724450 fallocate 0x00f1db thru 0x01481f (0x5645 bytes) 000448 1787599074.729419 write 0x01bc55 thru 0x028ef4 (0xd2a0 bytes) 000449 1787599074.737592 MIRROR_RESYNC 3 mirrors 000450 1787599075.502213 fallocate 0x015ad5 thru 0x019e18 (0x4344 bytes) 000451 1787599075.536809 read 0x00d535 thru 0x00fe34 (0x2900 bytes) 000452 1787599075.543452 trunc from 0x01f795 to 0x022b1b (0x3387 bytes) 000453 1787599075.549379 trunc from 0x02145a to 0x028ef4 (0x7a9b bytes) 000454 1787599075.554558 trunc from 0x01584d to 0x0232f2 (0xdaa6 bytes) 000455 1787599075.563805 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28ef5): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ef5): Mirror 11: 0x15300e65 (stale) Mirror 12: 0x15300e65 (stale) CRC-32 checksum value for chunk [0, 0x28ef5): Mirror 1: 0xd1b34577 Mirror 13: 0xd1b34577 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ef5: skipped 000456 1787599076.997951 punch from 0x019e66 to 0x01d4ec (0x3687 bytes) 000457 1787599077.094879 read 0x008065 thru 0x01443b (0xc3d7 bytes) 000458 1787599077.115375 trunc from 0x028ef5 to 0x01f432 (0x9ac2 bytes) 000459 1787599077.159064 trunc from 0x008ec1 to 0x016cbd (0xddfd bytes) 000460 1787599077.176307 trunc from 0x01f433 to 0x03ca5b (0x1d629 bytes) 000461 1787599077.224376 write 0x001ad0 thru 0x0058d7 (0x3e08 bytes) 000462 1787599077.236722 mapread 0x03623d thru 0x03a931 (0x46f5 bytes) 000463 1787599077.278921 MIRROR_RESYNC 4 mirrors 000464 1787599078.489327 trunc from 0x03ca5c to 0x028bc5 (0x13e96 bytes) 000465 1787599078.526400 punch from 0x016bd0 to 0x0189cb (0x1dfc bytes) 000466 1787599078.533390 trunc from 0x011288 to 0x0186a7 (0x7420 bytes) 000467 1787599078.541142 trunc from 0x028bc6 to 0x017d12 (0x10eb3 bytes) 000469 1787599078.576746 trunc from 0x017d13 to 0x014213 (0x3aff bytes) 000471 1787599078.615943 mapread 0x011019 thru 0x011184 (0x16c bytes) 000472 1787599078.628270 trunc from 0x014214 to 0x03c9ab (0x28798 bytes) 000474 1787599078.676407 write 0x021786 thru 0x02effe (0xd879 bytes) 000475 1787599078.699189 trunc from 0x014509 to 0x0240a3 (0xfb9b bytes) 000476 1787599078.723211 mapread 0x01e003 thru 0x021994 (0x3992 bytes) 000477 1787599078.743611 trunc from 0x01b601 to 0x02b410 (0xfe10 bytes) 000480 1787599078.754347 read 0x015659 thru 0x01f423 (0x9dcb bytes) 000481 1787599078.767752 mapread 0x0075b9 thru 0x00e91a (0x7362 bytes) 000482 1787599078.795761 mapwrite 0x036da6 thru 0x03ffff (0x925a bytes) 000483 1787599078.861009 read 0x037204 thru 0x03ffff (0x8dfc bytes) 000484 1787599078.862002 read 0x039746 thru 0x03ffff (0x68ba bytes) 000485 1787599078.865545 read 0x018760 thru 0x01aa99 (0x233a bytes) 000486 1787599078.872015 trunc from 0x018bf0 to 0x01be8c (0x329d bytes) 000487 1787599078.887879 mapwrite 0x0096c6 thru 0x013574 (0x9eaf bytes) 000488 1787599078.915379 read 0x02022a thru 0x02f5e4 (0xf3bb bytes) 000489 1787599078.920320 MIRROR_RESYNC 4 mirrors 000490 1787599080.214651 mapwrite 0x02b8a5 thru 0x02fd60 (0x44bc bytes) 000491 1787599080.343073 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x4d4a90ae (stale) Mirror 12: 0x4d4a90ae (stale) Mirror 13: 0x4d4a90ae (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7f88e21e Mirror 14: 0x7f88e21e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000492 1787599081.732093 trunc from 0x0158fd to 0x01c9b2 (0x70b6 bytes) 000493 1787599081.812540 trunc from 0x03ebf3 to 0x03ffff (0x140d bytes) 000494 1787599081.830965 fallocate 0x038c35 thru 0x03ffff (0x73cb bytes) 000495 1787599081.840141 mapread 0x006762 thru 0x00c387 (0x5c26 bytes) 000496 1787599081.878290 mapwrite 0x01d4e9 thru 0x028c38 (0xb750 bytes) 000497 1787599081.981226 trunc from 0x040000 to 0x002a30 (0x3d5cf bytes) 000498 1787599082.040012 mapwrite 0x0055a9 thru 0x01550c (0xff64 bytes) 000499 1787599082.191398 write 0x015dfd thru 0x01d51c (0x7720 bytes) 000500 1787599082.219825 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d51d): Mirror 11: 0x1341b0e (stale) Mirror 13: 0x1341b0e (stale) Mirror 14: 0x1341b0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d51d): Mirror 11: 0x1341b0e (stale) Mirror 13: 0x1341b0e (stale) Mirror 14: 0x1341b0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d51d: skipped 000501 1787599082.836082 mapwrite 0x01a1f7 thru 0x021088 (0x6e92 bytes) 000502 1787599082.895396 mapwrite 0x03163e thru 0x03c93b (0xb2fe bytes) 000503 1787599082.988818 read 0x02eb0c thru 0x0331e3 (0x46d8 bytes) 000504 1787599082.991150 fallocate 0x005209 thru 0x007e37 (0x2c2f bytes) 000505 1787599083.001172 read 0x02c5d8 thru 0x030f03 (0x492c bytes) 000506 1787599083.009259 MIRROR_RESYNC 4 mirrors 000507 1787599084.255885 punch from 0x02b8f5 to 0x032526 (0x6c32 bytes) 000508 1787599084.303239 trunc from 0x01ad3d to 0x026d17 (0xbfdb bytes) 000509 1787599084.311405 write 0x02d243 thru 0x03a991 (0xd74f bytes) 000510 1787599084.322484 punch from 0x00a639 to 0x00adfb (0x7c3 bytes) 000511 1787599084.326841 write 0x03a704 thru 0x03ffff (0x58fc bytes) 000512 1787599084.328929 mapread 0x00149e thru 0x00678f (0x52f2 bytes) 000513 1787599084.357920 fallocate 0x03c792 thru 0x03ffff (0x386e bytes) 000516 1787599084.365414 trunc from 0x008c40 to 0x0135d4 (0xa995 bytes) 000518 1787599084.373447 mapwrite 0x011c1a thru 0x012478 (0x85f bytes) 000519 1787599084.408369 punch from 0x026042 to 0x031ab2 (0xba71 bytes) 000520 1787599084.420669 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x25dd3812 (stale) Mirror 14: 0x25dd3812 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x25dd3812 (stale) Mirror 14: 0x25dd3812 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000521 1787599084.966910 trunc from 0x023811 to 0x02b5b6 (0x7da6 bytes) 000522 1787599084.985484 trunc from 0x02459b to 0x02b15c (0x6bc2 bytes) 000523 1787599085.003820 punch from 0x036519 to 0x03ffff (0x9ae7 bytes) 000524 1787599085.011230 read 0x01132f thru 0x014d05 (0x39d7 bytes) 000525 1787599085.021986 punch from 0x038c95 to 0x03bbb7 (0x2f23 bytes) 000526 1787599085.053234 trunc from 0x0377a1 to 0x03ffff (0x885f bytes) 000527 1787599085.062684 mapwrite 0x024670 thru 0x0345aa (0xff3b bytes) 000528 1787599085.164085 read 0x030fc4 thru 0x03f37a (0xe3b7 bytes) 000529 1787599085.178020 mapread 0x021b9c thru 0x02c25f (0xa6c4 bytes) 000530 1787599085.180936 read 0x032830 thru 0x03be18 (0x95e9 bytes) 000531 1787599085.197167 MIRROR_RESYNC 3 mirrors 000532 1787599086.654657 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) 000533 1787599087.182712 trunc from 0x040000 to 0x01164a (0x2e9b5 bytes) 000534 1787599087.261244 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1164b): Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1164b): Mirror 13: 0x30cfa5fc (stale) CRC-32 checksum value for chunk [0, 0x1164b): Mirror 1: 0x30cfa5fc Mirror 14: 0x30cfa5fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1164b: skipped 000535 1787599088.412691 trunc from 0x00657e to 0x00d809 (0x728c bytes) 000536 1787599088.506013 write 0x034a6a thru 0x039a7c (0x5013 bytes) 000537 1787599088.525278 trunc from 0x01eb22 to 0x02b780 (0xcc5f bytes) 000538 1787599088.537415 trunc from 0x01e8ac to 0x021a8d (0x31e2 bytes) 000539 1787599088.551381 punch from 0x035f67 to 0x039a7c (0x3b16 bytes) 000540 1787599088.595472 punch from 0x01b080 to 0x01e868 (0x37e9 bytes) 000541 1787599088.609386 mapread 0x010a69 thru 0x01f33c (0xe8d4 bytes) 000542 1787599088.689002 fallocate 0x03f597 thru 0x03ffff (0xa69 bytes) 000543 1787599088.714567 MIRROR_RESYNC 3 mirrors 000544 1787599090.623454 punch from 0x0220da to 0x030295 (0xe1bc bytes) 000545 1787599090.671928 read 0x00766b thru 0x0089dc (0x1372 bytes) 000546 1787599090.682285 read 0x01c032 thru 0x021275 (0x5244 bytes) 000547 1787599090.744893 mapread 0x02addc thru 0x02bfc5 (0x11ea bytes) 000548 1787599090.746332 mapread 0x016610 thru 0x01b5f5 (0x4fe6 bytes) 000549 1787599090.748394 read 0x023f3c thru 0x02637f (0x2444 bytes) 000550 1787599090.749603 mapread 0x011a6d thru 0x01c03a (0xa5ce bytes) 000551 1787599090.753657 punch from 0x00aaae to 0x019a0d (0xef60 bytes) 000552 1787599090.765160 fallocate 0x017063 thru 0x01efd3 (0x7f71 bytes) 000553 1787599090.770410 write 0x03c42b thru 0x03ffff (0x3bd5 bytes) 000554 1787599090.771356 write 0x01b4df thru 0x0205ec (0x510e bytes) 000555 1787599090.772281 write 0x03f9be thru 0x03ffff (0x642 bytes) 000556 1787599090.775676 trunc from 0x00c518 to 0x011111 (0x4bfa bytes) 000557 1787599090.791406 trunc from 0x025c42 to 0x02dab9 (0x7e78 bytes) 000558 1787599090.797668 write 0x031061 thru 0x038e53 (0x7df3 bytes) 000559 1787599090.799249 punch from 0x03da76 to 0x03ffff (0x258a bytes) 000560 1787599090.829343 mapread 0x0347e1 thru 0x038d36 (0x4556 bytes) 000561 1787599090.837052 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xae35c212 (stale) Mirror 14: 0xae35c212 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee92d229 Mirror 15: 0xee92d229 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000562 1787599092.161021 mapwrite 0x037bab thru 0x03ffff (0x8455 bytes) 000563 1787599092.339095 mapread 0x01dfb1 thru 0x02a29d (0xc2ed bytes) 000564 1787599092.394191 read 0x013964 thru 0x01a783 (0x6e20 bytes) 000566 1787599092.413168 read 0x03460b thru 0x03ffff (0xb9f5 bytes) 000567 1787599092.420261 fallocate 0x02f6a3 thru 0x030795 (0x10f3 bytes) 000568 1787599092.434948 write 0x0121ec thru 0x015784 (0x3599 bytes) 000569 1787599092.467799 write 0x012abf thru 0x01327c (0x7be bytes) 000570 1787599092.473570 read 0x025b3c thru 0x0348a7 (0xed6c bytes) 000571 1787599092.474190 punch from 0x035ba1 to 0x03fce9 (0xa149 bytes) 000572 1787599092.484114 mapread 0x037bee thru 0x03ffff (0x8412 bytes) 000573 1787599092.506031 write 0x020711 thru 0x02df38 (0xd828 bytes) 000574 1787599092.512486 punch from 0x006a9f to 0x012c74 (0xc1d6 bytes) 000575 1787599092.532117 trunc from 0x040000 to 0x000e5b (0x3f1a4 bytes) 000576 1787599092.577243 mapread 0x0002d8 thru 0x000e5b (0xb84 bytes) 000577 1787599092.578162 mapwrite 0x019d13 thru 0x01a5d9 (0x8c7 bytes) 000578 1787599092.706379 trunc from 0x016c3f to 0x01825b (0x161d bytes) 000579 1787599092.712634 punch from 0x0072d2 to 0x011d4a (0xaa79 bytes) 000580 1787599092.724236 trunc from 0x01a5da to 0x030960 (0x16387 bytes) 000581 1787599092.752078 mapwrite 0x018522 thru 0x020931 (0x8410 bytes) 000582 1787599092.881126 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30961): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30961): Mirror 13: 0xb0f0b79 (stale) Mirror 14: 0xb0f0b79 (stale) Mirror 15: 0x3ea9a92c (stale) CRC-32 checksum value for chunk [0, 0x30961): Mirror 1: 0x72e0250a Mirror 16: 0x72e0250a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30961: skipped 000583 1787599094.237694 punch from 0x01e0a6 to 0x022790 (0x46eb bytes) 000584 1787599094.288039 trunc from 0x030961 to 0x028daf (0x7bb1 bytes) 000585 1787599094.317433 punch from 0x026389 to 0x028daf (0x2a27 bytes) 000586 1787599094.324164 punch from 0x000e00 to 0x005d04 (0x4f05 bytes) 000587 1787599094.331060 write 0x038cf3 thru 0x03d781 (0x4a8f bytes) 000588 1787599094.332387 trunc from 0x025268 to 0x031f52 (0xcceb bytes) 000589 1787599094.341869 mapread 0x026fd6 thru 0x03350f (0xc53a bytes) 000591 1787599094.374203 fallocate 0x01d8ef thru 0x029ce4 (0xc3f6 bytes) 000593 1787599094.381064 read 0x000042 thru 0x00fe52 (0xfe11 bytes) 000594 1787599094.398637 fallocate 0x03b749 thru 0x03ffff (0x48b7 bytes) 000595 1787599094.410282 MIRROR_RESYNC 5 mirrors 000596 1787599095.974765 write 0x014b2a thru 0x01ea65 (0x9f3c bytes) 000597 1787599096.052301 mapwrite 0x02e027 thru 0x0335d5 (0x55af bytes) 000598 1787599096.145237 punch from 0x021639 to 0x022762 (0x112a bytes) 000599 1787599096.165541 write 0x01a4fb thru 0x02060a (0x6110 bytes) 000600 1787599096.174670 trunc from 0x0171d8 to 0x01ae86 (0x3caf bytes) 000601 1787599096.190155 punch from 0x00c1e3 to 0x015085 (0x8ea3 bytes) 000602 1787599096.195079 read 0x0146a3 thru 0x0162e6 (0x1c44 bytes) 000603 1787599096.200680 mapread 0x028508 thru 0x02bd72 (0x386b bytes) 000604 1787599096.210725 trunc from 0x03d782 to 0x000ba8 (0x3cbd9 bytes) 000605 1787599096.231155 punch from 0x0004ef to 0x000ba8 (0x6ba bytes) 000606 1787599096.236833 fallocate 0x03c440 thru 0x03ffff (0x3bc0 bytes) 000607 1787599096.243974 trunc from 0x01536a to 0x01bfe7 (0x6c7e bytes) 000609 1787599096.250794 fallocate 0x00b5da thru 0x017562 (0xbf89 bytes) 000611 1787599096.257449 read 0x0223c9 thru 0x02e580 (0xc1b8 bytes) 000612 1787599096.279102 mapwrite 0x031578 thru 0x037bf9 (0x6682 bytes) 000613 1787599096.299671 mapwrite 0x03006c thru 0x037490 (0x7425 bytes) 000614 1787599096.329726 trunc from 0x03eade to 0x03ffff (0x1522 bytes) 000615 1787599096.343150 write 0x00e9a4 thru 0x014b2a (0x6187 bytes) 000616 1787599096.344319 trunc from 0x02cf62 to 0x03c75d (0xf7fc bytes) 000617 1787599096.353716 trunc from 0x03877c to 0x03ffff (0x7884 bytes) 000618 1787599096.373821 MIRROR_RESYNC 5 mirrors 000619 1787599097.510193 mapread 0x025952 thru 0x0263a4 (0xa53 bytes) 000620 1787599097.529863 punch from 0x016f44 to 0x024ef4 (0xdfb1 bytes) 000621 1787599097.593489 trunc from 0x040000 to 0x03dc52 (0x23ad bytes) 000622 1787599097.624777 mapread 0x001230 thru 0x004724 (0x34f5 bytes) 000623 1787599097.638184 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3dc53): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc53): Mirror 13: 0xfd38dbdf (stale) Mirror 14: 0xfd38dbdf (stale) Mirror 15: 0xfd38dbdf (stale) Mirror 16: 0xfd38dbdf (stale) CRC-32 checksum value for chunk [0, 0x3dc53): Mirror 1: 0xfd38dbdf Mirror 17: 0xfd38dbdf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc53: skipped 000624 1787599098.821719 mapwrite 0x02fbd7 thru 0x032c2d (0x3057 bytes) 000625 1787599098.965627 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc53): Mirror 13: 0xfd38dbdf (stale) Mirror 14: 0xfd38dbdf (stale) Mirror 15: 0xfd38dbdf (stale) Mirror 16: 0xfd38dbdf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc53): Mirror 13: 0xfd38dbdf (stale) Mirror 14: 0xfd38dbdf (stale) Mirror 15: 0xfd38dbdf (stale) Mirror 16: 0xfd38dbdf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc53: skipped 000626 1787599099.531358 trunc from 0x03dc53 to 0x019489 (0x247c9 bytes) 000627 1787599099.571618 mapread 0x019023 thru 0x019489 (0x467 bytes) 000628 1787599099.586247 mapread 0x01390f thru 0x016c23 (0x3315 bytes) 000629 1787599099.616897 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1948a): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1948a): Mirror 13: 0xd8261e80 (stale) Mirror 14: 0xd8261e80 (stale) Mirror 15: 0xd8261e80 (stale) Mirror 16: 0xd8261e80 (stale) CRC-32 checksum value for chunk [0, 0x1948a): Mirror 1: 0xd8261e80 Mirror 17: 0xd8261e80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1948a: skipped 000630 1787599100.787144 write 0x01d5c3 thru 0x0281c0 (0xabfe bytes) 000631 1787599100.829687 read 0x011736 thru 0x011bc8 (0x493 bytes) 000633 1787599100.835861 fallocate 0x02583f thru 0x02eebf (0x9681 bytes) 000634 1787599100.840813 punch from 0x00b14c to 0x01a504 (0xf3b9 bytes) 000635 1787599100.847271 write 0x03d92f thru 0x03ffff (0x26d1 bytes) 000636 1787599100.849814 trunc from 0x040000 to 0x002c6a (0x3d395 bytes) 000637 1787599100.870856 trunc from 0x002c6b to 0x02510e (0x224a4 bytes) 000638 1787599100.889994 trunc from 0x02510f to 0x035aed (0x109df bytes) 000639 1787599100.905950 trunc from 0x035aee to 0x004b86 (0x30f67 bytes) 000641 1787599100.939730 trunc from 0x003f3d to 0x004b86 (0xc4a bytes) 000642 1787599100.948379 read 0x00037e thru 0x004b86 (0x4809 bytes) 000643 1787599100.959585 write 0x0165a2 thru 0x01b951 (0x53b0 bytes) 000644 1787599100.960800 trunc from 0x01b952 to 0x001ca7 (0x19caa bytes) 000645 1787599100.975860 write 0x03fb97 thru 0x03ffff (0x469 bytes) 000646 1787599100.977336 trunc from 0x02adcb to 0x03470f (0x9945 bytes) 000647 1787599100.984043 fallocate 0x01c3d8 thru 0x025420 (0x9049 bytes) 000648 1787599100.989161 mapwrite 0x03639a thru 0x03ffff (0x9c66 bytes) 000649 1787599101.046379 trunc from 0x040000 to 0x022c3a (0x1d3c5 bytes) 000650 1787599101.068809 mapread 0x018bbf thru 0x019b61 (0xfa3 bytes) 000651 1787599101.070402 trunc from 0x00a92d to 0x0131a7 (0x887b bytes) 000652 1787599101.082136 write 0x02d20a thru 0x02e3e7 (0x11de bytes) 000653 1787599101.113085 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e3e8): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e3e8): Mirror 13: 0x39556cf6 (stale) Mirror 14: 0x39556cf6 (stale) Mirror 15: 0x39556cf6 (stale) Mirror 16: 0x39556cf6 (stale) Mirror 17: 0x39556cf6 (stale) CRC-32 checksum value for chunk [0, 0x2e3e8): Mirror 1: 0x3cdba38d Mirror 18: 0x3cdba38d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e3e8: skipped 000654 1787599102.051731 punch from 0x0160ed to 0x01c006 (0x5f1a bytes) 000655 1787599102.142369 mapread 0x026301 thru 0x02d8d0 (0x75d0 bytes) 000656 1787599102.183288 mapread 0x0172c9 thru 0x026eb7 (0xfbef bytes) 000657 1787599102.210485 trunc from 0x000379 to 0x0029c6 (0x264e bytes) 000658 1787599102.233756 trunc from 0x015a1d to 0x0230d6 (0xd6ba bytes) 000659 1787599102.240343 punch from 0x00d681 to 0x0153bb (0x7d3b bytes) 000660 1787599102.252017 mapread 0x004482 thru 0x0096ee (0x526d bytes) 000661 1787599102.271987 trunc from 0x02e3e8 to 0x02fe5a (0x1a73 bytes) 000662 1787599102.361380 trunc from 0x00a7e4 to 0x0118d6 (0x70f3 bytes) 000663 1787599102.371077 trunc from 0x02fe5b to 0x0174fb (0x1895f bytes) 000664 1787599102.398340 fallocate 0x0038c3 thru 0x00b5b2 (0x7cf0 bytes) 000665 1787599102.405752 trunc from 0x01325d to 0x015db4 (0x2b58 bytes) 000666 1787599102.410801 trunc from 0x002827 to 0x002ef0 (0x6ca bytes) 000667 1787599102.422927 mapread 0x000f75 thru 0x009834 (0x88c0 bytes) 000668 1787599102.432864 trunc from 0x0174fc to 0x03f56d (0x28072 bytes) 000669 1787599102.455659 read 0x002ead thru 0x010426 (0xd57a bytes) 000670 1787599102.465676 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f56e): Mirror 13: 0xa537ff51 (stale) Mirror 14: 0xa537ff51 (stale) Mirror 15: 0xa537ff51 (stale) Mirror 16: 0xa537ff51 (stale) Mirror 18: 0x8a79d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f56e): Mirror 13: 0xa537ff51 (stale) Mirror 14: 0xa537ff51 (stale) Mirror 15: 0xa537ff51 (stale) Mirror 16: 0xa537ff51 (stale) Mirror 18: 0x8a79d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f56e: skipped 000671 1787599102.982419 trunc from 0x02b02b to 0x030b82 (0x5b58 bytes) 000672 1787599102.987835 trunc from 0x032d66 to 0x03f56d (0xc808 bytes) 000673 1787599102.992587 punch from 0x00ece7 to 0x0137b2 (0x4acc bytes) 000674 1787599102.997305 trunc from 0x00dfa2 to 0x0121e5 (0x4244 bytes) 000675 1787599103.002243 mapread 0x008b31 thru 0x012bb6 (0xa086 bytes) 000676 1787599103.020069 punch from 0x03003b to 0x03292c (0x28f2 bytes) 000678 1787599103.025688 read 0x03007c thru 0x03cdaa (0xcd2f bytes) 000679 1787599103.034140 fallocate 0x01d0c8 thru 0x01ef01 (0x1e3a bytes) 000680 1787599103.038590 write 0x024ae5 thru 0x03030f (0xb82b bytes) 000681 1787599103.049048 trunc from 0x03f56e to 0x0268c9 (0x18ca4 bytes) 000682 1787599103.067297 punch from 0x01efed to 0x02192a (0x293e bytes) 000683 1787599103.074017 write 0x03a5cf thru 0x03ffff (0x5a31 bytes) 000684 1787599103.075344 trunc from 0x040000 to 0x02b1f0 (0x14e0f bytes) 000685 1787599103.097228 fallocate 0x03b300 thru 0x03ffff (0x4d00 bytes) 000686 1787599103.107579 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b1f1): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1f1): Mirror 13: 0xf049aa14 (stale) Mirror 14: 0xf049aa14 (stale) Mirror 15: 0xf049aa14 (stale) Mirror 16: 0xf049aa14 (stale) Mirror 18: 0x147c93a1 (stale) CRC-32 checksum value for chunk [0, 0x2b1f1): Mirror 1: 0xfcd4d9be Mirror 19: 0xfcd4d9be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1f1: skipped 000687 1787599104.095126 mapwrite 0x032216 thru 0x03ffff (0xddea bytes) 000688 1787599104.314750 mapwrite 0x03530d thru 0x03ffff (0xacf3 bytes) 000690 1787599104.384240 punch from 0x02d031 to 0x03bb75 (0xeb45 bytes) 000691 1787599104.393464 trunc from 0x040000 to 0x031b20 (0xe4df bytes) 000692 1787599104.453375 mapread 0x00dbb5 thru 0x016b74 (0x8fc0 bytes) 000693 1787599104.506158 read 0x01d4b3 thru 0x022b63 (0x56b1 bytes) 000694 1787599104.519734 mapwrite 0x016048 thru 0x01657b (0x534 bytes) 000695 1787599104.552335 mapread 0x024527 thru 0x0315fd (0xd0d7 bytes) 000696 1787599104.563880 fallocate 0x0363a6 thru 0x03ffff (0x9c5a bytes) 000697 1787599104.569866 write 0x012885 thru 0x0140f2 (0x186e bytes) 000698 1787599104.571860 trunc from 0x02d893 to 0x031b20 (0x428e bytes) 000699 1787599104.580852 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31b21): Mirror 14: 0x730f8e25 (stale) Mirror 15: 0x730f8e25 (stale) Mirror 16: 0x730f8e25 (stale) Mirror 18: 0xcea64a2d (stale) Mirror 19: 0xcd0206 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b21): Mirror 14: 0x730f8e25 (stale) Mirror 15: 0x730f8e25 (stale) Mirror 16: 0x730f8e25 (stale) Mirror 18: 0xcea64a2d (stale) Mirror 19: 0xcd0206 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b21: skipped 000700 1787599105.242262 trunc from 0x031b21 to 0x0348fc (0x2ddc bytes) 000702 1787599105.339788 mapwrite 0x00b4cb thru 0x01a521 (0xf057 bytes) 000703 1787599105.467544 punch from 0x01007c to 0x0141b9 (0x413e bytes) 000704 1787599105.475157 punch from 0x017fa7 to 0x02402f (0xc089 bytes) 000705 1787599105.482383 punch from 0x02c5af to 0x0348fc (0x834e bytes) 000706 1787599105.489816 trunc from 0x0348fd to 0x007071 (0x2d88b bytes) 000707 1787599105.515486 fallocate 0x02b8e4 thru 0x0349df (0x90fc bytes) 000708 1787599105.522564 fallocate 0x011455 thru 0x018aab (0x7657 bytes) 000709 1787599105.530186 read 0x00a013 thru 0x018aab (0xea99 bytes) 000710 1787599105.545612 punch from 0x00a036 to 0x010d2d (0x6cf8 bytes) 000711 1787599105.552632 fallocate 0x022ff7 thru 0x02b29e (0x82a8 bytes) 000712 1787599105.559956 trunc from 0x018aac to 0x010733 (0x8378 bytes) 000713 1787599105.581886 trunc from 0x010734 to 0x00b9a6 (0x4d8d bytes) 000714 1787599105.630986 mapwrite 0x02ffae thru 0x03a0ec (0xa13f bytes) 000715 1787599105.739895 trunc from 0x003f70 to 0x00b86d (0x78fe bytes) 000716 1787599105.763455 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a0ed): Mirror 14: 0xb4e1b6ad (stale) Mirror 15: 0xb4e1b6ad (stale) Mirror 18: 0xe41328d0 (stale) Mirror 19: 0xc8c19682 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0ed): Mirror 14: 0xb4e1b6ad (stale) Mirror 15: 0xb4e1b6ad (stale) Mirror 18: 0xe41328d0 (stale) Mirror 19: 0xc8c19682 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0ed: skipped 000717 1787599106.258873 mapwrite 0x007213 thru 0x00f762 (0x8550 bytes) 000718 1787599106.315204 trunc from 0x018a35 to 0x01ff16 (0x74e2 bytes) 000719 1787599106.324760 mapread 0x00330f thru 0x00b7fa (0x84ec bytes) 000720 1787599106.327702 fallocate 0x00ab9c thru 0x00e917 (0x3d7c bytes) 000721 1787599106.337161 mapread 0x018461 thru 0x01f44d (0x6fed bytes) 000723 1787599106.353051 fallocate 0x0137d1 thru 0x0218c7 (0xe0f7 bytes) 000724 1787599106.359305 trunc from 0x03a0ed to 0x03333f (0x6dad bytes) 000725 1787599106.390027 mapwrite 0x00a66c thru 0x015147 (0xaadc bytes) 000728 1787599106.439338 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33340): Mirror 15: 0x2c57de3f (stale) Mirror 18: 0x6ba52b72 (stale) Mirror 19: 0x2d9bc604 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33340): Mirror 15: 0x2c57de3f (stale) Mirror 18: 0x6ba52b72 (stale) Mirror 19: 0x2d9bc604 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33340: skipped 000729 1787599106.886264 mapread 0x000cf6 thru 0x00c366 (0xb671 bytes) 000731 1787599106.918714 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33340): Mirror 15: 0x2c57de3f (stale) Mirror 18: 0x6ba52b72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33340): Mirror 15: 0x2c57de3f (stale) Mirror 18: 0x6ba52b72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33340: skipped 000732 1787599107.292003 fallocate 0x032839 thru 0x03ae1b (0x85e3 bytes) 000733 1787599107.305037 trunc from 0x03ae1c to 0x02f27f (0xbb9c bytes) 000734 1787599107.342327 punch from 0x02bb02 to 0x02f27f (0x377e bytes) 000735 1787599107.348981 trunc from 0x02f280 to 0x02d62f (0x1c50 bytes) 000736 1787599107.380010 mapwrite 0x02b218 thru 0x02c57d (0x1366 bytes) 000738 1787599107.471870 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d630): Mirror 18: 0xbaeb0ded (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d630): Mirror 18: 0xbaeb0ded (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d630: skipped 000739 1787599107.911474 write 0x03d4ef thru 0x03ffff (0x2b11 bytes) 000740 1787599107.918045 write 0x020f1c thru 0x0228a5 (0x198a bytes) 000741 1787599107.950997 punch from 0x022618 to 0x03086b (0xe254 bytes) 000742 1787599107.981859 mapwrite 0x025973 thru 0x02be44 (0x64d2 bytes) 000743 1787599108.066582 punch from 0x038260 to 0x03ffff (0x7da0 bytes) 000744 1787599108.084249 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) 000745 1787599108.250953 mapread 0x00e151 thru 0x011851 (0x3701 bytes) 000746 1787599108.270965 trunc from 0x040000 to 0x01d670 (0x2298f bytes) 000748 1787599108.292463 write 0x01dc20 thru 0x02d173 (0xf554 bytes) 000749 1787599108.297878 mapread 0x02b9fd thru 0x02d173 (0x1777 bytes) 000750 1787599108.302710 punch from 0x000593 to 0x008925 (0x8393 bytes) 000751 1787599108.307921 read 0x001915 thru 0x00caf9 (0xb1e5 bytes) 000753 1787599108.323007 MIRROR_EXTEND to 2 mirrors 000754 1787599109.092395 MIRROR_RESYNC 2 mirrors 000755 1787599109.490942 MIRROR_RESYNC 2 mirrors 000756 1787599110.018130 mapwrite 0x0277d4 thru 0x036a0e (0xf23b bytes) 000758 1787599110.275794 trunc from 0x036a0f to 0x014e05 (0x21c09 bytes) 000759 1787599110.322564 trunc from 0x00c5de to 0x00f43e (0x2e61 bytes) 000760 1787599110.336166 mapread 0x010605 thru 0x014e05 (0x4801 bytes) 000762 1787599110.352056 trunc from 0x00ef27 to 0x010489 (0x1563 bytes) 000763 1787599110.364361 mapwrite 0x0269ab thru 0x02944f (0x2aa5 bytes) 000764 1787599110.419562 write 0x01182d thru 0x01c5ef (0xadc3 bytes) 000765 1787599110.431014 MIRROR_RESYNC 2 mirrors 000766 1787599111.565081 trunc from 0x005cf3 to 0x015467 (0xf775 bytes) 000767 1787599111.608271 mapread 0x00c495 thru 0x012bfd (0x6769 bytes) 000768 1787599111.637493 trunc from 0x00cc02 to 0x01a7b8 (0xdbb7 bytes) 000770 1787599111.646569 trunc from 0x023460 to 0x02944f (0x5ff0 bytes) 000771 1787599111.654147 punch from 0x01ec8d to 0x028762 (0x9ad6 bytes) 000772 1787599111.661380 punch from 0x010ab8 to 0x017750 (0x6c99 bytes) 000773 1787599111.673018 MIRROR_RESYNC 2 mirrors 000775 1787599112.638965 mapwrite 0x00fe4d thru 0x01c94f (0xcb03 bytes) 000776 1787599112.811015 write 0x01fa1e thru 0x0226ce (0x2cb1 bytes) 000777 1787599112.851187 MIRROR_RESYNC 2 mirrors 000778 1787599113.829407 fallocate 0x0086db thru 0x011b9c (0x94c2 bytes) 000779 1787599113.887252 trunc from 0x000760 to 0x0076ab (0x6f4c bytes) 000781 1787599113.897121 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29450): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29450): Mirror 2: 0xd921571e (stale) CRC-32 checksum value for chunk [0, 0x29450): Mirror 1: 0xd921571e Mirror 3: 0xd921571e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29450: skipped 000782 1787599115.083116 trunc from 0x029450 to 0x01dc23 (0xb82c bytes) 000783 1787599115.170092 trunc from 0x01dc24 to 0x012d0c (0xaf17 bytes) 000784 1787599115.196888 read 0x00909c thru 0x00a967 (0x18cc bytes) 000785 1787599115.203910 punch from 0x006fb1 to 0x00e92a (0x797a bytes) 000786 1787599115.210031 fallocate 0x026559 thru 0x02d572 (0x701a bytes) 000787 1787599115.222950 trunc from 0x00de21 to 0x012d0c (0x4eec bytes) 000788 1787599115.237358 mapread 0x0076e6 thru 0x012d0c (0xb627 bytes) 000789 1787599115.297948 mapwrite 0x03b558 thru 0x03ffff (0x4aa8 bytes) 000790 1787599115.403074 write 0x02e3d3 thru 0x03cd5c (0xe98a bytes) 000791 1787599115.419013 read 0x02cd7c thru 0x0338c3 (0x6b48 bytes) 000793 1787599115.461619 read 0x02bf6f thru 0x032bf4 (0x6c86 bytes) 000794 1787599115.462647 trunc from 0x03f55c to 0x03ffff (0xaa4 bytes) 000795 1787599115.468937 read 0x03f2db thru 0x03ffff (0xd25 bytes) 000796 1787599115.475111 write 0x034529 thru 0x03ffff (0xbad7 bytes) 000797 1787599115.477483 write 0x033df2 thru 0x03fb94 (0xbda3 bytes) 000798 1787599115.479641 punch from 0x038512 to 0x03ffff (0x7aee bytes) 000799 1787599115.514492 mapwrite 0x037e94 thru 0x03ffff (0x816c bytes) 000801 1787599115.587464 mapwrite 0x0186c6 thru 0x020250 (0x7b8b bytes) 000802 1787599115.631968 trunc from 0x040000 to 0x01b50e (0x24af1 bytes) 000803 1787599115.680147 fallocate 0x00919a thru 0x009bc5 (0xa2c bytes) 000804 1787599115.695631 punch from 0x01610e to 0x01b50e (0x5401 bytes) 000805 1787599115.706417 punch from 0x00fe46 to 0x011e71 (0x202c bytes) 000806 1787599115.724407 punch from 0x003750 to 0x004815 (0x10c6 bytes) 000807 1787599115.739932 punch from 0x0110af to 0x014bdd (0x3b2f bytes) 000808 1787599115.747492 mapread 0x0193cd thru 0x01b50e (0x2142 bytes) 000809 1787599115.767674 read 0x008d6e thru 0x00b582 (0x2815 bytes) 000810 1787599115.774022 MIRROR_RESYNC 3 mirrors 000811 1787599116.615425 fallocate 0x00b029 thru 0x01510e (0xa0e6 bytes) 000812 1787599116.642374 mapwrite 0x031953 thru 0x039d1b (0x83c9 bytes) 000813 1787599116.728725 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39d1c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d1c): Mirror 2: 0x8937f7a7 (stale) Mirror 3: 0x8937f7a7 (stale) CRC-32 checksum value for chunk [0, 0x39d1c): Mirror 1: 0x337cccf4 Mirror 4: 0x337cccf4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d1c: skipped 000814 1787599117.731487 punch from 0x01b12e to 0x0237f4 (0x86c7 bytes) 000816 1787599117.772178 trunc from 0x039d1c to 0x0356af (0x466c bytes) 000817 1787599117.791865 trunc from 0x0356b0 to 0x00c4d0 (0x291df bytes) 000818 1787599117.802788 mapwrite 0x008ce0 thru 0x013f14 (0xb235 bytes) 000820 1787599117.863244 read 0x00788d thru 0x00b297 (0x3a0b bytes) 000821 1787599117.865656 trunc from 0x013f15 to 0x03f05b (0x2b147 bytes) 000822 1787599117.891373 trunc from 0x02f365 to 0x030201 (0xe9d bytes) 000823 1787599117.915983 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 2: 0x8e91586c (stale) Mirror 3: 0x8e91586c (stale) Mirror 4: 0x4ec72d4e (stale) CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 1: 0xa9b65b7c Mirror 5: 0xa9b65b7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f05c: skipped 000824 1787599118.969003 mapread 0x0228e4 thru 0x0242ce (0x19eb bytes) 000825 1787599119.010865 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 2: 0x8e91586c (stale) Mirror 3: 0x8e91586c (stale) Mirror 4: 0x4ec72d4e (stale) CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 1: 0xa9b65b7c Mirror 5: 0xa9b65b7c Mirror 6: 0xa9b65b7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f05c: skipped 000826 1787599120.256917 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 2: 0x8e91586c (stale) Mirror 3: 0x8e91586c (stale) Mirror 4: 0x4ec72d4e (stale) CRC-32 checksum value for chunk [0, 0x3f05c): Mirror 1: 0xa9b65b7c Mirror 5: 0xa9b65b7c Mirror 6: 0xa9b65b7c Mirror 7: 0xa9b65b7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f05c: skipped 000827 1787599121.595993 read 0x0113ee thru 0x018630 (0x7243 bytes) 000828 1787599121.609372 write 0x039bf6 thru 0x03ffff (0x640a bytes) 000829 1787599121.673851 fallocate 0x014b60 thru 0x022dc9 (0xe26a bytes) 000830 1787599121.683358 trunc from 0x0049fa to 0x011381 (0xc988 bytes) 000831 1787599121.694727 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0x9376f28f (stale) Mirror 5: 0xfa4d53c4 (stale) Mirror 6: 0xfa4d53c4 (stale) Mirror 7: 0xfa4d53c4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xedfea69 Mirror 8: 0xedfea69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000832 1787599123.019784 trunc from 0x03c47b to 0x03ffff (0x3b85 bytes) 000833 1787599123.126957 fallocate 0x011739 thru 0x01cfc5 (0xb88d bytes) 000834 1787599123.148340 trunc from 0x040000 to 0x02a8bb (0x15744 bytes) 000835 1787599123.214018 trunc from 0x0164a2 to 0x018c2f (0x278e bytes) 000836 1787599123.220755 trunc from 0x01c503 to 0x027c9b (0xb799 bytes) 000837 1787599123.235057 write 0x0280dd thru 0x030d9f (0x8cc3 bytes) 000838 1787599123.273665 fallocate 0x02692c thru 0x027eed (0x15c2 bytes) 000839 1787599123.282961 write 0x0128a5 thru 0x012c1a (0x376 bytes) 000840 1787599123.298977 mapwrite 0x00ea37 thru 0x0160f8 (0x76c2 bytes) 000841 1787599123.387307 trunc from 0x030da0 to 0x003f2f (0x2ce70 bytes) 000842 1787599123.429865 trunc from 0x0021e8 to 0x003b0a (0x1923 bytes) 000843 1787599123.442224 read 0x001366 thru 0x003f2f (0x2bca bytes) 000844 1787599123.484065 trunc from 0x003f30 to 0x025c88 (0x21d59 bytes) 000846 1787599123.521890 mapread 0x015821 thru 0x016e46 (0x1626 bytes) 000847 1787599123.549845 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25c89): Mirror 2: 0xbc836bbe (stale) Mirror 3: 0xbc836bbe (stale) Mirror 5: 0xcf7fcabb (stale) Mirror 6: 0xcf7fcabb (stale) Mirror 7: 0xcf7fcabb (stale) Mirror 8: 0xe54fdc6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c89): Mirror 2: 0xbc836bbe (stale) Mirror 3: 0xbc836bbe (stale) Mirror 5: 0xcf7fcabb (stale) Mirror 6: 0xcf7fcabb (stale) Mirror 7: 0xcf7fcabb (stale) Mirror 8: 0xe54fdc6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c89: skipped 000848 1787599124.201435 punch from 0x000dcd to 0x00bb34 (0xad68 bytes) 000849 1787599124.209707 fallocate 0x03ac04 thru 0x03ffff (0x53fc bytes) 000850 1787599124.215374 fallocate 0x032cba thru 0x032d1c (0x63 bytes) 000851 1787599124.225299 MIRROR_RESYNC 7 mirrors 000852 1787599125.775108 trunc from 0x040000 to 0x018a85 (0x2757a bytes) 000853 1787599125.820882 mapwrite 0x03b1da thru 0x03bf70 (0xd97 bytes) 000854 1787599125.877121 trunc from 0x02e136 to 0x0356ce (0x7599 bytes) 000855 1787599125.892566 mapread 0x002d45 thru 0x0081fb (0x54b7 bytes) 000856 1787599125.910723 mapwrite 0x01c4af thru 0x01e00f (0x1b61 bytes) 000857 1787599125.953457 trunc from 0x03531e to 0x03bf70 (0x6c53 bytes) 000858 1787599125.961434 punch from 0x006feb to 0x00fa2d (0x8a43 bytes) 000859 1787599125.969449 read 0x020f1a thru 0x02a656 (0x973d bytes) 000860 1787599125.978476 mapwrite 0x0330aa thru 0x034e7d (0x1dd4 bytes) 000862 1787599126.005807 MIRROR_RESYNC 7 mirrors 000863 1787599128.417855 MIRROR_RESYNC 7 mirrors 000864 1787599129.281330 punch from 0x0120ac to 0x01c150 (0xa0a5 bytes) 000865 1787599129.361491 mapread 0x035232 thru 0x035b81 (0x950 bytes) 000866 1787599129.376435 mapwrite 0x0274f1 thru 0x02e161 (0x6c71 bytes) 000868 1787599129.429382 write 0x0280ec thru 0x033f5a (0xbe6f bytes) 000869 1787599129.442993 write 0x0064e0 thru 0x011bc3 (0xb6e4 bytes) 000870 1787599129.467356 trunc from 0x03bf71 to 0x024562 (0x17a0e bytes) 000871 1787599129.515080 write 0x020f59 thru 0x02247a (0x1522 bytes) 000872 1787599129.535689 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24563): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24563): Mirror 2: 0x6b6a91f1 (stale) Mirror 3: 0x6b6a91f1 (stale) Mirror 5: 0x6b6a91f1 (stale) Mirror 6: 0x6b6a91f1 (stale) Mirror 7: 0x6b6a91f1 (stale) Mirror 8: 0x6b6a91f1 (stale) CRC-32 checksum value for chunk [0, 0x24563): Mirror 1: 0x9a91d237 Mirror 9: 0x9a91d237 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24563: skipped 000873 1787599131.097383 fallocate 0x01fdf1 thru 0x02abe1 (0xadf1 bytes) 000874 1787599131.158520 write 0x0071ae thru 0x0105ce (0x9421 bytes) 000875 1787599131.194932 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x24563): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24563): Mirror 2: 0x6b6a91f1 (stale) Mirror 3: 0x6b6a91f1 (stale) Mirror 5: 0x6b6a91f1 (stale) Mirror 6: 0x6b6a91f1 (stale) Mirror 7: 0x6b6a91f1 (stale) Mirror 8: 0x6b6a91f1 (stale) Mirror 9: 0x9a91d237 (stale) CRC-32 checksum value for chunk [0, 0x24563): Mirror 1: 0xd9e749c9 Mirror 10: 0xd9e749c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24563: skipped 000876 1787599132.916089 write 0x004b33 thru 0x00a214 (0x56e2 bytes) 000877 1787599133.040011 trunc from 0x0045aa to 0x0065a1 (0x1ff8 bytes) 000878 1787599133.074970 write 0x030577 thru 0x0310ba (0xb44 bytes) 000880 1787599133.076600 mapread 0x024960 thru 0x02e6a1 (0x9d42 bytes) 000881 1787599133.108162 trunc from 0x016246 to 0x016ca4 (0xa5f bytes) 000882 1787599133.124621 write 0x02fc90 thru 0x03334e (0x36bf bytes) 000884 1787599133.134614 trunc from 0x007159 to 0x015dbc (0xec64 bytes) 000885 1787599133.145847 mapread 0x02ce6c thru 0x032456 (0x55eb bytes) 000886 1787599133.147009 mapwrite 0x037e8b thru 0x03c328 (0x449e bytes) 000887 1787599133.281254 write 0x032900 thru 0x034341 (0x1a42 bytes) 000888 1787599133.284164 trunc from 0x03853a to 0x03c328 (0x3def bytes) 000889 1787599133.297791 read 0x013d94 thru 0x01cd06 (0x8f73 bytes) 000890 1787599133.322154 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c329): Mirror 3: 0xf64cfef8 (stale) Mirror 5: 0xf64cfef8 (stale) Mirror 6: 0xf64cfef8 (stale) Mirror 7: 0xf64cfef8 (stale) Mirror 8: 0xf64cfef8 (stale) Mirror 9: 0x9b8d3c81 (stale) Mirror 10: 0xa97aea66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c329): Mirror 3: 0xf64cfef8 (stale) Mirror 5: 0xf64cfef8 (stale) Mirror 6: 0xf64cfef8 (stale) Mirror 7: 0xf64cfef8 (stale) Mirror 8: 0xf64cfef8 (stale) Mirror 9: 0x9b8d3c81 (stale) Mirror 10: 0xa97aea66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c329: skipped 000891 1787599134.379826 write 0x01d2a8 thru 0x028dad (0xbb06 bytes) 000892 1787599134.426793 read 0x02daa5 thru 0x035feb (0x8547 bytes) 000893 1787599134.462117 trunc from 0x03c329 to 0x03ea33 (0x270b bytes) 000895 1787599134.491643 trunc from 0x03ea34 to 0x014255 (0x2a7de bytes) 000896 1787599134.527362 trunc from 0x014256 to 0x0165c9 (0x2374 bytes) 000897 1787599134.564885 MIRROR_RESYNC 8 mirrors 000898 1787599137.189851 write 0x03d40a thru 0x03e366 (0xf5d bytes) 000899 1787599137.220956 trunc from 0x03e367 to 0x019edf (0x24487 bytes) 000901 1787599137.242834 fallocate 0x0236d9 thru 0x024ab8 (0x13e0 bytes) 000902 1787599137.248292 mapwrite 0x01371c thru 0x01fa0e (0xc2f3 bytes) 000903 1787599137.322255 read 0x0072a4 thru 0x009734 (0x2491 bytes) 000904 1787599137.327995 trunc from 0x01ad20 to 0x01b1b0 (0x491 bytes) 000906 1787599137.334892 trunc from 0x01fa0f to 0x003bc4 (0x1be4a bytes) 000907 1787599137.349751 read 0x001389 thru 0x003bc4 (0x283c bytes) 000908 1787599137.356130 mapread 0x0030f5 thru 0x003bc4 (0xad0 bytes) 000909 1787599137.356946 punch from 0x0017d0 to 0x003bc4 (0x23f5 bytes) 000910 1787599137.361353 trunc from 0x003bc5 to 0x00a6b3 (0x6aef bytes) 000911 1787599137.377704 trunc from 0x00a6b4 to 0x026b8a (0x1c4d7 bytes) 000912 1787599137.399362 MIRROR_RESYNC 8 mirrors 000913 1787599138.512942 mapwrite 0x010d1a thru 0x01de21 (0xd108 bytes) 000914 1787599138.641186 fallocate 0x004a25 thru 0x00abc2 (0x619e bytes) 000915 1787599138.657924 MIRROR_RESYNC 8 mirrors 000916 1787599140.513974 trunc from 0x00bde6 to 0x01719b (0xb3b6 bytes) 000917 1787599140.553583 read 0x00e9fb thru 0x010d52 (0x2358 bytes) 000918 1787599140.561693 trunc from 0x008a0f to 0x00c183 (0x3775 bytes) 000919 1787599140.566776 mapread 0x002dc5 thru 0x00c11a (0x9356 bytes) 000920 1787599140.586844 fallocate 0x035e6f thru 0x0398d8 (0x3a6a bytes) 000921 1787599140.594482 trunc from 0x0398d9 to 0x02bd94 (0xdb44 bytes) 000922 1787599140.620413 punch from 0x00d735 to 0x015d95 (0x8661 bytes) 000923 1787599140.626968 fallocate 0x0270e6 thru 0x02c41f (0x533a bytes) 000924 1787599140.632831 mapread 0x0055b4 thru 0x00b40e (0x5e5b bytes) 000925 1787599140.633900 mapread 0x00562a thru 0x008d01 (0x36d8 bytes) 000926 1787599140.635956 write 0x00d752 thru 0x00dbcd (0x47c bytes) 000927 1787599140.641828 mapwrite 0x020cb0 thru 0x0290ef (0x8440 bytes) 000928 1787599140.688936 mapwrite 0x03b3b0 thru 0x03eb80 (0x37d1 bytes) 000929 1787599140.736025 trunc from 0x023770 to 0x0301f3 (0xca84 bytes) 000930 1787599140.744278 punch from 0x005c6d to 0x00f3c1 (0x9755 bytes) 000931 1787599140.751941 fallocate 0x014acd thru 0x01a0e1 (0x5615 bytes) 000932 1787599140.758415 fallocate 0x021926 thru 0x02c89b (0xaf76 bytes) 000934 1787599140.766652 read 0x0362d3 thru 0x03eb80 (0x88ae bytes) 000935 1787599140.772267 punch from 0x00f9d5 to 0x013ea0 (0x44cc bytes) 000936 1787599140.782954 trunc from 0x03eb81 to 0x02c304 (0x1287c bytes) 000937 1787599140.801252 punch from 0x0207af to 0x029d50 (0x95a2 bytes) 000938 1787599140.809736 mapread 0x006f48 thru 0x0121fd (0xb2b6 bytes) 000939 1787599140.829278 mapread 0x0214a0 thru 0x02c304 (0xae65 bytes) 000940 1787599140.833904 mapwrite 0x000aec thru 0x00f7df (0xecf4 bytes) 000941 1787599140.875005 mapread 0x00f9d1 thru 0x01311c (0x374c bytes) 000942 1787599140.885881 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c305): Mirror 3: 0x37601ff (stale) Mirror 5: 0x37601ff (stale) Mirror 7: 0x37601ff (stale) Mirror 8: 0x37601ff (stale) Mirror 9: 0x37601ff (stale) Mirror 10: 0x37601ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c305): Mirror 3: 0x37601ff (stale) Mirror 5: 0x37601ff (stale) Mirror 7: 0x37601ff (stale) Mirror 8: 0x37601ff (stale) Mirror 9: 0x37601ff (stale) Mirror 10: 0x37601ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c305: skipped 000944 1787599141.562487 trunc from 0x0257b8 to 0x02ad39 (0x5582 bytes) 000945 1787599141.569452 trunc from 0x02c305 to 0x007590 (0x24d74 bytes) 000946 1787599141.594915 mapread 0x005a2f thru 0x007590 (0x1b62 bytes) 000947 1787599141.615340 mapwrite 0x0171d4 thru 0x02177e (0xa5ab bytes) 000948 1787599141.673873 mapread 0x00eaae thru 0x014ac1 (0x6014 bytes) 000949 1787599141.688042 trunc from 0x014173 to 0x01f6d3 (0xb561 bytes) 000950 1787599141.699727 read 0x01c4c2 thru 0x01f84d (0x338c bytes) 000951 1787599141.706758 mapwrite 0x011727 thru 0x01caf5 (0xb3cf bytes) 000952 1787599141.775804 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2177f): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2177f): Mirror 3: 0xb0a1cc12 (stale) Mirror 5: 0xb0a1cc12 (stale) Mirror 7: 0xb0a1cc12 (stale) Mirror 8: 0xb0a1cc12 (stale) Mirror 9: 0xb0a1cc12 (stale) Mirror 10: 0xb0a1cc12 (stale) CRC-32 checksum value for chunk [0, 0x2177f): Mirror 1: 0xf132fa81 Mirror 11: 0xf132fa81 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2177f: skipped 000953 1787599143.057108 punch from 0x00adff to 0x00f0f5 (0x42f7 bytes) 000954 1787599143.119936 fallocate 0x01333b thru 0x017c3d (0x4903 bytes) 000955 1787599143.131909 trunc from 0x0045e4 to 0x0058bd (0x12da bytes) 000956 1787599143.154016 punch from 0x01ebc6 to 0x02177e (0x2bb9 bytes) 000957 1787599143.171132 fallocate 0x021e1d thru 0x02aab3 (0x8c97 bytes) 000958 1787599143.185601 read 0x0036fd thru 0x00bcc5 (0x85c9 bytes) 000959 1787599143.206867 fallocate 0x01fc44 thru 0x02f632 (0xf9ef bytes) 000961 1787599143.213826 read 0x01729b thru 0x0211ed (0x9f53 bytes) 000963 1787599143.235900 read 0x013a4f thru 0x02074d (0xccff bytes) 000964 1787599143.238146 trunc from 0x015b94 to 0x019bfa (0x4067 bytes) 000966 1787599143.254865 fallocate 0x02e8d7 thru 0x02ec8f (0x3b9 bytes) 000967 1787599143.270906 trunc from 0x02f633 to 0x022012 (0xd620 bytes) 000968 1787599143.360927 punch from 0x0009da to 0x00696c (0x5f93 bytes) 000969 1787599143.373071 trunc from 0x022013 to 0x014bb8 (0xd45a bytes) 000970 1787599143.390429 trunc from 0x014bb9 to 0x005067 (0xfb51 bytes) 000971 1787599143.408022 read 0x000177 thru 0x005067 (0x4ef1 bytes) 000972 1787599143.420168 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x5068): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5068): Mirror 3: 0xf6cb4732 (stale) Mirror 5: 0xf6cb4732 (stale) Mirror 7: 0xf6cb4732 (stale) Mirror 8: 0xf6cb4732 (stale) Mirror 9: 0xf6cb4732 (stale) Mirror 10: 0xf6cb4732 (stale) Mirror 11: 0xef29e735 (stale) CRC-32 checksum value for chunk [0, 0x5068): Mirror 1: 0xf6cb4732 Mirror 12: 0xf6cb4732 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5068: skipped 000973 1787599144.685911 trunc from 0x005068 to 0x029f7c (0x24f15 bytes) 000974 1787599144.834049 trunc from 0x017e32 to 0x025235 (0xd404 bytes) 000975 1787599144.843617 MIRROR_RESYNC 9 mirrors 000976 1787599146.792255 read 0x00ea7a thru 0x00eb44 (0xcb bytes) 000977 1787599146.824732 MIRROR_EXTEND to 10 mirrors 000978 1787599147.713930 mapread 0x00c465 thru 0x00c5c1 (0x15d bytes) 000979 1787599147.720082 trunc from 0x0174a7 to 0x023257 (0xbdb1 bytes) 000980 1787599147.769585 fallocate 0x023591 thru 0x026003 (0x2a73 bytes) 000981 1787599147.775922 read 0x00d02a thru 0x01ca5f (0xfa36 bytes) 000982 1787599147.790299 trunc from 0x029f7d to 0x01e2a4 (0xbcd8 bytes) 000983 1787599147.823563 trunc from 0x01e2a5 to 0x006211 (0x18093 bytes) 000984 1787599147.842493 trunc from 0x006212 to 0x02849f (0x2228e bytes) 000985 1787599147.889165 read 0x00c391 thru 0x01a0c7 (0xdd37 bytes) 000986 1787599147.901440 trunc from 0x0284a0 to 0x0208a3 (0x7bfc bytes) 000987 1787599147.943030 punch from 0x010ffd to 0x011a9c (0xaa0 bytes) 000988 1787599147.971939 write 0x024061 thru 0x0289eb (0x498b bytes) 000989 1787599147.984262 read 0x026173 thru 0x026eb3 (0xd41 bytes) 000990 1787599147.985121 trunc from 0x0289ec to 0x0143a1 (0x1464a bytes) 000991 1787599148.026478 trunc from 0x00e8d8 to 0x0143a1 (0x5aca bytes) 000993 1787599148.040395 mapread 0x008c30 thru 0x00e3c9 (0x579a bytes) 000995 1787599148.064058 trunc from 0x00d56a to 0x0143a1 (0x6e38 bytes) 000997 1787599148.089273 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x143a2): Mirror 3: 0xb366c975 (stale) Mirror 5: 0xb366c975 (stale) Mirror 7: 0xb366c975 (stale) Mirror 9: 0xb366c975 (stale) Mirror 10: 0xb366c975 (stale) Mirror 11: 0xb366c975 (stale) Mirror 12: 0xb366c975 (stale) Mirror 13: 0xb366c975 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x143a2): Mirror 3: 0xb366c975 (stale) Mirror 5: 0xb366c975 (stale) Mirror 7: 0xb366c975 (stale) Mirror 9: 0xb366c975 (stale) Mirror 10: 0xb366c975 (stale) Mirror 11: 0xb366c975 (stale) Mirror 12: 0xb366c975 (stale) Mirror 13: 0xb366c975 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x143a2: skipped 000998 1787599148.874961 write 0x02bead thru 0x0319df (0x5b33 bytes) 000999 1787599148.884385 write 0x015d2e thru 0x01c3f8 (0x66cb bytes) 001000 1787599148.897754 mapread 0x006e2a thru 0x0117b3 (0xa98a bytes) All operations completed A-OK! PASS 70a (134s) == sanity-flr test 71: check mirror extend parent fid ==== 15:19:14 (1787599154) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1a6:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 15:19:25 (1787599165) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195457 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145982 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b1:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 15:19:35 (1787599175) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7693408 .. Extending file size to 7648928 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8126208 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7850624 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8829824 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8346944 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8483552 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61346 61347 61348 61352 61354 done 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:20:57 (1787599257) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4774368 .. Extending file size to 7505664 .. Extending file size to 9107168 .. Extending file size to 8998816 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9067712 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62816 62817 62819 62821 62824 done 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:22:20 (1787599340) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1268892 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2044 1269260 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17824 3589196 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25592 3581428 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43416 7170624 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1268892 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2044 1269260 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17824 3589196 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25592 3581428 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43416 7170624 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121657 s, 17.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208617 s, 10.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x55df3e9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:22:42 (1787599362) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136293 s, 15.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 15:22:49 (1787599369) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c6:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:22:57 (1787599377) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105529 s, 19.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:23:05 (1787599385) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101421 s, 20.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:23:12 (1787599392) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12503 s, 16.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ca:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:23:48 (1787599428) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891332 s, 11.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:24:40 (1787599480) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166946 s, 12.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:25:32 (1787599532) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg308-server: Pool lustre.test_204f created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11791 s, 17.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg308-server: Pool lustre.test_204f destroyed PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:26:43 (1787599603) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:26:51 (1787599611) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:26:59 (1787599619) oleg308-server: Pool lustre.test_206 created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg308-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:27:29 (1787599649) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107209 s, 9.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072164 s, 14.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8698c79c Mirror 3: 0x8698c79c PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:27:43 (1787599663) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:27:47 (1787599667) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:27:52 (1787599672) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0873175 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x11d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x11d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x11d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1330000 Read this (4 bytes): BBBB 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:28:30 (1787599710) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0315985 s, 259 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:28:38 (1787599718) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367539 s, 28.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1d4 lmm_fid: [0x200000405:0x1d4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1d4 lmm_fid: [0x200000405:0x1d4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d6:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611604 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 15:28:39.000000000 -0400 Modify: 2026-08-24 15:28:39.000000000 -0400 Change: 2026-08-24 15:28:41.000000000 -0400 Birth: 2026-08-24 15:28:39.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:28:47 (1787599727) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302139 s, 34.7 MB/s oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg308-server: O/280000401/d20/2f4: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:29:11 (1787599751) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.282602 s, 145 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0461662 s, 88.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 15:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 15:29 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:29:20 (1787599760) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:29:30 (1787599770) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1938 sec =========== 15:29:42 (1787599782) === sanity-flr: start cleanup 15:29:44 (1787599784) === === sanity-flr: finish cleanup 15:29:55 (1787599795) ===