-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 06:09:18 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg305-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:09:36 (1787393376) === oleg305-client.virtnet: executing check_config_client /mnt/lustre oleg305-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg305-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9bba07cf2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9bba07cf2000.idle_timeout=debug disable quota as required oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:10:14 (1787393414) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8233] == sanity-flr test 0a: lfs mirror create with -N option == 06:10:16 (1787393416) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:10:33 (1787393433) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:10:37 (1787393437) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:10:40 (1787393440) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:11:03 (1787393463) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:11:06 (1787393466) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:11:10 (1787393470) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9bba07cf2000.stats=clear osc.lustre-OST0001-osc-ffff9bba07cf2000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:11:19 (1787393479) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:11:29 (1787393489) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:11:36 (1787393496) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:11:40 (1787393500) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:11:48 (1787393508) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:11:57 (1787393517) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:12:04 (1787393524) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:12:13 (1787393533) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:12:22 (1787393542) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:12:32 (1787393552) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:12:39 (1787393559) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.22605 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846143 s, 12.4 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:12:47 (1787393567) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.267947 s, 31.3 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:12:56 (1787393576) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:13:07 (1787393587) Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg305-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg305-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (132s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:15:19 (1787393719) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0991711 s, 31.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:15:53 (1787393753) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg305-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (67s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:17:00 (1787393820) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.5535 s, 20.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 06:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.859s user 0m0.001s sys 0m0.202s Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bba07cf2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.007s user 0m0.001s sys 0m0.179s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.972s user 0m0.000s sys 0m0.182s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg305-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bba07cf2000.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:18:30 (1787393910) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:18:33 (1787393913) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:18:37 (1787393917) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:18:40 (1787393920) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 06:18:46 (1787393926) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.539811 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:18:59 (1787393939) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362093 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309096 s, 33.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:19:06 (1787393946) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:19:14 (1787393954) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00157083 s, 652 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108822 s, 941 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123469 s, 829 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00215164 s, 476 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 06:19:24 (1787393964) create test files with size 15760511 .. fdbb2550faa45fc5f1decba6415cecd4 - 67b11eab1834ea2e5480596625d801fd - 3164d67256635a540f12bca3e3446846 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 06:19:46 (1787393986) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:19:58 (1787393998) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:20:05 (1787394005) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692467 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612849 s, 17.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 06:20:15 (1787394015) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0622558 s, 33.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.360477 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.210202 s, 10.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 06:20:33 (1787394033) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:20:38 (1787394038) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:20:43 (1787394043) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342562 s, 30.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 06:20:53 (1787394053) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.362208 s, 8.7 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:21:04 (1787394064) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:21:14 (1787394074) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.352788 s, 29.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:21:27 (1787394087) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9bba185bd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.453608 s, 23.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9bba269d4800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:21:43 (1787394103) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:22:20 (1787394140) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:22:56 (1787394176) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:23:00 (1787394180) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.28158 s, 11.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196983 s, 16.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfa9d35bd (nosync,stale) Mirror 2: 0xfa9d35bd (nosync,stale) ** verify mirror 2 doesn't change original checksum: e265ea7e2e50ee108606301c0eeb3b79 - mirror 1 checksum: 47c383919053da0318cab095a4c219f0 - mirror 2 checksum: e265ea7e2e50ee108606301c0eeb3b79 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:23:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:23:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e265ea7e2e50ee108606301c0eeb3b79 - mirror 1 checksum: 47c383919053da0318cab095a4c219f0 - mirror 2 checksum: 47c383919053da0318cab095a4c219f0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:23:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:23:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:23:11 (1787394191) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453389 s, 23.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:23:20 (1787394200) Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 06:23:33 (1787394213) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 06:24:05 (1787394245) targets are mounted 06:24:05 (1787394245) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209053 s, 50.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:24:23 (1787394263) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.051076 s, 401 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0338138 s, 606 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:24:33 (1787394273) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0594458 s, 345 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0263424 s, 777 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:24:43 (1787394283) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00714189 s, 2.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:24:50 (1787394290) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0402731 s, 509 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:24:58 (1787394298) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00511941 s, 800 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.05388 s, 76.0 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:25:05 (1787394305) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:25:28 (1787394328) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:25:56 (1787394356) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:26:20 (1787394380) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:26:45 (1787394405) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:26:52 (1787394412) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57389 mirror split pid 57390 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57389 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57390 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 06:27:59 (1787394479) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4481 000001 1787394481.918182 write 0x01e484 thru 0x0293b1 (0xaf2e bytes) 000002 1787394481.949337 trunc from 0x008fba to 0x00eb7f (0x5bc6 bytes) 000003 1787394481.959193 trunc from 0x019b38 to 0x028c18 (0xf0e1 bytes) 000005 1787394481.992806 mapread 0x01bff7 thru 0x02531e (0x9328 bytes) 000006 1787394482.032143 punch from 0x01dacf to 0x0293b1 (0xb8e3 bytes) truncating to largest ever: 0x2879d 000007 1787394482.039141 trunc from 0x0293b2 to 0x02879c (0xc15 bytes) 000008 1787394482.068501 write 0x032c80 thru 0x03901d (0x639e bytes) truncating to largest ever: 0x35fd8 000009 1787394482.074381 trunc from 0x03901e to 0x035fd7 (0x3046 bytes) 000010 1787394482.121754 read 0x02631f thru 0x035726 (0xf408 bytes) 000011 1787394482.136513 mapread 0x009ac0 thru 0x00fcf4 (0x6235 bytes) 000012 1787394482.181654 write 0x0380ff thru 0x038770 (0x672 bytes) 000013 1787394482.184453 mapwrite 0x008239 thru 0x016ad9 (0xe8a1 bytes) 000014 1787394482.212718 read 0x0144d1 thru 0x01f687 (0xb1b7 bytes) 000015 1787394482.224203 mapwrite 0x005026 thru 0x00de53 (0x8e2e bytes) 000016 1787394482.242177 mapread 0x0256ce thru 0x03055e (0xae91 bytes) 000017 1787394482.244076 read 0x028da6 thru 0x034fb4 (0xc20f bytes) 000018 1787394482.245612 trunc from 0x038771 to 0x004b94 (0x33bdc bytes) 000019 1787394482.260814 mapread 0x002276 thru 0x004b94 (0x291f bytes) 000022 1787394482.264664 trunc from 0x002b46 to 0x004b94 (0x204f bytes) 000023 1787394482.271513 trunc from 0x004b95 to 0x014903 (0xfd6f bytes) 000024 1787394482.285488 trunc from 0x014904 to 0x02e849 (0x19f46 bytes) 000025 1787394482.301801 fallocate 0x02db1a thru 0x0342ed (0x67d4 bytes) 000026 1787394482.307308 mapwrite 0x0113fe thru 0x01c597 (0xb19a bytes) 000027 1787394482.362686 read 0x02154f thru 0x02e849 (0xd2fb bytes) 000028 1787394482.363586 mapread 0x023fcf thru 0x029b69 (0x5b9b bytes) 000030 1787394482.366818 trunc from 0x0256c0 to 0x0263ba (0xcfb bytes) 000031 1787394482.383172 write 0x011ca0 thru 0x020b25 (0xee86 bytes) 000032 1787394482.392268 write 0x01f073 thru 0x02ae27 (0xbdb5 bytes) 000033 1787394482.397291 punch from 0x0242e8 to 0x02cbc0 (0x88d9 bytes) 000034 1787394482.427789 mapwrite 0x01e34e thru 0x01f914 (0x15c7 bytes) 000035 1787394482.447373 mapwrite 0x03d53a thru 0x03ffff (0x2ac6 bytes) 000036 1787394482.531271 mapwrite 0x01a6a4 thru 0x0200c6 (0x5a23 bytes) 000038 1787394482.559330 trunc from 0x0039ce to 0x009392 (0x59c5 bytes) 000039 1787394482.575355 write 0x00e8d1 thru 0x0148ca (0x5ffa bytes) 000040 1787394482.584342 read 0x01e5f8 thru 0x02bbea (0xd5f3 bytes) 000042 1787394482.595210 mapwrite 0x02c070 thru 0x03928f (0xd220 bytes) 000043 1787394482.646628 mapread 0x037cc3 thru 0x03eb3d (0x6e7b bytes) 000044 1787394482.650693 mapwrite 0x03626f thru 0x03ffff (0x9d91 bytes) 000045 1787394482.687622 read 0x038609 thru 0x03ffff (0x79f7 bytes) 000046 1787394482.688663 trunc from 0x040000 to 0x018c54 (0x273ab bytes) 000048 1787394482.719122 trunc from 0x00eb62 to 0x01305f (0x44fe bytes) 000049 1787394482.726510 trunc from 0x018c55 to 0x01cd2f (0x40db bytes) 000050 1787394482.760684 write 0x01f650 thru 0x02b6ba (0xc06b bytes) 000051 1787394482.766100 fallocate 0x01befa thru 0x020d13 (0x4e1a bytes) 000053 1787394482.774167 read 0x0083fa thru 0x015cc7 (0xd8ce bytes) 000054 1787394482.785363 punch from 0x0271c8 to 0x02b6ba (0x44f3 bytes) 000055 1787394482.805280 read 0x018fe5 thru 0x024f4a (0xbf66 bytes) 000057 1787394482.816156 write 0x00dab9 thru 0x01b5be (0xdb06 bytes) 000058 1787394482.819282 read 0x002f04 thru 0x006847 (0x3944 bytes) fallocating to largest ever: 0x40000 000059 1787394482.826715 fallocate 0x036eb8 thru 0x03ffff (0x9148 bytes) 000060 1787394482.843078 write 0x0320f4 thru 0x03ffff (0xdf0c bytes) 000063 1787394482.859127 trunc from 0x021c1b to 0x029a3d (0x7e23 bytes) 000064 1787394482.868386 mapwrite 0x0270d3 thru 0x034a49 (0xd977 bytes) 000065 1787394482.929123 trunc from 0x01c5bf to 0x024b5b (0x859d bytes) 000066 1787394482.935773 mapwrite 0x027e48 thru 0x0326df (0xa898 bytes) 000067 1787394482.963789 write 0x0193db thru 0x023e93 (0xaab9 bytes) 000068 1787394482.973559 trunc from 0x040000 to 0x0054ad (0x3ab52 bytes) 000069 1787394482.999273 trunc from 0x00418c to 0x0054ad (0x1322 bytes) 000070 1787394483.006103 trunc from 0x004f33 to 0x0054ad (0x57b bytes) 000073 1787394483.014619 punch from 0x003129 to 0x0054ad (0x2385 bytes) 000074 1787394483.022073 fallocate 0x006706 thru 0x00d7fd (0x70f8 bytes) 000075 1787394483.028240 fallocate 0x03defe thru 0x03ffff (0x2102 bytes) 000076 1787394483.034795 write 0x01831a thru 0x023d5c (0xba43 bytes) 000077 1787394483.050455 write 0x038b5d thru 0x039504 (0x9a8 bytes) 000078 1787394483.070280 read 0x01a584 thru 0x025b82 (0xb5ff bytes) 000079 1787394483.089984 write 0x038d8d thru 0x03ffff (0x7273 bytes) 000080 1787394483.093001 mapwrite 0x01b885 thru 0x021da2 (0x651e bytes) 000082 1787394483.121756 fallocate 0x0300fc thru 0x030c26 (0xb2b bytes) 000085 1787394483.135176 trunc from 0x0139be to 0x018c2b (0x526e bytes) 000087 1787394483.142093 punch from 0x02df5f to 0x03bc20 (0xdcc2 bytes) 000088 1787394483.147666 punch from 0x018947 to 0x01e370 (0x5a2a bytes) 000089 1787394483.158165 mapwrite 0x02a716 thru 0x0362ea (0xbbd5 bytes) 000091 1787394483.290986 trunc from 0x040000 to 0x036cc6 (0x9339 bytes) 000092 1787394483.314115 trunc from 0x008c20 to 0x010f6d (0x834e bytes) 000094 1787394483.323102 mapwrite 0x00d24d thru 0x0185da (0xb38e bytes) 000095 1787394483.414414 read 0x01da16 thru 0x02125d (0x3848 bytes) 000096 1787394483.427005 punch from 0x02f2c4 to 0x036cc6 (0x7a03 bytes) 000097 1787394483.433681 read 0x001335 thru 0x00b276 (0x9f42 bytes) 000099 1787394483.435451 trunc from 0x036cc7 to 0x026544 (0x10782 bytes) 000100 1787394483.458423 fallocate 0x01dfe9 thru 0x026f89 (0x8fa1 bytes) 000101 1787394483.467294 write 0x02b6f3 thru 0x030279 (0x4b87 bytes) 000103 1787394483.474329 fallocate 0x01c9f6 thru 0x02376a (0x6d75 bytes) 000104 1787394483.480774 mapread 0x029e19 thru 0x030279 (0x6461 bytes) 000105 1787394483.489714 trunc from 0x03027a to 0x00f9f3 (0x20886 bytes) 000106 1787394483.512366 MIRROR_EXTEND to 2 mirrors 000107 1787394484.233327 trunc from 0x00a69d to 0x00d394 (0x2cf8 bytes) 000108 1787394484.296702 write 0x0011bf thru 0x00a97a (0x97bc bytes) 000109 1787394484.324967 read 0x00d7ed thru 0x00f9f3 (0x2207 bytes) 000110 1787394484.340310 MIRROR_RESYNC 2 mirrors 000111 1787394485.263168 punch from 0x00c1b2 to 0x00f9f3 (0x3842 bytes) 000112 1787394485.329261 MIRROR_RESYNC 2 mirrors 000113 1787394485.930864 fallocate 0x008db5 thru 0x00dfc1 (0x520d bytes) 000114 1787394485.987809 write 0x033995 thru 0x03ffff (0xc66b bytes) 000115 1787394485.989677 mapwrite 0x004db3 thru 0x0145ff (0xf84d bytes) 000117 1787394486.091983 punch from 0x013fcd to 0x01ce5e (0x8e92 bytes) 000118 1787394486.101608 mapwrite 0x028bb8 thru 0x02ab60 (0x1fa9 bytes) 000119 1787394486.137691 mapread 0x014ecc thru 0x0183ba (0x34ef bytes) 000120 1787394486.155569 write 0x00e13b thru 0x015b20 (0x79e6 bytes) 000121 1787394486.157391 mapwrite 0x032ba4 thru 0x03f2e9 (0xc746 bytes) 000122 1787394486.186273 read 0x0255f5 thru 0x0280bb (0x2ac7 bytes) 000123 1787394486.187112 mapread 0x035d39 thru 0x03a092 (0x435a bytes) 000124 1787394486.202722 trunc from 0x000829 to 0x00da20 (0xd1f8 bytes) 000125 1787394486.223166 write 0x03efbf thru 0x03ffff (0x1041 bytes) 000127 1787394486.224996 mapread 0x00f69c thru 0x019f66 (0xa8cb bytes) 000128 1787394486.232505 read 0x02a7af thru 0x033c19 (0x946b bytes) 000129 1787394486.237360 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) 000130 1787394486.560543 write 0x02f0a8 thru 0x03c861 (0xd7ba bytes) 000131 1787394486.576103 write 0x016aac thru 0x02679d (0xfcf2 bytes) 000132 1787394486.620356 MIRROR_EXTEND to 2 mirrors 000133 1787394487.743649 trunc from 0x0267d6 to 0x034cd3 (0xe4fe bytes) 000134 1787394487.771529 punch from 0x01844b to 0x021e7c (0x9a32 bytes) 000135 1787394487.792192 MIRROR_RESYNC 2 mirrors 000136 1787394488.665393 MIRROR_RESYNC 2 mirrors 000137 1787394489.137431 fallocate 0x001d03 thru 0x003292 (0x1590 bytes) 000138 1787394489.199104 fallocate 0x00e4da thru 0x012b55 (0x467c bytes) 000140 1787394489.206436 trunc from 0x040000 to 0x031871 (0xe78e bytes) 000141 1787394489.255154 trunc from 0x00882d to 0x00c6a1 (0x3e75 bytes) 000142 1787394489.270654 trunc from 0x031872 to 0x0327d0 (0xf5f bytes) 000143 1787394489.308415 trunc from 0x0327d1 to 0x00eb9e (0x23c32 bytes) 000144 1787394489.365093 write 0x03e821 thru 0x03ffff (0x17df bytes) 000145 1787394489.367599 mapread 0x001fe9 thru 0x008780 (0x6798 bytes) 000146 1787394489.418316 trunc from 0x040000 to 0x03d285 (0x2d7a bytes) 000147 1787394489.458334 read 0x0161b6 thru 0x02232d (0xc178 bytes) 000148 1787394489.508627 read 0x0225f9 thru 0x027694 (0x509c bytes) 000149 1787394489.522024 mapwrite 0x037717 thru 0x03ffff (0x88e9 bytes) 000150 1787394489.602171 trunc from 0x0160b3 to 0x018418 (0x2366 bytes) 000151 1787394489.612597 trunc from 0x022bdc to 0x02bdd5 (0x91fa bytes) 000152 1787394489.622309 read 0x022680 thru 0x02f98a (0xd30b bytes) 000153 1787394489.635514 trunc from 0x040000 to 0x03895b (0x76a4 bytes) 000154 1787394489.666338 mapread 0x0038f4 thru 0x0114ff (0xdc0c bytes) 000156 1787394489.668716 punch from 0x024548 to 0x02b5bf (0x7078 bytes) 000157 1787394489.694477 MIRROR_RESYNC 2 mirrors 000159 1787394490.767473 trunc from 0x00b30c to 0x013577 (0x826c bytes) 000160 1787394490.821670 punch from 0x01420e to 0x017179 (0x2f6c bytes) 000162 1787394490.829728 punch from 0x015205 to 0x01b61c (0x6418 bytes) 000163 1787394490.836676 trunc from 0x0077ae to 0x00fdf8 (0x864b bytes) 000164 1787394490.847722 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3895c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3895c): Mirror 2: 0x95ebedd8 (stale) CRC-32 checksum value for chunk [0, 0x3895c): Mirror 1: 0x4f2986cb Mirror 3: 0x4f2986cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3895c: skipped 000165 1787394492.100478 MIRROR_RESYNC 3 mirrors 000166 1787394492.857356 read 0x03718d thru 0x03895b (0x17cf bytes) 000167 1787394492.863956 read 0x01ab42 thru 0x0258d5 (0xad94 bytes) 000169 1787394492.994676 MIRROR_EXTEND to 4 mirrors 000170 1787394493.853143 write 0x011aeb thru 0x01394c (0x1e62 bytes) 000171 1787394493.897522 trunc from 0x007ee9 to 0x017c03 (0xfd1b bytes) 000172 1787394493.908333 mapwrite 0x01b387 thru 0x027252 (0xbecc bytes) 000173 1787394493.968685 read 0x002203 thru 0x009498 (0x7296 bytes) 000174 1787394493.980243 mapread 0x012007 thru 0x01d576 (0xb570 bytes) 000175 1787394493.994635 mapread 0x031614 thru 0x0316cd (0xba bytes) 000179 1787394494.020713 punch from 0x01093c to 0x01c775 (0xbe3a bytes) 000180 1787394494.027669 trunc from 0x03895c to 0x032d7c (0x5bdf bytes) 000181 1787394494.058341 MIRROR_RESYNC 4 mirrors 000182 1787394495.023649 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) 000183 1787394495.501289 trunc from 0x00b684 to 0x01a972 (0xf2ef bytes) 000184 1787394495.575434 MIRROR_RESYNC 3 mirrors 000185 1787394496.418200 mapwrite 0x008800 thru 0x00f4ab (0x6cac bytes) 000186 1787394496.546614 write 0x01de4b thru 0x026a42 (0x8bf8 bytes) 000187 1787394496.579531 punch from 0x006e5d to 0x016bcb (0xfd6f bytes) 000188 1787394496.586128 mapwrite 0x03f893 thru 0x03ffff (0x76d bytes) 000189 1787394496.647101 trunc from 0x011b53 to 0x012b24 (0xfd2 bytes) 000190 1787394496.654333 write 0x008b4f thru 0x00facf (0x6f81 bytes) 000192 1787394496.670509 read 0x03da7a thru 0x03ffff (0x2586 bytes) 000193 1787394496.672430 fallocate 0x0265ab thru 0x02d45a (0x6eb0 bytes) 000195 1787394496.678871 punch from 0x025832 to 0x030b39 (0xb308 bytes) 000196 1787394496.687165 mapwrite 0x0005a3 thru 0x00717b (0x6bd9 bytes) 000197 1787394496.742925 mapread 0x03307e thru 0x037325 (0x42a8 bytes) 000198 1787394496.769629 read 0x016416 thru 0x01839a (0x1f85 bytes) 000199 1787394496.791354 mapread 0x01ce15 thru 0x0200d5 (0x32c1 bytes) 000200 1787394496.792561 mapread 0x000226 thru 0x008b44 (0x891f bytes) 000202 1787394496.801498 fallocate 0x012edc thru 0x014f48 (0x206d bytes) 000203 1787394496.812187 fallocate 0x01ede1 thru 0x01ef39 (0x159 bytes) 000205 1787394496.830654 punch from 0x00905c to 0x0166b2 (0xd657 bytes) 000206 1787394496.844519 mapwrite 0x02716a thru 0x0333d9 (0xc270 bytes) 000207 1787394496.900534 mapwrite 0x031a21 thru 0x03c2c5 (0xa8a5 bytes) 000208 1787394496.939265 mapread 0x03c8dc thru 0x03ffff (0x3724 bytes) 000209 1787394496.940716 fallocate 0x014155 thru 0x022621 (0xe4cd bytes) 000210 1787394496.946692 trunc from 0x040000 to 0x01092d (0x2f6d2 bytes) 000211 1787394496.982603 trunc from 0x00525d to 0x00a426 (0x51ca bytes) 000212 1787394497.006178 mapread 0x00e3ee thru 0x01092d (0x2540 bytes) 000213 1787394497.054702 read 0x000f59 thru 0x006a32 (0x5ada bytes) 000214 1787394497.087508 trunc from 0x00b267 to 0x01092d (0x56c7 bytes) 000215 1787394497.102454 trunc from 0x00bdb1 to 0x00c286 (0x4d6 bytes) 000216 1787394497.110794 fallocate 0x03c0e1 thru 0x03ffff (0x3f1f bytes) 000217 1787394497.127686 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1092e): Mirror 3: 0x55e2235d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1092e): Mirror 3: 0x55e2235d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1092e: skipped 000219 1787394497.563024 read 0x01047c thru 0x01092d (0x4b2 bytes) 000220 1787394497.591241 punch from 0x001d49 to 0x00e997 (0xcc4f bytes) 000221 1787394497.610891 mapwrite 0x00eebe thru 0x01b529 (0xc66c bytes) 000222 1787394497.712888 write 0x031531 thru 0x03ffff (0xeacf bytes) 000223 1787394497.714840 read 0x006261 thru 0x010175 (0x9f15 bytes) 000224 1787394497.738675 write 0x00d8f8 thru 0x00ee32 (0x153b bytes) 000225 1787394497.741615 mapread 0x016b6b thru 0x01a66b (0x3b01 bytes) 000226 1787394497.744833 punch from 0x028a52 to 0x0369e2 (0xdf91 bytes) 000227 1787394497.770362 punch from 0x02ca71 to 0x03c3fc (0xf98c bytes) 000228 1787394497.779505 punch from 0x02d3d9 to 0x03a322 (0xcf4a bytes) 000229 1787394497.787447 trunc from 0x03b3cf to 0x03ffff (0x4c31 bytes) 000230 1787394497.805966 trunc from 0x040000 to 0x014e39 (0x2b1c6 bytes) 000231 1787394497.854844 punch from 0x01322a to 0x014e39 (0x1c10 bytes) 000233 1787394497.875725 write 0x00f85b thru 0x0178bb (0x8061 bytes) 000234 1787394497.878282 trunc from 0x0178bc to 0x00ccf8 (0xabc3 bytes) 000235 1787394497.904997 mapread 0x00a0ad thru 0x00c2a9 (0x21fd bytes) 000236 1787394497.908481 trunc from 0x00ccf9 to 0x03ed5c (0x32064 bytes) 000237 1787394497.941131 mapwrite 0x030ec5 thru 0x03beb3 (0xafef bytes) 000238 1787394498.056600 mapwrite 0x03b5ca thru 0x03ffff (0x4a36 bytes) 000239 1787394498.170722 mapread 0x01c896 thru 0x01d2ed (0xa58 bytes) 000240 1787394498.190050 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd70078e9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x38db0eb0 Mirror 4: 0x38db0eb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000242 1787394499.560435 trunc from 0x040000 to 0x02ef6a (0x11095 bytes) 000243 1787394499.652049 trunc from 0x02ef6b to 0x0165fb (0x1896f bytes) 000244 1787394499.683197 mapwrite 0x0356f4 thru 0x03c57d (0x6e8a bytes) 000245 1787394499.742577 mapread 0x013c5d thru 0x019cd6 (0x607a bytes) 000246 1787394499.759325 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c57e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c57e): Mirror 3: 0x568bbcde (stale) Mirror 4: 0xc1c5cc80 (stale) CRC-32 checksum value for chunk [0, 0x3c57e): Mirror 1: 0x33fb0469 Mirror 5: 0x33fb0469 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c57e: skipped 000247 1787394500.818265 mapread 0x002c27 thru 0x005700 (0x2ada bytes) 000248 1787394500.856660 fallocate 0x016580 thru 0x02458a (0xe00b bytes) 000249 1787394500.913975 trunc from 0x038259 to 0x03c57d (0x4325 bytes) 000250 1787394500.926250 trunc from 0x01a235 to 0x01fc8f (0x5a5b bytes) 000251 1787394500.939214 fallocate 0x009e70 thru 0x017878 (0xda09 bytes) 000252 1787394500.947611 fallocate 0x01c7b9 thru 0x022f49 (0x6791 bytes) 000253 1787394500.955811 write 0x03658b thru 0x03946a (0x2ee0 bytes) 000254 1787394500.974380 punch from 0x024779 to 0x03430a (0xfb92 bytes) 000255 1787394500.985194 trunc from 0x0322d8 to 0x0344dd (0x2206 bytes) 000256 1787394501.001118 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c57e): Mirror 3: 0x568bbcde (stale) Mirror 5: 0x33fb0469 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c57e): Mirror 3: 0x568bbcde (stale) Mirror 5: 0x33fb0469 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c57e: skipped 000257 1787394501.403757 write 0x004291 thru 0x0080c1 (0x3e31 bytes) 000258 1787394501.439294 trunc from 0x03c57e to 0x03a338 (0x2245 bytes) 000259 1787394501.466757 trunc from 0x00a9f8 to 0x00ea0a (0x4013 bytes) 000262 1787394501.484688 mapwrite 0x00bc35 thru 0x00eb36 (0x2f02 bytes) 000263 1787394501.563347 read 0x029a6e thru 0x038ffe (0xf591 bytes) 000264 1787394501.575855 write 0x016f35 thru 0x022117 (0xb1e3 bytes) 000265 1787394501.591665 write 0x007602 thru 0x00cede (0x58dd bytes) 000266 1787394501.597611 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a339): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a339): Mirror 3: 0x32a59920 (stale) Mirror 5: 0xac646445 (stale) CRC-32 checksum value for chunk [0, 0x3a339): Mirror 1: 0xb3a38430 Mirror 6: 0xb3a38430 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a339: skipped 000267 1787394503.086558 punch from 0x0032c2 to 0x00f06e (0xbdad bytes) 000273 1787394503.176631 trunc from 0x02cbfe to 0x032a86 (0x5e89 bytes) 000274 1787394503.200157 MIRROR_RESYNC 4 mirrors 000275 1787394504.707309 MIRROR_RESYNC 4 mirrors 000276 1787394505.168459 fallocate 0x011a92 thru 0x01a41f (0x898e bytes) 000277 1787394505.227065 trunc from 0x03a339 to 0x0103ac (0x29f8c bytes) 000278 1787394505.254241 write 0x038772 thru 0x03ffff (0x788e bytes) 000279 1787394505.259117 mapwrite 0x02a1b8 thru 0x02eb5e (0x49a7 bytes) 000280 1787394505.314355 mapwrite 0x02da1b thru 0x02e67c (0xc62 bytes) 000281 1787394505.332075 mapread 0x015248 thru 0x01ef68 (0x9d21 bytes) 000282 1787394505.363888 punch from 0x02d56f to 0x02fd5a (0x27ec bytes) 000283 1787394505.370582 mapread 0x02a371 thru 0x03141e (0x70ae bytes) 000285 1787394505.392497 read 0x025d6d thru 0x02b2ea (0x557e bytes) 000286 1787394505.393350 mapread 0x03efc6 thru 0x03f0c7 (0x102 bytes) 000287 1787394505.400968 mapread 0x020540 thru 0x02c3ea (0xbeab bytes) 000288 1787394505.402404 mapwrite 0x02222c thru 0x025f37 (0x3d0c bytes) 000289 1787394505.421124 write 0x0010cf thru 0x001f31 (0xe63 bytes) 000291 1787394505.433305 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc92bfc93 (stale) Mirror 5: 0xc92bfc93 (stale) Mirror 6: 0xc92bfc93 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7fd3c31c Mirror 7: 0x7fd3c31c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000292 1787394506.605111 mapwrite 0x020ec6 thru 0x029b76 (0x8cb1 bytes) 000293 1787394506.737486 trunc from 0x02da08 to 0x035733 (0x7d2c bytes) 000294 1787394506.743589 mapread 0x01bb1b thru 0x022d15 (0x71fb bytes) 000295 1787394506.745039 read 0x03227e thru 0x03ffff (0xdd82 bytes) 000296 1787394506.754860 trunc from 0x00c3c2 to 0x00f473 (0x30b2 bytes) 000297 1787394506.759827 fallocate 0x002456 thru 0x00d870 (0xb41b bytes) 000298 1787394506.765247 fallocate 0x02adc6 thru 0x03ab4e (0xfd89 bytes) 000299 1787394506.769996 mapread 0x0160c9 thru 0x022a21 (0xc959 bytes) 000300 1787394506.784124 mapwrite 0x0017c0 thru 0x003952 (0x2193 bytes) 000301 1787394506.803162 trunc from 0x040000 to 0x005068 (0x3af97 bytes) 000302 1787394506.830554 fallocate 0x002816 thru 0x0088c9 (0x60b4 bytes) 000303 1787394506.836330 write 0x0350ce thru 0x0371fa (0x212d bytes) 000304 1787394506.844565 MIRROR_RESYNC 5 mirrors 000305 1787394508.354621 read 0x005dd8 thru 0x0061e7 (0x410 bytes) 000306 1787394508.359521 fallocate 0x0330ca thru 0x03ffff (0xcf36 bytes) 000307 1787394508.419767 mapwrite 0x01d7b8 thru 0x02adf2 (0xd63b bytes) 000308 1787394508.509441 punch from 0x0171a3 to 0x023eae (0xcd0c bytes) 000309 1787394508.523084 trunc from 0x03f20b to 0x03ffff (0xdf5 bytes) 000310 1787394508.542779 mapwrite 0x0382ab thru 0x03a20e (0x1f64 bytes) 000311 1787394508.612135 trunc from 0x040000 to 0x005773 (0x3a88c bytes) 000312 1787394508.646532 read 0x0012c1 thru 0x005773 (0x44b3 bytes) 000313 1787394508.665292 mapwrite 0x013cd1 thru 0x0209bc (0xccec bytes) 000314 1787394508.771048 write 0x031023 thru 0x03ffff (0xefdd bytes) 000315 1787394508.781440 mapwrite 0x0233a2 thru 0x031ee0 (0xeb3f bytes) 000317 1787394508.862252 write 0x000e4f thru 0x00d773 (0xc925 bytes) 000319 1787394508.866496 trunc from 0x00fc2b to 0x012743 (0x2b19 bytes) 000320 1787394508.873146 trunc from 0x040000 to 0x02d2e6 (0x12d19 bytes) 000321 1787394508.905774 write 0x02d411 thru 0x03bd2c (0xe91c bytes) 000322 1787394508.909895 read 0x005a51 thru 0x009dec (0x439c bytes) 000324 1787394508.919370 punch from 0x003b5f to 0x00e1e7 (0xa689 bytes) 000325 1787394508.942337 trunc from 0x02a1d7 to 0x02ab44 (0x96e bytes) 000326 1787394508.959508 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd2d): Mirror 5: 0xe8c63aa7 (stale) Mirror 6: 0xe8c63aa7 (stale) Mirror 7: 0xe8c63aa7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd2d): Mirror 5: 0xe8c63aa7 (stale) Mirror 6: 0xe8c63aa7 (stale) Mirror 7: 0xe8c63aa7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd2d: skipped 000328 1787394509.448383 mapwrite 0x02e475 thru 0x03ace6 (0xc872 bytes) 000329 1787394509.497497 read 0x0001bf thru 0x00e0d4 (0xdf16 bytes) 000330 1787394509.514075 mapread 0x02e5cf thru 0x03bbd8 (0xd60a bytes) 000331 1787394509.515688 trunc from 0x0186fc to 0x020121 (0x7a26 bytes) 000332 1787394509.521925 punch from 0x02db93 to 0x02e16b (0x5d9 bytes) 000333 1787394509.530696 mapread 0x01fd4a thru 0x02c664 (0xc91b bytes) 000334 1787394509.549495 read 0x01500d thru 0x01fb8a (0xab7e bytes) 000335 1787394509.559845 trunc from 0x000919 to 0x003410 (0x2af8 bytes) 000336 1787394509.565415 mapwrite 0x02037c thru 0x029ab2 (0x9737 bytes) 000337 1787394509.610605 punch from 0x01cc09 to 0x01f0a9 (0x24a1 bytes) 000338 1787394509.616993 write 0x009bec thru 0x015ef8 (0xc30d bytes) 000339 1787394509.624966 read 0x00e944 thru 0x012c93 (0x4350 bytes) 000340 1787394509.626096 trunc from 0x03bd2d to 0x03526e (0x6abe bytes) 000341 1787394509.648243 write 0x0030da thru 0x00af13 (0x7e3a bytes) 000342 1787394509.662129 write 0x02b668 thru 0x0328ae (0x7247 bytes) 000344 1787394509.665797 trunc from 0x03526f to 0x034bad (0x6c1 bytes) 000345 1787394509.689121 fallocate 0x015dfd thru 0x02274b (0xc94f bytes) 000346 1787394509.699626 trunc from 0x02042d to 0x02fa3f (0xf613 bytes) 000347 1787394509.717744 fallocate 0x031f9e thru 0x03f772 (0xd7d5 bytes) 000348 1787394509.724823 mapwrite 0x028162 thru 0x035390 (0xd22f bytes) 000349 1787394509.913585 write 0x018a6a thru 0x027137 (0xe6ce bytes) 000350 1787394509.919885 punch from 0x029070 to 0x02d7a8 (0x4739 bytes) 000351 1787394509.931884 read 0x00bedc thru 0x011da5 (0x5eca bytes) 000352 1787394509.934843 trunc from 0x035391 to 0x02de4c (0x7544 bytes) 000353 1787394509.956941 read 0x016f87 thru 0x01ddeb (0x6e65 bytes) 000354 1787394509.958839 trunc from 0x02d5f7 to 0x02de4c (0x856 bytes) 000355 1787394509.964489 read 0x01bc23 thru 0x02136b (0x5749 bytes) 000356 1787394509.966356 punch from 0x027797 to 0x02de4c (0x66b6 bytes) 000357 1787394510.008476 write 0x033227 thru 0x035f39 (0x2d13 bytes) 000358 1787394510.012004 fallocate 0x03d736 thru 0x03ffff (0x28ca bytes) 000359 1787394510.026111 punch from 0x00d413 to 0x0185f5 (0xb1e3 bytes) 000360 1787394510.036702 fallocate 0x0285d4 thru 0x035e4a (0xd877 bytes) 000361 1787394510.056859 punch from 0x02c870 to 0x032a8c (0x621d bytes) 000362 1787394510.080814 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35f3a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f3a): Mirror 5: 0x1230e5a (stale) Mirror 6: 0x1230e5a (stale) Mirror 7: 0x1230e5a (stale) CRC-32 checksum value for chunk [0, 0x35f3a): Mirror 1: 0x197480b5 Mirror 8: 0x197480b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f3a: skipped 000363 1787394511.323081 trunc from 0x008d24 to 0x0152ba (0xc597 bytes) 000364 1787394511.377969 trunc from 0x035f3a to 0x024f0a (0x1102f bytes) 000365 1787394511.428521 trunc from 0x0191ac to 0x024af0 (0xb945 bytes) 000366 1787394511.454686 mapread 0x01a7df thru 0x024f0a (0xa72c bytes) 000368 1787394511.507225 read 0x003ac3 thru 0x011bc3 (0xe101 bytes) 000369 1787394511.536109 MIRROR_RESYNC 5 mirrors 000370 1787394513.332415 mapread 0x01014b thru 0x01aabe (0xa974 bytes) 000371 1787394513.351439 trunc from 0x024f0b to 0x032f28 (0xe01e bytes) 000372 1787394513.391382 mapwrite 0x005ee3 thru 0x006c10 (0xd2e bytes) 000373 1787394513.451377 fallocate 0x000d6d thru 0x008767 (0x79fb bytes) 000374 1787394513.473482 trunc from 0x032f29 to 0x02adf9 (0x812f bytes) 000375 1787394513.507438 mapwrite 0x0030a7 thru 0x00b6bc (0x8616 bytes) 000376 1787394513.577811 read 0x01dd1b thru 0x02358f (0x5875 bytes) 000377 1787394513.590375 mapread 0x006d74 thru 0x010130 (0x93bd bytes) 000378 1787394513.593231 punch from 0x00a662 to 0x0169ce (0xc36d bytes) 000379 1787394513.603208 MIRROR_RESYNC 5 mirrors 000380 1787394515.373028 trunc from 0x02adfa to 0x03fb2e (0x14d35 bytes) 000382 1787394515.415971 mapwrite 0x0266d8 thru 0x02b8df (0x5208 bytes) 000383 1787394515.461445 trunc from 0x03fb2f to 0x0170ed (0x28a41 bytes) 000384 1787394515.485586 mapread 0x00fa1b thru 0x0170ed (0x76d3 bytes) 000385 1787394515.497867 punch from 0x014df2 to 0x0170ed (0x22fc bytes) 000387 1787394515.503893 trunc from 0x01079b to 0x0170ed (0x6953 bytes) 000388 1787394515.510154 trunc from 0x0170ee to 0x009f42 (0xd1ab bytes) 000389 1787394515.524384 mapread 0x000767 thru 0x005301 (0x4b9b bytes) 000390 1787394515.533118 mapwrite 0x01513c thru 0x018e41 (0x3d06 bytes) 000391 1787394515.575367 trunc from 0x00e953 to 0x0104b8 (0x1b66 bytes) 000392 1787394515.581367 trunc from 0x018e42 to 0x0304d6 (0x17695 bytes) 000393 1787394515.597001 read 0x0024ce thru 0x011c51 (0xf784 bytes) 000394 1787394515.606307 fallocate 0x021791 thru 0x0264a4 (0x4d14 bytes) 000395 1787394515.611597 trunc from 0x0304d7 to 0x0296da (0x6dfc bytes) 000396 1787394515.631181 trunc from 0x02048d to 0x024018 (0x3b8c bytes) 000397 1787394515.636560 trunc from 0x0296db to 0x0337b3 (0xa0d9 bytes) 000398 1787394515.658191 read 0x023ad4 thru 0x02d699 (0x9bc6 bytes) 000399 1787394515.672367 write 0x003f8b thru 0x005aaf (0x1b25 bytes) 000400 1787394515.674365 mapwrite 0x02e8bb thru 0x032c4b (0x4391 bytes) 000401 1787394515.692419 fallocate 0x004e73 thru 0x011061 (0xc1ef bytes) 000402 1787394515.698124 punch from 0x0220bc to 0x02b8e4 (0x9829 bytes) 000403 1787394515.703716 punch from 0x01a534 to 0x024f63 (0xaa30 bytes) 000404 1787394515.709565 mapwrite 0x020ec2 thru 0x02f897 (0xe9d6 bytes) 000405 1787394515.799295 punch from 0x01488f to 0x014f87 (0x6f9 bytes) 000406 1787394515.804789 trunc from 0x0337b4 to 0x00ea3a (0x24d79 bytes) 000407 1787394515.828167 read 0x008c6a thru 0x00aa50 (0x1de7 bytes) 000408 1787394515.838517 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xea3b): Mirror 5: 0x82a0259a (stale) Mirror 6: 0x82a0259a (stale) Mirror 8: 0x82a0259a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea3b): Mirror 5: 0x82a0259a (stale) Mirror 6: 0x82a0259a (stale) Mirror 8: 0x82a0259a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea3b: skipped 000409 1787394516.287394 fallocate 0x038f45 thru 0x03ffff (0x70bb bytes) 000410 1787394516.296682 punch from 0x0251d4 to 0x028499 (0x32c6 bytes) 000411 1787394516.303393 mapwrite 0x017b7a thru 0x01c6b6 (0x4b3d bytes) 000412 1787394516.350070 mapread 0x0256c0 thru 0x030051 (0xa992 bytes) 000413 1787394516.393461 write 0x02e3cd thru 0x036a9e (0x86d2 bytes) 000414 1787394516.405068 trunc from 0x040000 to 0x0202bc (0x1fd43 bytes) 000415 1787394516.427115 punch from 0x018910 to 0x0202bc (0x79ad bytes) 000416 1787394516.444309 trunc from 0x0202bd to 0x01b1af (0x510d bytes) 000417 1787394516.467991 trunc from 0x01b1b0 to 0x005c46 (0x15569 bytes) 000418 1787394516.499488 write 0x022444 thru 0x022e42 (0x9ff bytes) 000419 1787394516.502864 write 0x03f286 thru 0x03f56a (0x2e5 bytes) 000420 1787394516.505051 mapwrite 0x0386b7 thru 0x039e6a (0x17b4 bytes) 000421 1787394516.543711 mapwrite 0x0352b5 thru 0x038d4c (0x3a98 bytes) 000422 1787394516.574556 trunc from 0x01ee99 to 0x028dd9 (0x9f41 bytes) 000423 1787394516.593307 read 0x012161 thru 0x01256a (0x40a bytes) 000424 1787394516.602282 read 0x010d2f thru 0x016e96 (0x6168 bytes) 000425 1787394516.636863 trunc from 0x03f56b to 0x0217f9 (0x1dd71 bytes) 000426 1787394516.663332 read 0x020a3a thru 0x0217f9 (0xdc0 bytes) 000427 1787394516.669253 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x217fa): Mirror 5: 0x5efc6e2b (stale) Mirror 8: 0x5efc6e2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x217fa): Mirror 5: 0x5efc6e2b (stale) Mirror 8: 0x5efc6e2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x217fa: skipped 000428 1787394517.105411 MIRROR_RESYNC 3 mirrors 000430 1787394518.073455 write 0x0157a2 thru 0x01f418 (0x9c77 bytes) 000432 1787394518.123705 fallocate 0x016189 thru 0x024b17 (0xe98f bytes) 000433 1787394518.130421 trunc from 0x0217fa to 0x031d46 (0x1054d bytes) 000434 1787394518.160253 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31d47): Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d47): Mirror 5: 0xd7634744 (stale) Mirror 8: 0xd7634744 (stale) CRC-32 checksum value for chunk [0, 0x31d47): Mirror 1: 0xb9da3a5d Mirror 9: 0xb9da3a5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d47: skipped 000435 1787394519.532624 trunc from 0x00b582 to 0x017347 (0xbdc6 bytes) 000436 1787394519.585709 read 0x02c237 thru 0x02f6f3 (0x34bd bytes) 000437 1787394519.595827 mapread 0x016b3d thru 0x026419 (0xf8dd bytes) 000438 1787394519.652072 mapwrite 0x00c6cc thru 0x017f87 (0xb8bc bytes) 000439 1787394519.711216 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d47): Mirror 5: 0xd7634744 (stale) Mirror 8: 0xd7634744 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d47): Mirror 5: 0xd7634744 (stale) Mirror 8: 0xd7634744 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d47: skipped 000440 1787394520.101073 mapread 0x00e0a5 thru 0x016945 (0x88a1 bytes) 000441 1787394520.171011 punch from 0x015d7b to 0x0161bf (0x445 bytes) 000442 1787394520.180723 trunc from 0x031d47 to 0x029706 (0x8640 bytes) 000443 1787394520.235773 write 0x00aabb thru 0x018579 (0xdabf bytes) 000444 1787394520.243283 mapread 0x0121d1 thru 0x013ea1 (0x1cd1 bytes) 000445 1787394520.249142 trunc from 0x029707 to 0x028577 (0x118f bytes) 000446 1787394520.271229 mapwrite 0x033c63 thru 0x034006 (0x3a4 bytes) 000447 1787394520.376577 read 0x021b1d thru 0x022223 (0x707 bytes) 000448 1787394520.395818 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34007): Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34007): Mirror 5: 0x7dfb54ca (stale) Mirror 8: 0x7dfb54ca (stale) CRC-32 checksum value for chunk [0, 0x34007): Mirror 1: 0x967188da Mirror 9: 0x967188da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34007: skipped 000449 1787394521.438694 mapread 0x01c5b2 thru 0x02be61 (0xf8b0 bytes) 000451 1787394521.555646 punch from 0x02340b to 0x02ccaa (0x98a0 bytes) 000452 1787394521.608266 punch from 0x021c0d to 0x0277d6 (0x5bca bytes) 000453 1787394521.617579 punch from 0x02dd2f to 0x031cbc (0x3f8e bytes) 000454 1787394521.636398 write 0x0195e1 thru 0x01d9da (0x43fa bytes) 000455 1787394521.690327 write 0x011202 thru 0x01ed75 (0xdb74 bytes) 000456 1787394521.721295 trunc from 0x034007 to 0x00449d (0x2fb69 bytes) 000457 1787394521.778564 write 0x024b7f thru 0x02643f (0x18c1 bytes) 000459 1787394521.784071 mapread 0x015e4f thru 0x018fd8 (0x318a bytes) 000460 1787394521.811843 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26440): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26440): Mirror 5: 0x431dc5e4 (stale) Mirror 8: 0x431dc5e4 (stale) Mirror 9: 0xdf8cadcf (stale) CRC-32 checksum value for chunk [0, 0x26440): Mirror 1: 0x64070ac1 Mirror 10: 0x64070ac1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26440: skipped 000461 1787394522.888366 write 0x00e21d thru 0x00f89e (0x1682 bytes) 000462 1787394522.938276 mapread 0x015e7d thru 0x0222f6 (0xc47a bytes) 000463 1787394522.965290 trunc from 0x026440 to 0x03ac2b (0x147ec bytes) 000465 1787394522.984305 trunc from 0x03ac2c to 0x03c089 (0x145e bytes) 000466 1787394523.005335 trunc from 0x03449d to 0x034c1f (0x783 bytes) 000467 1787394523.010641 fallocate 0x026611 thru 0x02d3b4 (0x6da4 bytes) 000468 1787394523.016864 fallocate 0x03c6ed thru 0x03ffff (0x3913 bytes) 000469 1787394523.022067 fallocate 0x0240a4 thru 0x033ec4 (0xfe21 bytes) 000470 1787394523.027148 punch from 0x01d67f to 0x0267b5 (0x9137 bytes) 000471 1787394523.034187 trunc from 0x03cad4 to 0x03ffff (0x352c bytes) 000473 1787394523.039358 write 0x037dcd thru 0x03f909 (0x7b3d bytes) 000475 1787394523.049507 trunc from 0x028e35 to 0x0354e0 (0xc6ac bytes) 000476 1787394523.059716 MIRROR_RESYNC 5 mirrors 000477 1787394524.855691 trunc from 0x040000 to 0x033ba9 (0xc456 bytes) 000478 1787394524.932611 mapwrite 0x015114 thru 0x017204 (0x20f1 bytes) 000480 1787394524.981840 fallocate 0x017514 thru 0x024447 (0xcf34 bytes) 000481 1787394524.989481 mapread 0x00b6d4 thru 0x00d2ab (0x1bd8 bytes) 000482 1787394525.005662 fallocate 0x036db1 thru 0x03fa63 (0x8cb3 bytes) 000483 1787394525.013318 mapwrite 0x00a70f thru 0x00f404 (0x4cf6 bytes) 000484 1787394525.039292 trunc from 0x008575 to 0x00f925 (0x73b1 bytes) 000485 1787394525.046397 trunc from 0x00f3bf to 0x01c64b (0xd28d bytes) 000486 1787394525.052390 punch from 0x02a616 to 0x032ad0 (0x84bb bytes) 000487 1787394525.060345 mapwrite 0x02ff2a thru 0x03a485 (0xa55c bytes) 000488 1787394525.145640 mapwrite 0x027f69 thru 0x036f79 (0xf011 bytes) 000489 1787394525.195138 mapwrite 0x035856 thru 0x036594 (0xd3f bytes) 000490 1787394525.214560 mapread 0x032eba thru 0x038eb5 (0x5ffc bytes) 000491 1787394525.217290 punch from 0x031114 to 0x033d7a (0x2c67 bytes) 000492 1787394525.238449 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a486): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a486): Mirror 5: 0xee298333 (stale) Mirror 8: 0xee298333 (stale) Mirror 9: 0xee298333 (stale) Mirror 10: 0xee298333 (stale) CRC-32 checksum value for chunk [0, 0x3a486): Mirror 1: 0x6efd3f93 Mirror 11: 0x6efd3f93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a486: skipped 000493 1787394526.779403 read 0x032141 thru 0x03a485 (0x8345 bytes) 000494 1787394526.803799 fallocate 0x03d526 thru 0x03ffff (0x2ada bytes) 000495 1787394526.892054 punch from 0x00fa25 to 0x01e0e8 (0xe6c4 bytes) 000496 1787394526.903390 fallocate 0x03d8ec thru 0x03dbbf (0x2d4 bytes) 000497 1787394526.909245 mapwrite 0x025318 thru 0x0279cb (0x26b4 bytes) 000498 1787394526.998397 trunc from 0x02d34b to 0x0323b7 (0x506d bytes) 000499 1787394527.010331 punch from 0x008650 to 0x013d31 (0xb6e2 bytes) 000500 1787394527.016513 fallocate 0x002ca4 thru 0x006140 (0x349d bytes) 000501 1787394527.027951 mapread 0x012460 thru 0x01e4bf (0xc060 bytes) 000502 1787394527.084072 punch from 0x017355 to 0x017e6e (0xb1a bytes) 000503 1787394527.097103 mapwrite 0x021eca thru 0x026fe5 (0x511c bytes) 000505 1787394527.142241 mapread 0x001b5a thru 0x00cb69 (0xb010 bytes) 000506 1787394527.161291 read 0x011964 thru 0x01bda6 (0xa443 bytes) 000507 1787394527.177786 punch from 0x0290c3 to 0x035717 (0xc655 bytes) 000508 1787394527.193235 trunc from 0x017bb4 to 0x0221a6 (0xa5f3 bytes) 000509 1787394527.203683 punch from 0x03b8d0 to 0x03f917 (0x4048 bytes) 000510 1787394527.212111 mapwrite 0x001197 thru 0x002840 (0x16aa bytes) 000511 1787394527.231377 write 0x00f0bd thru 0x016dde (0x7d22 bytes) 000514 1787394527.242281 trunc from 0x040000 to 0x006aa4 (0x3955b bytes) 000515 1787394527.299591 trunc from 0x006aa5 to 0x034526 (0x2da82 bytes) 000516 1787394527.324326 trunc from 0x02cdfa to 0x034526 (0x772d bytes) 000517 1787394527.338443 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34527): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34527): Mirror 5: 0x8a7fe2c0 (stale) Mirror 8: 0x8a7fe2c0 (stale) Mirror 9: 0x8a7fe2c0 (stale) Mirror 10: 0x8a7fe2c0 (stale) Mirror 11: 0x43516909 (stale) CRC-32 checksum value for chunk [0, 0x34527): Mirror 1: 0x8e806454 Mirror 12: 0x8e806454 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34527: skipped 000518 1787394528.459074 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34527): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34527): Mirror 5: 0x8a7fe2c0 (stale) Mirror 8: 0x8a7fe2c0 (stale) Mirror 9: 0x8a7fe2c0 (stale) Mirror 10: 0x8a7fe2c0 (stale) Mirror 11: 0x43516909 (stale) CRC-32 checksum value for chunk [0, 0x34527): Mirror 1: 0x8e806454 Mirror 12: 0x8e806454 Mirror 13: 0x8e806454 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34527: skipped 000519 1787394529.444051 fallocate 0x007821 thru 0x0106d0 (0x8eb0 bytes) 000520 1787394529.495210 trunc from 0x034527 to 0x00ab68 (0x299be bytes) 000521 1787394529.549818 write 0x01f251 thru 0x02adbc (0xbb6c bytes) 000522 1787394529.552814 punch from 0x001904 to 0x0037a1 (0x1e9e bytes) 000523 1787394529.562110 trunc from 0x003223 to 0x00a681 (0x745f bytes) 000524 1787394529.569381 fallocate 0x0219c9 thru 0x03144a (0xfa82 bytes) 000525 1787394529.591701 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2adbd): Mirror 5: 0xcc0d6378 (stale) Mirror 8: 0xcc0d6378 (stale) Mirror 9: 0xcc0d6378 (stale) Mirror 10: 0xcc0d6378 (stale) Mirror 12: 0x83c5f447 (stale) Mirror 13: 0x83c5f447 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adbd): Mirror 5: 0xcc0d6378 (stale) Mirror 8: 0xcc0d6378 (stale) Mirror 9: 0xcc0d6378 (stale) Mirror 10: 0xcc0d6378 (stale) Mirror 12: 0x83c5f447 (stale) Mirror 13: 0x83c5f447 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adbd: skipped 000527 1787394530.315997 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2adbd): Mirror 5: 0xcc0d6378 (stale) Mirror 9: 0xcc0d6378 (stale) Mirror 10: 0xcc0d6378 (stale) Mirror 12: 0x83c5f447 (stale) Mirror 13: 0x83c5f447 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adbd): Mirror 5: 0xcc0d6378 (stale) Mirror 9: 0xcc0d6378 (stale) Mirror 10: 0xcc0d6378 (stale) Mirror 12: 0x83c5f447 (stale) Mirror 13: 0x83c5f447 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adbd: skipped 000528 1787394530.870842 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2adbd): Mirror 9: 0xcc0d6378 (stale) Mirror 10: 0xcc0d6378 (stale) Mirror 12: 0x83c5f447 (stale) Mirror 13: 0x83c5f447 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adbd): Mirror 9: 0xcc0d6378 (stale) Mirror 10: 0xcc0d6378 (stale) Mirror 12: 0x83c5f447 (stale) Mirror 13: 0x83c5f447 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adbd: skipped 000529 1787394531.305756 punch from 0x01f632 to 0x02adbc (0xb78b bytes) 000530 1787394531.312374 write 0x01031b thru 0x015546 (0x522c bytes) 000531 1787394531.326267 mapwrite 0x036b68 thru 0x03c18f (0x5628 bytes) 000532 1787394531.415182 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c190): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c190): Mirror 9: 0x21da4c64 (stale) Mirror 10: 0x21da4c64 (stale) Mirror 12: 0x81441fe9 (stale) Mirror 13: 0x81441fe9 (stale) CRC-32 checksum value for chunk [0, 0x3c190): Mirror 1: 0x59f2bda8 Mirror 14: 0x59f2bda8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c190: skipped 000533 1787394532.803602 MIRROR_RESYNC 6 mirrors 000534 1787394534.710035 trunc from 0x0181b4 to 0x02459c (0xc3e9 bytes) 000535 1787394534.775559 mapread 0x01dbf9 thru 0x027e93 (0xa29b bytes) 000536 1787394534.828707 punch from 0x02ba4e to 0x0395a9 (0xdb5c bytes) 000537 1787394534.841256 fallocate 0x000fa2 thru 0x00f595 (0xe5f4 bytes) 000538 1787394534.847076 mapwrite 0x00a8de thru 0x00e512 (0x3c35 bytes) 000539 1787394534.891401 mapwrite 0x03ab89 thru 0x03ffff (0x5477 bytes) 000540 1787394534.953336 read 0x00e437 thru 0x01872b (0xa2f5 bytes) 000542 1787394534.965553 trunc from 0x000990 to 0x00a53f (0x9bb0 bytes) 000543 1787394534.971808 mapwrite 0x02fe03 thru 0x038453 (0x8651 bytes) 000544 1787394535.002040 read 0x03f0ef thru 0x03ffff (0xf11 bytes) 000546 1787394535.003549 write 0x00f90f thru 0x01a8ec (0xafde bytes) 000547 1787394535.006356 read 0x014113 thru 0x014b78 (0xa66 bytes) 000548 1787394535.007322 mapread 0x037860 thru 0x03c81a (0x4fbb bytes) 000549 1787394535.009272 write 0x03908b thru 0x03db00 (0x4a76 bytes) 000550 1787394535.011762 read 0x01e584 thru 0x02c983 (0xe400 bytes) 000551 1787394535.017221 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1544e246 (stale) Mirror 10: 0x1544e246 (stale) Mirror 12: 0x1544e246 (stale) Mirror 13: 0x1544e246 (stale) Mirror 14: 0x1544e246 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x78e1082f Mirror 15: 0x78e1082f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000552 1787394536.272029 trunc from 0x040000 to 0x02cce4 (0x1331b bytes) 000553 1787394536.350101 punch from 0x0261b0 to 0x02aa5e (0x48af bytes) 000554 1787394536.360634 trunc from 0x02cce5 to 0x0210b0 (0xbc34 bytes) 000555 1787394536.387532 trunc from 0x0210b1 to 0x00a372 (0x16d3e bytes) 000556 1787394536.410094 mapread 0x004adc thru 0x00a372 (0x5897 bytes) 000557 1787394536.430477 trunc from 0x00a373 to 0x0390d2 (0x2ed60 bytes) 000558 1787394536.449101 read 0x00f610 thru 0x012ed9 (0x38ca bytes) 000559 1787394536.454342 mapwrite 0x019738 thru 0x0198af (0x178 bytes) 000561 1787394536.476234 punch from 0x0388e7 to 0x0390d2 (0x7ec bytes) 000562 1787394536.481836 mapwrite 0x0179b6 thru 0x01d5ae (0x5bf9 bytes) 000563 1787394536.507879 fallocate 0x005a04 thru 0x012af3 (0xd0f0 bytes) 000564 1787394536.516291 mapwrite 0x0152b9 thru 0x01f51f (0xa267 bytes) 000565 1787394536.557106 trunc from 0x0390d3 to 0x031c7d (0x7455 bytes) 000566 1787394536.604798 fallocate 0x01d254 thru 0x02924f (0xbffc bytes) 000567 1787394536.638317 MIRROR_RESYNC 7 mirrors 000568 1787394538.806544 mapwrite 0x0117d7 thru 0x02129c (0xfac6 bytes) 000570 1787394538.932198 punch from 0x030730 to 0x031720 (0xff1 bytes) 000571 1787394538.940737 fallocate 0x004f42 thru 0x0101b5 (0xb274 bytes) 000572 1787394538.952026 trunc from 0x031c7e to 0x00a268 (0x27a15 bytes) 000573 1787394538.982397 mapread 0x0091b1 thru 0x00a268 (0x10b8 bytes) 000574 1787394538.993775 read 0x00520c thru 0x00a268 (0x505d bytes) 000575 1787394538.994572 fallocate 0x037278 thru 0x03ffff (0x8d88 bytes) 000576 1787394539.001441 punch from 0x037f8f to 0x03f785 (0x77f7 bytes) 000577 1787394539.007210 fallocate 0x00bf6e thru 0x011327 (0x53ba bytes) 000578 1787394539.013392 punch from 0x013970 to 0x019f73 (0x6604 bytes) 000579 1787394539.019095 trunc from 0x040000 to 0x02c365 (0x13c9a bytes) 000580 1787394539.036777 trunc from 0x02c366 to 0x031c44 (0x58df bytes) 000581 1787394539.056381 write 0x03c13c thru 0x03ffff (0x3ec4 bytes) 000582 1787394539.059230 punch from 0x03af77 to 0x03e94d (0x39d7 bytes) 000583 1787394539.071216 write 0x013e89 thru 0x01cca1 (0x8e19 bytes) 000584 1787394539.082057 trunc from 0x040000 to 0x014e76 (0x2b189 bytes) 000585 1787394539.097103 trunc from 0x00a584 to 0x014bf9 (0xa676 bytes) 000586 1787394539.108566 punch from 0x002c2b to 0x012155 (0xf52b bytes) 000587 1787394539.116427 write 0x0217c7 thru 0x024ce9 (0x3523 bytes) 000588 1787394539.119687 punch from 0x02393e to 0x024ce9 (0x13ac bytes) 000589 1787394539.130028 fallocate 0x02e68a thru 0x03d407 (0xed7e bytes) 000590 1787394539.137007 trunc from 0x024cea to 0x0119dc (0x1330d bytes) 000591 1787394539.152583 punch from 0x00f5c1 to 0x0119dc (0x241c bytes) 000592 1787394539.158890 write 0x020ac7 thru 0x02968f (0x8bc9 bytes) 000593 1787394539.164823 fallocate 0x012f1c thru 0x018c88 (0x5d6d bytes) 000595 1787394539.171658 fallocate 0x01b616 thru 0x01ec9c (0x3687 bytes) 000596 1787394539.180228 read 0x013152 thru 0x019e12 (0x6cc1 bytes) 000597 1787394539.244648 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29690): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29690): Mirror 9: 0xf9719cbf (stale) Mirror 10: 0xf9719cbf (stale) Mirror 12: 0xf9719cbf (stale) Mirror 13: 0xf9719cbf (stale) Mirror 14: 0xf9719cbf (stale) Mirror 15: 0xf9719cbf (stale) CRC-32 checksum value for chunk [0, 0x29690): Mirror 1: 0xf607f71f Mirror 16: 0xf607f71f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29690: skipped 000598 1787394540.145766 punch from 0x00d92a to 0x015b64 (0x823b bytes) 000599 1787394540.195517 mapwrite 0x00d82e thru 0x00d8c1 (0x94 bytes) 000600 1787394540.211805 trunc from 0x029690 to 0x018384 (0x1130b bytes) 000601 1787394540.270238 punch from 0x0138a7 to 0x018384 (0x4ade bytes) 000602 1787394540.276702 write 0x0197eb thru 0x028571 (0xed87 bytes) 000603 1787394540.280085 mapread 0x01d13d thru 0x022e5a (0x5d1e bytes) 000604 1787394540.282255 trunc from 0x019db8 to 0x01a065 (0x2ae bytes) 000605 1787394540.298945 mapwrite 0x039747 thru 0x03ffff (0x68b9 bytes) 000606 1787394540.351960 trunc from 0x02e630 to 0x030b65 (0x2536 bytes) 000607 1787394540.360798 read 0x00fceb thru 0x0170b8 (0x73ce bytes) 000608 1787394540.384538 trunc from 0x0139cc to 0x014ce3 (0x1318 bytes) 000609 1787394540.396515 trunc from 0x040000 to 0x0228bc (0x1d743 bytes) 000610 1787394540.428596 write 0x012ffe thru 0x015a7b (0x2a7e bytes) 000611 1787394540.451390 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x228bd): Mirror 9: 0x96896e3d (stale) Mirror 10: 0x96896e3d (stale) Mirror 12: 0x96896e3d (stale) Mirror 13: 0x96896e3d (stale) Mirror 14: 0x96896e3d (stale) Mirror 15: 0x96896e3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x228bd): Mirror 9: 0x96896e3d (stale) Mirror 10: 0x96896e3d (stale) Mirror 12: 0x96896e3d (stale) Mirror 13: 0x96896e3d (stale) Mirror 14: 0x96896e3d (stale) Mirror 15: 0x96896e3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x228bd: skipped 000612 1787394541.020257 mapread 0x01a076 thru 0x01d50b (0x3496 bytes) 000613 1787394541.042139 fallocate 0x02f3a5 thru 0x03192d (0x2589 bytes) 000614 1787394541.048238 fallocate 0x009e7e thru 0x0167b8 (0xc93b bytes) 000615 1787394541.054198 mapwrite 0x019bef thru 0x025240 (0xb652 bytes) 000616 1787394541.119327 fallocate 0x014a30 thru 0x01d3e1 (0x89b2 bytes) 000617 1787394541.125298 trunc from 0x025241 to 0x026e27 (0x1be7 bytes) 000618 1787394541.143737 read 0x023103 thru 0x026e27 (0x3d25 bytes) 000620 1787394541.156747 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26e28): Mirror 9: 0x38eaa132 (stale) Mirror 10: 0x38eaa132 (stale) Mirror 12: 0x38eaa132 (stale) Mirror 13: 0x38eaa132 (stale) Mirror 14: 0x38eaa132 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e28): Mirror 9: 0x38eaa132 (stale) Mirror 10: 0x38eaa132 (stale) Mirror 12: 0x38eaa132 (stale) Mirror 13: 0x38eaa132 (stale) Mirror 14: 0x38eaa132 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e28: skipped 000621 1787394541.716198 read 0x00690a thru 0x00e7c7 (0x7ebe bytes) 000622 1787394541.740732 mapread 0x007c53 thru 0x01148f (0x983d bytes) 000623 1787394541.794286 read 0x001f92 thru 0x00e8a5 (0xc914 bytes) 000624 1787394541.808955 write 0x014f76 thru 0x0209e9 (0xba74 bytes) 000625 1787394541.818553 mapwrite 0x01c894 thru 0x0258c8 (0x9035 bytes) 000626 1787394541.842625 trunc from 0x026e28 to 0x015986 (0x114a1 bytes) 000627 1787394541.867158 punch from 0x00ee2b to 0x015986 (0x6b5c bytes) 000628 1787394541.874577 fallocate 0x006f15 thru 0x0100d3 (0x91bf bytes) 000629 1787394541.879083 write 0x01b952 thru 0x01ef65 (0x3614 bytes) 000630 1787394541.881709 write 0x01070b thru 0x016958 (0x624e bytes) 000631 1787394541.892191 write 0x038fc4 thru 0x03f1f3 (0x6230 bytes) 000633 1787394541.895813 trunc from 0x00ce5d to 0x0133d8 (0x657c bytes) 000634 1787394541.921027 mapread 0x00bf05 thru 0x013e74 (0x7f70 bytes) 000636 1787394541.980331 punch from 0x0223d2 to 0x02f983 (0xd5b2 bytes) 000637 1787394541.986219 read 0x00e3ee thru 0x014e28 (0x6a3b bytes) 000638 1787394541.987581 mapread 0x00784a thru 0x00fad7 (0x828e bytes) 000639 1787394541.989738 fallocate 0x03fc5f thru 0x03ffff (0x3a1 bytes) 000640 1787394541.995855 trunc from 0x02eef8 to 0x033426 (0x452f bytes) 000641 1787394542.001864 fallocate 0x013696 thru 0x0139ff (0x36a bytes) 000642 1787394542.007621 mapread 0x00c18c thru 0x01ad9a (0xec0f bytes) 000643 1787394542.034832 punch from 0x02e2b0 to 0x0372f7 (0x9048 bytes) 000644 1787394542.049804 write 0x0088a7 thru 0x009329 (0xa83 bytes) 000646 1787394542.055632 trunc from 0x002adb to 0x00db0e (0xb034 bytes) 000648 1787394542.069154 punch from 0x017a07 to 0x0248a8 (0xcea2 bytes) 000649 1787394542.074890 trunc from 0x030b78 to 0x03c9f7 (0xbe80 bytes) 000651 1787394542.081711 fallocate 0x02ff9e thru 0x036fae (0x7011 bytes) 000652 1787394542.088001 punch from 0x00aa63 to 0x00e440 (0x39de bytes) 000653 1787394542.093142 mapread 0x030d15 thru 0x037e20 (0x710c bytes) 000654 1787394542.161022 trunc from 0x03f1f4 to 0x0374b0 (0x7d43 bytes) 000655 1787394542.177593 read 0x0314a9 thru 0x0374b0 (0x6008 bytes) 000656 1787394542.178655 trunc from 0x01cfac to 0x02050d (0x3562 bytes) 000657 1787394542.183878 read 0x024a82 thru 0x02dfc3 (0x9542 bytes) 000658 1787394542.198073 write 0x00c804 thru 0x0165ef (0x9dec bytes) 000659 1787394542.199562 punch from 0x02e0a6 to 0x032836 (0x4791 bytes) 000660 1787394542.208593 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x374b1): Mirror 10: 0xbdd4ecc0 (stale) Mirror 12: 0xbdd4ecc0 (stale) Mirror 13: 0xbdd4ecc0 (stale) Mirror 14: 0xbdd4ecc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374b1): Mirror 10: 0xbdd4ecc0 (stale) Mirror 12: 0xbdd4ecc0 (stale) Mirror 13: 0xbdd4ecc0 (stale) Mirror 14: 0xbdd4ecc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374b1: skipped 000661 1787394542.582223 mapwrite 0x018e2e thru 0x0243fa (0xb5cd bytes) 000663 1787394542.669270 trunc from 0x0374b1 to 0x03f87e (0x83ce bytes) 000664 1787394542.703404 mapwrite 0x03be9b thru 0x03ffff (0x4165 bytes) 000665 1787394542.740951 fallocate 0x02ca98 thru 0x03a924 (0xde8d bytes) 000666 1787394542.746115 trunc from 0x019879 to 0x0240f5 (0xa87d bytes) 000667 1787394542.751553 mapwrite 0x00d42d thru 0x016f88 (0x9b5c bytes) 000668 1787394542.778597 read 0x01d26e thru 0x01f123 (0x1eb6 bytes) 000669 1787394542.789349 write 0x00bb9c thru 0x01486b (0x8cd0 bytes) 000670 1787394542.791704 mapread 0x02da43 thru 0x038148 (0xa706 bytes) 000671 1787394542.800981 mapread 0x00f699 thru 0x01315c (0x3ac4 bytes) 000672 1787394542.802143 write 0x00586f thru 0x006dca (0x155c bytes) 000673 1787394542.812840 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe68c5dea (stale) Mirror 12: 0xe68c5dea (stale) Mirror 13: 0xe68c5dea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe68c5dea (stale) Mirror 12: 0xe68c5dea (stale) Mirror 13: 0xe68c5dea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1787394543.130853 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe68c5dea (stale) Mirror 12: 0xe68c5dea (stale) Mirror 13: 0xe68c5dea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x50c599ba Mirror 14: 0x50c599ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000675 1787394544.398851 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xe68c5dea (stale) Mirror 13: 0xe68c5dea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xe68c5dea (stale) Mirror 13: 0xe68c5dea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x50c599ba Mirror 14: 0x50c599ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000676 1787394545.079101 trunc from 0x01fb1d to 0x0254de (0x59c2 bytes) 000677 1787394545.134479 fallocate 0x02632c thru 0x02d080 (0x6d55 bytes) 000679 1787394545.140940 write 0x007815 thru 0x00dcde (0x64ca bytes) 000680 1787394545.170863 trunc from 0x040000 to 0x035502 (0xaafd bytes) 000681 1787394545.222458 fallocate 0x008604 thru 0x01196f (0x936c bytes) 000682 1787394545.237450 trunc from 0x00ef71 to 0x016b52 (0x7be2 bytes) 000683 1787394545.267339 MIRROR_RESYNC 4 mirrors 000684 1787394547.041204 fallocate 0x006b66 thru 0x00885b (0x1cf6 bytes) 000686 1787394547.098165 MIRROR_RESYNC 4 mirrors 000687 1787394548.612787 punch from 0x026232 to 0x02b9bf (0x578e bytes) 000688 1787394548.683551 mapread 0x00af92 thru 0x00ee73 (0x3ee2 bytes) 000690 1787394548.728644 mapread 0x00be82 thru 0x013569 (0x76e8 bytes) 000691 1787394548.731080 trunc from 0x018499 to 0x027b8b (0xf6f3 bytes) 000692 1787394548.743337 write 0x024196 thru 0x02a217 (0x6082 bytes) 000693 1787394548.767348 fallocate 0x020e25 thru 0x028398 (0x7574 bytes) 000694 1787394548.783022 trunc from 0x00ca3f to 0x018cd9 (0xc29b bytes) 000695 1787394548.796071 mapwrite 0x006e90 thru 0x012434 (0xb5a5 bytes) 000696 1787394548.859546 fallocate 0x00c6be thru 0x0128df (0x6222 bytes) 000697 1787394548.866329 fallocate 0x00da2d thru 0x011e4e (0x4422 bytes) 000698 1787394548.875724 trunc from 0x035503 to 0x01db8e (0x17974 bytes) 000699 1787394548.907563 trunc from 0x01db8f to 0x01d01f (0xb6f bytes) 000700 1787394548.934269 read 0x00aa32 thru 0x00ff3d (0x550c bytes) 000701 1787394548.935309 write 0x024164 thru 0x024a37 (0x8d4 bytes) 000702 1787394548.936893 write 0x031f1c thru 0x034946 (0x2a2b bytes) 000703 1787394548.939089 write 0x022ec5 thru 0x02ee0e (0xbf4a bytes) 000704 1787394548.964538 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34947): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34947): Mirror 12: 0xe217e36d (stale) Mirror 13: 0xe217e36d (stale) Mirror 14: 0xe217e36d (stale) CRC-32 checksum value for chunk [0, 0x34947): Mirror 1: 0x6da42a43 Mirror 15: 0x6da42a43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34947: skipped 000705 1787394550.178832 fallocate 0x01ac9d thru 0x0291a4 (0xe508 bytes) 000706 1787394550.221894 mapread 0x0219fd thru 0x02ffb4 (0xe5b8 bytes) 000707 1787394550.267284 write 0x00b25c thru 0x017595 (0xc33a bytes) 000708 1787394550.290615 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34947): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34947): Mirror 12: 0xe217e36d (stale) Mirror 13: 0xe217e36d (stale) Mirror 14: 0xe217e36d (stale) Mirror 15: 0x6da42a43 (stale) CRC-32 checksum value for chunk [0, 0x34947): Mirror 1: 0x470cb51b Mirror 16: 0x470cb51b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34947: skipped 000709 1787394551.190218 mapread 0x023ba2 thru 0x0292e7 (0x5746 bytes) 000710 1787394551.224129 trunc from 0x034947 to 0x035f27 (0x15e1 bytes) 000712 1787394551.321530 trunc from 0x003b15 to 0x0067f9 (0x2ce5 bytes) 000713 1787394551.331132 mapwrite 0x001ed1 thru 0x010bcb (0xecfb bytes) 000714 1787394551.394326 punch from 0x02aa93 to 0x0355a1 (0xab0f bytes) 000716 1787394551.420174 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35f28): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f28): Mirror 12: 0x9e48d6 (stale) Mirror 13: 0x9e48d6 (stale) Mirror 14: 0x9e48d6 (stale) Mirror 15: 0x4263130d (stale) Mirror 16: 0xebb8eaec (stale) CRC-32 checksum value for chunk [0, 0x35f28): Mirror 1: 0x5adf434f Mirror 17: 0x5adf434f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f28: skipped 000717 1787394552.438380 punch from 0x001823 to 0x00d7b9 (0xbf97 bytes) 000718 1787394552.509038 mapwrite 0x010ced thru 0x01e897 (0xdbab bytes) 000719 1787394552.591183 mapwrite 0x00319f thru 0x00da42 (0xa8a4 bytes) 000720 1787394552.647869 mapread 0x01776d thru 0x01fb8b (0x841f bytes) 000722 1787394552.653876 trunc from 0x016f62 to 0x0170b7 (0x156 bytes) 000723 1787394552.673559 read 0x02bc96 thru 0x03565e (0x99c9 bytes) 000724 1787394552.695865 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35f28): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f28): Mirror 12: 0x9e48d6 (stale) Mirror 13: 0x9e48d6 (stale) Mirror 14: 0x9e48d6 (stale) Mirror 15: 0x4263130d (stale) Mirror 16: 0xebb8eaec (stale) Mirror 17: 0x5adf434f (stale) CRC-32 checksum value for chunk [0, 0x35f28): Mirror 1: 0x325fb7fb Mirror 18: 0x325fb7fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f28: skipped 000725 1787394554.128696 fallocate 0x00f941 thru 0x0192e4 (0x99a4 bytes) 000727 1787394554.190000 write 0x000876 thru 0x00c84d (0xbfd8 bytes) 000728 1787394554.206017 punch from 0x0209d0 to 0x020d2e (0x35f bytes) 000729 1787394554.211430 trunc from 0x029981 to 0x03598f (0xc00f bytes) 000730 1787394554.218227 trunc from 0x02244a to 0x02c12c (0x9ce3 bytes) 000731 1787394554.224192 read 0x02c730 thru 0x035f27 (0x97f8 bytes) 000732 1787394554.232716 mapread 0x007a7b thru 0x012254 (0xa7da bytes) 000733 1787394554.251801 mapwrite 0x0297b0 thru 0x02e237 (0x4a88 bytes) 000734 1787394554.287140 punch from 0x000d8b to 0x006a76 (0x5cec bytes) 000735 1787394554.293830 mapwrite 0x0081ba thru 0x00f0e9 (0x6f30 bytes) 000736 1787394554.312849 trunc from 0x028ac4 to 0x035f27 (0xd464 bytes) 000737 1787394554.318714 punch from 0x00bca1 to 0x011ca1 (0x6001 bytes) 000738 1787394554.327961 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x35f28): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f28): Mirror 12: 0x9e48d6 (stale) Mirror 13: 0x9e48d6 (stale) Mirror 14: 0x9e48d6 (stale) Mirror 15: 0x4263130d (stale) Mirror 16: 0xebb8eaec (stale) Mirror 17: 0x5adf434f (stale) Mirror 18: 0x325fb7fb (stale) CRC-32 checksum value for chunk [0, 0x35f28): Mirror 1: 0x69e1f9a7 Mirror 19: 0x69e1f9a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f28: skipped 000739 1787394555.858134 trunc from 0x020e7c to 0x02e563 (0xd6e8 bytes) 000740 1787394555.945125 write 0x0153e0 thru 0x01beef (0x6b10 bytes) 000741 1787394555.978598 trunc from 0x035f28 to 0x01cfd6 (0x18f51 bytes) 000742 1787394556.073844 write 0x020ea4 thru 0x02a4fa (0x9657 bytes) 000743 1787394556.077139 trunc from 0x02a4fb to 0x0335db (0x90e1 bytes) 000744 1787394556.115375 fallocate 0x03056a thru 0x031653 (0x10ea bytes) 000745 1787394556.133242 mapwrite 0x0254e4 thru 0x03485c (0xf379 bytes) 000746 1787394556.198854 read 0x00de89 thru 0x017871 (0x99e9 bytes) 000747 1787394556.208856 read 0x0113d3 thru 0x01377d (0x23ab bytes) 000748 1787394556.210089 punch from 0x005c14 to 0x00c47f (0x686c bytes) 000750 1787394556.216572 fallocate 0x011bc7 thru 0x02015e (0xe598 bytes) 000751 1787394556.221760 fallocate 0x033556 thru 0x0357bd (0x2268 bytes) 000752 1787394556.227540 mapwrite 0x00722d thru 0x0083f0 (0x11c4 bytes) 000753 1787394556.256569 write 0x023b84 thru 0x02d47c (0x98f9 bytes) 000754 1787394556.258341 write 0x03b314 thru 0x03ffff (0x4cec bytes) 000755 1787394556.261129 punch from 0x020e2f to 0x02bf27 (0xb0f9 bytes) 000756 1787394556.279515 trunc from 0x0068c4 to 0x01487e (0xdfbb bytes) 000758 1787394556.286114 mapwrite 0x03b88a thru 0x03ffff (0x4776 bytes) 000759 1787394556.304850 trunc from 0x000e9b to 0x00beba (0xb020 bytes) 000760 1787394556.310161 mapread 0x0114a7 thru 0x017c16 (0x6770 bytes) 000761 1787394556.333005 write 0x00c297 thru 0x013b5b (0x78c5 bytes) 000762 1787394556.340570 write 0x019521 thru 0x020415 (0x6ef5 bytes) 000764 1787394556.355581 trunc from 0x040000 to 0x01f8f2 (0x2070d bytes) 000765 1787394556.385105 trunc from 0x01813c to 0x01f8f2 (0x77b7 bytes) 000766 1787394556.413228 fallocate 0x01e49f thru 0x022057 (0x3bb9 bytes) 000767 1787394556.423116 MIRROR_RESYNC 9 mirrors 000768 1787394559.217955 read 0x01c0fd thru 0x01f8f2 (0x37f6 bytes) 000769 1787394559.225245 trunc from 0x00f6e0 to 0x01a595 (0xaeb6 bytes) 000770 1787394559.269650 fallocate 0x019582 thru 0x0280c3 (0xeb42 bytes) 000771 1787394559.277078 trunc from 0x00223e to 0x0077d2 (0x5595 bytes) 000772 1787394559.284144 read 0x0203c7 thru 0x0280c3 (0x7cfd bytes) 000773 1787394559.295964 mapwrite 0x013ab3 thru 0x0207a9 (0xccf7 bytes) 000775 1787394559.326507 read 0x020aaf thru 0x020ec0 (0x412 bytes) 000776 1787394559.328698 punch from 0x018d6b to 0x025860 (0xcaf6 bytes) 000777 1787394559.337399 mapwrite 0x03631c thru 0x036d4b (0xa30 bytes) 000778 1787394559.384251 write 0x016739 thru 0x020ea0 (0xa768 bytes) 000779 1787394559.386723 mapwrite 0x00f806 thru 0x016f83 (0x777e bytes) 000780 1787394559.416894 MIRROR_RESYNC 9 mirrors 000781 1787394561.602707 mapwrite 0x0122d5 thru 0x0162cb (0x3ff7 bytes) 000782 1787394561.723699 write 0x0386af thru 0x03b44c (0x2d9e bytes) 000783 1787394561.729037 mapwrite 0x020485 thru 0x023cd4 (0x3850 bytes) 000784 1787394561.766541 mapread 0x0296e9 thru 0x0298dc (0x1f4 bytes) 000785 1787394561.784151 mapread 0x031cf0 thru 0x038907 (0x6c18 bytes) 000787 1787394561.803464 punch from 0x024f48 to 0x02f9c1 (0xaa7a bytes) 000788 1787394561.809749 read 0x02b216 thru 0x02e37c (0x3167 bytes) 000789 1787394561.825039 mapwrite 0x00ce42 thru 0x0133a3 (0x6562 bytes) 000790 1787394561.871664 trunc from 0x03b44d to 0x0274d7 (0x13f75 bytes) 000791 1787394561.899113 mapwrite 0x029799 thru 0x02a414 (0xc7c bytes) 000792 1787394561.945651 MIRROR_RESYNC 9 mirrors 000793 1787394564.444643 read 0x008138 thru 0x01224e (0xa117 bytes) 000794 1787394564.455725 mapread 0x000918 thru 0x003b55 (0x323e bytes) 000795 1787394564.483695 MIRROR_EXTEND to 10 mirrors 000796 1787394565.383693 mapread 0x00a7d1 thru 0x017314 (0xcb44 bytes) 000797 1787394565.442978 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) 000798 1787394566.067603 trunc from 0x00abf4 to 0x00b799 (0xba6 bytes) 000799 1787394566.139461 mapread 0x0151d6 thru 0x01dca4 (0x8acf bytes) 000800 1787394566.161436 trunc from 0x02a415 to 0x03688e (0xc47a bytes) 000801 1787394566.189585 fallocate 0x0356c1 thru 0x03ffff (0xa93f bytes) 000802 1787394566.197054 mapread 0x02440b thru 0x030cab (0xc8a1 bytes) 000803 1787394566.219827 write 0x00e821 thru 0x00fb63 (0x1343 bytes) 000804 1787394566.231335 punch from 0x0355b6 to 0x03688e (0x12d9 bytes) 000806 1787394566.238189 trunc from 0x03688f to 0x0100ed (0x267a1 bytes) 000807 1787394566.260562 trunc from 0x0100ee to 0x02ec07 (0x1eb1a bytes) 000808 1787394566.287438 trunc from 0x01cb87 to 0x02a071 (0xd4eb bytes) 000809 1787394566.293969 read 0x0244e0 thru 0x029bea (0x570b bytes) 000810 1787394566.305383 mapread 0x008519 thru 0x01233d (0x9e25 bytes) 000811 1787394566.332859 write 0x032632 thru 0x03f1af (0xcb7e bytes) 000812 1787394566.337518 fallocate 0x00346c thru 0x009071 (0x5c06 bytes) 000813 1787394566.351475 fallocate 0x035693 thru 0x03d468 (0x7dd6 bytes) 000814 1787394566.357185 trunc from 0x00551a to 0x00a7fd (0x52e4 bytes) 000815 1787394566.361971 write 0x0058c1 thru 0x0119ef (0xc12f bytes) 000816 1787394566.373654 mapwrite 0x008c4e thru 0x012d36 (0xa0e9 bytes) 000817 1787394566.396448 read 0x02b5a6 thru 0x02caa3 (0x14fe bytes) 000818 1787394566.418506 fallocate 0x02a4ff thru 0x034031 (0x9b33 bytes) 000819 1787394566.425691 trunc from 0x016d54 to 0x023f7c (0xd229 bytes) 000820 1787394566.430273 trunc from 0x023e89 to 0x02a7ca (0x6942 bytes) 000821 1787394566.440531 punch from 0x02bf17 to 0x03b1b4 (0xf29e bytes) 000822 1787394566.449329 write 0x00286a thru 0x003481 (0xc18 bytes) 000823 1787394566.450598 write 0x0331e9 thru 0x03ffff (0xce17 bytes) 000824 1787394566.460707 mapread 0x004ead thru 0x00b000 (0x6154 bytes) 000825 1787394566.463128 mapwrite 0x03a901 thru 0x03f998 (0x5098 bytes) 000828 1787394566.494179 fallocate 0x025cf8 thru 0x034a70 (0xed79 bytes) 000829 1787394566.501450 trunc from 0x024ca4 to 0x0303e1 (0xb73e bytes) 000830 1787394566.515579 trunc from 0x034b82 to 0x039144 (0x45c3 bytes) 000831 1787394566.527815 mapwrite 0x0241d6 thru 0x0293b6 (0x51e1 bytes) 000832 1787394566.618509 read 0x0268c3 thru 0x0304c5 (0x9c03 bytes) 000833 1787394566.637107 read 0x03470d thru 0x03e5ea (0x9ede bytes) 000835 1787394566.638485 fallocate 0x000682 thru 0x00ce3c (0xc7bb bytes) 000836 1787394566.645129 trunc from 0x040000 to 0x014e2a (0x2b1d5 bytes) 000837 1787394566.677016 mapread 0x00e8f1 thru 0x014e2a (0x653a bytes) 000838 1787394566.678613 fallocate 0x01e210 thru 0x01f380 (0x1171 bytes) 000839 1787394566.685619 trunc from 0x0199d4 to 0x01f380 (0x59ad bytes) 000840 1787394566.698629 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1f381): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f381): Mirror 13: 0x3c6a02b7 (stale) Mirror 14: 0x3c6a02b7 (stale) Mirror 15: 0x3c6a02b7 (stale) Mirror 16: 0x3c6a02b7 (stale) Mirror 17: 0x3c6a02b7 (stale) Mirror 18: 0x3c6a02b7 (stale) Mirror 19: 0x3c6a02b7 (stale) Mirror 20: 0x3c6a02b7 (stale) CRC-32 checksum value for chunk [0, 0x1f381): Mirror 1: 0xe5f85a22 Mirror 21: 0xe5f85a22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f381: skipped 000841 1787394568.202206 write 0x029ba3 thru 0x031251 (0x76af bytes) 000842 1787394568.312774 mapwrite 0x03f109 thru 0x03ffff (0xef7 bytes) 000843 1787394568.386136 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xa79862d5 (stale) Mirror 15: 0xa79862d5 (stale) Mirror 16: 0xa79862d5 (stale) Mirror 17: 0xa79862d5 (stale) Mirror 18: 0xa79862d5 (stale) Mirror 19: 0xa79862d5 (stale) Mirror 20: 0xa79862d5 (stale) Mirror 21: 0x33af4900 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xa79862d5 (stale) Mirror 15: 0xa79862d5 (stale) Mirror 16: 0xa79862d5 (stale) Mirror 17: 0xa79862d5 (stale) Mirror 18: 0xa79862d5 (stale) Mirror 19: 0xa79862d5 (stale) Mirror 20: 0xa79862d5 (stale) Mirror 21: 0x33af4900 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000844 1787394569.282566 read 0x00aa88 thru 0x00cf10 (0x2489 bytes) 000845 1787394569.297627 trunc from 0x040000 to 0x00b81a (0x347e5 bytes) 000846 1787394569.335070 trunc from 0x003196 to 0x00b81a (0x8685 bytes) 000847 1787394569.341772 punch from 0x006874 to 0x00a889 (0x4016 bytes) 000848 1787394569.349751 mapread 0x004188 thru 0x008522 (0x439b bytes) 000849 1787394569.413122 trunc from 0x00b81b to 0x01c4cc (0x10cb2 bytes) 000850 1787394569.447520 fallocate 0x004283 thru 0x00f52d (0xb2ab bytes) 000851 1787394569.454965 punch from 0x011b3b to 0x01c4cc (0xa992 bytes) 000852 1787394569.466389 mapread 0x00df94 thru 0x01c4cc (0xe539 bytes) 000853 1787394569.505893 write 0x00ece4 thru 0x010c0a (0x1f27 bytes) 000854 1787394569.507340 fallocate 0x034f5d thru 0x03e20c (0x92b0 bytes) 000855 1787394569.525894 MIRROR_RESYNC 9 mirrors 000856 1787394571.942891 punch from 0x01528b to 0x01c4cc (0x7242 bytes) 000857 1787394571.990850 fallocate 0x02893c thru 0x02f6bb (0x6d80 bytes) 000858 1787394572.005008 trunc from 0x0138eb to 0x01c4cc (0x8be2 bytes) 000859 1787394572.014176 write 0x00452f thru 0x01314a (0xec1c bytes) 000860 1787394572.028713 punch from 0x015053 to 0x016e2f (0x1ddd bytes) 000861 1787394572.042126 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1c4cd): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c4cd): Mirror 13: 0xa7a17a64 (stale) Mirror 15: 0xa7a17a64 (stale) Mirror 16: 0xa7a17a64 (stale) Mirror 17: 0xa7a17a64 (stale) Mirror 18: 0xa7a17a64 (stale) Mirror 19: 0xa7a17a64 (stale) Mirror 20: 0xa7a17a64 (stale) Mirror 21: 0xa7a17a64 (stale) CRC-32 checksum value for chunk [0, 0x1c4cd): Mirror 1: 0x3b24b801 Mirror 22: 0x3b24b801 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c4cd: skipped 000862 1787394573.409878 mapread 0x00e55a thru 0x00fe9f (0x1946 bytes) 000863 1787394573.471724 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c4cd): Mirror 15: 0xa7a17a64 (stale) Mirror 16: 0xa7a17a64 (stale) Mirror 17: 0xa7a17a64 (stale) Mirror 18: 0xa7a17a64 (stale) Mirror 19: 0xa7a17a64 (stale) Mirror 20: 0xa7a17a64 (stale) Mirror 21: 0xa7a17a64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c4cd): Mirror 15: 0xa7a17a64 (stale) Mirror 16: 0xa7a17a64 (stale) Mirror 17: 0xa7a17a64 (stale) Mirror 18: 0xa7a17a64 (stale) Mirror 19: 0xa7a17a64 (stale) Mirror 20: 0xa7a17a64 (stale) Mirror 21: 0xa7a17a64 (stale) CRC-32 checksum value for chunk [0, 0x1c4cd): Mirror 1: 0x3b24b801 Mirror 22: 0x3b24b801 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c4cd: skipped 000864 1787394574.538445 trunc from 0x01c4cd to 0x017906 (0x4bc6 bytes) 000865 1787394574.605555 read 0x00f644 thru 0x0120b2 (0x2a6f bytes) 000866 1787394574.621921 punch from 0x001d3b to 0x0095c8 (0x788e bytes) 000867 1787394574.641581 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17907): Mirror 16: 0xa25f29da (stale) Mirror 17: 0xa25f29da (stale) Mirror 18: 0xa25f29da (stale) Mirror 19: 0xa25f29da (stale) Mirror 20: 0xa25f29da (stale) Mirror 21: 0xa25f29da (stale) Mirror 22: 0xb2a03325 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17907): Mirror 16: 0xa25f29da (stale) Mirror 17: 0xa25f29da (stale) Mirror 18: 0xa25f29da (stale) Mirror 19: 0xa25f29da (stale) Mirror 20: 0xa25f29da (stale) Mirror 21: 0xa25f29da (stale) Mirror 22: 0xb2a03325 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17907: skipped 000869 1787394575.180572 write 0x01a781 thru 0x01ea47 (0x42c7 bytes) 000871 1787394575.188075 trunc from 0x010914 to 0x01d404 (0xcaf1 bytes) 000873 1787394575.213970 trunc from 0x01ea48 to 0x038ce6 (0x1a29f bytes) 000874 1787394575.256664 mapread 0x024bab thru 0x02eea9 (0xa2ff bytes) 000876 1787394575.292915 trunc from 0x038ce7 to 0x0103b8 (0x2892e bytes) 000877 1787394575.313935 punch from 0x00dc53 to 0x0103b8 (0x2766 bytes) 000878 1787394575.324551 read 0x0054b9 thru 0x00fc54 (0xa79c bytes) 000879 1787394575.343773 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x103b9): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x103b9): Mirror 16: 0xe5b1ea7e (stale) Mirror 17: 0xe5b1ea7e (stale) Mirror 18: 0xe5b1ea7e (stale) Mirror 19: 0xe5b1ea7e (stale) Mirror 20: 0xe5b1ea7e (stale) Mirror 21: 0xe5b1ea7e (stale) Mirror 22: 0x5b0de3c7 (stale) CRC-32 checksum value for chunk [0, 0x103b9): Mirror 1: 0xa96fc543 Mirror 23: 0xa96fc543 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x103b9: skipped 000880 1787394576.437839 trunc from 0x0103b9 to 0x002e16 (0xd5a2 bytes) 000882 1787394576.544382 fallocate 0x0102dc thru 0x0122b0 (0x1fd5 bytes) 000883 1787394576.549548 write 0x03ee2f thru 0x03ffff (0x11d1 bytes) 000884 1787394576.551241 read 0x024981 thru 0x0291e7 (0x4867 bytes) 000885 1787394576.558875 read 0x0157bb thru 0x01b104 (0x594a bytes) 000886 1787394576.572281 MIRROR_RESYNC 9 mirrors 000887 1787394578.355410 write 0x033805 thru 0x03b162 (0x795e bytes) 000888 1787394578.415344 mapwrite 0x01d979 thru 0x02acba (0xd342 bytes) 000889 1787394578.478174 mapwrite 0x01a10e thru 0x01cbfb (0x2aee bytes) 000890 1787394578.508742 trunc from 0x040000 to 0x023a3e (0x1c5c1 bytes) 000891 1787394578.546247 MIRROR_RESYNC 9 mirrors 000892 1787394580.305514 mapread 0x00b143 thru 0x01420e (0x90cc bytes) 000893 1787394580.353420 MIRROR_EXTEND to 10 mirrors 000894 1787394581.339798 punch from 0x006a26 to 0x00b008 (0x45e3 bytes) 000895 1787394581.382393 MIRROR_RESYNC 10 mirrors 000896 1787394583.077019 read 0x00a2e2 thru 0x00c54a (0x2269 bytes) 000897 1787394583.091992 mapwrite 0x02b322 thru 0x039e20 (0xeaff bytes) 000899 1787394583.200811 punch from 0x024e37 to 0x02cd2a (0x7ef4 bytes) 000900 1787394583.207770 mapread 0x00a635 thru 0x013f7e (0x994a bytes) 000901 1787394583.228595 trunc from 0x005054 to 0x007255 (0x2202 bytes) 000903 1787394583.235712 read 0x029a83 thru 0x02ae23 (0x13a1 bytes) 000905 1787394583.246335 mapread 0x01d398 thru 0x023d70 (0x69d9 bytes) 000906 1787394583.247675 trunc from 0x039e21 to 0x0290c5 (0x10d5b bytes) 000907 1787394583.286456 punch from 0x02352b to 0x0290c5 (0x5b9b bytes) 000908 1787394583.303411 trunc from 0x0290c6 to 0x038a70 (0xf9ab bytes) 000909 1787394583.327306 read 0x00d84f thru 0x011b86 (0x4338 bytes) 000910 1787394583.328451 write 0x024d0d thru 0x02feb2 (0xb1a6 bytes) 000914 1787394583.372649 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a71): Mirror 16: 0x8f7301d7 (stale) Mirror 18: 0x8f7301d7 (stale) Mirror 19: 0x8f7301d7 (stale) Mirror 20: 0x8f7301d7 (stale) Mirror 21: 0x8f7301d7 (stale) Mirror 22: 0x8f7301d7 (stale) Mirror 23: 0x8f7301d7 (stale) Mirror 24: 0x8f7301d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a71): Mirror 16: 0x8f7301d7 (stale) Mirror 18: 0x8f7301d7 (stale) Mirror 19: 0x8f7301d7 (stale) Mirror 20: 0x8f7301d7 (stale) Mirror 21: 0x8f7301d7 (stale) Mirror 22: 0x8f7301d7 (stale) Mirror 23: 0x8f7301d7 (stale) Mirror 24: 0x8f7301d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a71: skipped 000916 1787394584.046664 punch from 0x02adad to 0x02ba09 (0xc5d bytes) 000917 1787394584.052844 fallocate 0x01ee5d thru 0x02c50b (0xd6af bytes) 000918 1787394584.069870 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38a71): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a71): Mirror 16: 0x8f7301d7 (stale) Mirror 18: 0x8f7301d7 (stale) Mirror 19: 0x8f7301d7 (stale) Mirror 20: 0x8f7301d7 (stale) Mirror 21: 0x8f7301d7 (stale) Mirror 22: 0x8f7301d7 (stale) Mirror 23: 0x8f7301d7 (stale) Mirror 24: 0x8f7301d7 (stale) CRC-32 checksum value for chunk [0, 0x38a71): Mirror 1: 0xf3c2e0bb Mirror 25: 0xf3c2e0bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a71: skipped 000919 1787394585.591198 write 0x0363a3 thru 0x03ffff (0x9c5d bytes) 000920 1787394585.722330 MIRROR_RESYNC 10 mirrors 000921 1787394588.565704 mapwrite 0x001a8a thru 0x0039bf (0x1f36 bytes) 000922 1787394588.684541 mapwrite 0x03553d thru 0x03ffff (0xaac3 bytes) 000923 1787394588.719758 mapread 0x03c4ff thru 0x03e480 (0x1f82 bytes) 000924 1787394588.721832 trunc from 0x024c28 to 0x02a987 (0x5d60 bytes) 000925 1787394588.728709 punch from 0x038d7f to 0x03ffff (0x7281 bytes) 000927 1787394588.735415 mapwrite 0x03d1b6 thru 0x03ffff (0x2e4a bytes) 000928 1787394588.760840 trunc from 0x028b54 to 0x036185 (0xd632 bytes) 000929 1787394588.767721 mapwrite 0x0372c8 thru 0x03ffff (0x8d38 bytes) 000930 1787394588.790429 fallocate 0x027224 thru 0x02752c (0x309 bytes) 000931 1787394588.795859 punch from 0x01be9e to 0x02688a (0xa9ed bytes) 000932 1787394588.802492 punch from 0x013a41 to 0x01e602 (0xabc2 bytes) 000933 1787394588.807569 mapwrite 0x029179 thru 0x037be3 (0xea6b bytes) 000934 1787394588.849269 fallocate 0x03f3d9 thru 0x03ffff (0xc27 bytes) 000935 1787394588.854637 mapwrite 0x033255 thru 0x03c0c1 (0x8e6d bytes) 000937 1787394588.882810 punch from 0x02c116 to 0x038a3e (0xc929 bytes) 000938 1787394588.891056 punch from 0x00c07a to 0x0180af (0xc036 bytes) 000939 1787394588.898818 mapwrite 0x00749e thru 0x00fc90 (0x87f3 bytes) 000940 1787394588.939718 punch from 0x01eba7 to 0x026bd0 (0x802a bytes) 000941 1787394588.956597 mapwrite 0x038b91 thru 0x03ffff (0x746f bytes) 000942 1787394589.022598 mapread 0x000393 thru 0x00bd28 (0xb996 bytes) 000943 1787394589.031306 mapwrite 0x03401a thru 0x03ec62 (0xac49 bytes) 000944 1787394589.082716 write 0x03d2ee thru 0x03dde8 (0xafb bytes) 000946 1787394589.091633 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xa67f682f (stale) Mirror 18: 0xa67f682f (stale) Mirror 20: 0xa67f682f (stale) Mirror 21: 0xa67f682f (stale) Mirror 22: 0xa67f682f (stale) Mirror 23: 0xa67f682f (stale) Mirror 24: 0xa67f682f (stale) Mirror 25: 0xa67f682f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xa67f682f (stale) Mirror 18: 0xa67f682f (stale) Mirror 20: 0xa67f682f (stale) Mirror 21: 0xa67f682f (stale) Mirror 22: 0xa67f682f (stale) Mirror 23: 0xa67f682f (stale) Mirror 24: 0xa67f682f (stale) Mirror 25: 0xa67f682f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000947 1787394589.927268 read 0x02f5a3 thru 0x030c70 (0x16ce bytes) 000948 1787394589.935603 mapread 0x000d08 thru 0x00cfd7 (0xc2d0 bytes) 000949 1787394589.988221 fallocate 0x032e87 thru 0x036135 (0x32af bytes) 000950 1787394589.995115 trunc from 0x040000 to 0x026e1a (0x191e5 bytes) 000951 1787394590.035063 trunc from 0x026e1b to 0x029b9c (0x2d82 bytes) 000952 1787394590.071063 fallocate 0x0015d8 thru 0x001ad5 (0x4fe bytes) 000953 1787394590.080827 trunc from 0x020506 to 0x0273e8 (0x6ee3 bytes) 000954 1787394590.094700 mapwrite 0x0150d8 thru 0x01df23 (0x8e4c bytes) 000955 1787394590.151250 trunc from 0x0150b1 to 0x024346 (0xf296 bytes) 000957 1787394590.171701 fallocate 0x0196fe thru 0x024430 (0xad33 bytes) 000958 1787394590.186121 fallocate 0x003334 thru 0x011b02 (0xe7cf bytes) 000959 1787394590.192531 mapwrite 0x0075e6 thru 0x00c107 (0x4b22 bytes) 000960 1787394590.231001 mapwrite 0x00830b thru 0x015417 (0xd10d bytes) 000961 1787394590.279823 read 0x016359 thru 0x022317 (0xbfbf bytes) 000962 1787394590.303002 mapwrite 0x002aa3 thru 0x004bd0 (0x212e bytes) 000963 1787394590.320619 trunc from 0x018b99 to 0x02608e (0xd4f6 bytes) 000964 1787394590.326522 mapread 0x0043dd thru 0x00e91f (0xa543 bytes) 000965 1787394590.328184 trunc from 0x002f2a to 0x00f530 (0xc607 bytes) 000968 1787394590.335291 fallocate 0x01989c thru 0x0281f3 (0xe958 bytes) 000969 1787394590.340635 punch from 0x00faee to 0x014698 (0x4bab bytes) 000970 1787394590.349559 write 0x00c4bc thru 0x0180a3 (0xbbe8 bytes) 000971 1787394590.360482 mapwrite 0x024751 thru 0x033003 (0xe8b3 bytes) 000972 1787394590.460590 read 0x01445f thru 0x022c9f (0xe841 bytes) 000973 1787394590.475562 trunc from 0x02bfa9 to 0x02e5f3 (0x264b bytes) 000974 1787394590.481384 trunc from 0x02cac2 to 0x02e922 (0x1e61 bytes) 000975 1787394590.488273 trunc from 0x033004 to 0x00e6ea (0x24919 bytes) 000976 1787394590.513424 MIRROR_RESYNC 9 mirrors 000977 1787394591.847966 fallocate 0x00dd8d thru 0x017d18 (0x9f8c bytes) 000978 1787394591.882968 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xe6eb): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe6eb): Mirror 16: 0xea2987bd (stale) Mirror 18: 0xea2987bd (stale) Mirror 20: 0xea2987bd (stale) Mirror 21: 0xea2987bd (stale) Mirror 22: 0xea2987bd (stale) Mirror 23: 0xea2987bd (stale) Mirror 24: 0xea2987bd (stale) Mirror 25: 0xea2987bd (stale) CRC-32 checksum value for chunk [0, 0xe6eb): Mirror 1: 0xea2987bd Mirror 26: 0xea2987bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe6eb: skipped 000980 1787394593.222986 MIRROR_RESYNC 10 mirrors 000981 1787394594.818672 read 0x0077c3 thru 0x00e6ea (0x6f28 bytes) 000982 1787394594.830654 write 0x021f3c thru 0x02a5b4 (0x8679 bytes) 000983 1787394594.895622 write 0x02e5e8 thru 0x038d03 (0xa71c bytes) 000984 1787394594.905135 trunc from 0x025681 to 0x027e95 (0x2815 bytes) 000985 1787394594.975397 fallocate 0x018cfc thru 0x023dd7 (0xb0dc bytes) 000986 1787394594.994474 mapwrite 0x00efe9 thru 0x014dc8 (0x5de0 bytes) 000987 1787394595.062549 mapread 0x033256 thru 0x038d03 (0x5aae bytes) 000988 1787394595.064530 fallocate 0x0167a3 thru 0x01a1f3 (0x3a51 bytes) 000989 1787394595.076196 fallocate 0x010c03 thru 0x01303d (0x243b bytes) 000991 1787394595.090727 write 0x017cf2 thru 0x0264ee (0xe7fd bytes) 000992 1787394595.105675 read 0x010138 thru 0x018d7e (0x8c47 bytes) 000993 1787394595.109419 mapread 0x037bd9 thru 0x038d03 (0x112b bytes) 000994 1787394595.111708 fallocate 0x01e1e4 thru 0x026830 (0x864d bytes) 000995 1787394595.129635 trunc from 0x038d04 to 0x01be28 (0x1cedb bytes) 000996 1787394595.159521 read 0x016333 thru 0x01bc38 (0x5906 bytes) 000997 1787394595.160411 punch from 0x01b193 to 0x01be28 (0xc96 bytes) 000998 1787394595.201838 fallocate 0x00f665 thru 0x0198be (0xa25a bytes) 000999 1787394595.210533 fallocate 0x01ab99 thru 0x01eadb (0x3f43 bytes) 001000 1787394595.225481 punch from 0x00c3e2 to 0x0138eb (0x750a bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 06:30:00 (1787394600) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 06:30:10 (1787394610) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254794 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125509 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 06:30:18 (1787394618) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3829184 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3453152 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6205536 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8380064 .. Extending file size to 7792768 .. Extending file size to 8450976 .. Extending file size to 8210272 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9121696 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8393088 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8431968 .. Extending file size to 8622400 .. Waiting 61128 61129 61131 61133 61135 done 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) 192.168.203.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:31:41 (1787394701) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1318336 .. Extending file size to 766112 .. Extending file size to 1578688 .. failed Extending file size to 7619488 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7531872 .. Extending file size to 7803840 .. Extending file size to 8008064 .. Extending file size to 8851264 .. Extending file size to 7947456 .. Extending file size to 8268576 .. Extending file size to 9141440 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8891520 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62437 62439 62440 62441 62444 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) 192.168.203.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (87s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:33:08 (1787394788) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18860 3588160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24580 3582440 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43440 7170600 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18860 3588160 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24580 3582440 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43440 7170600 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1045 s, 20.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261215 s, 8.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x370facc8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:33:32 (1787394812) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131802 s, 15.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 06:33:39 (1787394819) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:33:46 (1787394826) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0923594 s, 22.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x221:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x220:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x221:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:33:54 (1787394834) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122752 s, 17.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x222:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x223:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x222:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x222:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x223:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x222:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x223:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:34:03 (1787394843) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114398 s, 18.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x224:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x224:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x225:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:34:37 (1787394877) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120698 s, 8.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:35:31 (1787394931) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117992 s, 17.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:36:18 (1787394978) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg305-server: Pool lustre.test_204f created Adding targets to pool oleg305-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg305-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158315 s, 13.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x225:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x225:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg305-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg305-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg305-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:37:28 (1787395048) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:37:36 (1787395056) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:37:44 (1787395064) oleg305-server: Pool lustre.test_206 created Adding targets to pool oleg305-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg305-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg305-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg305-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg305-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:38:14 (1787395094) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462803 s, 22.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374233 s, 28.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8f775a07 Mirror 3: 0x8f775a07 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:38:26 (1787395106) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:38:31 (1787395111) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:38:34 (1787395114) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0738649 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x14d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) PASS 209a (46s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:39:21 (1787395161) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.025492 s, 321 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:39:30 (1787395170) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100396 s, 10.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 06:39:32.000000000 -0400 Modify: 2026-08-22 06:39:32.000000000 -0400 Change: 2026-08-22 06:39:33.000000000 -0400 Birth: 2026-08-22 06:39:31.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:39:39 (1787395179) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874195 s, 12.0 MB/s oleg305-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg305-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg305-server: O/280000401/d30/13e: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:40:06 (1787395206) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.191105 s, 214 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0245671 s, 167 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 06:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 06:40 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:40:16 (1787395216) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:40:26 (1787395226) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1875 sec =========== 06:40:35 (1787395235) === sanity-flr: start cleanup 06:40:37 (1787395237) === === sanity-flr: finish cleanup 06:40:45 (1787395245) ===