-----============= acceptance-small: sanity-flr ============----- Mon Aug 17 02:16:02 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg309-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:16:23 (1786947383) === oleg309-client.virtnet: executing check_config_client /mnt/lustre oleg309-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg309-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89680653c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff89680653c000.idle_timeout=debug disable quota as required oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:17:07 (1786947427) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8221] == sanity-flr test 0a: lfs mirror create with -N option == 02:17:09 (1786947429) PASS 0a (24s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:17:33 (1786947453) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:17:37 (1786947457) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:17:41 (1786947461) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:18:07 (1786947487) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:18:11 (1786947491) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 02:18:15 (1786947495) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89680653c000.stats=clear osc.lustre-OST0001-osc-ffff89680653c000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:18:23 (1786947503) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:18:34 (1786947514) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:18:43 (1786947523) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:18:47 (1786947527) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:18:57 (1786947537) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 02:19:07 (1786947547) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:19:15 (1786947555) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:19:23 (1786947563) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:19:30 (1786947570) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:19:39 (1786947579) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:19:44 (1786947584) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173873 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101919 s, 10.3 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:19:52 (1786947592) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.518412 s, 38.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:20:00 (1786947600) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:20:10 (1786947610) Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg309-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:22:27 (1786947747) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.55012 s, 17.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:23:12 (1786947792) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:24:36 (1786947876) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.40773 s, 15.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 17 02:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.133s user 0m0.001s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89680653c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.151s user 0m0.001s sys 0m0.190s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.181s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg309-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89680653c000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:26:14 (1786947974) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:26:18 (1786947978) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:26:21 (1786947981) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 02:26:24 (1786947984) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 02:26:31 (1786947991) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.553145 s, 22.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:26:44 (1786948004) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451947 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510874 s, 20.5 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:26:50 (1786948010) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:26:58 (1786948018) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00106605 s, 961 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0052701 s, 194 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00273877 s, 374 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00429956 s, 238 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 02:27:08 (1786948028) create test files with size 15737313 .. a6dcb8b4f992077604dfb1c8059dd63f - 3320a42bbffe2e073e2596b5e48a7bad - 7205213ab6610014cd5a8270c9d57f46 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 02:27:30 (1786948050) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:27:43 (1786948063) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:27:50 (1786948070) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975715 s, 10.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322662 s, 32.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb1:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb2:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 02:27:59 (1786948079) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130222 s, 16.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.452946 s, 9.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265406 s, 7.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 02:28:15 (1786948095) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:28:18 (1786948098) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:28:22 (1786948102) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284799 s, 36.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 02:28:30 (1786948110) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.44446 s, 7.1 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:28:42 (1786948122) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:28:53 (1786948133) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.328133 s, 32.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:29:05 (1786948145) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff896812cd3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.356751 s, 29.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff89681044d800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:29:20 (1786948160) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:29:58 (1786948198) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:30:34 (1786948234) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 02:30:38 (1786948238) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.32127 s, 9.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.332494 s, 9.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf44ae000 (nosync,stale) Mirror 2: 0xf44ae000 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 28ab0672c0b73ea7a2ce327626b91d96 - mirror 1 checksum: 3a7183bbb4e491d5126a1975386b2b42 - mirror 2 checksum: 28ab0672c0b73ea7a2ce327626b91d96 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T02:30:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T02:30:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 28ab0672c0b73ea7a2ce327626b91d96 - mirror 1 checksum: 3a7183bbb4e491d5126a1975386b2b42 - mirror 2 checksum: 3a7183bbb4e491d5126a1975386b2b42 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T02:30:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T02:30:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:30:49 (1786948249) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611422 s, 17.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:30:58 (1786948258) Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 02:31:11 (1786948271) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds1 to oleg309-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 02:31:42 (1786948302) targets are mounted 02:31:42 (1786948302) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02782 s, 37.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:31:58 (1786948318) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0312567 s, 655 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0193302 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:32:08 (1786948328) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0842666 s, 243 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0115669 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:32:17 (1786948337) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00546465 s, 3.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:32:24 (1786948344) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0623697 s, 328 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:32:32 (1786948352) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00543187 s, 754 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0423585 s, 96.7 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:32:39 (1786948359) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:33:00 (1786948380) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:33:27 (1786948407) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:33:50 (1786948430) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:34:13 (1786948453) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 02:34:19 (1786948459) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57557 mirror split pid 57559 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57557 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57559 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 02:35:26 (1786948526) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8528 skipping zero length zero range truncating to largest ever: 0x25c0e 000003 1786948529.027326 trunc from 00000000 to 0x025c0d (0x25c0e bytes) 000004 1786948529.072874 punch from 0x01712a to 0x01f5d3 (0x84aa bytes) 000005 1786948529.081125 mapread 0x005d84 thru 0x00ec35 (0x8eb2 bytes) 000006 1786948529.117274 mapread 0x022d7a thru 0x025c0d (0x2e94 bytes) 000007 1786948529.141992 mapwrite 0x030334 thru 0x03a920 (0xa5ed bytes) 000008 1786948529.238679 trunc from 0x008c92 to 0x00ccd6 (0x4045 bytes) 000009 1786948529.247833 mapwrite 0x020faf thru 0x027d8c (0x6dde bytes) 000010 1786948529.307989 mapwrite 0x00ae6b thru 0x019fab (0xf141 bytes) 000011 1786948529.406454 trunc from 0x03a921 to 0x00a65b (0x302c5 bytes) 000012 1786948529.435861 fallocate 0x013eaa thru 0x01a348 (0x649f bytes) 000013 1786948529.445151 mapread 0x00d639 thru 0x017641 (0xa009 bytes) 000014 1786948529.480614 fallocate 0x01121a thru 0x016f22 (0x5d09 bytes) 000015 1786948529.494417 write 0x0069dc thru 0x015ed4 (0xf4f9 bytes) 000016 1786948529.499610 mapwrite 0x02ef2d thru 0x031ae3 (0x2bb7 bytes) 000017 1786948529.577472 mapwrite 0x027a4f thru 0x02c10a (0x46bc bytes) 000018 1786948529.614024 fallocate 0x002465 thru 0x01110b (0xeca7 bytes) truncating to largest ever: 0x270c8 000020 1786948529.628059 trunc from 0x031ae4 to 0x0270c7 (0xaa1c bytes) 000021 1786948529.657307 punch from 0x0036b0 to 0x010f34 (0xd885 bytes) 000022 1786948529.667257 mapwrite 0x02decd thru 0x03c897 (0xe9cb bytes) 000023 1786948529.747150 trunc from 0x03c898 to 0x001fee (0x3a8a9 bytes) 000024 1786948529.778985 mapread 0x000f75 thru 0x001fee (0x107a bytes) 000025 1786948529.801740 write 0x01e1ca thru 0x01ec31 (0xa68 bytes) fallocating to largest ever: 0x3f5c8 000026 1786948529.809571 fallocate 0x0349ad thru 0x03f5c7 (0xac1b bytes) 000027 1786948529.823860 mapwrite 0x03e472 thru 0x03ffff (0x1b8e bytes) 000028 1786948529.936660 trunc from 0x040000 to 0x027d9a (0x18265 bytes) 000029 1786948529.969074 punch from 0x01bc5e to 0x01fbfb (0x3f9e bytes) 000030 1786948529.979570 mapread 0x013678 thru 0x014961 (0x12ea bytes) 000031 1786948530.000601 write 0x023e97 thru 0x026e59 (0x2fc3 bytes) 000032 1786948530.013587 trunc from 0x027d9b to 0x014b04 (0x13296 bytes) 000033 1786948530.040516 fallocate 0x01337e thru 0x0199cc (0x664f bytes) 000034 1786948530.056756 read 0x010e21 thru 0x016747 (0x5927 bytes) 000035 1786948530.071376 punch from 0x00fa16 to 0x0185bb (0x8ba6 bytes) 000036 1786948530.089363 mapwrite 0x01b14c thru 0x029c14 (0xeac9 bytes) 000037 1786948530.181260 trunc from 0x029c15 to 0x005bd8 (0x2403c bytes) 000038 1786948530.210556 trunc from 0x005bd9 to 0x0114b6 (0xb8de bytes) 000039 1786948530.230610 trunc from 0x0114b7 to 0x0171dd (0x5d27 bytes) 000040 1786948530.262915 mapread 0x008cc1 thru 0x00a6a6 (0x19e6 bytes) 000041 1786948530.291478 mapread 0x004a26 thru 0x013c47 (0xf222 bytes) 000044 1786948530.322857 punch from 0x012a1a to 0x0171dd (0x47c4 bytes) 000046 1786948530.330366 fallocate 0x018dc3 thru 0x026731 (0xd96f bytes) 000047 1786948530.336962 write 0x01d6c1 thru 0x02100c (0x394c bytes) 000048 1786948530.339974 read 0x01021b thru 0x01df9e (0xdd84 bytes) 000049 1786948530.355599 trunc from 0x00fbd4 to 0x01b0d4 (0xb501 bytes) 000050 1786948530.364442 trunc from 0x02100d to 0x005bbc (0x1b450 bytes) 000051 1786948530.383930 trunc from 0x002338 to 0x005bbc (0x3885 bytes) 000052 1786948530.391865 read 0x0032ce thru 0x005bbc (0x28ef bytes) 000054 1786948530.400275 fallocate 0x03043f thru 0x0307d1 (0x393 bytes) 000056 1786948530.413325 trunc from 0x005bbd to 0x000ddc (0x4de0 bytes) 000057 1786948530.430996 trunc from 0x00050a to 0x000ddc (0x8d3 bytes) 000058 1786948530.437958 trunc from 0x000ddd to 0x02d995 (0x2cbb9 bytes) 000060 1786948530.463232 mapwrite 0x031aeb thru 0x03e971 (0xce87 bytes) 000061 1786948530.564068 write 0x015ca6 thru 0x025860 (0xfbbb bytes) 000062 1786948530.566769 write 0x0055a7 thru 0x01025c (0xacb6 bytes) 000063 1786948530.577281 write 0x004e4a thru 0x00bc36 (0x6ded bytes) fallocating to largest ever: 0x40000 000064 1786948530.590890 fallocate 0x0323c5 thru 0x03ffff (0xdc3b bytes) 000065 1786948530.599001 trunc from 0x02a904 to 0x03a5bf (0xfcbc bytes) 000066 1786948530.604926 read 0x024c41 thru 0x02c90d (0x7ccd bytes) 000067 1786948530.629276 fallocate 0x02eced thru 0x0379f5 (0x8d09 bytes) 000068 1786948530.640650 mapwrite 0x03caf7 thru 0x03ffff (0x3509 bytes) 000070 1786948530.691926 read 0x014c8e thru 0x01dd52 (0x90c5 bytes) 000071 1786948530.694499 punch from 0x018501 to 0x01b404 (0x2f04 bytes) 000072 1786948530.703378 trunc from 0x040000 to 0x00e75a (0x318a5 bytes) 000073 1786948530.728830 trunc from 0x0002fd to 0x00a96a (0xa66e bytes) 000075 1786948530.736354 write 0x0107ef thru 0x018ea0 (0x86b2 bytes) 000076 1786948530.738122 punch from 0x009ed5 to 0x017e0c (0xdf38 bytes) 000077 1786948530.755242 punch from 0x0118bb to 0x01561f (0x3d65 bytes) 000078 1786948530.760803 fallocate 0x00a646 thru 0x01a1eb (0xfba6 bytes) 000082 1786948530.770143 read 0x0075fd thru 0x010c65 (0x9669 bytes) 000084 1786948530.789445 trunc from 0x0126a4 to 0x014ba8 (0x2505 bytes) 000085 1786948530.797268 mapread 0x00fe21 thru 0x018750 (0x8930 bytes) 000086 1786948530.817896 mapread 0x015aaf thru 0x018ea0 (0x33f2 bytes) 000087 1786948530.819308 read 0x00a87f thru 0x00c215 (0x1997 bytes) 000089 1786948530.822098 read 0x005535 thru 0x00c481 (0x6f4d bytes) 000090 1786948530.823226 punch from 0x0040dd to 0x006b72 (0x2a96 bytes) 000091 1786948530.832847 MIRROR_EXTEND to 2 mirrors 000092 1786948531.563747 write 0x01f870 thru 0x02241a (0x2bab bytes) 000095 1786948531.608183 trunc from 0x02241b to 0x03b216 (0x18dfc bytes) 000096 1786948531.637891 write 0x01a383 thru 0x01d25e (0x2edc bytes) 000097 1786948531.652037 mapread 0x02440f thru 0x027e3f (0x3a31 bytes) 000098 1786948531.674765 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b217): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b217): Mirror 2: 0xb925d439 (stale) CRC-32 checksum value for chunk [0, 0x3b217): Mirror 1: 0xc21eeeb7 Mirror 3: 0xc21eeeb7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b217: skipped 000099 1786948532.906753 trunc from 0x03b217 to 0x03efe8 (0x3dd2 bytes) 000100 1786948533.011982 fallocate 0x014dca thru 0x017eb1 (0x30e8 bytes) 000101 1786948533.017604 punch from 0x03df37 to 0x03efe8 (0x10b2 bytes) 000102 1786948533.023055 read 0x032ba4 thru 0x039eb0 (0x730d bytes) 000103 1786948533.035412 trunc from 0x00ea86 to 0x0120b8 (0x3633 bytes) 000105 1786948533.042603 trunc from 0x037172 to 0x03835e (0x11ed bytes) 000106 1786948533.056512 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3efe9): Mirror 3: 0xd51773e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efe9): Mirror 3: 0xd51773e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efe9: skipped 000107 1786948533.475838 trunc from 0x0076e7 to 0x007874 (0x18e bytes) 000108 1786948533.480408 mapwrite 0x01f67d thru 0x0247be (0x5142 bytes) 000109 1786948533.531923 mapread 0x0053e9 thru 0x00e7da (0x93f2 bytes) 000111 1786948533.550444 mapwrite 0x02d947 thru 0x03a626 (0xcce0 bytes) 000112 1786948533.608394 mapwrite 0x01e322 thru 0x02323b (0x4f1a bytes) 000113 1786948533.645214 mapwrite 0x000065 thru 0x00fbf1 (0xfb8d bytes) 000114 1786948533.691539 mapwrite 0x02f24e thru 0x0317ba (0x256d bytes) 000116 1786948533.710099 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3efe9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efe9): Mirror 3: 0xd51773e0 (stale) CRC-32 checksum value for chunk [0, 0x3efe9): Mirror 1: 0x18b253f2 Mirror 4: 0x18b253f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efe9: skipped 000117 1786948534.854778 MIRROR_RESYNC 3 mirrors 000118 1786948535.813239 fallocate 0x006c4b thru 0x00bed2 (0x5288 bytes) 000119 1786948535.852337 mapread 0x02923c thru 0x02b24f (0x2014 bytes) 000120 1786948535.871717 write 0x01489b thru 0x01d08c (0x87f2 bytes) 000121 1786948535.882476 fallocate 0x00e22f thru 0x01b21b (0xcfed bytes) 000122 1786948535.887337 trunc from 0x03efe9 to 0x011697 (0x2d951 bytes) 000123 1786948535.907395 write 0x003fef thru 0x00c8be (0x88d0 bytes) 000124 1786948535.917963 fallocate 0x01c658 thru 0x02bf38 (0xf8e1 bytes) 000126 1786948535.924898 punch from 0x00a34b to 0x00bb1a (0x17d0 bytes) 000127 1786948535.936562 write 0x01b4a6 thru 0x01e26f (0x2dca bytes) 000128 1786948535.944183 mapread 0x01291f thru 0x01f85c (0xcf3e bytes) 000129 1786948535.953819 read 0x028e11 thru 0x02bf38 (0x3128 bytes) 000130 1786948535.979170 MIRROR_RESYNC 3 mirrors 000131 1786948536.919402 trunc from 0x02bf39 to 0x013a62 (0x184d6 bytes) 000132 1786948536.987939 fallocate 0x03a032 thru 0x03ffff (0x5fce bytes) 000133 1786948537.004692 write 0x02a290 thru 0x037caf (0xda20 bytes) 000134 1786948537.014521 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37cb0): Mirror 4: 0xd589a426 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37cb0): Mirror 4: 0xd589a426 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37cb0: skipped 000135 1786948537.442757 write 0x03d7ae thru 0x03ffff (0x2852 bytes) 000136 1786948537.451704 fallocate 0x00179a thru 0x00c444 (0xacab bytes) 000137 1786948537.465902 read 0x004ea4 thru 0x00b943 (0x6aa0 bytes) 000138 1786948537.475612 trunc from 0x01516c to 0x01f559 (0xa3ee bytes) 000140 1786948537.488384 fallocate 0x02b090 thru 0x038dae (0xdd1f bytes) 000141 1786948537.496942 mapread 0x02f397 thru 0x033daa (0x4a14 bytes) 000142 1786948537.559173 MIRROR_RESYNC 2 mirrors 000143 1786948538.261620 trunc from 0x040000 to 0x00b286 (0x34d79 bytes) 000144 1786948538.316970 trunc from 0x0048a7 to 0x00b286 (0x69e0 bytes) 000145 1786948538.326945 punch from 0x00a457 to 0x00b286 (0xe30 bytes) 000146 1786948538.337854 punch from 0x001565 to 0x00b286 (0x9d22 bytes) 000147 1786948538.346601 trunc from 0x0044b0 to 0x00b286 (0x6dd7 bytes) 000148 1786948538.355907 mapwrite 0x03fd7e thru 0x03ffff (0x282 bytes) 000149 1786948538.439028 read 0x011613 thru 0x0180a3 (0x6a91 bytes) 000150 1786948538.459946 punch from 0x03dd04 to 0x03ffff (0x22fc bytes) 000151 1786948538.470993 fallocate 0x03a04b thru 0x03ffff (0x5fb5 bytes) 000152 1786948538.477986 write 0x007eb3 thru 0x00f0a4 (0x71f2 bytes) 000153 1786948538.502898 trunc from 0x040000 to 0x03c4e2 (0x3b1d bytes) 000154 1786948538.528688 mapwrite 0x0387cd thru 0x03ffff (0x7833 bytes) 000155 1786948538.605614 trunc from 0x040000 to 0x00f212 (0x30ded bytes) 000156 1786948538.634892 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf213): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf213): Mirror 4: 0xb8b9978a (stale) CRC-32 checksum value for chunk [0, 0xf213): Mirror 1: 0xe7151c1f Mirror 5: 0xe7151c1f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf213: skipped 000157 1786948539.658672 MIRROR_RESYNC 3 mirrors 000158 1786948540.318546 mapwrite 0x01502d thru 0x0170ba (0x208e bytes) 000159 1786948540.418341 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x170bb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x170bb): Mirror 4: 0xd9f22a38 (stale) Mirror 5: 0xd9f22a38 (stale) CRC-32 checksum value for chunk [0, 0x170bb): Mirror 1: 0x219f1bcf Mirror 6: 0x219f1bcf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x170bb: skipped 000160 1786948541.274260 trunc from 0x0170bb to 0x00c74e (0xa96c bytes) 000161 1786948541.350886 write 0x00087d thru 0x008858 (0x7fdc bytes) 000162 1786948541.365276 trunc from 0x0031a7 to 0x009440 (0x629a bytes) 000163 1786948541.388826 MIRROR_RESYNC 4 mirrors 000164 1786948542.503146 MIRROR_RESYNC 4 mirrors 000165 1786948542.989257 fallocate 0x01b91a thru 0x027391 (0xba78 bytes) 000166 1786948543.026332 trunc from 0x012f05 to 0x01717f (0x427b bytes) 000167 1786948543.031705 read 0x013c96 thru 0x020fc0 (0xd32b bytes) 000168 1786948543.044596 trunc from 0x027392 to 0x02eb05 (0x7774 bytes) 000169 1786948543.068021 write 0x011218 thru 0x01e7cd (0xd5b6 bytes) 000170 1786948543.081430 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2eb06): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb06): Mirror 4: 0x3b54509 (stale) Mirror 5: 0x3b54509 (stale) Mirror 6: 0x3b54509 (stale) CRC-32 checksum value for chunk [0, 0x2eb06): Mirror 1: 0x840999c8 Mirror 7: 0x840999c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb06: skipped 000171 1786948544.215766 trunc from 0x024b30 to 0x02b70d (0x6bde bytes) 000172 1786948544.255074 trunc from 0x02eb06 to 0x02e1f5 (0x910 bytes) 000173 1786948544.277014 fallocate 0x005d7f thru 0x00d829 (0x7aab bytes) 000174 1786948544.283561 mapwrite 0x021fec thru 0x02c3fe (0xa413 bytes) 000176 1786948544.352570 trunc from 0x0027cd to 0x00c51b (0x9d4f bytes) 000177 1786948544.369164 MIRROR_RESYNC 5 mirrors 000179 1786948546.065843 mapread 0x00b370 thru 0x01725b (0xbeec bytes) 000180 1786948546.101027 trunc from 0x02e1f6 to 0x03c649 (0xe454 bytes) 000181 1786948546.159081 trunc from 0x03c45c to 0x03c649 (0x1ee bytes) 000182 1786948546.171368 fallocate 0x02e37e thru 0x036635 (0x82b8 bytes) 000183 1786948546.176994 read 0x01190c thru 0x01cf8c (0xb681 bytes) 000184 1786948546.185349 read 0x026524 thru 0x02a0d1 (0x3bae bytes) 000186 1786948546.200367 punch from 0x037573 to 0x038f31 (0x19bf bytes) 000187 1786948546.212408 punch from 0x02b2f5 to 0x03863d (0xd349 bytes) 000188 1786948546.232571 MIRROR_RESYNC 5 mirrors 000189 1786948547.860969 trunc from 0x03c64a to 0x010957 (0x2bcf2 bytes) 000190 1786948547.920684 fallocate 0x00c90e thru 0x0134c2 (0x6bb5 bytes) 000191 1786948547.928353 mapwrite 0x031ddc thru 0x03a0d3 (0x82f8 bytes) 000192 1786948548.002024 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a0d4): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0d4): Mirror 4: 0x97df872b (stale) Mirror 5: 0x97df872b (stale) Mirror 6: 0x97df872b (stale) Mirror 7: 0x97df872b (stale) CRC-32 checksum value for chunk [0, 0x3a0d4): Mirror 1: 0xb9c85225 Mirror 8: 0xb9c85225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0d4: skipped 000193 1786948549.095725 read 0x000acc thru 0x007eb5 (0x73ea bytes) 000194 1786948549.114219 write 0x008eda thru 0x00c1ea (0x3311 bytes) 000195 1786948549.179496 trunc from 0x03a0d4 to 0x03bfe8 (0x1f15 bytes) 000196 1786948549.226479 MIRROR_RESYNC 6 mirrors 000198 1786948550.731765 trunc from 0x02f8ec to 0x032ea7 (0x35bc bytes) 000199 1786948550.765259 mapread 0x037543 thru 0x039df1 (0x28af bytes) 000200 1786948550.798611 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfe9): Mirror 5: 0x9d288add (stale) Mirror 6: 0x9d288add (stale) Mirror 7: 0x9d288add (stale) Mirror 8: 0x9d288add (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfe9): Mirror 5: 0x9d288add (stale) Mirror 6: 0x9d288add (stale) Mirror 7: 0x9d288add (stale) Mirror 8: 0x9d288add (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfe9: skipped 000201 1786948551.288154 MIRROR_RESYNC 5 mirrors 000202 1786948552.733265 write 0x0087ad thru 0x01776b (0xefbf bytes) 000203 1786948552.817118 mapwrite 0x015ec6 thru 0x0197de (0x3919 bytes) 000205 1786948552.887267 fallocate 0x00eb3b thru 0x00f96b (0xe31 bytes) 000206 1786948552.899190 read 0x01f54d thru 0x02a9dd (0xb491 bytes) 000207 1786948552.913803 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bfe9): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfe9): Mirror 5: 0x790303a (stale) Mirror 6: 0x790303a (stale) Mirror 7: 0x790303a (stale) Mirror 8: 0x790303a (stale) CRC-32 checksum value for chunk [0, 0x3bfe9): Mirror 1: 0x1069f512 Mirror 9: 0x1069f512 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfe9: skipped 000208 1786948554.165991 trunc from 0x03bfe9 to 0x010b99 (0x2b44f bytes) 000209 1786948554.246104 punch from 0x005bc0 to 0x0082a7 (0x26e8 bytes) 000211 1786948554.252314 write 0x0258d5 thru 0x034455 (0xeb81 bytes) 000212 1786948554.262596 read 0x0009c4 thru 0x002cd1 (0x230e bytes) 000213 1786948554.268834 fallocate 0x027380 thru 0x031200 (0x9e81 bytes) 000214 1786948554.277474 trunc from 0x034456 to 0x015c2e (0x1e827 bytes) 000215 1786948554.298537 mapwrite 0x032b58 thru 0x0381bf (0x5668 bytes) 000216 1786948554.353288 mapwrite 0x03f425 thru 0x03ffff (0xbdb bytes) 000217 1786948554.386686 write 0x01b053 thru 0x0294cb (0xe479 bytes) 000218 1786948554.404948 fallocate 0x0310ce thru 0x03ffff (0xef32 bytes) 000219 1786948554.410675 mapwrite 0x01f7ae thru 0x02917e (0x99d1 bytes) 000220 1786948554.435697 write 0x030666 thru 0x03ad2d (0xa6c8 bytes) 000221 1786948554.441447 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x81ef58b0 (stale) Mirror 6: 0x81ef58b0 (stale) Mirror 7: 0x81ef58b0 (stale) Mirror 8: 0x81ef58b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x81ef58b0 (stale) Mirror 6: 0x81ef58b0 (stale) Mirror 7: 0x81ef58b0 (stale) Mirror 8: 0x81ef58b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000223 1786948554.906106 write 0x034d32 thru 0x03ffff (0xb2ce bytes) 000224 1786948554.926520 mapread 0x027ec5 thru 0x0349b3 (0xcaef bytes) 000225 1786948554.986869 mapread 0x00ceb9 thru 0x01bc48 (0xed90 bytes) 000226 1786948555.011841 read 0x02e581 thru 0x02f33e (0xdbe bytes) 000227 1786948555.012756 punch from 0x02e7aa to 0x03118f (0x29e6 bytes) 000228 1786948555.024754 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x81ef58b0 (stale) Mirror 6: 0x81ef58b0 (stale) Mirror 8: 0x81ef58b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x81ef58b0 (stale) Mirror 6: 0x81ef58b0 (stale) Mirror 8: 0x81ef58b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000229 1786948555.378688 write 0x023115 thru 0x02b3ff (0x82eb bytes) 000230 1786948555.411287 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x81ef58b0 (stale) Mirror 8: 0x81ef58b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x81ef58b0 (stale) Mirror 8: 0x81ef58b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000231 1786948555.813358 trunc from 0x040000 to 0x00a74d (0x358b2 bytes) 000232 1786948555.865804 punch from 0x001808 to 0x00a74d (0x8f46 bytes) 000233 1786948555.876176 mapwrite 0x013c61 thru 0x019a55 (0x5df5 bytes) 000235 1786948555.979517 fallocate 0x02f680 thru 0x038066 (0x89e7 bytes) 000236 1786948555.987703 mapwrite 0x013f62 thru 0x014a4f (0xaee bytes) 000237 1786948556.012524 mapwrite 0x032274 thru 0x03f079 (0xce06 bytes) 000238 1786948556.089530 mapwrite 0x039e8e thru 0x03ffff (0x6172 bytes) 000239 1786948556.149845 mapread 0x000e31 thru 0x00deaa (0xd07a bytes) 000240 1786948556.161611 read 0x037d6d thru 0x03ecc2 (0x6f56 bytes) 000241 1786948556.163943 fallocate 0x016211 thru 0x0226ba (0xc4aa bytes) 000242 1786948556.170267 punch from 0x034562 to 0x03d3e7 (0x8e86 bytes) 000243 1786948556.176828 fallocate 0x029a80 thru 0x02e305 (0x4886 bytes) 000244 1786948556.185139 trunc from 0x01e716 to 0x02765c (0x8f47 bytes) 000246 1786948556.193525 punch from 0x03e19d to 0x03ffff (0x1e63 bytes) 000249 1786948556.200960 trunc from 0x040000 to 0x01bf3d (0x240c2 bytes) 000250 1786948556.216041 mapread 0x002bbf thru 0x011fbf (0xf401 bytes) 000251 1786948556.222046 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1bf3e): Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bf3e): Mirror 5: 0xe5e6d67d (stale) Mirror 8: 0xe5e6d67d (stale) CRC-32 checksum value for chunk [0, 0x1bf3e): Mirror 1: 0xfa04b816 Mirror 9: 0xfa04b816 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bf3e: skipped 000252 1786948557.187698 punch from 0x0177d1 to 0x0185b3 (0xde3 bytes) 000255 1786948557.234978 mapwrite 0x03da1c thru 0x03e35c (0x941 bytes) 000256 1786948557.302109 write 0x0349e4 thru 0x03677d (0x1d9a bytes) 000257 1786948557.313840 punch from 0x009424 to 0x00a0e6 (0xcc3 bytes) 000258 1786948557.320233 read 0x0115a7 thru 0x020ec2 (0xf91c bytes) 000259 1786948557.339921 write 0x035043 thru 0x03ea34 (0x99f2 bytes) 000260 1786948557.348526 MIRROR_RESYNC 4 mirrors 000261 1786948558.494369 punch from 0x01969c to 0x024524 (0xae89 bytes) 000262 1786948558.536759 read 0x0387ae thru 0x03ea34 (0x6287 bytes) 000263 1786948558.546713 trunc from 0x00bc6c to 0x00efd6 (0x336b bytes) 000264 1786948558.554659 fallocate 0x012c2e thru 0x01696f (0x3d42 bytes) 000265 1786948558.573275 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 5: 0x6e5aef71 (stale) Mirror 8: 0x6e5aef71 (stale) Mirror 9: 0x6e5aef71 (stale) CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 1: 0xf1d7c96c Mirror 10: 0xf1d7c96c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea35: skipped 000266 1786948559.604465 trunc from 0x01c3f9 to 0x0243e2 (0x7fea bytes) 000267 1786948559.651701 write 0x00c4a1 thru 0x01332a (0x6e8a bytes) 000268 1786948559.672322 write 0x018681 thru 0x019bec (0x156c bytes) 000269 1786948559.691380 trunc from 0x03ea35 to 0x00baad (0x32f87 bytes) 000270 1786948559.717265 mapread 0x009596 thru 0x00baad (0x2518 bytes) 000271 1786948559.752461 mapwrite 0x02fcf0 thru 0x03f32c (0xf63d bytes) 000272 1786948559.867077 read 0x027b5a thru 0x031c03 (0xa0aa bytes) 000273 1786948559.882312 mapwrite 0x014e83 thru 0x022d76 (0xdef4 bytes) 000274 1786948559.968964 MIRROR_RESYNC 5 mirrors 000275 1786948561.511062 mapwrite 0x02cef1 thru 0x02d241 (0x351 bytes) 000276 1786948561.631583 trunc from 0x01a8a6 to 0x0291e2 (0xe93d bytes) 000277 1786948561.636587 trunc from 0x006b57 to 0x00c348 (0x57f2 bytes) 000278 1786948561.641165 trunc from 0x03cf8b to 0x03f32c (0x23a2 bytes) 000279 1786948561.646569 mapwrite 0x015e8e thru 0x019f3e (0x40b1 bytes) 000280 1786948561.720903 trunc from 0x00e1fc to 0x010ac1 (0x28c6 bytes) 000282 1786948561.736535 mapwrite 0x03a8ed thru 0x03ffff (0x5713 bytes) 000283 1786948561.822516 trunc from 0x040000 to 0x01d009 (0x22ff6 bytes) 000284 1786948561.839695 read 0x00137f thru 0x0086fb (0x737d bytes) 000285 1786948561.861675 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1d00a): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d00a): Mirror 5: 0x2d1f6ab (stale) Mirror 8: 0x2d1f6ab (stale) Mirror 9: 0x2d1f6ab (stale) Mirror 10: 0x2d1f6ab (stale) CRC-32 checksum value for chunk [0, 0x1d00a): Mirror 1: 0x248ab634 Mirror 11: 0x248ab634 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d00a: skipped 000286 1786948562.722120 mapwrite 0x03694c thru 0x03def4 (0x75a9 bytes) 000287 1786948562.815518 read 0x01794f thru 0x025770 (0xde22 bytes) 000288 1786948562.824826 trunc from 0x03def5 to 0x02e7ff (0xf6f5 bytes) 000289 1786948562.846319 mapwrite 0x01c7fa thru 0x021db7 (0x55be bytes) 000290 1786948562.864260 trunc from 0x02e800 to 0x01aa9c (0x13d63 bytes) 000291 1786948562.884108 read 0x00f9fb thru 0x00fdaa (0x3b0 bytes) 000292 1786948562.893667 punch from 0x00e38e to 0x014e14 (0x6a87 bytes) 000294 1786948562.900242 read 0x0100cf thru 0x011cef (0x1c21 bytes) 000297 1786948562.910529 trunc from 0x00c3d7 to 0x010305 (0x3f2f bytes) 000298 1786948562.915104 mapwrite 0x003e85 thru 0x00bc23 (0x7d9f bytes) 000299 1786948562.932963 fallocate 0x03890f thru 0x03ffff (0x76f1 bytes) 000300 1786948562.942228 fallocate 0x0255cd thru 0x02d37f (0x7db3 bytes) 000301 1786948562.950030 mapwrite 0x02b3cb thru 0x03441a (0x9050 bytes) 000303 1786948563.011615 trunc from 0x03441b to 0x0079eb (0x2ca2f bytes) 000304 1786948563.037222 read 0x000bd4 thru 0x0079eb (0x6e18 bytes) 000305 1786948563.039121 trunc from 0x0079ec to 0x006f86 (0xa65 bytes) 000306 1786948563.060086 punch from 0x00496e to 0x006f86 (0x2619 bytes) 000307 1786948563.067644 mapwrite 0x008eff thru 0x018a00 (0xfb02 bytes) 000308 1786948563.160375 trunc from 0x00dd9d to 0x0150b9 (0x731d bytes) 000309 1786948563.165316 mapread 0x002fce thru 0x00352d (0x560 bytes) 000310 1786948563.166558 fallocate 0x00ed01 thru 0x01e75d (0xfa5d bytes) 000311 1786948563.191127 fallocate 0x0383b0 thru 0x03ffff (0x7c50 bytes) 000312 1786948563.200989 trunc from 0x018a01 to 0x006b6e (0x11e92 bytes) 000313 1786948563.249523 trunc from 0x000cd5 to 0x006b6e (0x5e9a bytes) 000315 1786948563.275056 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6b6f): Mirror 5: 0x280d324d (stale) Mirror 8: 0x280d324d (stale) Mirror 9: 0x280d324d (stale) Mirror 10: 0x280d324d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b6f): Mirror 5: 0x280d324d (stale) Mirror 8: 0x280d324d (stale) Mirror 9: 0x280d324d (stale) Mirror 10: 0x280d324d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b6f: skipped 000316 1786948563.707180 trunc from 0x001f7f to 0x006b6e (0x4bf0 bytes) 000318 1786948563.715495 write 0x009b27 thru 0x00f041 (0x551b bytes) 000319 1786948563.723304 punch from 0x00e24a to 0x00f041 (0xdf8 bytes) 000320 1786948563.749851 MIRROR_RESYNC 5 mirrors 000322 1786948565.052374 mapwrite 0x037614 thru 0x03ffff (0x89ec bytes) 000324 1786948565.180067 mapread 0x026582 thru 0x02fef9 (0x9978 bytes) 000325 1786948565.210025 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8351020f (stale) Mirror 8: 0x8351020f (stale) Mirror 10: 0x8351020f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8351020f (stale) Mirror 8: 0x8351020f (stale) Mirror 10: 0x8351020f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000327 1786948565.816486 mapwrite 0x0241a5 thru 0x031ecf (0xdd2b bytes) 000328 1786948565.872790 trunc from 0x02319d to 0x0251f2 (0x2056 bytes) 000329 1786948565.877700 punch from 0x0210d6 to 0x030ba6 (0xfad1 bytes) 000330 1786948565.883053 punch from 0x03559f to 0x03ae63 (0x58c5 bytes) 000332 1786948565.890510 trunc from 0x024357 to 0x0284ab (0x4155 bytes) 000333 1786948565.900534 fallocate 0x02792b thru 0x028745 (0xe1b bytes) 000334 1786948565.907144 punch from 0x030147 to 0x03c6e7 (0xc5a1 bytes) 000336 1786948565.917003 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x8351020f (stale) Mirror 10: 0x8351020f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x8351020f (stale) Mirror 10: 0x8351020f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000337 1786948566.245141 read 0x036190 thru 0x03bb6b (0x59dc bytes) 000338 1786948566.258371 trunc from 0x040000 to 0x01c509 (0x23af6 bytes) 000339 1786948566.303074 mapread 0x0166d0 thru 0x0173ed (0xd1e bytes) 000341 1786948566.328778 trunc from 0x01b559 to 0x01c509 (0xfb1 bytes) 000342 1786948566.341554 trunc from 0x00839f to 0x017d5f (0xf9c1 bytes) 000343 1786948566.350163 trunc from 0x01c50a to 0x039221 (0x1cd18 bytes) 000344 1786948566.369565 write 0x035d5c thru 0x037962 (0x1c07 bytes) 000345 1786948566.384443 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39222): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39222): Mirror 8: 0xfe13ea49 (stale) Mirror 10: 0xfe13ea49 (stale) CRC-32 checksum value for chunk [0, 0x39222): Mirror 1: 0xd33184f5 Mirror 11: 0xd33184f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39222: skipped 000346 1786948567.421186 trunc from 0x01e6e4 to 0x02428a (0x5ba7 bytes) 000348 1786948567.509004 mapread 0x0150d2 thru 0x020a0c (0xb93b bytes) 000350 1786948567.572396 mapwrite 0x023248 thru 0x02f90a (0xc6c3 bytes) 000351 1786948567.619668 mapwrite 0x03a2e4 thru 0x03ffff (0x5d1c bytes) 000352 1786948567.727693 mapread 0x0039e0 thru 0x004b64 (0x1185 bytes) 000353 1786948567.743504 mapread 0x02bb91 thru 0x0387f5 (0xcc65 bytes) 000354 1786948567.758184 read 0x00a6fd thru 0x00cdd0 (0x26d4 bytes) 000355 1786948567.759209 write 0x00cdcc thru 0x0158e9 (0x8b1e bytes) 000356 1786948567.762176 mapread 0x037f1f thru 0x03ffff (0x80e1 bytes) 000357 1786948567.763828 mapread 0x0232f6 thru 0x027b39 (0x4844 bytes) 000358 1786948567.765799 write 0x008daf thru 0x00b95e (0x2bb0 bytes) 000359 1786948567.768200 mapwrite 0x0326c5 thru 0x038ed7 (0x6813 bytes) 000360 1786948567.788591 fallocate 0x00d324 thru 0x0180e8 (0xadc5 bytes) 000361 1786948567.799803 write 0x036c52 thru 0x03be7d (0x522c bytes) 000363 1786948567.803223 trunc from 0x040000 to 0x0304e5 (0xfb1a bytes) 000364 1786948567.835191 read 0x02619c thru 0x0304e5 (0xa34a bytes) 000365 1786948567.836228 mapread 0x00065b thru 0x00709d (0x6a43 bytes) 000366 1786948567.844681 fallocate 0x0103ff thru 0x01c2df (0xbee1 bytes) 000367 1786948567.850545 fallocate 0x011374 thru 0x0147a1 (0x342e bytes) 000368 1786948567.858465 punch from 0x003b49 to 0x012835 (0xeced bytes) 000369 1786948567.868581 write 0x020de5 thru 0x028245 (0x7461 bytes) 000370 1786948567.870996 mapread 0x0007a8 thru 0x0022bc (0x1b15 bytes) 000371 1786948567.874261 fallocate 0x01e49a thru 0x022eaf (0x4a16 bytes) 000372 1786948567.880238 read 0x023f91 thru 0x02c0d7 (0x8147 bytes) 000373 1786948567.881574 write 0x035689 thru 0x03ffff (0xa977 bytes) 000374 1786948567.889705 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x8351020f (stale) Mirror 11: 0xc09ccf98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x8351020f (stale) Mirror 11: 0xc09ccf98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000375 1786948568.222583 trunc from 0x001e8b to 0x003ea5 (0x201b bytes) 000376 1786948568.229800 punch from 0x0305cd to 0x036c29 (0x665d bytes) 000377 1786948568.235167 read 0x001e85 thru 0x0118f4 (0xfa70 bytes) 000378 1786948568.249887 trunc from 0x040000 to 0x01ff09 (0x200f6 bytes) 000379 1786948568.279682 trunc from 0x01af57 to 0x01b1f5 (0x29f bytes) 000380 1786948568.288278 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ff0a): Mirror 8: 0x4554f903 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff0a): Mirror 8: 0x4554f903 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff0a: skipped 000381 1786948568.530049 fallocate 0x019aed thru 0x022135 (0x8649 bytes) 000382 1786948568.535561 punch from 0x0154ee to 0x01f0d8 (0x9beb bytes) 000384 1786948568.553006 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22136): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22136): Mirror 8: 0xa17ae04e (stale) CRC-32 checksum value for chunk [0, 0x22136): Mirror 1: 0x1e36a25f Mirror 9: 0x1e36a25f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22136: skipped 000386 1786948569.844709 read 0x01425f thru 0x01c4a4 (0x8246 bytes) 000387 1786948569.889110 trunc from 0x00c007 to 0x0160c2 (0xa0bc bytes) 000388 1786948569.969725 write 0x0030ce thru 0x0105f0 (0xd523 bytes) 000389 1786948570.003513 trunc from 0x022136 to 0x005497 (0x1cc9e bytes) 000390 1786948570.073161 mapread 0x00421b thru 0x005497 (0x127d bytes) 000391 1786948570.074488 mapread 0x003f6b thru 0x005497 (0x152d bytes) 000392 1786948570.086570 trunc from 0x002e3b to 0x005497 (0x265d bytes) 000394 1786948570.119261 trunc from 0x005498 to 0x003bdb (0x18bc bytes) 000395 1786948570.146066 write 0x039fac thru 0x03e9c1 (0x4a16 bytes) 000396 1786948570.155126 punch from 0x03935b to 0x03e9c1 (0x5667 bytes) 000397 1786948570.189007 trunc from 0x015a66 to 0x023f3b (0xe4d6 bytes) 000398 1786948570.204734 trunc from 0x03e9c2 to 0x011565 (0x2d45c bytes) 000399 1786948570.237615 fallocate 0x0170a0 thru 0x021863 (0xa7c4 bytes) 000400 1786948570.253429 mapread 0x021806 thru 0x021863 (0x5e bytes) 000401 1786948570.263436 write 0x02187c thru 0x025c31 (0x43b6 bytes) 000402 1786948570.267739 punch from 0x001d7d to 0x006198 (0x441c bytes) 000403 1786948570.290025 MIRROR_RESYNC 3 mirrors 000404 1786948571.224286 mapwrite 0x0390b9 thru 0x03ffff (0x6f47 bytes) 000405 1786948571.348502 punch from 0x03912e to 0x039dac (0xc7f bytes) 000406 1786948571.376225 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x606bdece (stale) Mirror 9: 0x606bdece (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x14f20b54 Mirror 10: 0x14f20b54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000407 1786948572.296517 trunc from 0x040000 to 0x00b773 (0x3488c bytes) 000409 1786948572.372228 fallocate 0x01d66b thru 0x028221 (0xabb7 bytes) 000410 1786948572.380978 trunc from 0x00b774 to 0x00719a (0x45d9 bytes) 000411 1786948572.401585 write 0x01f4ea thru 0x023640 (0x4157 bytes) 000412 1786948572.408203 punch from 0x021b3e to 0x023640 (0x1b03 bytes) 000413 1786948572.438240 fallocate 0x01e1c4 thru 0x024642 (0x647f bytes) 000414 1786948572.456665 write 0x013784 thru 0x017182 (0x39ff bytes) 000415 1786948572.481454 trunc from 0x023641 to 0x005006 (0x1e63a bytes) 000416 1786948572.508673 write 0x026e6c thru 0x035fef (0xf184 bytes) 000417 1786948572.513973 mapwrite 0x02ed52 thru 0x02ef8b (0x23a bytes) 000418 1786948572.544531 mapread 0x016e7e thru 0x022db8 (0xbf3b bytes) 000419 1786948572.602607 MIRROR_RESYNC 4 mirrors 000420 1786948573.645764 write 0x033a2f thru 0x034510 (0xae2 bytes) 000421 1786948573.688443 trunc from 0x035ff0 to 0x006d8c (0x2f263 bytes) 000422 1786948573.716514 write 0x0213be thru 0x024517 (0x315a bytes) 000423 1786948573.717889 write 0x02482d thru 0x02a18a (0x595e bytes) 000424 1786948573.719181 fallocate 0x0375e2 thru 0x03f73b (0x815a bytes) 000425 1786948573.725960 fallocate 0x03e657 thru 0x03f4fc (0xea6 bytes) 000426 1786948573.731052 fallocate 0x01a165 thru 0x026a89 (0xc925 bytes) 000427 1786948573.736273 write 0x009b90 thru 0x00cee1 (0x3352 bytes) 000428 1786948573.745546 mapwrite 0x03fcff thru 0x03ffff (0x301 bytes) 000429 1786948573.820141 read 0x0225e6 thru 0x02f966 (0xd381 bytes) 000430 1786948573.836212 read 0x029456 thru 0x03864a (0xf1f5 bytes) 000432 1786948573.850147 fallocate 0x00f153 thru 0x01cdbe (0xdc6c bytes) 000434 1786948573.858618 mapwrite 0x02a793 thru 0x036d4b (0xc5b9 bytes) 000436 1786948573.885988 read 0x011148 thru 0x020d39 (0xfbf2 bytes) 000437 1786948573.886823 trunc from 0x034af9 to 0x038828 (0x3d30 bytes) 000438 1786948573.897474 punch from 0x03a679 to 0x03ffff (0x5987 bytes) 000439 1786948573.904878 punch from 0x0071a3 to 0x00cf27 (0x5d85 bytes) 000440 1786948573.916487 read 0x031e7e thru 0x03af8b (0x910e bytes) 000442 1786948573.934687 trunc from 0x040000 to 0x036f8d (0x9072 bytes) 000443 1786948573.968715 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36f8e): Mirror 9: 0x9d6c9313 (stale) Mirror 10: 0x9d6c9313 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f8e): Mirror 9: 0x9d6c9313 (stale) Mirror 10: 0x9d6c9313 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f8e: skipped 000444 1786948574.401439 mapwrite 0x02600d thru 0x032b69 (0xcb5d bytes) 000445 1786948574.451194 trunc from 0x002024 to 0x00719b (0x5178 bytes) 000447 1786948574.460293 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36f8e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f8e): Mirror 9: 0x9d6c9313 (stale) Mirror 10: 0x9d6c9313 (stale) CRC-32 checksum value for chunk [0, 0x36f8e): Mirror 1: 0x27f74d30 Mirror 11: 0x27f74d30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f8e: skipped 000448 1786948575.456700 punch from 0x031e86 to 0x036f8d (0x5108 bytes) 000450 1786948575.507047 fallocate 0x00f3b4 thru 0x011b3e (0x278b bytes) 000451 1786948575.513750 trunc from 0x02cf84 to 0x032dca (0x5e47 bytes) 000452 1786948575.521605 trunc from 0x001168 to 0x00f7a2 (0xe63b bytes) 000453 1786948575.528409 trunc from 0x028636 to 0x032384 (0x9d4f bytes) 000454 1786948575.537580 mapwrite 0x017a47 thru 0x0262fb (0xe8b5 bytes) 000455 1786948575.621944 read 0x034acc thru 0x036f8d (0x24c2 bytes) 000456 1786948575.629052 trunc from 0x036f8e to 0x00516c (0x31e21 bytes) 000457 1786948575.651145 punch from 0x0012df to 0x00516c (0x3e8e bytes) 000458 1786948575.657400 trunc from 0x0038bc to 0x00516c (0x18b1 bytes) 000459 1786948575.662548 punch from 0x002aae to 0x00516c (0x26bf bytes) 000460 1786948575.669135 trunc from 0x0000a0 to 0x00516c (0x50cd bytes) 000461 1786948575.676808 trunc from 0x002543 to 0x00516c (0x2c2a bytes) 000462 1786948575.682525 trunc from 0x004d63 to 0x00516c (0x40a bytes) 000463 1786948575.687960 fallocate 0x011b00 thru 0x01a39d (0x889e bytes) 000465 1786948575.695594 write 0x0048d6 thru 0x00ab1b (0x6246 bytes) 000466 1786948575.715200 punch from 0x00014c to 0x00a50d (0xa3c2 bytes) 000467 1786948575.733290 read 0x0112ee thru 0x012384 (0x1097 bytes) 000470 1786948575.761441 mapread 0x002c28 thru 0x0110b7 (0xe490 bytes) 000471 1786948575.764450 read 0x01072d thru 0x01a39d (0x9c71 bytes) 000472 1786948575.765414 write 0x0198e6 thru 0x01d428 (0x3b43 bytes) 000474 1786948575.767725 mapread 0x01b2a9 thru 0x01d428 (0x2180 bytes) 000475 1786948575.768764 fallocate 0x03158b thru 0x033244 (0x1cba bytes) 000476 1786948575.781104 write 0x02848d thru 0x02d38e (0x4f02 bytes) 000478 1786948575.800601 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d38f): Mirror 10: 0x7833ae65 (stale) Mirror 11: 0x15dcb52e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d38f): Mirror 10: 0x7833ae65 (stale) Mirror 11: 0x15dcb52e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d38f: skipped 000479 1786948576.352281 trunc from 0x027041 to 0x02d38e (0x634e bytes) 000480 1786948576.385717 MIRROR_RESYNC 3 mirrors 000481 1786948577.429258 mapread 0x0078fa thru 0x00a36f (0x2a76 bytes) 000482 1786948577.458367 fallocate 0x008779 thru 0x00cb7b (0x4403 bytes) 000483 1786948577.490776 write 0x0178b0 thru 0x02541d (0xdb6e bytes) 000484 1786948577.507506 fallocate 0x037d7b thru 0x03ffff (0x8285 bytes) 000485 1786948577.516473 fallocate 0x00dc12 thru 0x011738 (0x3b27 bytes) 000486 1786948577.523756 trunc from 0x00f869 to 0x014815 (0x4fad bytes) 000487 1786948577.528530 punch from 0x027d4d to 0x02c434 (0x46e8 bytes) 000488 1786948577.536937 fallocate 0x022ea7 thru 0x02845e (0x55b8 bytes) 000489 1786948577.549352 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x230f2ebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x230f2ebf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000491 1786948578.092899 mapread 0x02f454 thru 0x03e2fa (0xeea7 bytes) 000492 1786948578.166575 trunc from 0x01bd1a to 0x02a855 (0xeb3c bytes) 000493 1786948578.187513 read 0x00361f thru 0x009833 (0x6215 bytes) 000494 1786948578.198280 fallocate 0x03b43c thru 0x03ffff (0x4bc4 bytes) 000495 1786948578.206151 read 0x02c3b9 thru 0x03ae59 (0xeaa1 bytes) 000496 1786948578.208057 trunc from 0x040000 to 0x005d3e (0x3a2c1 bytes) 000497 1786948578.241977 fallocate 0x019318 thru 0x027877 (0xe560 bytes) 000498 1786948578.248658 trunc from 0x005d3f to 0x026fc8 (0x2128a bytes) 000499 1786948578.277426 read 0x006644 thru 0x00b81f (0x51dc bytes) 000500 1786948578.299341 mapread 0x0037ec thru 0x0136ed (0xff02 bytes) 000501 1786948578.301319 mapwrite 0x029f3f thru 0x039e93 (0xff55 bytes) 000502 1786948578.354294 mapread 0x030221 thru 0x039e93 (0x9c73 bytes) 000503 1786948578.357246 punch from 0x006400 to 0x01591a (0xf51b bytes) 000504 1786948578.366557 punch from 0x013b09 to 0x0183b5 (0x48ad bytes) 000505 1786948578.372590 trunc from 0x01384b to 0x023601 (0xfdb7 bytes) 000506 1786948578.378563 write 0x016cea thru 0x0231a7 (0xc4be bytes) 000507 1786948578.397113 punch from 0x00ae11 to 0x0190a8 (0xe298 bytes) 000508 1786948578.421456 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) 000509 1786948578.561953 read 0x01883e thru 0x01901d (0x7e0 bytes) 000510 1786948578.572312 punch from 0x0179d3 to 0x01bb0a (0x4138 bytes) 000513 1786948578.588762 trunc from 0x029f1c to 0x02ba06 (0x1aeb bytes) 000514 1786948578.604645 trunc from 0x038858 to 0x039e93 (0x163c bytes) 000516 1786948578.624183 fallocate 0x0202ad thru 0x027769 (0x74bd bytes) 000517 1786948578.641282 read 0x00f9dc thru 0x014198 (0x47bd bytes) 000519 1786948578.672251 fallocate 0x03d812 thru 0x03ffff (0x27ee bytes) 000522 1786948578.682183 write 0x017f8a thru 0x01c4cd (0x4544 bytes) 000523 1786948578.693866 punch from 0x02cb32 to 0x032fbc (0x648b bytes) 000526 1786948578.703000 mapwrite 0x025b26 thru 0x032e25 (0xd300 bytes) 000528 1786948578.844723 trunc from 0x0046b1 to 0x0117c7 (0xd117 bytes) 000529 1786948578.854071 read 0x02da27 thru 0x02e092 (0x66c bytes) 000531 1786948578.855345 trunc from 0x039e94 to 0x033ab4 (0x63df bytes) 000532 1786948578.880076 read 0x02ab71 thru 0x033ab4 (0x8f44 bytes) 000533 1786948578.882308 trunc from 0x017b39 to 0x01ddb0 (0x6278 bytes) 000534 1786948578.897452 read 0x00194d thru 0x00f6fd (0xddb1 bytes) 000535 1786948578.918452 mapwrite 0x003051 thru 0x011b4c (0xeafc bytes) 000537 1786948578.945082 punch from 0x00053e to 0x00fae7 (0xf5aa bytes) 000538 1786948578.953952 fallocate 0x036ae3 thru 0x038443 (0x1961 bytes) 000539 1786948578.966513 trunc from 0x038444 to 0x0057f2 (0x32c51 bytes) 000541 1786948579.000611 trunc from 0x0057f3 to 0x01114b (0xb959 bytes) 000544 1786948579.033706 mapread 0x00a280 thru 0x010834 (0x65b5 bytes) 000545 1786948579.107723 mapread 0x00ac64 thru 0x00b264 (0x601 bytes) 000546 1786948579.108650 trunc from 0x00cfb6 to 0x01114b (0x4196 bytes) 000548 1786948579.132591 mapread 0x001364 thru 0x008154 (0x6df1 bytes) 000550 1786948579.230789 MIRROR_EXTEND to 2 mirrors 000551 1786948580.031787 trunc from 0x0071e7 to 0x01114b (0x9f65 bytes) 000552 1786948580.082601 write 0x0029fc thru 0x00bfe8 (0x95ed bytes) 000553 1786948580.100797 mapwrite 0x023206 thru 0x029df9 (0x6bf4 bytes) 000554 1786948580.227788 trunc from 0x029dfa to 0x038af3 (0xecfa bytes) 000555 1786948580.251218 trunc from 0x038af4 to 0x02665c (0x12497 bytes) 000556 1786948580.286360 punch from 0x013126 to 0x0142d8 (0x11b3 bytes) 000557 1786948580.297465 punch from 0x0074e5 to 0x012cae (0xb7ca bytes) 000558 1786948580.303063 fallocate 0x00af7a thru 0x0188d6 (0xd95d bytes) 000559 1786948580.311737 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) 000560 1786948580.508476 write 0x01839f thru 0x01ad9a (0x29fc bytes) 000562 1786948580.531105 mapread 0x023e17 thru 0x025dae (0x1f98 bytes) 000564 1786948580.556414 trunc from 0x01c6b8 to 0x02538b (0x8cd4 bytes) 000565 1786948580.562504 read 0x0262d7 thru 0x02665c (0x386 bytes) 000566 1786948580.563269 trunc from 0x011a71 to 0x01a709 (0x8c99 bytes) 000567 1786948580.574000 mapwrite 0x00f5b1 thru 0x0179dd (0x842d bytes) 000568 1786948580.607387 write 0x000d4a thru 0x000ea4 (0x15b bytes) 000569 1786948580.614458 trunc from 0x0027dd to 0x0071db (0x49ff bytes) 000570 1786948580.620456 mapwrite 0x001e29 thru 0x009e4a (0x8022 bytes) 000571 1786948580.652168 mapread 0x0007eb thru 0x004f6f (0x4785 bytes) 000572 1786948580.654370 write 0x0223b3 thru 0x027fd4 (0x5c22 bytes) 000574 1786948580.660000 mapread 0x012ec8 thru 0x019a49 (0x6b82 bytes) 000575 1786948580.676912 write 0x030fce thru 0x03add2 (0x9e05 bytes) 000576 1786948580.686020 MIRROR_EXTEND to 2 mirrors 000577 1786948581.637866 trunc from 0x01721a to 0x02702e (0xfe15 bytes) 000578 1786948581.700165 mapwrite 0x009603 thru 0x014778 (0xb176 bytes) 000579 1786948581.784890 fallocate 0x03b756 thru 0x03ffff (0x48aa bytes) 000580 1786948581.794157 mapread 0x035b7d thru 0x03f690 (0x9b14 bytes) 000581 1786948581.808913 trunc from 0x001899 to 0x0113f6 (0xfb5e bytes) 000582 1786948581.814021 punch from 0x029207 to 0x029326 (0x120 bytes) 000583 1786948581.818494 fallocate 0x02e621 thru 0x02ec18 (0x5f8 bytes) 000584 1786948581.823161 write 0x00b324 thru 0x0155ae (0xa28b bytes) 000585 1786948581.837130 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x466c2d1e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6aef244 Mirror 3: 0x6aef244 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000586 1786948583.055273 fallocate 0x01b3a3 thru 0x023b83 (0x87e1 bytes) 000587 1786948583.098948 mapwrite 0x039ce4 thru 0x03ffff (0x631c bytes) 000588 1786948583.141375 write 0x036e24 thru 0x03acd1 (0x3eae bytes) 000590 1786948583.154684 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x466c2d1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x466c2d1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000591 1786948583.597371 write 0x0228d7 thru 0x02cf9b (0xa6c5 bytes) 000592 1786948583.636481 mapwrite 0x010a56 thru 0x01b7d1 (0xad7c bytes) 000593 1786948583.733835 write 0x01b3bb thru 0x01d895 (0x24db bytes) 000594 1786948583.735354 fallocate 0x02f3e5 thru 0x0320c0 (0x2cdc bytes) 000595 1786948583.749827 fallocate 0x03be3f thru 0x03ffff (0x41c1 bytes) 000596 1786948583.775201 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x466c2d1e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x739e742 Mirror 3: 0x739e742 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1786948584.888085 write 0x01ba2d thru 0x02854b (0xcb1f bytes) 000598 1786948584.969164 mapwrite 0x00c277 thru 0x00ec30 (0x29ba bytes) 000599 1786948585.020518 trunc from 0x01453a to 0x0167f8 (0x22bf bytes) 000600 1786948585.031336 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x739e742 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x739e742 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000601 1786948585.402915 mapread 0x0394c7 thru 0x03ffff (0x6b39 bytes) 000603 1786948585.422299 mapwrite 0x01b684 thru 0x0255be (0x9f3b bytes) 000604 1786948585.462383 mapread 0x02c2c4 thru 0x02ecf5 (0x2a32 bytes) 000605 1786948585.487499 mapwrite 0x019ecc thru 0x026c3c (0xcd71 bytes) 000606 1786948585.544485 trunc from 0x040000 to 0x03bff8 (0x4007 bytes) 000608 1786948585.578330 mapread 0x031ce8 thru 0x03bff8 (0xa311 bytes) 000609 1786948585.579757 read 0x00f82a thru 0x01e4c8 (0xec9f bytes) 000610 1786948585.589547 write 0x022206 thru 0x028f0c (0x6d07 bytes) 000611 1786948585.591494 mapread 0x01a68a thru 0x021453 (0x6dca bytes) 000612 1786948585.593058 mapwrite 0x017c8b thru 0x01b878 (0x3bee bytes) 000613 1786948585.613616 punch from 0x003f38 to 0x0098d8 (0x59a1 bytes) 000614 1786948585.623498 write 0x008fd0 thru 0x00c4f3 (0x3524 bytes) 000615 1786948585.639426 write 0x0326d8 thru 0x03a005 (0x792e bytes) 000616 1786948585.641068 mapread 0x0034e3 thru 0x01048b (0xcfa9 bytes) 000617 1786948585.657261 write 0x01e434 thru 0x020ba6 (0x2773 bytes) 000618 1786948585.658597 fallocate 0x02907d thru 0x033beb (0xab6f bytes) 000619 1786948585.664148 read 0x02fa16 thru 0x038d17 (0x9302 bytes) 000621 1786948585.670674 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bff9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bff9): Mirror 3: 0x45399bc8 (stale) CRC-32 checksum value for chunk [0, 0x3bff9): Mirror 1: 0x986f4da Mirror 4: 0x986f4da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bff9: skipped 000623 1786948586.708232 fallocate 0x0287d9 thru 0x02d9ed (0x5215 bytes) 000624 1786948586.748704 fallocate 0x03e7a7 thru 0x03fda7 (0x1601 bytes) 000625 1786948586.753808 read 0x034fc0 thru 0x038420 (0x3461 bytes) 000626 1786948586.766138 punch from 0x00fbba to 0x016019 (0x6460 bytes) 000627 1786948586.772340 fallocate 0x0373fe thru 0x03e77e (0x7381 bytes) 000628 1786948586.777652 mapwrite 0x018b8d thru 0x023c56 (0xb0ca bytes) 000629 1786948586.825426 fallocate 0x00256c thru 0x010f20 (0xe9b5 bytes) 000630 1786948586.835130 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 4: 0xfcb677f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 4: 0xfcb677f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e77f: skipped 000631 1786948587.116600 trunc from 0x030f7d to 0x037de7 (0x6e6b bytes) 000632 1786948587.127007 MIRROR_RESYNC 2 mirrors 000633 1786948588.156406 mapwrite 0x039ad5 thru 0x03ffff (0x652b bytes) 000635 1786948588.293799 fallocate 0x02d63d thru 0x039578 (0xbf3c bytes) 000636 1786948588.304890 mapwrite 0x0233d5 thru 0x02c6ae (0x92da bytes) 000637 1786948588.357189 trunc from 0x040000 to 0x0185a5 (0x27a5a bytes) 000638 1786948588.378949 fallocate 0x00b4b6 thru 0x00c7f9 (0x1344 bytes) 000639 1786948588.385920 mapread 0x00e402 thru 0x0185a5 (0xa1a4 bytes) 000641 1786948588.402467 punch from 0x001275 to 0x00c7b4 (0xb540 bytes) 000642 1786948588.408191 punch from 0x014caf to 0x0173d1 (0x2723 bytes) 000643 1786948588.417869 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) 000644 1786948588.578005 trunc from 0x00c42e to 0x0185a5 (0xc178 bytes) 000645 1786948588.589502 mapread 0x008863 thru 0x010e8d (0x862b bytes) 000646 1786948588.626718 mapread 0x0141ad thru 0x0185a5 (0x43f9 bytes) 000647 1786948588.645075 fallocate 0x030eb2 thru 0x03d963 (0xcab2 bytes) 000648 1786948588.653881 read 0x018e29 thru 0x01adc0 (0x1f98 bytes) 000649 1786948588.660651 read 0x034d19 thru 0x03a2eb (0x55d3 bytes) 000650 1786948588.675357 read 0x028b53 thru 0x02e11f (0x55cd bytes) 000651 1786948588.676296 write 0x02efe2 thru 0x035f71 (0x6f90 bytes) 000652 1786948588.685687 mapread 0x014e58 thru 0x0205cf (0xb778 bytes) 000653 1786948588.687040 trunc from 0x038b94 to 0x03d963 (0x4dd0 bytes) 000654 1786948588.693090 read 0x025309 thru 0x02e82c (0x9524 bytes) 000655 1786948588.694183 mapwrite 0x020875 thru 0x024b23 (0x42af bytes) 000656 1786948588.716347 fallocate 0x029d21 thru 0x035790 (0xba70 bytes) 000659 1786948588.727245 mapwrite 0x0333e6 thru 0x03e60d (0xb228 bytes) 000660 1786948588.786545 punch from 0x01baf2 to 0x0275e7 (0xbaf6 bytes) 000661 1786948588.792184 mapread 0x0182af thru 0x01dfd6 (0x5d28 bytes) 000662 1786948588.818341 trunc from 0x023169 to 0x02b1c1 (0x8059 bytes) 000663 1786948588.831449 fallocate 0x01a894 thru 0x01bdd8 (0x1545 bytes) 000664 1786948588.840312 punch from 0x00502c to 0x0095fa (0x45cf bytes) 000665 1786948588.863507 MIRROR_EXTEND to 2 mirrors 000666 1786948589.740884 fallocate 0x004e23 thru 0x00a7da (0x59b8 bytes) 000667 1786948589.786648 punch from 0x038521 to 0x03e60d (0x60ed bytes) 000668 1786948589.802331 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e60e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e60e): Mirror 2: 0x6d06d13c (stale) CRC-32 checksum value for chunk [0, 0x3e60e): Mirror 1: 0x7d39fcba Mirror 3: 0x7d39fcba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e60e: skipped 000669 1786948590.947496 read 0x019000 thru 0x01bfd9 (0x2fda bytes) 000670 1786948590.965962 read 0x013d53 thru 0x01b507 (0x77b5 bytes) 000671 1786948590.999389 fallocate 0x02bbf9 thru 0x03a9bc (0xedc4 bytes) 000672 1786948591.051836 mapread 0x0002f6 thru 0x00ed7a (0xea85 bytes) 000673 1786948591.149692 mapwrite 0x00d303 thru 0x01cfbf (0xfcbd bytes) 000674 1786948591.291890 trunc from 0x03e60e to 0x02846e (0x1619f bytes) 000675 1786948591.326368 read 0x01d942 thru 0x0210c9 (0x3788 bytes) 000676 1786948591.338583 mapwrite 0x009473 thru 0x00c65e (0x31ec bytes) 000677 1786948591.353265 fallocate 0x03ee54 thru 0x03ffff (0x11ac bytes) 000678 1786948591.358326 fallocate 0x0041e8 thru 0x004e41 (0xc5a bytes) 000679 1786948591.363874 fallocate 0x03d2a0 thru 0x03da97 (0x7f8 bytes) 000680 1786948591.368472 read 0x00126f thru 0x004367 (0x30f9 bytes) 000681 1786948591.369090 punch from 0x0047a7 to 0x00a0c0 (0x591a bytes) 000682 1786948591.374294 mapwrite 0x024353 thru 0x028ce3 (0x4991 bytes) 000684 1786948591.421286 trunc from 0x028ce4 to 0x03c342 (0x1365f bytes) 000685 1786948591.456766 mapwrite 0x01f972 thru 0x02c7c2 (0xce51 bytes) 000686 1786948591.493977 punch from 0x0390df to 0x03c342 (0x3264 bytes) 000687 1786948591.499542 fallocate 0x0124ce thru 0x02154e (0xf081 bytes) 000689 1786948591.504931 trunc from 0x035888 to 0x03c342 (0x6abb bytes) 000690 1786948591.513839 mapread 0x036e5e thru 0x038526 (0x16c9 bytes) 000691 1786948591.534470 mapwrite 0x03a5bf thru 0x03ffff (0x5a41 bytes) 000692 1786948591.598099 trunc from 0x00529e to 0x015110 (0xfe73 bytes) 000693 1786948591.603044 read 0x038f7b thru 0x03ffff (0x7085 bytes) 000694 1786948591.616508 mapread 0x039633 thru 0x03ffff (0x69cd bytes) 000695 1786948591.618464 read 0x02f8cb thru 0x033ee7 (0x461d bytes) 000696 1786948591.619346 trunc from 0x0330c2 to 0x03fda2 (0xcce1 bytes) 000697 1786948591.625290 write 0x00c339 thru 0x010d7d (0x4a45 bytes) 000698 1786948591.633202 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5edd865b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5edd865b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000699 1786948591.955146 trunc from 0x040000 to 0x002c18 (0x3d3e7 bytes) 000701 1786948591.985522 trunc from 0x00239f to 0x002c18 (0x87a bytes) 000702 1786948591.995877 write 0x028ea6 thru 0x037d0f (0xee6a bytes) 000703 1786948592.002269 punch from 0x0332c0 to 0x037d0f (0x4a50 bytes) 000704 1786948592.019361 read 0x0357b2 thru 0x037d0f (0x255e bytes) 000705 1786948592.025620 read 0x01c6c3 thru 0x02314a (0x6a88 bytes) 000706 1786948592.040538 trunc from 0x00ae79 to 0x018657 (0xd7df bytes) 000707 1786948592.046399 fallocate 0x00e122 thru 0x01a8b3 (0xc792 bytes) 000708 1786948592.055763 mapread 0x011b1c thru 0x01686b (0x4d50 bytes) 000709 1786948592.108942 fallocate 0x004c49 thru 0x00c64d (0x7a05 bytes) 000710 1786948592.115206 read 0x00f352 thru 0x01dd4a (0xe9f9 bytes) 000712 1786948592.126861 write 0x02509e thru 0x02bdf5 (0x6d58 bytes) 000713 1786948592.128586 trunc from 0x037d10 to 0x0112eb (0x26a24 bytes) 000714 1786948592.146984 fallocate 0x019085 thru 0x023a34 (0xa9b0 bytes) 000716 1786948592.161294 mapwrite 0x022f96 thru 0x02a7db (0x7846 bytes) 000717 1786948592.277861 read 0x0118ca thru 0x01d33a (0xba71 bytes) 000718 1786948592.293743 read 0x024b97 thru 0x02600c (0x1476 bytes) 000719 1786948592.294561 mapread 0x000bb2 thru 0x0033a2 (0x27f1 bytes) 000720 1786948592.301286 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) 000722 1786948592.465899 trunc from 0x02a7dc to 0x013e18 (0x169c3 bytes) 000723 1786948592.491733 read 0x00668d thru 0x00ca78 (0x63ec bytes) 000724 1786948592.512535 mapread 0x00ffea thru 0x013e18 (0x3e2f bytes) 000725 1786948592.562690 fallocate 0x00a576 thru 0x018f71 (0xe9fc bytes) 000727 1786948592.577792 mapwrite 0x009f47 thru 0x0178e2 (0xd99c bytes) 000728 1786948592.661453 mapwrite 0x017056 thru 0x01ed2a (0x7cd5 bytes) 000729 1786948592.758161 MIRROR_EXTEND to 2 mirrors 000731 1786948593.509595 write 0x01761b thru 0x01eb71 (0x7557 bytes) 000732 1786948593.583924 punch from 0x01411c to 0x01cb99 (0x8a7e bytes) 000733 1786948593.610613 punch from 0x019b6d to 0x01ed2a (0x51be bytes) 000734 1786948593.617743 trunc from 0x01ed2b to 0x03b8da (0x1cbb0 bytes) 000735 1786948593.654884 mapread 0x02f865 thru 0x034019 (0x47b5 bytes) 000736 1786948593.688478 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) 000737 1786948593.835443 punch from 0x011ed6 to 0x01ae40 (0x8f6b bytes) 000738 1786948593.840465 write 0x03ff20 thru 0x03ffff (0xe0 bytes) 000739 1786948593.851854 trunc from 0x03e3ac to 0x03ffff (0x1c54 bytes) 000740 1786948593.872255 trunc from 0x040000 to 0x0132c5 (0x2cd3a bytes) 000741 1786948593.892661 mapread 0x001d3b thru 0x00f5b2 (0xd878 bytes) 000742 1786948593.919136 read 0x005dda thru 0x00d944 (0x7b6b bytes) 000743 1786948593.920046 trunc from 0x0132c6 to 0x03d86d (0x2a5a8 bytes) 000744 1786948593.941510 punch from 0x02984d to 0x02ed20 (0x54d4 bytes) 000745 1786948593.949489 trunc from 0x03d86e to 0x011ebd (0x2b9b0 bytes) 000746 1786948593.974031 trunc from 0x00ae20 to 0x011ebd (0x709e bytes) 000747 1786948593.984538 trunc from 0x00cecf to 0x010784 (0x38b6 bytes) 000748 1786948593.998429 read 0x0091a4 thru 0x011ebd (0x8d1a bytes) 000749 1786948594.019274 fallocate 0x01d2f8 thru 0x02a732 (0xd43b bytes) 000750 1786948594.025381 mapwrite 0x0275f9 thru 0x02f205 (0x7c0d bytes) 000752 1786948594.124530 fallocate 0x037252 thru 0x03ffff (0x8dae bytes) 000753 1786948594.136577 read 0x01dbf1 thru 0x020c37 (0x3047 bytes) 000754 1786948594.137405 write 0x00c3e6 thru 0x01ba53 (0xf66e bytes) 000755 1786948594.140818 mapwrite 0x01e83c thru 0x025791 (0x6f56 bytes) 000756 1786948594.163022 trunc from 0x01d414 to 0x01fc9e (0x288b bytes) 000757 1786948594.179767 mapread 0x016212 thru 0x021650 (0xb43f bytes) 000758 1786948594.188967 punch from 0x0211c0 to 0x0299cc (0x880d bytes) 000759 1786948594.198175 trunc from 0x02f206 to 0x0287fb (0x6a0a bytes) 000760 1786948594.230752 MIRROR_EXTEND to 2 mirrors 000761 1786948594.985806 read 0x010122 thru 0x01b664 (0xb543 bytes) 000762 1786948594.996877 write 0x008626 thru 0x012770 (0xa14b bytes) 000763 1786948595.049040 punch from 0x0173cc to 0x023af9 (0xc72e bytes) 000764 1786948595.054414 mapread 0x0053dd thru 0x005798 (0x3bc bytes) 000765 1786948595.061389 trunc from 0x0287fc to 0x02c7e9 (0x3fee bytes) 000766 1786948595.086708 fallocate 0x023565 thru 0x02b906 (0x83a2 bytes) 000768 1786948595.092508 read 0x014480 thru 0x01b216 (0x6d97 bytes) 000769 1786948595.105038 write 0x01006a thru 0x01bef6 (0xbe8d bytes) 000770 1786948595.106699 fallocate 0x02e2a4 thru 0x031648 (0x33a5 bytes) 000771 1786948595.112009 write 0x006df1 thru 0x0139ee (0xcbfe bytes) 000772 1786948595.113593 mapread 0x002d0b thru 0x00c568 (0x985e bytes) 000773 1786948595.115468 read 0x00815c thru 0x008a4f (0x8f4 bytes) 000774 1786948595.116497 write 0x021672 thru 0x022dab (0x173a bytes) 000775 1786948595.118476 mapwrite 0x00fdb5 thru 0x01ed24 (0xef70 bytes) 000776 1786948595.139106 read 0x024669 thru 0x02cc43 (0x85db bytes) 000777 1786948595.140084 write 0x02c189 thru 0x02f541 (0x33b9 bytes) 000780 1786948595.147564 trunc from 0x024867 to 0x026932 (0x20cc bytes) 000781 1786948595.152611 mapread 0x00e224 thru 0x013613 (0x53f0 bytes) 000782 1786948595.158132 MIRROR_RESYNC 2 mirrors 000783 1786948596.062223 trunc from 0x010b70 to 0x019356 (0x87e7 bytes) 000784 1786948596.115620 punch from 0x0017ba to 0x009b79 (0x83c0 bytes) 000785 1786948596.124478 trunc from 0x02f5e3 to 0x031648 (0x2066 bytes) 000786 1786948596.135086 read 0x01672d thru 0x02475f (0xe033 bytes) 000787 1786948596.151402 mapwrite 0x00216e thru 0x010619 (0xe4ac bytes) 000788 1786948596.206508 punch from 0x0290b6 to 0x029b8d (0xad8 bytes) 000789 1786948596.211366 read 0x006697 thru 0x013ae1 (0xd44b bytes) 000790 1786948596.227988 punch from 0x02ce54 to 0x031648 (0x47f5 bytes) 000791 1786948596.233114 fallocate 0x007df2 thru 0x014a6d (0xcc7c bytes) 000792 1786948596.238352 mapwrite 0x00710e thru 0x00a0c2 (0x2fb5 bytes) 000793 1786948596.261682 trunc from 0x01bc21 to 0x0264a6 (0xa886 bytes) 000795 1786948596.274065 mapread 0x011a23 thru 0x01cdac (0xb38a bytes) 000797 1786948596.298288 trunc from 0x031649 to 0x00b263 (0x263e5 bytes) 000798 1786948596.339157 trunc from 0x0093c9 to 0x00b263 (0x1e9b bytes) 000799 1786948596.345074 write 0x0277a9 thru 0x033b52 (0xc3aa bytes) 000800 1786948596.346787 write 0x028e44 thru 0x03048f (0x764c bytes) 000801 1786948596.350590 mapread 0x005d35 thru 0x011ace (0xbd9a bytes) 000803 1786948596.424968 fallocate 0x016965 thru 0x024dff (0xe49b bytes) 000804 1786948596.432346 mapwrite 0x01e419 thru 0x024c33 (0x681b bytes) 000805 1786948596.511768 trunc from 0x020f08 to 0x025168 (0x4261 bytes) 000806 1786948596.521278 fallocate 0x03f4cf thru 0x03ffff (0xb31 bytes) 000807 1786948596.530571 read 0x03e91e thru 0x03ffff (0x16e2 bytes) 000808 1786948596.543687 mapwrite 0x038b2c thru 0x03ffff (0x74d4 bytes) 000809 1786948596.577524 trunc from 0x040000 to 0x01145c (0x2eba3 bytes) 000811 1786948596.603134 mapwrite 0x01a416 thru 0x01cdce (0x29b9 bytes) 000812 1786948596.658781 read 0x002252 thru 0x0034f0 (0x129f bytes) 000813 1786948596.659995 trunc from 0x01ba80 to 0x01cdce (0x134f bytes) 000814 1786948596.667964 read 0x01ac58 thru 0x01cdce (0x2177 bytes) 000815 1786948596.682355 punch from 0x01cc73 to 0x01cdce (0x15c bytes) 000816 1786948596.690363 trunc from 0x01cdcf to 0x019889 (0x3545 bytes) 000817 1786948596.711915 trunc from 0x011406 to 0x019889 (0x8484 bytes) 000818 1786948596.721363 trunc from 0x0167e1 to 0x01927c (0x2a9c bytes) 000819 1786948596.729840 fallocate 0x03004b thru 0x0309d8 (0x98e bytes) 000820 1786948596.759045 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1988a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1988a): Mirror 2: 0x58e97490 (stale) CRC-32 checksum value for chunk [0, 0x1988a): Mirror 1: 0x983865be Mirror 3: 0x983865be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1988a: skipped 000821 1786948597.766173 trunc from 0x01988a to 0x0003ef (0x1949a bytes) 000822 1786948597.817921 read 0x0000c7 thru 0x0003ef (0x329 bytes) 000823 1786948597.830113 mapwrite 0x02cb9a thru 0x03bd08 (0xf16f bytes) 000824 1786948597.909350 mapwrite 0x01fbb1 thru 0x02d1eb (0xd63b bytes) 000825 1786948597.946414 mapwrite 0x0363b7 thru 0x03a800 (0x444a bytes) 000826 1786948597.962432 trunc from 0x023b6a to 0x02e082 (0xa519 bytes) 000827 1786948597.975880 trunc from 0x03bd09 to 0x038bc0 (0x3148 bytes) 000828 1786948598.005928 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38bc1): Mirror 3: 0xc6feadf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bc1): Mirror 3: 0xc6feadf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bc1: skipped 000829 1786948598.387727 mapread 0x02c733 thru 0x02d891 (0x115f bytes) 000830 1786948598.403545 write 0x022298 thru 0x022bb4 (0x91d bytes) 000831 1786948598.430300 MIRROR_RESYNC 2 mirrors 000832 1786948599.204118 punch from 0x01afc4 to 0x027f43 (0xcf80 bytes) 000833 1786948599.244985 mapread 0x009886 thru 0x0174ab (0xdc26 bytes) 000834 1786948599.273163 write 0x00f414 thru 0x012315 (0x2f02 bytes) 000835 1786948599.275190 fallocate 0x01b333 thru 0x0230cb (0x7d99 bytes) 000836 1786948599.282345 trunc from 0x02bcc9 to 0x02f322 (0x365a bytes) 000837 1786948599.287539 write 0x00977d thru 0x014421 (0xaca5 bytes) 000838 1786948599.289511 mapwrite 0x02a54e thru 0x03666c (0xc11f bytes) 000839 1786948599.332439 trunc from 0x038bc1 to 0x00f181 (0x29a3f bytes) 000841 1786948599.355351 trunc from 0x005b1e to 0x009ce5 (0x41c8 bytes) 000842 1786948599.360409 mapwrite 0x0288f6 thru 0x03181b (0x8f26 bytes) 000843 1786948599.425108 write 0x00f7d0 thru 0x012a65 (0x3296 bytes) 000844 1786948599.434827 fallocate 0x030964 thru 0x03f821 (0xeebe bytes) 000845 1786948599.449325 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3181c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3181c): Mirror 3: 0xdcc7c948 (stale) CRC-32 checksum value for chunk [0, 0x3181c): Mirror 1: 0x9f54a7b5 Mirror 4: 0x9f54a7b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3181c: skipped 000847 1786948600.442922 trunc from 0x03181c to 0x01559f (0x1c27c bytes) 000848 1786948600.512139 trunc from 0x00b1f2 to 0x01559f (0xa3ae bytes) 000849 1786948600.533053 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x155a0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x155a0): Mirror 3: 0xf671cc0e (stale) Mirror 4: 0xa1450e8b (stale) CRC-32 checksum value for chunk [0, 0x155a0): Mirror 1: 0xd98dfaa0 Mirror 5: 0xd98dfaa0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x155a0: skipped 000850 1786948601.483083 mapread 0x00d05a thru 0x014855 (0x77fc bytes) 000851 1786948601.539388 fallocate 0x03424f thru 0x03892b (0x46dd bytes) 000852 1786948601.581391 write 0x023456 thru 0x02a926 (0x74d1 bytes) 000853 1786948601.588090 mapwrite 0x027264 thru 0x031a9b (0xa838 bytes) 000854 1786948601.638081 fallocate 0x03083f thru 0x0327e4 (0x1fa6 bytes) 000855 1786948601.644946 write 0x01795d thru 0x01b438 (0x3adc bytes) 000856 1786948601.656331 write 0x00814b thru 0x00b843 (0x36f9 bytes) 000857 1786948601.666968 fallocate 0x005122 thru 0x013d44 (0xec23 bytes) 000858 1786948601.671862 fallocate 0x0121ff thru 0x01e511 (0xc313 bytes) 000859 1786948601.677288 read 0x02bd65 thru 0x02ced2 (0x116e bytes) 000860 1786948601.678572 punch from 0x009118 to 0x00a66c (0x1555 bytes) 000861 1786948601.695894 trunc from 0x0212b2 to 0x02d9ab (0xc6fa bytes) 000862 1786948601.702354 trunc from 0x0149a3 to 0x020e41 (0xc49f bytes) 000863 1786948601.709172 trunc from 0x02344a to 0x0239aa (0x561 bytes) 000864 1786948601.715938 mapwrite 0x0023f5 thru 0x010b45 (0xe751 bytes) 000865 1786948601.768737 mapwrite 0x012bc4 thru 0x013679 (0xab6 bytes) 000866 1786948601.803443 fallocate 0x0313ac thru 0x03ffff (0xec54 bytes) 000867 1786948601.818596 mapread 0x02920a thru 0x0327e4 (0x95db bytes) 000868 1786948601.842870 trunc from 0x00df7a to 0x0115cf (0x3656 bytes) 000869 1786948601.850867 read 0x018b5a thru 0x01966e (0xb15 bytes) 000871 1786948601.852345 write 0x00acd9 thru 0x01833f (0xd667 bytes) 000873 1786948601.857077 write 0x02e916 thru 0x03999f (0xb08a bytes) 000874 1786948601.859332 read 0x005620 thru 0x006600 (0xfe1 bytes) 000875 1786948601.863459 punch from 0x01e863 to 0x02e7ac (0xff4a bytes) 000876 1786948601.887207 read 0x0330de thru 0x03999f (0x68c2 bytes) 000877 1786948601.888255 mapread 0x002c11 thru 0x0105cf (0xd9bf bytes) 000878 1786948601.895390 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x399a0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399a0): Mirror 3: 0xd45a1850 (stale) Mirror 4: 0x1d6b0c85 (stale) Mirror 5: 0x3f612de1 (stale) CRC-32 checksum value for chunk [0, 0x399a0): Mirror 1: 0x13e542d6 Mirror 6: 0x13e542d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399a0: skipped 000879 1786948603.041440 write 0x02c53b thru 0x03bd5c (0xf822 bytes) 000880 1786948603.096103 trunc from 0x03bd5d to 0x020ec3 (0x1ae99 bytes) 000881 1786948603.130114 trunc from 0x00b4c9 to 0x00bf02 (0xa3a bytes) 000882 1786948603.136016 mapwrite 0x01a393 thru 0x026461 (0xc0cf bytes) 000883 1786948603.215953 write 0x00baaa thru 0x015ac4 (0xa01b bytes) 000884 1786948603.227525 mapread 0x024b9c thru 0x026461 (0x18c6 bytes) 000885 1786948603.234296 read 0x025560 thru 0x026461 (0xf02 bytes) 000886 1786948603.236445 trunc from 0x012938 to 0x01ea66 (0xc12f bytes) 000888 1786948603.265974 read 0x010081 thru 0x0152bb (0x523b bytes) 000889 1786948603.290432 mapread 0x00a241 thru 0x010fab (0x6d6b bytes) 000891 1786948603.292367 punch from 0x005c6a to 0x00c4a8 (0x683f bytes) 000892 1786948603.301433 mapwrite 0x01b508 thru 0x0225b9 (0x70b2 bytes) 000893 1786948603.328021 trunc from 0x005db9 to 0x00c584 (0x67cc bytes) 000894 1786948603.333370 trunc from 0x026462 to 0x01cfbf (0x94a2 bytes) 000896 1786948603.370353 fallocate 0x03f8a1 thru 0x03ffff (0x75f bytes) 000897 1786948603.387507 trunc from 0x01142e to 0x01cfbf (0xbb92 bytes) 000898 1786948603.401651 trunc from 0x014e5b to 0x01cfbf (0x8165 bytes) 000899 1786948603.406478 write 0x01eddd thru 0x02550b (0x672f bytes) 000900 1786948603.410409 mapread 0x015bb3 thru 0x0221a8 (0xc5f6 bytes) 000901 1786948603.490815 fallocate 0x03b1f4 thru 0x03fcea (0x4af7 bytes) 000902 1786948603.497511 punch from 0x0246c1 to 0x02550b (0xe4b bytes) 000903 1786948603.526258 punch from 0x0138cc to 0x0213e7 (0xdb1c bytes) 000905 1786948603.537251 mapwrite 0x02a64a thru 0x030250 (0x5c07 bytes) 000906 1786948603.609182 mapwrite 0x03b2a6 thru 0x03ffff (0x4d5a bytes) 000907 1786948603.694369 mapread 0x0065ff thru 0x011bd8 (0xb5da bytes) 000909 1786948603.767235 punch from 0x00b0b2 to 0x01adf1 (0xfd40 bytes) 000910 1786948603.772621 mapread 0x006e6e thru 0x0150c6 (0xe259 bytes) 000911 1786948603.838281 fallocate 0x02bacd thru 0x02ddd5 (0x2309 bytes) 000912 1786948603.842825 fallocate 0x03e2fb thru 0x03f0c2 (0xdc8 bytes) 000913 1786948603.847318 trunc from 0x040000 to 0x013b3b (0x2c4c4 bytes) 000914 1786948603.863861 punch from 0x00d1e4 to 0x013b3b (0x6958 bytes) 000915 1786948603.871816 mapread 0x007f12 thru 0x00ae5e (0x2f4d bytes) 000916 1786948603.873588 mapwrite 0x01bf2b thru 0x01ecff (0x2dd5 bytes) 000917 1786948603.938320 mapread 0x01949a thru 0x01b12b (0x1c92 bytes) 000918 1786948603.958037 read 0x00a914 thru 0x00c0f1 (0x17de bytes) 000920 1786948603.963490 write 0x035f18 thru 0x039e97 (0x3f80 bytes) 000921 1786948603.967069 trunc from 0x002857 to 0x005d65 (0x350f bytes) 000922 1786948603.980085 mapread 0x0122aa thru 0x012b2c (0x883 bytes) 000923 1786948603.999222 punch from 0x01d883 to 0x021ff0 (0x476e bytes) 000924 1786948604.005808 trunc from 0x039e98 to 0x02613c (0x13d5b bytes) 000925 1786948604.027094 read 0x011907 thru 0x017f2f (0x6629 bytes) 000926 1786948604.028514 trunc from 0x014f13 to 0x01a437 (0x5525 bytes) 000927 1786948604.035026 fallocate 0x030d67 thru 0x032dbf (0x2059 bytes) 000928 1786948604.040336 mapread 0x002cfe thru 0x0116e7 (0xe9ea bytes) 000929 1786948604.046245 mapread 0x000a7a thru 0x00cd7c (0xc303 bytes) 000930 1786948604.047580 trunc from 0x00fb86 to 0x012f70 (0x33eb bytes) 000931 1786948604.060120 MIRROR_RESYNC 5 mirrors 000932 1786948605.891699 punch from 0x016a2e to 0x022fd4 (0xc5a7 bytes) 000933 1786948605.944486 trunc from 0x0251fd to 0x0295ac (0x43b0 bytes) 000934 1786948605.953883 mapwrite 0x02f433 thru 0x033538 (0x4106 bytes) 000935 1786948606.054985 trunc from 0x033539 to 0x00d050 (0x264e8 bytes) 000936 1786948606.075821 mapread 0x000a93 thru 0x00d050 (0xc5be bytes) 000937 1786948606.091288 read 0x00c515 thru 0x00d050 (0xb3c bytes) 000938 1786948606.092460 read 0x00290a thru 0x0049a9 (0x20a0 bytes) 000939 1786948606.093752 fallocate 0x03f084 thru 0x03ffff (0xf7c bytes) 000940 1786948606.103470 punch from 0x003f2f to 0x00cb9c (0x8c6e bytes) 000941 1786948606.117155 fallocate 0x0143ca thru 0x01980e (0x5445 bytes) 000942 1786948606.127403 trunc from 0x01980f to 0x00c40b (0xd403 bytes) 000944 1786948606.148960 punch from 0x005c31 to 0x0079fa (0x1dca bytes) 000945 1786948606.156167 mapread 0x002dcf thru 0x0072fb (0x452d bytes) 000946 1786948606.169058 mapwrite 0x003d09 thru 0x006026 (0x231e bytes) 000947 1786948606.193188 trunc from 0x00c40c to 0x03b799 (0x2f38e bytes) 000948 1786948606.207377 mapwrite 0x03f9e6 thru 0x03ffff (0x61a bytes) 000949 1786948606.244544 mapwrite 0x01345f thru 0x016736 (0x32d8 bytes) 000950 1786948606.260220 trunc from 0x01ce3a to 0x020d3b (0x3f02 bytes) 000951 1786948606.266477 write 0x02ad8c thru 0x02bd07 (0xf7c bytes) 000952 1786948606.269615 mapread 0x028e52 thru 0x033d46 (0xaef5 bytes) 000953 1786948606.272525 write 0x0175ae thru 0x023794 (0xc1e7 bytes) 000954 1786948606.275191 punch from 0x00c70b to 0x00fed6 (0x37cc bytes) 000955 1786948606.281612 read 0x03489a thru 0x03ffff (0xb766 bytes) 000956 1786948606.284998 trunc from 0x040000 to 0x011df8 (0x2e207 bytes) 000957 1786948606.308886 write 0x03a7b2 thru 0x03ffff (0x584e bytes) 000958 1786948606.311559 punch from 0x03cceb to 0x03ce3d (0x153 bytes) 000959 1786948606.329113 trunc from 0x040000 to 0x033d6c (0xc293 bytes) 000960 1786948606.349095 trunc from 0x00e570 to 0x00f3cf (0xe60 bytes) 000961 1786948606.355550 trunc from 0x0238c4 to 0x02b04f (0x778c bytes) 000962 1786948606.367266 trunc from 0x033d6d to 0x03c12a (0x83be bytes) 000963 1786948606.387455 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c12b): Mirror 3: 0x1e57ae05 (stale) Mirror 4: 0x1e57ae05 (stale) Mirror 6: 0x1e57ae05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c12b): Mirror 3: 0x1e57ae05 (stale) Mirror 4: 0x1e57ae05 (stale) Mirror 6: 0x1e57ae05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c12b: skipped 000965 1786948606.741273 trunc from 0x03c12b to 0x001655 (0x3aad5 bytes) 000966 1786948606.777786 write 0x008b96 thru 0x00c37f (0x37ea bytes) 000967 1786948606.780359 fallocate 0x037b0e thru 0x038943 (0xe36 bytes) 000968 1786948606.787242 mapread 0x0369a2 thru 0x038943 (0x1fa2 bytes) 000969 1786948606.806935 mapwrite 0x01bc83 thru 0x026a4c (0xadca bytes) 000970 1786948606.860409 read 0x030e60 thru 0x035e7d (0x501e bytes) 000971 1786948606.873348 write 0x024b9d thru 0x031ec7 (0xd32b bytes) 000972 1786948606.877537 mapwrite 0x01e9c4 thru 0x0257c1 (0x6dfe bytes) 000974 1786948606.926404 MIRROR_RESYNC 4 mirrors 000975 1786948608.218096 punch from 0x0083b6 to 0x00e74b (0x6396 bytes) 000976 1786948608.246819 read 0x010463 thru 0x014558 (0x40f6 bytes) 000977 1786948608.261165 punch from 0x00c7ba to 0x011a24 (0x526b bytes) 000978 1786948608.270933 read 0x00d191 thru 0x012f43 (0x5db3 bytes) 000979 1786948608.286313 trunc from 0x038944 to 0x0333c4 (0x557f bytes) 000980 1786948608.334379 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x333c5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333c5): Mirror 3: 0xbaae2280 (stale) Mirror 4: 0xbaae2280 (stale) Mirror 6: 0xbaae2280 (stale) CRC-32 checksum value for chunk [0, 0x333c5): Mirror 1: 0xeca7f7f1 Mirror 7: 0xeca7f7f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333c5: skipped 000981 1786948609.374425 mapwrite 0x03054d thru 0x037fb3 (0x7a67 bytes) 000982 1786948609.518381 trunc from 0x009374 to 0x0097cf (0x45c bytes) 000983 1786948609.530535 fallocate 0x0384c3 thru 0x03ffff (0x7b3d bytes) 000984 1786948609.538551 fallocate 0x03f32d thru 0x03ffff (0xcd3 bytes) 000985 1786948609.560972 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fb4): Mirror 3: 0xc46259f9 (stale) Mirror 6: 0xc46259f9 (stale) Mirror 7: 0x3b16feeb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb4): Mirror 3: 0xc46259f9 (stale) Mirror 6: 0xc46259f9 (stale) Mirror 7: 0x3b16feeb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb4: skipped 000986 1786948609.971253 read 0x03709a thru 0x037fb3 (0xf1a bytes) 000987 1786948609.978329 punch from 0x01e03d to 0x02d8d8 (0xf89c bytes) 000988 1786948609.984130 punch from 0x028e22 to 0x035b50 (0xcd2f bytes) 000989 1786948609.989294 read 0x033075 thru 0x036f66 (0x3ef2 bytes) 000991 1786948610.005837 write 0x000b56 thru 0x00be2e (0xb2d9 bytes) 000992 1786948610.012440 write 0x01f2f4 thru 0x02d425 (0xe132 bytes) 000993 1786948610.016386 mapread 0x02ebc6 thru 0x037fb3 (0x93ee bytes) 000994 1786948610.019047 write 0x037c53 thru 0x03ffff (0x83ad bytes) 000995 1786948610.024877 MIRROR_RESYNC 4 mirrors 000996 1786948611.311344 MIRROR_EXTEND to 5 mirrors 000997 1786948612.258672 trunc from 0x040000 to 0x01f1ef (0x20e10 bytes) 000998 1786948612.353292 mapwrite 0x009cc0 thru 0x019b4f (0xfe90 bytes) 000999 1786948612.452466 trunc from 0x0167d1 to 0x01c8ad (0x60dd bytes) 001000 1786948612.472233 trunc from 0x006d3d to 0x00a2c5 (0x3589 bytes) All operations completed A-OK! PASS 70a (91s) == sanity-flr test 71: check mirror extend parent fid ==== 02:36:57 (1786948617) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 02:37:07 (1786948627) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180565 s, 11.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x322:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144326 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x322:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x340:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x322:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x340:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x33f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x323:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 02:37:15 (1786948635) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6323680 .. Extending file size to 7660736 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7237792 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7654368 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8535904 .. Extending file size to 7696064 .. Extending file size to 7755968 .. Extending file size to 8766912 .. Extending file size to 8778272 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9160992 .. Extending file size to 9071232 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61408 61410 61411 61413 61415 done 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) 192.168.203.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:38:35 (1786948715) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8790176 .. Extending file size to 8408832 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8896096 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9070944 .. Extending file size to 9087968 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62875 62876 62877 62880 62881 done 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) 192.168.203.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:39:57 (1786948797) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2568 1268736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17780 3589240 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25528 3581492 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43308 7170732 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2568 1268736 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17780 3589240 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25528 3581492 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43308 7170732 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159247 s, 13.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170675 s, 12.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x86eadc3c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:40:12 (1786948812) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10172 s, 20.6 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 02:40:19 (1786948819) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x228:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x338:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:40:25 (1786948825) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0623788 s, 33.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x358:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x339:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:40:32 (1786948832) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0840525 s, 25.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:40:39 (1786948839) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0875052 s, 24.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33c:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:41:11 (1786948871) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754719 s, 13.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x31:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:42:02 (1786948922) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106723 s, 19.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:42:49 (1786948969) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg309-server: Pool lustre.test_204f created Adding targets to pool oleg309-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg309-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11963 s, 17.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x33e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg309-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg309-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg309-server: Pool lustre.test_204f destroyed PASS 204f (66s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:43:56 (1786949036) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:44:04 (1786949044) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:44:13 (1786949053) oleg309-server: Pool lustre.test_206 created Adding targets to pool oleg309-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg309-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg309-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg309-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg309-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:44:41 (1786949081) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788375 s, 13.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871851 s, 12.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9c868388 Mirror 3: 0x9c868388 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:44:53 (1786949093) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:44:56 (1786949096) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:45:00 (1786949100) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0834692 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x22f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x22f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x22f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd40000 Read this (4 bytes): BBBB 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:45:37 (1786949137) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0155535 s, 527 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:45:44 (1786949144) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434223 s, 24.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x249 lmm_fid: [0x200000405:0x249:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x249 lmm_fid: [0x200000405:0x249:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x348:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611721 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-17 02:45:46.000000000 -0400 Modify: 2026-08-17 02:45:46.000000000 -0400 Change: 2026-08-17 02:45:47.807000000 -0400 Birth: 2026-08-17 02:45:46.883000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:45:54 (1786949154) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682594 s, 15.4 MB/s oleg309-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg309-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg309-server: O/280000401/d7/367: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:46:15 (1786949175) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.233859 s, 175 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0507314 s, 80.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 17 02:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x24e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 17 02:46 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:46:23 (1786949183) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:46:31 (1786949191) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1837 sec =========== 02:46:40 (1786949200) === sanity-flr: start cleanup 02:46:42 (1786949202) === === sanity-flr: finish cleanup 02:46:52 (1786949212) ===