-----============= acceptance-small: sanity-flr ============----- Thu Sep 10 01:01:39 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg222-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:02:02 (1789016522) === oleg222-client.virtnet: executing check_config_client /mnt/lustre oleg222-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg222-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9521c7ff9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9521c7ff9000.idle_timeout=debug disable quota as required oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:02:53 (1789016573) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8239] == sanity-flr test 0a: lfs mirror create with -N option == 01:02:56 (1789016576) PASS 0a (26s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:03:22 (1789016602) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:03:26 (1789016606) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:03:30 (1789016610) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:04:01 (1789016641) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:04:06 (1789016646) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:04:10 (1789016650) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9521c7ff9000.stats=clear osc.lustre-OST0001-osc-ffff9521c7ff9000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:04:20 (1789016660) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:04:34 (1789016674) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:04:45 (1789016685) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:04:49 (1789016689) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (12s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:05:01 (1789016701) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 01:05:13 (1789016713) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:05:22 (1789016722) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:05:31 (1789016731) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:05:41 (1789016741) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:05:52 (1789016752) PASS 7 (10s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:06:02 (1789016762) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.13721 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160491 s, 6.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:06:12 (1789016772) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0997243 s, 42.1 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:06:19 (1789016779) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:06:32 (1789016792) Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg222-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (140s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:08:53 (1789016933) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.811937 s, 12.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (57s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:09:50 (1789016990) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:11:11 (1789017071) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.4924 s, 15.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 10 01:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.332s user 0m0.000s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9521c7ff9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.281s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.143s user 0m0.000s sys 0m0.187s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9521c7ff9000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:12:48 (1789017168) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:12:52 (1789017172) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:12:56 (1789017176) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:12:59 (1789017179) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 01:13:07 (1789017187) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.798401 s, 15.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:13:20 (1789017200) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223979 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344138 s, 30.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:13:28 (1789017208) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:13:37 (1789017217) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114537 s, 894 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113471 s, 902 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000837625 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000948573 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 01:13:48 (1789017228) create test files with size 15736320 .. 1fdf3ee74ce8a26c83d5dbcfe3cd281c - caa4e67a83cd3488fa6fc8b364444530 - 933668ec1c73b0c1c26a9cc97874007c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 01:14:16 (1789017256) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:14:31 (1789017271) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:14:38 (1789017278) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119919 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844146 s, 12.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 01:14:49 (1789017289) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0851437 s, 24.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.339616 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216877 s, 9.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 01:15:03 (1789017303) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:15:07 (1789017307) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:15:10 (1789017310) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396685 s, 26.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 01:15:18 (1789017318) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.38569 s, 8.2 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:15:28 (1789017328) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:15:38 (1789017338) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.223168 s, 47.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:15:49 (1789017349) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9521d11ce000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.375842 s, 27.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9521c83c5000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:16:08 (1789017368) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:16:42 (1789017402) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:17:14 (1789017434) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:17:17 (1789017437) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.17194 s, 18.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.175125 s, 18.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe651ba0e (nosync,stale) Mirror 2: 0xe651ba0e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 49caa9d097f63192282e131fa04c3978 - mirror 1 checksum: c425d7e6d8c7963f2fd7d3f5020c2559 - mirror 2 checksum: 49caa9d097f63192282e131fa04c3978 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T01:17:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T01:17:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 49caa9d097f63192282e131fa04c3978 - mirror 1 checksum: c425d7e6d8c7963f2fd7d3f5020c2559 - mirror 2 checksum: c425d7e6d8c7963f2fd7d3f5020c2559 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T01:17:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T01:17:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:17:29 (1789017449) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035554 s, 29.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:17:37 (1789017457) Failing mds1 on oleg222-server Stopping /mnt/lustre-mds1 (opts:) on oleg222-server 01:17:52 (1789017472) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 01:18:29 (1789017509) targets are mounted 01:18:29 (1789017509) facet_failover done oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532474 s, 19.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } PASS 50A (71s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:18:48 (1789017528) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0916142 s, 224 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0211521 s, 968 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:18:58 (1789017538) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.082989 s, 247 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0164949 s, 1.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:19:10 (1789017550) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0067567 s, 2.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:19:19 (1789017559) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0857567 s, 239 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (12s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:19:31 (1789017571) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0150179 s, 273 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0527758 s, 77.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:19:39 (1789017579) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:20:05 (1789017605) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:20:36 (1789017636) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:21:01 (1789017661) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:21:27 (1789017687) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 01:21:37 (1789017697) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57613 mirror split pid 57615 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57613 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57615 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 01:22:46 (1789017766) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7768 skipping zero size read 000002 1789017769.464094 write 0x01d422 thru 0x02bdf0 (0xe9cf bytes) 000003 1789017769.472493 mapread 0x00a6fa thru 0x01819d (0xdaa4 bytes) truncating to largest ever: 0xe96e 000005 1789017769.620081 trunc from 0x02bdf1 to 0x00e96d (0x1d483 bytes) 000006 1789017769.665390 read 0x000f99 thru 0x00b69d (0xa705 bytes) 000007 1789017769.686405 punch from 0x003f0e to 0x00e96d (0xaa60 bytes) 000008 1789017769.697443 trunc from 0x001fe2 to 0x00e96d (0xc98c bytes) 000010 1789017769.705499 read 0x00d579 thru 0x00dab1 (0x539 bytes) 000011 1789017769.722869 write 0x00e76d thru 0x018def (0xa683 bytes) 000012 1789017769.724306 mapwrite 0x00bd46 thru 0x019136 (0xd3f1 bytes) 000013 1789017769.831219 mapwrite 0x01cfc9 thru 0x01ed64 (0x1d9c bytes) 000014 1789017769.918811 punch from 0x0153a4 to 0x018997 (0x35f4 bytes) 000015 1789017769.935608 punch from 0x00e562 to 0x01e0a8 (0xfb47 bytes) 000016 1789017769.949519 write 0x00fa6e thru 0x018388 (0x891b bytes) 000017 1789017770.010199 MIRROR_EXTEND to 2 mirrors 000018 1789017771.381784 mapwrite 0x001ac5 thru 0x00d688 (0xbbc4 bytes) 000019 1789017771.539863 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ed65): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed65): Mirror 2: 0xb7f89192 (stale) CRC-32 checksum value for chunk [0, 0x1ed65): Mirror 1: 0x4308f2c Mirror 3: 0x4308f2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed65: skipped 000020 1789017773.150400 mapwrite 0x0156e5 thru 0x01d828 (0x8144 bytes) fallocating to largest ever: 0x12485 000021 1789017773.283776 fallocate 0x00b42f thru 0x012484 (0x7056 bytes) 000022 1789017773.291236 punch from 0x01e89f to 0x01ed64 (0x4c6 bytes) 000023 1789017773.313449 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x1d579 000024 1789017774.598171 trunc from 0x01ed65 to 0x01d578 (0x17ec bytes) 000025 1789017774.702245 read 0x00b5a0 thru 0x00da6a (0x24cb bytes) 000026 1789017774.711802 punch from 0x0152a7 to 0x01bb9a (0x68f4 bytes) truncating to largest ever: 0x3b823 000027 1789017774.729265 trunc from 0x01d579 to 0x03b822 (0x1e2aa bytes) 000028 1789017774.785880 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b823): Mirror 3: 0x784cc3b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b823): Mirror 3: 0x784cc3b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b823: skipped 000029 1789017775.634511 write 0x000682 thru 0x00ea8c (0xe40b bytes) 000030 1789017775.651644 read 0x03971d thru 0x03b822 (0x2106 bytes) 000031 1789017775.658536 trunc from 0x0288e7 to 0x02e81c (0x5f36 bytes) 000032 1789017775.664547 trunc from 0x0227eb to 0x02beb3 (0x96c9 bytes) 000033 1789017775.673891 trunc from 0x03b823 to 0x023ebe (0x17964 bytes) 000034 1789017775.705817 read 0x01fab7 thru 0x023ebe (0x4408 bytes) 000035 1789017775.727713 trunc from 0x023ebf to 0x0346cd (0x1080f bytes) 000036 1789017775.785161 punch from 0x020e23 to 0x02ed96 (0xdf74 bytes) 000037 1789017775.791516 read 0x0200dd thru 0x025782 (0x56a6 bytes) 000038 1789017775.813623 punch from 0x02f73d to 0x0346cd (0x4f91 bytes) 000039 1789017775.818584 read 0x0173ec thru 0x024438 (0xd04d bytes) 000040 1789017775.819673 trunc from 0x0310db to 0x0346cd (0x35f3 bytes) 000041 1789017775.851513 read 0x01a072 thru 0x01a2cf (0x25e bytes) 000042 1789017775.861232 trunc from 0x0346ce to 0x004780 (0x2ff4d bytes) 000043 1789017775.930618 trunc from 0x0022ea to 0x004780 (0x2497 bytes) 000044 1789017775.994842 MIRROR_RESYNC 2 mirrors 000045 1789017777.139606 punch from 0x004612 to 0x004780 (0x16f bytes) 000046 1789017777.174007 trunc from 0x001586 to 0x004780 (0x31fb bytes) 000049 1789017777.184295 trunc from 0x004781 to 0x025670 (0x20ef0 bytes) fallocating to largest ever: 0x3ff28 000050 1789017777.216600 fallocate 0x031825 thru 0x03ff27 (0xe703 bytes) 000051 1789017777.225426 mapread 0x01e3ba thru 0x028f72 (0xabb9 bytes) 000052 1789017777.258498 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ff28): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff28): Mirror 3: 0xcef579cc (stale) CRC-32 checksum value for chunk [0, 0x3ff28): Mirror 1: 0x2bd3e186 Mirror 4: 0x2bd3e186 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff28: skipped 000053 1789017778.599375 trunc from 0x0087f1 to 0x00caa6 (0x42b6 bytes) 000054 1789017778.747149 mapread 0x0355ac thru 0x03c886 (0x72db bytes) 000055 1789017778.799821 trunc from 0x03ff28 to 0x01f407 (0x20b20 bytes) 000056 1789017778.936284 read 0x01ca99 thru 0x01f407 (0x296f bytes) 000057 1789017778.945896 trunc from 0x018ffc to 0x01e12f (0x5134 bytes) 000058 1789017778.950940 read 0x0104ca thru 0x01f407 (0xef3e bytes) 000059 1789017778.981666 fallocate 0x01ce8d thru 0x0217d4 (0x4948 bytes) 000060 1789017779.002669 punch from 0x006669 to 0x00e051 (0x79e9 bytes) 000062 1789017779.013180 fallocate 0x01ba8c thru 0x02818d (0xc702 bytes) 000063 1789017779.029918 read 0x00eeae thru 0x01b1be (0xc311 bytes) 000064 1789017779.050089 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2818e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2818e): Mirror 3: 0xbe82f19d (stale) Mirror 4: 0x78abf125 (stale) CRC-32 checksum value for chunk [0, 0x2818e): Mirror 1: 0x78abf125 Mirror 5: 0x78abf125 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2818e: skipped 000065 1789017780.427446 read 0x001bb7 thru 0x0038aa (0x1cf4 bytes) 000066 1789017780.467296 read 0x000e4b thru 0x0024e9 (0x169f bytes) 000067 1789017780.526415 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2818e): Mirror 3: 0xbe82f19d (stale) Mirror 4: 0x78abf125 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2818e): Mirror 3: 0xbe82f19d (stale) Mirror 4: 0x78abf125 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2818e: skipped 000068 1789017781.199840 punch from 0x00dc0f to 0x016f2b (0x931d bytes) 000069 1789017781.238833 read 0x0039f9 thru 0x01155d (0xdb65 bytes) 000070 1789017781.259449 mapwrite 0x0398f7 thru 0x03ffff (0x6709 bytes) 000071 1789017781.349950 write 0x03cdd3 thru 0x03d191 (0x3bf bytes) 000072 1789017781.350911 read 0x01fc8e thru 0x02f474 (0xf7e7 bytes) 000073 1789017781.374745 read 0x00c994 thru 0x011087 (0x46f4 bytes) 000074 1789017781.375945 read 0x039aa4 thru 0x03bcdf (0x223c bytes) 000075 1789017781.376770 mapwrite 0x037eec thru 0x03ffff (0x8114 bytes) 000076 1789017781.438050 punch from 0x00744f to 0x0121a9 (0xad5b bytes) 000077 1789017781.449812 trunc from 0x003b1d to 0x00ad5b (0x723f bytes) 000078 1789017781.454864 write 0x03d807 thru 0x03ffff (0x27f9 bytes) 000079 1789017781.469567 MIRROR_RESYNC 3 mirrors 000080 1789017782.784909 MIRROR_RESYNC 3 mirrors 000081 1789017783.242702 punch from 0x012440 to 0x01d404 (0xafc5 bytes) 000082 1789017783.285930 trunc from 0x040000 to 0x015b09 (0x2a4f6 bytes) 000083 1789017783.315836 trunc from 0x01556f to 0x015b09 (0x59b bytes) 000085 1789017783.324978 trunc from 0x002149 to 0x007b55 (0x5a0d bytes) 000086 1789017783.329748 fallocate 0x02df41 thru 0x02f798 (0x1858 bytes) 000087 1789017783.336061 mapwrite 0x01a67b thru 0x029b70 (0xf4f6 bytes) 000088 1789017783.431079 mapwrite 0x02f347 thru 0x03ecaa (0xf964 bytes) 000090 1789017783.525082 trunc from 0x03ecab to 0x037e23 (0x6e87 bytes) 000091 1789017783.550201 write 0x00edb1 thru 0x01a2a5 (0xb4f5 bytes) 000092 1789017783.568527 mapread 0x034534 thru 0x037e23 (0x38f0 bytes) 000093 1789017783.574496 read 0x022a79 thru 0x02c880 (0x9e08 bytes) 000094 1789017783.575869 read 0x019f7e thru 0x029c49 (0xfccc bytes) 000095 1789017783.579318 punch from 0x01ee4f to 0x029388 (0xa53a bytes) 000096 1789017783.615866 fallocate 0x024893 thru 0x0252f3 (0xa61 bytes) 000097 1789017783.627969 write 0x0241c6 thru 0x025f37 (0x1d72 bytes) 000098 1789017783.652972 read 0x00aa51 thru 0x012cea (0x829a bytes) 000099 1789017783.654083 read 0x01e234 thru 0x026598 (0x8365 bytes) 000100 1789017783.657889 mapwrite 0x01c73b thru 0x023400 (0x6cc6 bytes) 000101 1789017783.708426 punch from 0x036d51 to 0x0375fd (0x8ad bytes) 000102 1789017783.719394 read 0x01d0b7 thru 0x022f63 (0x5ead bytes) 000103 1789017783.719997 punch from 0x0207c5 to 0x02a343 (0x9b7f bytes) 000104 1789017783.737410 read 0x01847e thru 0x018f01 (0xa84 bytes) 000105 1789017783.738706 punch from 0x021045 to 0x0298a6 (0x8862 bytes) 000106 1789017783.751602 trunc from 0x037e24 to 0x0239be (0x14465 bytes) 000107 1789017783.775537 trunc from 0x018ced to 0x02376e (0xaa82 bytes) 000108 1789017783.782914 mapwrite 0x030308 thru 0x03d7a7 (0xd4a0 bytes) 000109 1789017783.860988 punch from 0x013ab5 to 0x014bc1 (0x110d bytes) 000110 1789017783.869319 punch from 0x01a319 to 0x01d5fc (0x32e4 bytes) 000111 1789017783.880512 punch from 0x03b000 to 0x03d7a7 (0x27a8 bytes) 000113 1789017783.892571 mapread 0x01ceab thru 0x0256b5 (0x880b bytes) 000115 1789017783.906852 mapread 0x02c0ed thru 0x0399ee (0xd902 bytes) 000116 1789017783.907970 trunc from 0x010823 to 0x010a8d (0x26b bytes) 000117 1789017783.942247 MIRROR_RESYNC 3 mirrors 000118 1789017785.176932 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000120 1789017785.659484 write 0x0254e2 thru 0x02e26a (0x8d89 bytes) 000121 1789017785.762440 read 0x03d2a3 thru 0x03d7a7 (0x505 bytes) 000122 1789017785.775582 punch from 0x02f799 to 0x0370f1 (0x7959 bytes) 000123 1789017785.783254 write 0x01e295 thru 0x02c844 (0xe5b0 bytes) 000124 1789017785.797286 punch from 0x010299 to 0x01cdf7 (0xcb5f bytes) 000125 1789017785.811557 trunc from 0x0116d3 to 0x0175c7 (0x5ef5 bytes) 000126 1789017785.823485 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000127 1789017786.094657 mapread 0x03a48a thru 0x03d7a7 (0x331e bytes) 000128 1789017786.120890 trunc from 0x03d7a8 to 0x024288 (0x1951f bytes) 000129 1789017786.151940 fallocate 0x035085 thru 0x03f126 (0xa0a2 bytes) 000130 1789017786.157064 punch from 0x01d07d to 0x024288 (0x720c bytes) 000131 1789017786.166086 fallocate 0x023596 thru 0x024323 (0xd8e bytes) 000132 1789017786.177650 read 0x00d030 thru 0x0172d6 (0xa2a7 bytes) 000134 1789017786.201722 write 0x0306a5 thru 0x0345b1 (0x3f0d bytes) 000135 1789017786.203890 trunc from 0x0345b2 to 0x02e366 (0x624b bytes) 000136 1789017786.233857 punch from 0x01b682 to 0x028cfe (0xd67d bytes) 000137 1789017786.239103 fallocate 0x019779 thru 0x02903e (0xf8c6 bytes) 000138 1789017786.245111 read 0x02466f thru 0x02d96d (0x92ff bytes) 000139 1789017786.255334 write 0x00c865 thru 0x013acb (0x7267 bytes) 000140 1789017786.256538 write 0x02e127 thru 0x036982 (0x885c bytes) 000141 1789017786.257700 trunc from 0x01be5d to 0x028c61 (0xce05 bytes) 000142 1789017786.266656 punch from 0x0244d0 to 0x02ae2b (0x695c bytes) 000143 1789017786.273472 trunc from 0x036983 to 0x016638 (0x2034a bytes) 000145 1789017786.306974 mapwrite 0x02047e thru 0x02cc76 (0xc7f9 bytes) 000146 1789017786.501200 write 0x0038e3 thru 0x00bd47 (0x8465 bytes) 000148 1789017786.505590 punch from 0x004b98 to 0x005edf (0x1348 bytes) 000149 1789017786.527880 write 0x015008 thru 0x023720 (0xe719 bytes) 000150 1789017786.529425 mapread 0x00b5e6 thru 0x00cb90 (0x15ab bytes) 000151 1789017786.529966 trunc from 0x02cc77 to 0x034424 (0x77ae bytes) 000152 1789017786.551259 mapwrite 0x021004 thru 0x024ca7 (0x3ca4 bytes) fallocating to largest ever: 0x40000 000153 1789017786.581055 fallocate 0x035570 thru 0x03ffff (0xaa90 bytes) 000155 1789017786.597169 trunc from 0x034425 to 0x005a63 (0x2e9c1 bytes) 000156 1789017786.627052 mapread 0x002b35 thru 0x005a63 (0x2f2f bytes) 000158 1789017786.656903 mapwrite 0x00e76e thru 0x014f52 (0x67e5 bytes) 000159 1789017786.741022 read 0x004040 thru 0x0103ca (0xc38b bytes) 000161 1789017786.755738 mapwrite 0x008911 thru 0x013be0 (0xb2d0 bytes) 000163 1789017786.784073 fallocate 0x018ede thru 0x021b46 (0x8c69 bytes) 000164 1789017786.791598 mapwrite 0x000787 thru 0x00a2ef (0x9b69 bytes) 000166 1789017786.818783 fallocate 0x02926d thru 0x036899 (0xd62d bytes) 000167 1789017786.825222 fallocate 0x034b6e thru 0x03a95e (0x5df1 bytes) 000168 1789017786.833482 read 0x0197aa thru 0x020642 (0x6e99 bytes) 000169 1789017786.850215 mapread 0x0100eb thru 0x018940 (0x8856 bytes) 000170 1789017786.854889 write 0x0146ad thru 0x01c759 (0x80ad bytes) 000171 1789017786.858103 punch from 0x001c13 to 0x004ab3 (0x2ea1 bytes) 000172 1789017786.868104 mapwrite 0x01c6dc thru 0x02a79d (0xe0c2 bytes) 000173 1789017786.977524 mapread 0x01f975 thru 0x024d37 (0x53c3 bytes) 000176 1789017786.993188 read 0x023eb0 thru 0x02a79d (0x68ee bytes) 000177 1789017786.993911 punch from 0x001190 to 0x001343 (0x1b4 bytes) 000178 1789017787.007102 mapwrite 0x0264f9 thru 0x031a6f (0xb577 bytes) 000179 1789017787.135092 fallocate 0x024dc5 thru 0x03441c (0xf658 bytes) 000180 1789017787.169302 write 0x03f71f thru 0x03ffff (0x8e1 bytes) 000181 1789017787.171941 trunc from 0x01bce7 to 0x02a5cd (0xe8e7 bytes) 000182 1789017787.181314 fallocate 0x022597 thru 0x024d8b (0x27f5 bytes) 000183 1789017787.193665 read 0x032602 thru 0x033efc (0x18fb bytes) 000184 1789017787.223536 mapwrite 0x02eb6d thru 0x034ca3 (0x6137 bytes) 000189 1789017787.302973 trunc from 0x040000 to 0x037ed2 (0x812d bytes) 000190 1789017787.344869 fallocate 0x014f0e thru 0x01c07b (0x716e bytes) 000191 1789017787.357562 trunc from 0x01fcb2 to 0x024834 (0x4b83 bytes) 000192 1789017787.367269 mapread 0x036ba6 thru 0x037ed2 (0x132d bytes) 000193 1789017787.368681 mapread 0x022cc4 thru 0x028493 (0x57d0 bytes) 000194 1789017787.445570 punch from 0x01d9e6 to 0x01f4bd (0x1ad8 bytes) 000195 1789017787.469261 read 0x0041f7 thru 0x009191 (0x4f9b bytes) 000196 1789017787.471716 fallocate 0x038f37 thru 0x03ffff (0x70c9 bytes) 000197 1789017787.491089 trunc from 0x00a7d7 to 0x00c011 (0x183b bytes) 000198 1789017787.501581 trunc from 0x037ed3 to 0x000ed1 (0x37001 bytes) 000199 1789017787.526032 write 0x02e3d1 thru 0x03ab24 (0xc754 bytes) 000200 1789017787.530043 mapwrite 0x03c46c thru 0x03ffff (0x3b94 bytes) 000201 1789017787.680328 trunc from 0x040000 to 0x00764e (0x389b1 bytes) 000202 1789017787.735218 mapread 0x006126 thru 0x00764e (0x1529 bytes) 000203 1789017787.737632 mapread 0x001ca1 thru 0x00764e (0x59ae bytes) 000204 1789017787.739044 fallocate 0x01db15 thru 0x020691 (0x2b7d bytes) 000205 1789017787.747465 trunc from 0x01b7dc to 0x020691 (0x4eb6 bytes) 000206 1789017787.752195 trunc from 0x01f994 to 0x020691 (0xcfe bytes) 000207 1789017787.760095 mapwrite 0x034e94 thru 0x03a678 (0x57e5 bytes) 000208 1789017787.891009 mapread 0x00d02c thru 0x01c50d (0xf4e2 bytes) 000210 1789017788.091115 fallocate 0x0281eb thru 0x02a5f4 (0x240a bytes) 000211 1789017788.102420 read 0x021024 thru 0x021bbc (0xb99 bytes) 000212 1789017788.115128 fallocate 0x029ea7 thru 0x032463 (0x85bd bytes) 000213 1789017788.125013 fallocate 0x02d7e0 thru 0x0353e3 (0x7c04 bytes) 000214 1789017788.138609 mapwrite 0x00002a thru 0x00ba8d (0xba64 bytes) 000217 1789017788.181226 mapwrite 0x02656e thru 0x02a0c4 (0x3b57 bytes) 000218 1789017788.199390 mapread 0x005992 thru 0x00b11f (0x578e bytes) 000219 1789017788.200228 mapread 0x033d00 thru 0x03a678 (0x6979 bytes) 000220 1789017788.201045 write 0x0229bd thru 0x02892c (0x5f70 bytes) 000221 1789017788.203953 read 0x032ad9 thru 0x03a678 (0x7ba0 bytes) 000222 1789017788.204628 punch from 0x02bcbc to 0x02eda2 (0x30e7 bytes) 000223 1789017788.209920 trunc from 0x03a679 to 0x033ccb (0x69ad bytes) 000224 1789017788.228308 mapread 0x028f95 thru 0x02c8d3 (0x393f bytes) 000225 1789017788.237075 trunc from 0x015439 to 0x018537 (0x30ff bytes) 000226 1789017788.241445 punch from 0x02addb to 0x033ccb (0x8ef1 bytes) 000227 1789017788.247157 read 0x00a842 thru 0x013c66 (0x9425 bytes) 000228 1789017788.247908 mapread 0x02c90f thru 0x033ccb (0x73bd bytes) 000229 1789017788.258214 fallocate 0x033d2a thru 0x0352ef (0x15c6 bytes) 000230 1789017788.263052 trunc from 0x0352f0 to 0x01436a (0x20f85 bytes) 000231 1789017788.281431 mapread 0x012068 thru 0x01436a (0x2303 bytes) 000232 1789017788.288097 read 0x00b188 thru 0x0103e9 (0x5262 bytes) 000234 1789017788.289854 fallocate 0x02c283 thru 0x0399d7 (0xd755 bytes) 000235 1789017788.295743 trunc from 0x01436b to 0x014aab (0x741 bytes) 000236 1789017788.314596 mapwrite 0x0331b7 thru 0x03c576 (0x93c0 bytes) 000238 1789017788.362697 write 0x0111c2 thru 0x015cd6 (0x4b15 bytes) 000239 1789017788.363987 punch from 0x022a5a to 0x02ee4b (0xc3f2 bytes) 000241 1789017788.374882 MIRROR_EXTEND to 2 mirrors 000242 1789017789.291690 read 0x034b25 thru 0x03c576 (0x7a52 bytes) 000243 1789017789.310174 punch from 0x002ff0 to 0x003859 (0x86a bytes) 000244 1789017789.353863 mapread 0x026c08 thru 0x02beb9 (0x52b2 bytes) 000245 1789017789.375947 trunc from 0x03c577 to 0x01cd45 (0x1f831 bytes) 000246 1789017789.448941 fallocate 0x023eec thru 0x033299 (0xf3ae bytes) 000247 1789017789.458023 read 0x03092f thru 0x033299 (0x296b bytes) 000249 1789017789.472547 read 0x010586 thru 0x01776c (0x71e7 bytes) 000250 1789017789.479296 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000251 1789017789.644163 trunc from 0x0048b0 to 0x00ca2f (0x8180 bytes) 000253 1789017789.655809 MIRROR_EXTEND to 2 mirrors 000254 1789017790.516222 punch from 0x00a622 to 0x019eae (0xf88d bytes) 000255 1789017790.558890 read 0x01734c thru 0x01c277 (0x4f2c bytes) 000256 1789017790.575892 read 0x02a244 thru 0x033299 (0x9056 bytes) 000257 1789017790.610924 punch from 0x00843a to 0x0109ee (0x85b5 bytes) 000258 1789017790.618178 punch from 0x027c21 to 0x02b81a (0x3bfa bytes) 000259 1789017790.628295 mapread 0x014970 thru 0x0234a1 (0xeb32 bytes) 000260 1789017790.630900 write 0x03f12e thru 0x03ffff (0xed2 bytes) 000261 1789017790.638717 trunc from 0x00541d to 0x010120 (0xad04 bytes) 000262 1789017790.645581 trunc from 0x012632 to 0x021c1b (0xf5ea bytes) 000263 1789017790.652304 mapread 0x03eeee thru 0x03ffff (0x1112 bytes) 000264 1789017790.661616 mapread 0x00601d thru 0x00affc (0x4fe0 bytes) 000265 1789017790.692389 trunc from 0x040000 to 0x023d58 (0x1c2a7 bytes) 000266 1789017790.722690 fallocate 0x03a1b9 thru 0x03ffff (0x5e47 bytes) 000267 1789017790.728499 punch from 0x0007c1 to 0x007124 (0x6964 bytes) 000268 1789017790.735144 punch from 0x004955 to 0x0063f6 (0x1aa2 bytes) 000269 1789017790.739905 punch from 0x030c52 to 0x03b439 (0xa7e8 bytes) 000270 1789017790.746690 write 0x006f31 thru 0x007255 (0x325 bytes) 000271 1789017790.758960 fallocate 0x02d5cc thru 0x03726f (0x9ca4 bytes) 000272 1789017790.769684 MIRROR_RESYNC 2 mirrors 000273 1789017791.596080 trunc from 0x03f9bf to 0x03ffff (0x641 bytes) 000274 1789017791.625935 trunc from 0x0263e2 to 0x03296f (0xc58e bytes) 000275 1789017791.632033 read 0x01a6fa thru 0x01a8f5 (0x1fc bytes) 000276 1789017791.639011 write 0x0295af thru 0x02e280 (0x4cd2 bytes) 000277 1789017791.663612 trunc from 0x03b70b to 0x03fb67 (0x445d bytes) 000278 1789017791.680508 mapwrite 0x004065 thru 0x0076ef (0x368b bytes) 000279 1789017791.757858 trunc from 0x040000 to 0x01367c (0x2c983 bytes) 000280 1789017791.805676 mapwrite 0x02c07f thru 0x036964 (0xa8e6 bytes) 000281 1789017791.921802 trunc from 0x036965 to 0x014c53 (0x21d11 bytes) 000282 1789017791.951080 read 0x00e6a6 thru 0x014c53 (0x65ae bytes) 000284 1789017791.979518 mapwrite 0x0255a2 thru 0x02ef07 (0x9966 bytes) 000285 1789017792.070253 read 0x02ae30 thru 0x02c98c (0x1b5d bytes) 000286 1789017792.070815 fallocate 0x03bfa6 thru 0x03ffff (0x405a bytes) 000287 1789017792.078750 punch from 0x015237 to 0x017d83 (0x2b4d bytes) 000288 1789017792.086633 read 0x01d29c thru 0x020f51 (0x3cb6 bytes) 000289 1789017792.087324 mapwrite 0x031f1d thru 0x03c49c (0xa580 bytes) 000290 1789017792.140047 fallocate 0x00f453 thru 0x0170ee (0x7c9c bytes) 000291 1789017792.146118 trunc from 0x029667 to 0x03520a (0xbba4 bytes) 000292 1789017792.156247 mapwrite 0x00681e thru 0x012b20 (0xc303 bytes) 000293 1789017792.183512 trunc from 0x03c49d to 0x00e733 (0x2dd69 bytes) 000294 1789017792.206993 punch from 0x0055d3 to 0x00e733 (0x9161 bytes) 000295 1789017792.215934 punch from 0x00af70 to 0x00e733 (0x37c4 bytes) 000296 1789017792.221121 trunc from 0x004e90 to 0x00e733 (0x98a4 bytes) 000297 1789017792.226775 mapread 0x00176e thru 0x00341d (0x1cb0 bytes) 000298 1789017792.227781 trunc from 0x00e734 to 0x0328bd (0x2418a bytes) 000299 1789017792.244196 write 0x00e0b6 thru 0x019e33 (0xbd7e bytes) 000300 1789017792.256440 trunc from 0x0328be to 0x02e2ed (0x45d0 bytes) 000301 1789017792.272549 read 0x00efdc thru 0x018cbe (0x9ce3 bytes) 000302 1789017792.273231 punch from 0x019413 to 0x01a95b (0x1549 bytes) 000303 1789017792.288991 read 0x00bae0 thru 0x019815 (0xdd36 bytes) 000304 1789017792.309999 mapread 0x02ba60 thru 0x02e2ed (0x288e bytes) 000306 1789017792.312142 mapwrite 0x011028 thru 0x0129e7 (0x19c0 bytes) 000307 1789017792.324983 mapread 0x00c398 thru 0x017477 (0xb0e0 bytes) 000310 1789017792.328942 read 0x00493c thru 0x00660b (0x1cd0 bytes) 000311 1789017792.330159 write 0x003a65 thru 0x003f92 (0x52e bytes) 000312 1789017792.332202 trunc from 0x02e2ee to 0x0008af (0x2da3e bytes) 000313 1789017792.361432 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x8b0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b0): Mirror 2: 0xc8e01805 (stale) CRC-32 checksum value for chunk [0, 0x8b0): Mirror 1: 0xc8e01805 Mirror 3: 0xc8e01805 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b0: skipped 000314 1789017793.413660 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8b0): Mirror 2: 0xc8e01805 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b0): Mirror 2: 0xc8e01805 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b0: skipped 000315 1789017793.772552 fallocate 0x017439 thru 0x01df1e (0x6ae6 bytes) 000316 1789017793.791923 mapwrite 0x02189e thru 0x027ec5 (0x6628 bytes) 000319 1789017793.897819 read 0x01fd72 thru 0x0243d9 (0x4668 bytes) 000320 1789017793.898557 punch from 0x024847 to 0x0265ba (0x1d74 bytes) 000321 1789017793.904247 trunc from 0x027ec6 to 0x01cdf9 (0xb0cc bytes) 000322 1789017793.924036 fallocate 0x0398a3 thru 0x03ebdf (0x533d bytes) 000323 1789017793.930512 trunc from 0x004615 to 0x013876 (0xf262 bytes) 000325 1789017793.939553 trunc from 0x01cdfa to 0x0237b4 (0x69bb bytes) 000326 1789017793.955458 trunc from 0x01f311 to 0x0237b4 (0x44a4 bytes) 000327 1789017793.961374 write 0x012ccd thru 0x020c67 (0xdf9b bytes) 000328 1789017793.973175 trunc from 0x0237b5 to 0x01e699 (0x511b bytes) 000329 1789017793.999479 mapread 0x010ba4 thru 0x01ccc2 (0xc11f bytes) 000330 1789017794.012146 fallocate 0x026c45 thru 0x02737a (0x736 bytes) 000331 1789017794.019325 punch from 0x004762 to 0x013a48 (0xf2e7 bytes) 000333 1789017794.044882 read 0x024851 thru 0x02737a (0x2b2a bytes) 000334 1789017794.057014 mapwrite 0x00743b thru 0x015031 (0xdbf7 bytes) 000335 1789017794.087779 read 0x0269df thru 0x02737a (0x99c bytes) 000336 1789017794.088734 fallocate 0x0392f8 thru 0x03ffff (0x6d08 bytes) 000337 1789017794.094636 trunc from 0x040000 to 0x039893 (0x676c bytes) 000339 1789017794.110893 mapwrite 0x019340 thru 0x025a66 (0xc727 bytes) 000340 1789017794.127861 mapread 0x000519 thru 0x00b390 (0xae78 bytes) 000341 1789017794.128914 mapwrite 0x01cb20 thru 0x020efd (0x43de bytes) 000342 1789017794.142435 write 0x0365a3 thru 0x03ffff (0x9a5d bytes) 000343 1789017794.155179 fallocate 0x035ba3 thru 0x03ffff (0xa45d bytes) 000344 1789017794.166860 MIRROR_RESYNC 2 mirrors 000345 1789017795.156219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000348 1789017795.384189 fallocate 0x0188d4 thru 0x026945 (0xe072 bytes) 000350 1789017795.422765 fallocate 0x012d7f thru 0x0187b7 (0x5a39 bytes) 000351 1789017795.432943 trunc from 0x03519e to 0x03ffff (0xae62 bytes) 000352 1789017795.463801 trunc from 0x03bfc9 to 0x03ffff (0x4037 bytes) 000353 1789017795.486938 fallocate 0x037ae7 thru 0x03fe25 (0x833f bytes) 000354 1789017795.498153 write 0x039de3 thru 0x03d98f (0x3bad bytes) 000355 1789017795.530876 mapwrite 0x014a8e thru 0x014ea3 (0x416 bytes) 000356 1789017795.565565 mapread 0x009be4 thru 0x018bc2 (0xefdf bytes) 000357 1789017795.629828 punch from 0x03978a to 0x03ffff (0x6876 bytes) 000358 1789017795.649186 read 0x037262 thru 0x03ffff (0x8d9e bytes) 000359 1789017795.698668 trunc from 0x040000 to 0x025882 (0x1a77d bytes) 000360 1789017795.765734 punch from 0x00d735 to 0x014114 (0x69e0 bytes) 000361 1789017795.776907 punch from 0x023574 to 0x025882 (0x230f bytes) 000363 1789017795.783528 write 0x010409 thru 0x015380 (0x4f78 bytes) 000364 1789017795.789957 trunc from 0x011415 to 0x01f389 (0xdf75 bytes) 000366 1789017795.815140 trunc from 0x025883 to 0x0285bb (0x2d39 bytes) 000367 1789017795.859308 mapread 0x021aeb thru 0x0285bb (0x6ad1 bytes) 000368 1789017795.875006 trunc from 0x01f122 to 0x0285bb (0x949a bytes) 000369 1789017795.888874 trunc from 0x027371 to 0x0285bb (0x124b bytes) 000371 1789017795.897636 write 0x033d56 thru 0x035716 (0x19c1 bytes) 000372 1789017795.899842 write 0x028eb7 thru 0x0354de (0xc628 bytes) 000373 1789017795.908747 read 0x006f2e thru 0x00a98f (0x3a62 bytes) 000374 1789017795.909576 mapread 0x022f82 thru 0x025dde (0x2e5d bytes) 000375 1789017795.950742 trunc from 0x008bcf to 0x014341 (0xb773 bytes) 000376 1789017795.962252 mapread 0x028f27 thru 0x0350e0 (0xc1ba bytes) 000377 1789017795.977199 trunc from 0x013f56 to 0x023956 (0xfa01 bytes) 000378 1789017795.985227 mapread 0x01df53 thru 0x02cf70 (0xf01e bytes) 000379 1789017796.077675 trunc from 0x00d611 to 0x0164aa (0x8e9a bytes) 000380 1789017796.101155 punch from 0x00147b to 0x0046ff (0x3285 bytes) 000381 1789017796.115981 write 0x0302e4 thru 0x03035b (0x78 bytes) 000382 1789017796.116866 trunc from 0x01abb9 to 0x01d288 (0x26d0 bytes) 000383 1789017796.133978 read 0x006e2c thru 0x009c26 (0x2dfb bytes) 000384 1789017796.161118 punch from 0x01bf94 to 0x0210ef (0x515c bytes) 000386 1789017796.168213 mapwrite 0x02e19a thru 0x0358db (0x7742 bytes) 000388 1789017796.215887 fallocate 0x037f88 thru 0x03ffff (0x8078 bytes) 000389 1789017796.221740 mapwrite 0x02c00c thru 0x0381fe (0xc1f3 bytes) 000390 1789017796.290739 fallocate 0x03ec53 thru 0x03ffff (0x13ad bytes) 000391 1789017796.299723 mapread 0x03da20 thru 0x03ffff (0x25e0 bytes) 000392 1789017796.301134 read 0x0245db thru 0x03322a (0xec50 bytes) 000393 1789017796.301826 fallocate 0x0366f7 thru 0x03ffff (0x9909 bytes) 000394 1789017796.309062 trunc from 0x02304d to 0x02aa2f (0x79e3 bytes) 000395 1789017796.327978 read 0x01fc1d thru 0x02a5d7 (0xa9bb bytes) 000396 1789017796.349738 MIRROR_EXTEND to 2 mirrors 000397 1789017797.112307 fallocate 0x0200f4 thru 0x024267 (0x4174 bytes) 000398 1789017797.158272 read 0x035b12 thru 0x03ffff (0xa4ee bytes) 000399 1789017797.176757 trunc from 0x032846 to 0x0336d6 (0xe91 bytes) 000401 1789017797.186447 mapread 0x0393d2 thru 0x03ffff (0x6c2e bytes) 000402 1789017797.188863 trunc from 0x040000 to 0x0132d5 (0x2cd2a bytes) 000403 1789017797.254028 punch from 0x00b73b to 0x0132d5 (0x7b9b bytes) 000404 1789017797.274569 punch from 0x0017b5 to 0x002c2c (0x1478 bytes) 000406 1789017797.280902 read 0x001f15 thru 0x002795 (0x881 bytes) 000407 1789017797.301179 trunc from 0x00d2d3 to 0x0132d5 (0x6003 bytes) 000408 1789017797.309480 mapread 0x000166 thru 0x00aff2 (0xae8d bytes) 000409 1789017797.310484 trunc from 0x01313a to 0x0132d5 (0x19c bytes) 000410 1789017797.316775 mapwrite 0x02a84b thru 0x032d5c (0x8512 bytes) 000411 1789017797.385686 mapwrite 0x01406d thru 0x017ec0 (0x3e54 bytes) 000412 1789017797.400492 trunc from 0x00a25c to 0x0181d7 (0xdf7c bytes) 000413 1789017797.410812 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000414 1789017797.601989 trunc from 0x02c7c8 to 0x032d5c (0x6595 bytes) 000415 1789017797.609678 punch from 0x026724 to 0x032d5c (0xc639 bytes) 000416 1789017797.620806 read 0x0119de thru 0x011dd4 (0x3f7 bytes) 000417 1789017797.630751 punch from 0x006295 to 0x009413 (0x317f bytes) 000418 1789017797.635795 punch from 0x01b615 to 0x026db2 (0xb79e bytes) 000419 1789017797.640289 read 0x024aab thru 0x032d5c (0xe2b2 bytes) 000420 1789017797.651565 mapread 0x01f0b8 thru 0x02d160 (0xe0a9 bytes) 000421 1789017797.653714 fallocate 0x02269e thru 0x02541a (0x2d7d bytes) 000422 1789017797.660311 fallocate 0x03e278 thru 0x03ffff (0x1d88 bytes) 000423 1789017797.666008 punch from 0x00f47f to 0x01be4e (0xc9d0 bytes) 000424 1789017797.677711 MIRROR_EXTEND to 2 mirrors 000426 1789017798.623189 trunc from 0x00ad32 to 0x015d77 (0xb046 bytes) 000428 1789017798.660971 read 0x0076a4 thru 0x016c61 (0xf5be bytes) 000430 1789017798.682141 mapwrite 0x021c8b thru 0x026163 (0x44d9 bytes) 000431 1789017798.730720 mapwrite 0x03420d thru 0x035657 (0x144b bytes) 000432 1789017798.814100 punch from 0x0011cf to 0x00da4f (0xc881 bytes) 000433 1789017798.823732 fallocate 0x004e10 thru 0x007943 (0x2b34 bytes) 000434 1789017798.842490 MIRROR_RESYNC 2 mirrors 000435 1789017799.960901 punch from 0x00e8dc to 0x011026 (0x274b bytes) 000436 1789017800.026731 MIRROR_RESYNC 2 mirrors 000437 1789017801.050752 write 0x035089 thru 0x03ffff (0xaf77 bytes) 000438 1789017801.115310 punch from 0x00215a to 0x00edbe (0xcc65 bytes) 000439 1789017801.131314 punch from 0x01a239 to 0x01ee71 (0x4c39 bytes) 000441 1789017801.140255 fallocate 0x02bc2b thru 0x02d223 (0x15f9 bytes) 000442 1789017801.158418 trunc from 0x040000 to 0x016211 (0x29dee bytes) 000443 1789017801.194447 fallocate 0x0202b5 thru 0x02784f (0x759b bytes) 000444 1789017801.213704 MIRROR_RESYNC 2 mirrors 000445 1789017802.216127 mapread 0x00a08f thru 0x016211 (0xc183 bytes) 000446 1789017802.257650 mapwrite 0x0349e3 thru 0x0350d1 (0x6ef bytes) 000447 1789017802.352927 fallocate 0x025f1b thru 0x02bd82 (0x5e68 bytes) 000448 1789017802.359462 fallocate 0x003302 thru 0x006f3c (0x3c3b bytes) 000449 1789017802.371375 trunc from 0x0350d2 to 0x00d42d (0x27ca4 bytes) 000450 1789017802.403698 write 0x026470 thru 0x0277a2 (0x1333 bytes) 000452 1789017802.406963 mapread 0x0116e0 thru 0x01f6ae (0xdfcf bytes) 000454 1789017802.434686 fallocate 0x00d552 thru 0x00e231 (0xce0 bytes) 000455 1789017802.442900 punch from 0x02335b to 0x026c2d (0x38d3 bytes) 000456 1789017802.463205 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000457 1789017802.618381 trunc from 0x025cc6 to 0x0277a2 (0x1add bytes) 000458 1789017802.630704 trunc from 0x0277a3 to 0x00b0e8 (0x1c6ba bytes) 000459 1789017802.688814 fallocate 0x03028f thru 0x03551b (0x528d bytes) 000460 1789017802.708789 punch from 0x000c4a to 0x008af0 (0x7ea7 bytes) 000463 1789017802.719828 trunc from 0x006bbc to 0x00b0e8 (0x452d bytes) 000464 1789017802.728911 read 0x007091 thru 0x00b0e8 (0x4058 bytes) 000466 1789017802.738207 punch from 0x00935d to 0x00b0e8 (0x1d8c bytes) 000467 1789017802.743078 mapread 0x0028a8 thru 0x00a7ee (0x7f47 bytes) 000468 1789017802.770597 write 0x01a760 thru 0x01b0f4 (0x995 bytes) 000469 1789017802.772738 mapread 0x010033 thru 0x01b0f4 (0xb0c2 bytes) 000470 1789017802.799582 write 0x00d836 thru 0x01af46 (0xd711 bytes) 000471 1789017802.803691 read 0x01afae thru 0x01b0f4 (0x147 bytes) 000472 1789017802.804761 punch from 0x009aaf to 0x015721 (0xbc73 bytes) 000473 1789017802.837327 trunc from 0x00a2be to 0x017659 (0xd39c bytes) 000474 1789017802.855138 write 0x02a486 thru 0x03108d (0x6c08 bytes) 000476 1789017802.862004 write 0x02eaec thru 0x033a84 (0x4f99 bytes) 000477 1789017802.864013 fallocate 0x022180 thru 0x02c5c1 (0xa442 bytes) 000478 1789017802.879761 mapwrite 0x03f55d thru 0x03ffff (0xaa3 bytes) 000479 1789017803.009316 mapread 0x0243dc thru 0x02ca0a (0x862f bytes) 000480 1789017803.010167 mapwrite 0x00e538 thru 0x00f3b7 (0xe80 bytes) 000482 1789017803.030633 punch from 0x03f284 to 0x03ffff (0xd7c bytes) 000483 1789017803.038102 mapread 0x012cd3 thru 0x01fbac (0xceda bytes) 000484 1789017803.039213 mapwrite 0x01ccc5 thru 0x02c12f (0xf46b bytes) 000485 1789017803.065738 read 0x027d7b thru 0x032c4c (0xaed2 bytes) 000487 1789017803.066809 trunc from 0x021fc9 to 0x02674f (0x4787 bytes) 000489 1789017803.084577 write 0x00e249 thru 0x014396 (0x614e bytes) 000490 1789017803.085694 punch from 0x01f1c6 to 0x027fdb (0x8e16 bytes) 000491 1789017803.103914 mapread 0x038e15 thru 0x03ffff (0x71eb bytes) 000492 1789017803.119158 write 0x035802 thru 0x03ffff (0xa7fe bytes) 000493 1789017803.120638 mapread 0x00ff3b thru 0x01b6e6 (0xb7ac bytes) 000495 1789017803.143213 MIRROR_EXTEND to 2 mirrors 000496 1789017804.107981 trunc from 0x040000 to 0x023847 (0x1c7b8 bytes) 000497 1789017804.255047 trunc from 0x022a2c to 0x023847 (0xe1c bytes) 000498 1789017804.260748 trunc from 0x00dcf7 to 0x013601 (0x590b bytes) 000500 1789017804.289633 MIRROR_RESYNC 2 mirrors 000501 1789017805.363523 fallocate 0x006f38 thru 0x008c3c (0x1d05 bytes) 000502 1789017805.410061 read 0x004aba thru 0x00dab6 (0x8ffd bytes) 000503 1789017805.438601 trunc from 0x023848 to 0x037f94 (0x1474d bytes) 000504 1789017805.472771 read 0x00820a thru 0x0106c4 (0x84bb bytes) 000505 1789017805.501538 mapread 0x022115 thru 0x028c5c (0x6b48 bytes) 000506 1789017805.502735 punch from 0x02860d to 0x02de1b (0x580f bytes) 000507 1789017805.507559 trunc from 0x01b2d5 to 0x025da3 (0xaacf bytes) 000508 1789017805.513140 fallocate 0x0071b2 thru 0x00b37c (0x41cb bytes) 000510 1789017805.518681 trunc from 0x02bcbe to 0x02f63d (0x3980 bytes) 000511 1789017805.523894 mapread 0x01d9ba thru 0x02d343 (0xf98a bytes) 000512 1789017805.597805 write 0x01cb92 thru 0x01e8d6 (0x1d45 bytes) 000513 1789017805.611099 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37f95): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f95): Mirror 2: 0xf0de221b (stale) CRC-32 checksum value for chunk [0, 0x37f95): Mirror 1: 0x7e805c38 Mirror 3: 0x7e805c38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f95: skipped 000514 1789017807.164643 trunc from 0x008ba2 to 0x014fee (0xc44d bytes) 000515 1789017807.238559 mapwrite 0x013c72 thru 0x01ad78 (0x7107 bytes) 000516 1789017807.399751 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f95): Mirror 2: 0xf0de221b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f95): Mirror 2: 0xf0de221b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f95: skipped 000517 1789017807.785308 mapwrite 0x0193f2 thru 0x01dee2 (0x4af1 bytes) 000518 1789017807.865209 punch from 0x01cc24 to 0x025028 (0x8405 bytes) 000519 1789017807.872118 mapwrite 0x01718b thru 0x025066 (0xdedc bytes) 000520 1789017807.938003 trunc from 0x005aab to 0x00d40e (0x7964 bytes) 000521 1789017807.946121 fallocate 0x026345 thru 0x02a39f (0x405b bytes) 000522 1789017807.957129 trunc from 0x01fa27 to 0x026185 (0x675f bytes) 000523 1789017807.967910 mapread 0x003999 thru 0x0117f3 (0xde5b bytes) 000524 1789017807.989018 write 0x02bad7 thru 0x031023 (0x554d bytes) 000525 1789017808.013019 mapwrite 0x02dd8f thru 0x0355fa (0x786c bytes) 000526 1789017808.039221 trunc from 0x037f95 to 0x01cb28 (0x1b46c bytes) 000527 1789017808.071073 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000530 1789017808.262000 punch from 0x01639c to 0x01cb28 (0x678d bytes) 000531 1789017808.270092 fallocate 0x013c21 thru 0x014efe (0x12de bytes) 000532 1789017808.279842 mapwrite 0x009c1a thru 0x0119fd (0x7de4 bytes) 000533 1789017808.338062 mapwrite 0x01ad54 thru 0x02aa41 (0xfcee bytes) 000534 1789017808.417337 mapwrite 0x031319 thru 0x03b4b4 (0xa19c bytes) 000535 1789017808.485428 punch from 0x003654 to 0x010f29 (0xd8d6 bytes) 000536 1789017808.492280 mapread 0x02e849 thru 0x03b4b4 (0xcc6c bytes) 000538 1789017808.495337 mapwrite 0x0226dd thru 0x025f8c (0x38b0 bytes) 000539 1789017808.514760 trunc from 0x01e16d to 0x01f37b (0x120f bytes) 000540 1789017808.520197 read 0x03511a thru 0x03b4b4 (0x639b bytes) 000541 1789017808.520727 read 0x025b70 thru 0x02d2df (0x7770 bytes) 000542 1789017808.521116 trunc from 0x03b4b5 to 0x028dc1 (0x126f3 bytes) 000544 1789017808.543738 read 0x020152 thru 0x0260ee (0x5f9d bytes) 000546 1789017808.545285 fallocate 0x02c88d thru 0x035dc2 (0x9536 bytes) 000547 1789017808.559447 write 0x01462b thru 0x01d243 (0x8c19 bytes) 000549 1789017808.580673 mapread 0x00085b thru 0x004db7 (0x455d bytes) 000551 1789017808.619791 trunc from 0x028dc2 to 0x0124a4 (0x1691d bytes) 000552 1789017808.653835 read 0x006066 thru 0x00917b (0x3116 bytes) 000553 1789017808.660172 MIRROR_EXTEND to 2 mirrors 000554 1789017809.840623 trunc from 0x011ada to 0x0124a4 (0x9cb bytes) 000555 1789017809.934328 mapwrite 0x02edd0 thru 0x036e07 (0x8038 bytes) 000556 1789017810.068570 punch from 0x011709 to 0x01f427 (0xdd1f bytes) 000557 1789017810.092202 mapwrite 0x03b508 thru 0x03ffff (0x4af8 bytes) 000558 1789017810.179124 trunc from 0x01efe3 to 0x028212 (0x9230 bytes) 000560 1789017810.189682 punch from 0x0214a2 to 0x023fa4 (0x2b03 bytes) 000561 1789017810.199099 mapread 0x0100e3 thru 0x01e48e (0xe3ac bytes) 000562 1789017810.271335 mapread 0x039482 thru 0x03ffff (0x6b7e bytes) 000563 1789017810.273030 trunc from 0x00d748 to 0x01a664 (0xcf1d bytes) 000564 1789017810.283332 trunc from 0x02e1c4 to 0x03d6c2 (0xf4ff bytes) 000565 1789017810.291259 trunc from 0x004ca6 to 0x008823 (0x3b7e bytes) 000566 1789017810.306293 fallocate 0x02bff3 thru 0x03be24 (0xfe32 bytes) 000567 1789017810.318377 mapwrite 0x009bba thru 0x0190f0 (0xf537 bytes) 000569 1789017810.380150 mapread 0x03999e thru 0x03b31f (0x1982 bytes) 000570 1789017810.403410 trunc from 0x040000 to 0x0195a4 (0x26a5b bytes) 000571 1789017810.437918 fallocate 0x013eff thru 0x01803a (0x413c bytes) 000572 1789017810.450338 write 0x03be33 thru 0x03ec17 (0x2de5 bytes) 000573 1789017810.452549 trunc from 0x03ec18 to 0x00927b (0x3599c bytes) 000574 1789017810.477159 mapread 0x00279a thru 0x00927b (0x6ae2 bytes) 000575 1789017810.498617 trunc from 0x006dd8 to 0x006e02 (0x2b bytes) 000576 1789017810.512297 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000577 1789017810.731715 trunc from 0x00927c to 0x038454 (0x2f1d9 bytes) 000578 1789017810.759164 read 0x022600 thru 0x025e5c (0x385d bytes) 000579 1789017810.772536 trunc from 0x038455 to 0x021ee5 (0x1656f bytes) 000580 1789017810.797201 mapwrite 0x008b07 thru 0x013235 (0xa72f bytes) 000581 1789017810.865474 mapread 0x01d8f9 thru 0x021ee5 (0x45ed bytes) 000582 1789017810.878385 trunc from 0x01061a to 0x013773 (0x315a bytes) 000583 1789017810.886704 write 0x01ad43 thru 0x01d618 (0x28d6 bytes) 000584 1789017810.904002 MIRROR_EXTEND to 2 mirrors 000585 1789017812.095962 mapread 0x01845b thru 0x021ee5 (0x9a8b bytes) 000586 1789017812.141230 trunc from 0x01eded to 0x021ee5 (0x30f9 bytes) 000587 1789017812.174317 trunc from 0x003e01 to 0x00d9f6 (0x9bf6 bytes) 000588 1789017812.181710 mapwrite 0x0198d0 thru 0x01f2e0 (0x5a11 bytes) 000589 1789017812.211537 write 0x035b5d thru 0x039ab9 (0x3f5d bytes) 000590 1789017812.217274 trunc from 0x039aba to 0x01c28f (0x1d82a bytes) 000591 1789017812.242314 read 0x016919 thru 0x01c28f (0x5977 bytes) 000592 1789017812.244259 trunc from 0x01c290 to 0x004196 (0x180f9 bytes) 000593 1789017812.269086 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000594 1789017812.429994 mapread 0x00050e thru 0x004196 (0x3c89 bytes) 000595 1789017812.471591 trunc from 0x000546 to 0x004196 (0x3c51 bytes) 000596 1789017812.476035 fallocate 0x017629 thru 0x024d74 (0xd74c bytes) 000597 1789017812.480084 mapread 0x001856 thru 0x004196 (0x2941 bytes) 000599 1789017812.485926 mapwrite 0x01bf78 thru 0x021efd (0x5f86 bytes) 000601 1789017812.539496 mapwrite 0x022a31 thru 0x02dcac (0xb27c bytes) 000602 1789017812.593561 trunc from 0x02c282 to 0x02dcac (0x1a2b bytes) 000603 1789017812.599935 trunc from 0x02dcad to 0x0338ee (0x5c42 bytes) 000604 1789017812.633569 trunc from 0x00d883 to 0x018ccf (0xb44d bytes) 000606 1789017812.638754 write 0x012c86 thru 0x016848 (0x3bc3 bytes) 000608 1789017812.652107 punch from 0x0056f8 to 0x00d53f (0x7e48 bytes) 000609 1789017812.656862 write 0x01b717 thru 0x029cd1 (0xe5bb bytes) 000610 1789017812.658638 trunc from 0x016433 to 0x01ed83 (0x8951 bytes) 000611 1789017812.682333 trunc from 0x0338ef to 0x007a2d (0x2bec1 bytes) 000612 1789017812.695544 read 0x005492 thru 0x007a2d (0x259c bytes) 000613 1789017812.702159 trunc from 0x0019df to 0x007a2d (0x604f bytes) 000614 1789017812.710059 mapread 0x002d09 thru 0x007a2d (0x4d25 bytes) 000615 1789017812.723648 fallocate 0x035484 thru 0x037061 (0x1bde bytes) 000616 1789017812.730562 mapread 0x00ea10 thru 0x00f65c (0xc4d bytes) 000617 1789017812.748270 fallocate 0x033be4 thru 0x03e443 (0xa860 bytes) 000618 1789017812.766593 MIRROR_EXTEND to 2 mirrors 000620 1789017813.805638 trunc from 0x03e444 to 0x00bb60 (0x328e3 bytes) 000621 1789017813.866630 MIRROR_RESYNC 2 mirrors 000622 1789017814.577611 trunc from 0x000c6e to 0x0037ff (0x2b92 bytes) 000623 1789017814.606671 punch from 0x002265 to 0x00872c (0x64c8 bytes) 000624 1789017814.613603 trunc from 0x00bb61 to 0x0326cd (0x26b6d bytes) 000625 1789017814.641675 mapread 0x020f89 thru 0x02fbb4 (0xec2c bytes) 000626 1789017814.679497 fallocate 0x023248 thru 0x02b89e (0x8657 bytes) 000627 1789017814.685677 fallocate 0x03810a thru 0x03ffff (0x7ef6 bytes) 000628 1789017814.692824 mapread 0x016014 thru 0x016300 (0x2ed bytes) 000629 1789017814.713517 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x326ce): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ce): Mirror 2: 0x65a84892 (stale) CRC-32 checksum value for chunk [0, 0x326ce): Mirror 1: 0x65a84892 Mirror 3: 0x65a84892 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ce: skipped 000630 1789017815.903928 trunc from 0x025534 to 0x02cf00 (0x79cd bytes) 000631 1789017815.956127 read 0x006032 thru 0x00ea16 (0x89e5 bytes) 000632 1789017815.967940 write 0x030838 thru 0x0323d9 (0x1ba2 bytes) 000633 1789017815.984680 fallocate 0x039c57 thru 0x03ffff (0x63a9 bytes) 000634 1789017816.003268 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326ce): Mirror 3: 0x65a84892 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ce): Mirror 3: 0x65a84892 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ce: skipped 000635 1789017816.416136 read 0x00aacc thru 0x016131 (0xb666 bytes) 000637 1789017816.431100 read 0x01c4d6 thru 0x0241ee (0x7d19 bytes) 000638 1789017816.446560 mapread 0x0262f5 thru 0x02f782 (0x948e bytes) 000639 1789017816.448779 mapread 0x00ac49 thru 0x0162a9 (0xb661 bytes) 000640 1789017816.451288 fallocate 0x0178f3 thru 0x023962 (0xc070 bytes) 000642 1789017816.458718 fallocate 0x025293 thru 0x026efe (0x1c6c bytes) 000643 1789017816.463780 mapwrite 0x02dd26 thru 0x03cd85 (0xf060 bytes) 000644 1789017816.534786 trunc from 0x0209a9 to 0x028702 (0x7d5a bytes) 000645 1789017816.549744 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 3: 0xe67cc2a4 (stale) CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 1: 0xb421c30a Mirror 4: 0xb421c30a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd86: skipped 000646 1789017817.716873 trunc from 0x003471 to 0x00f517 (0xc0a7 bytes) 000647 1789017817.784085 mapwrite 0x00563b thru 0x00568d (0x53 bytes) 000648 1789017817.842579 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 4: 0xb421c30a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 4: 0xb421c30a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd86: skipped 000650 1789017818.150953 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 4: 0xb421c30a (stale) CRC-32 checksum value for chunk [0, 0x3cd86): Mirror 1: 0x503ddad5 Mirror 5: 0x503ddad5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd86: skipped 000651 1789017819.084915 fallocate 0x01f761 thru 0x021706 (0x1fa6 bytes) 000652 1789017819.142107 mapwrite 0x01e99f thru 0x025015 (0x6677 bytes) 000653 1789017819.234428 fallocate 0x03b087 thru 0x03c848 (0x17c2 bytes) 000654 1789017819.244031 mapwrite 0x02414e thru 0x032eee (0xeda1 bytes) 000655 1789017819.286492 punch from 0x037ecd to 0x03b1a9 (0x32dd bytes) 000657 1789017819.293761 punch from 0x021236 to 0x02d565 (0xc330 bytes) 000658 1789017819.302490 write 0x0059d5 thru 0x012e95 (0xd4c1 bytes) 000659 1789017819.329614 trunc from 0x029bc3 to 0x02ee56 (0x5294 bytes) 000660 1789017819.336097 read 0x037657 thru 0x03cd85 (0x572f bytes) 000661 1789017819.349596 mapread 0x00619b thru 0x00bc5a (0x5ac0 bytes) 000662 1789017819.350285 read 0x01b7d8 thru 0x027b80 (0xc3a9 bytes) 000664 1789017819.357699 trunc from 0x02a70c to 0x031e32 (0x7727 bytes) 000665 1789017819.362712 trunc from 0x03cd86 to 0x03e3b0 (0x162b bytes) 000666 1789017819.383948 mapwrite 0x000aaa thru 0x004322 (0x3879 bytes) 000667 1789017819.409867 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e3b1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3b1): Mirror 4: 0x8fe1eb8c (stale) Mirror 5: 0x3b3d0d08 (stale) CRC-32 checksum value for chunk [0, 0x3e3b1): Mirror 1: 0x3b4e49c5 Mirror 6: 0x3b4e49c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3b1: skipped 000669 1789017820.425537 trunc from 0x018c77 to 0x028bc3 (0xff4d bytes) 000670 1789017820.448530 mapread 0x012a35 thru 0x018405 (0x59d1 bytes) 000671 1789017820.468896 mapwrite 0x02488c thru 0x02dd40 (0x94b5 bytes) 000672 1789017820.506105 fallocate 0x039950 thru 0x03a67d (0xd2e bytes) 000673 1789017820.512145 write 0x025555 thru 0x026357 (0xe03 bytes) 000675 1789017820.518683 trunc from 0x020cab to 0x02550b (0x4861 bytes) 000677 1789017820.529469 punch from 0x015e0b to 0x02516b (0xf361 bytes) 000678 1789017820.534171 read 0x0041d2 thru 0x00d97d (0x97ac bytes) 000679 1789017820.548089 trunc from 0x027e2c to 0x02e08d (0x6262 bytes) 000680 1789017820.554048 mapwrite 0x0346a2 thru 0x03a39c (0x5cfb bytes) 000681 1789017820.580313 read 0x02059c thru 0x025cea (0x574f bytes) 000682 1789017820.593643 mapwrite 0x023b9e thru 0x027eca (0x432d bytes) 000683 1789017820.610286 punch from 0x00d9e1 to 0x016c94 (0x92b4 bytes) 000684 1789017820.619482 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e3b1): Mirror 4: 0x8fe1eb8c (stale) Mirror 5: 0x3b3d0d08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3b1): Mirror 4: 0x8fe1eb8c (stale) Mirror 5: 0x3b3d0d08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3b1: skipped 000685 1789017820.910059 fallocate 0x00da46 thru 0x01c778 (0xed33 bytes) 000686 1789017820.923456 trunc from 0x022f47 to 0x02ff4c (0xd006 bytes) 000687 1789017820.945739 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e3b1): Mirror 5: 0x3b3d0d08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3b1): Mirror 5: 0x3b3d0d08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3b1: skipped 000688 1789017821.219839 trunc from 0x03e3b1 to 0x037a45 (0x696b bytes) 000690 1789017821.273414 fallocate 0x034570 thru 0x035dbe (0x184f bytes) 000691 1789017821.279415 trunc from 0x037a46 to 0x00fcb0 (0x27d95 bytes) 000692 1789017821.294281 read 0x00b4c4 thru 0x00bb1b (0x658 bytes) 000693 1789017821.300709 read 0x00a76b thru 0x00fcb0 (0x5546 bytes) 000694 1789017821.309798 mapread 0x0078c2 thru 0x0085eb (0xd2a bytes) 000695 1789017821.311287 trunc from 0x007d1a to 0x00fcb0 (0x7f97 bytes) 000696 1789017821.317023 read 0x007d03 thru 0x00aa65 (0x2d63 bytes) 000697 1789017821.327504 trunc from 0x003e63 to 0x0049a5 (0xb43 bytes) 000698 1789017821.332904 trunc from 0x00fcb1 to 0x0116a9 (0x19f9 bytes) 000699 1789017821.353288 mapread 0x00c74c thru 0x00d5b4 (0xe69 bytes) 000700 1789017821.355030 mapread 0x00b56c thru 0x0116a9 (0x613e bytes) 000701 1789017821.361463 fallocate 0x01f290 thru 0x020897 (0x1608 bytes) 000702 1789017821.366132 mapread 0x00f6fd thru 0x0116a9 (0x1fad bytes) 000703 1789017821.367262 fallocate 0x008ae6 thru 0x00d411 (0x492c bytes) 000704 1789017821.372388 punch from 0x00e69e to 0x0116a9 (0x300c bytes) 000705 1789017821.376969 mapread 0x00dfac thru 0x0116a8 (0x36fd bytes) 000706 1789017821.394042 trunc from 0x0116aa to 0x027f18 (0x1686f bytes) 000707 1789017821.410616 trunc from 0x016fa9 to 0x018e4a (0x1ea2 bytes) 000708 1789017821.416999 punch from 0x00c525 to 0x011093 (0x4b6f bytes) 000709 1789017821.422244 read 0x009e55 thru 0x017ba8 (0xdd54 bytes) 000710 1789017821.432528 trunc from 0x00d66d to 0x012ded (0x5781 bytes) 000711 1789017821.441539 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000714 1789017821.638039 punch from 0x021ea4 to 0x027f18 (0x6075 bytes) 000715 1789017821.645630 trunc from 0x020cd3 to 0x027f18 (0x7246 bytes) 000716 1789017821.649993 fallocate 0x019feb thru 0x01f9fa (0x5a10 bytes) 000717 1789017821.653848 trunc from 0x027f19 to 0x00595d (0x225bb bytes) 000718 1789017821.674457 mapread 0x0048b7 thru 0x00595d (0x10a7 bytes) 000719 1789017821.684902 fallocate 0x0128b4 thru 0x01c489 (0x9bd6 bytes) 000721 1789017821.690136 mapread 0x00dde1 thru 0x013b66 (0x5d86 bytes) 000724 1789017821.701088 trunc from 0x01c48a to 0x02a80e (0xe385 bytes) 000725 1789017821.714997 read 0x00c2de thru 0x0123a8 (0x60cb bytes) 000727 1789017821.717693 trunc from 0x02a80f to 0x03c732 (0x11f24 bytes) 000728 1789017821.731737 fallocate 0x026bc6 thru 0x02e642 (0x7a7d bytes) 000729 1789017821.737614 trunc from 0x03c733 to 0x028f07 (0x1382b bytes) 000730 1789017821.758012 read 0x005981 thru 0x005d2d (0x3ad bytes) 000732 1789017821.761509 read 0x023c7f thru 0x028f07 (0x5289 bytes) 000733 1789017821.773145 read 0x022504 thru 0x028f07 (0x6a04 bytes) 000734 1789017821.773896 fallocate 0x0206a1 thru 0x02a3af (0x9d0f bytes) 000735 1789017821.782650 trunc from 0x028f08 to 0x007f16 (0x20ff1 bytes) 000737 1789017821.800538 mapwrite 0x01a77a thru 0x029531 (0xedb8 bytes) 000738 1789017821.852718 fallocate 0x03bf03 thru 0x03ffff (0x40fd bytes) 000739 1789017821.858284 trunc from 0x029532 to 0x02b25b (0x1d2a bytes) 000740 1789017821.879122 punch from 0x01383d to 0x01e9be (0xb182 bytes) 000741 1789017821.884619 trunc from 0x02b25c to 0x00a897 (0x209c4 bytes) 000742 1789017821.896997 trunc from 0x00a796 to 0x00a897 (0x102 bytes) 000743 1789017821.902286 trunc from 0x00a898 to 0x01591d (0xb086 bytes) 000744 1789017821.916002 write 0x02fe8b thru 0x038da6 (0x8f1c bytes) 000746 1789017821.917959 mapwrite 0x02a590 thru 0x02afab (0xa1c bytes) 000747 1789017821.940989 mapread 0x008337 thru 0x011089 (0x8d53 bytes) 000748 1789017821.977527 write 0x03f76f thru 0x03ffff (0x891 bytes) 000749 1789017821.978587 mapread 0x014116 thru 0x015ed7 (0x1dc2 bytes) 000750 1789017821.991075 mapread 0x035b69 thru 0x039b66 (0x3ffe bytes) 000751 1789017821.992177 write 0x029782 thru 0x02e3af (0x4c2e bytes) 000752 1789017821.998268 trunc from 0x01dc34 to 0x02b8c6 (0xdc93 bytes) 000753 1789017822.013039 mapwrite 0x02f0ab thru 0x032787 (0x36dd bytes) 000754 1789017822.027762 mapwrite 0x0251ac thru 0x02c8cc (0x7721 bytes) 000755 1789017822.098000 trunc from 0x009b85 to 0x0136e5 (0x9b61 bytes) 000756 1789017822.113999 read 0x017261 thru 0x017b52 (0x8f2 bytes) 000757 1789017822.141610 fallocate 0x013a50 thru 0x014dac (0x135d bytes) 000758 1789017822.156448 trunc from 0x040000 to 0x02c426 (0x13bd9 bytes) 000759 1789017822.191320 trunc from 0x02c427 to 0x02e8a4 (0x247e bytes) 000761 1789017822.233705 punch from 0x0212ca to 0x02e8a4 (0xd5db bytes) 000762 1789017822.239670 fallocate 0x0371bd thru 0x03ffff (0x8e43 bytes) 000763 1789017822.246820 mapwrite 0x0396fe thru 0x03ffff (0x6902 bytes) 000765 1789017822.312953 read 0x01e4a0 thru 0x023d36 (0x5897 bytes) 000766 1789017822.344726 mapread 0x02a75e thru 0x03a2e0 (0xfb83 bytes) 000767 1789017822.345796 punch from 0x0259b7 to 0x02b0f0 (0x573a bytes) 000768 1789017822.364871 write 0x0028e1 thru 0x003317 (0xa37 bytes) 000769 1789017822.368954 fallocate 0x017de8 thru 0x024811 (0xca2a bytes) 000771 1789017822.397704 read 0x0075ef thru 0x00e560 (0x6f72 bytes) 000772 1789017822.402341 write 0x021e04 thru 0x023bc0 (0x1dbd bytes) 000773 1789017822.404702 read 0x007620 thru 0x00a94f (0x3330 bytes) 000774 1789017822.405731 punch from 0x0310ec to 0x0359a0 (0x48b5 bytes) 000775 1789017822.413467 mapwrite 0x038ad4 thru 0x03ffff (0x752c bytes) 000776 1789017822.458852 mapwrite 0x010f13 thru 0x020c1a (0xfd08 bytes) 000777 1789017822.492283 read 0x01a9c4 thru 0x02218b (0x77c8 bytes) 000778 1789017822.493754 mapwrite 0x02ff48 thru 0x03955b (0x9614 bytes) 000779 1789017822.532082 mapread 0x02b71f thru 0x02e645 (0x2f27 bytes) 000780 1789017822.533701 mapwrite 0x0385da thru 0x03a92f (0x2356 bytes) 000781 1789017822.559066 mapread 0x03c7b7 thru 0x03ffff (0x3849 bytes) 000782 1789017822.560815 write 0x037674 thru 0x03ca9b (0x5428 bytes) 000784 1789017822.562426 mapwrite 0x029295 thru 0x0381e2 (0xef4e bytes) 000785 1789017822.595094 read 0x00dd36 thru 0x0195e5 (0xb8b0 bytes) 000786 1789017822.596091 trunc from 0x040000 to 0x021fe8 (0x1e017 bytes) 000787 1789017822.634549 fallocate 0x02dbe9 thru 0x030f50 (0x3368 bytes) 000788 1789017822.661927 write 0x00f0b0 thru 0x019618 (0xa569 bytes) 000790 1789017822.666217 mapread 0x02eaee thru 0x030f50 (0x2463 bytes) 000792 1789017822.686860 punch from 0x01ec24 to 0x023fc1 (0x539e bytes) 000793 1789017822.692217 fallocate 0x03a5b7 thru 0x03ffff (0x5a49 bytes) 000795 1789017822.706900 trunc from 0x02d40d to 0x030f50 (0x3b44 bytes) 000798 1789017822.727068 trunc from 0x030f51 to 0x016475 (0x1aadb bytes) 000799 1789017822.749772 mapread 0x0163a2 thru 0x016475 (0xd4 bytes) 000800 1789017822.751026 punch from 0x00c245 to 0x0117ec (0x55a8 bytes) 000801 1789017822.785900 trunc from 0x00120f to 0x00b092 (0x9e84 bytes) 000802 1789017822.807469 read 0x002e5b thru 0x00770f (0x48b5 bytes) 000803 1789017822.830095 mapread 0x0030d8 thru 0x00ec5a (0xbb83 bytes) 000804 1789017822.836871 mapwrite 0x012cf4 thru 0x01f998 (0xcca5 bytes) 000805 1789017822.945153 read 0x003e34 thru 0x00b7d2 (0x799f bytes) 000806 1789017822.947142 read 0x006a29 thru 0x00d2e5 (0x68bd bytes) 000807 1789017822.950045 trunc from 0x01f999 to 0x03d95b (0x1dfc3 bytes) 000808 1789017822.987783 MIRROR_EXTEND to 2 mirrors 000809 1789017824.105961 write 0x028925 thru 0x0361d6 (0xd8b2 bytes) 000811 1789017824.195543 write 0x03b2ce thru 0x03ffff (0x4d32 bytes) 000812 1789017824.214215 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x259737f8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x56882077 Mirror 3: 0x56882077 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000813 1789017826.010795 trunc from 0x02d27c to 0x02f0d6 (0x1e5b bytes) 000814 1789017826.077300 mapread 0x0221af thru 0x030ce1 (0xeb33 bytes) 000815 1789017826.124011 punch from 0x036370 to 0x03ffff (0x9c90 bytes) 000816 1789017826.132422 write 0x01cfba thru 0x02c8c7 (0xf90e bytes) 000818 1789017826.148986 fallocate 0x03c28d thru 0x03ffff (0x3d73 bytes) 000819 1789017826.154804 punch from 0x0242b9 to 0x02d60b (0x9353 bytes) 000820 1789017826.170983 fallocate 0x011e30 thru 0x01bde6 (0x9fb7 bytes) 000821 1789017826.176018 punch from 0x01bc65 to 0x029eda (0xe276 bytes) 000823 1789017826.183211 fallocate 0x0197bf thru 0x01c031 (0x2873 bytes) 000824 1789017826.189898 write 0x001602 thru 0x00c1b7 (0xabb6 bytes) 000825 1789017826.205187 trunc from 0x040000 to 0x0016b0 (0x3e94f bytes) 000826 1789017826.231444 fallocate 0x0228d3 thru 0x031cd7 (0xf405 bytes) 000827 1789017826.243720 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16b1): Mirror 2: 0x65e022a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16b1): Mirror 2: 0x65e022a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16b1: skipped 000828 1789017826.599925 trunc from 0x0016b1 to 0x01d2f1 (0x1bc41 bytes) 000829 1789017826.621055 write 0x00de47 thru 0x01c084 (0xe23e bytes) 000830 1789017826.631862 punch from 0x00b9fa to 0x0101d4 (0x47db bytes) 000831 1789017826.647050 fallocate 0x03b8b5 thru 0x03ffff (0x474b bytes) 000832 1789017826.651948 punch from 0x01a9ad to 0x01d2f1 (0x2945 bytes) 000833 1789017826.657217 punch from 0x006386 to 0x009845 (0x34c0 bytes) 000835 1789017826.666592 write 0x01ad78 thru 0x028dc6 (0xe04f bytes) 000836 1789017826.673682 trunc from 0x0187c8 to 0x028258 (0xfa91 bytes) 000837 1789017826.687454 write 0x017888 thru 0x01b18c (0x3905 bytes) 000838 1789017826.700543 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28dc7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28dc7): Mirror 2: 0x7d13ccfa (stale) CRC-32 checksum value for chunk [0, 0x28dc7): Mirror 1: 0x25cb405f Mirror 3: 0x25cb405f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28dc7: skipped 000839 1789017827.971319 mapread 0x015b94 thru 0x01b924 (0x5d91 bytes) 000840 1789017828.015161 read 0x0027ad thru 0x003799 (0xfed bytes) 000841 1789017828.025043 fallocate 0x023b2c thru 0x02d588 (0x9a5d bytes) 000842 1789017828.072789 read 0x028166 thru 0x02b2d2 (0x316d bytes) 000843 1789017828.088609 read 0x022972 thru 0x02b474 (0x8b03 bytes) 000844 1789017828.089543 write 0x0240ee thru 0x032f6b (0xee7e bytes) 000845 1789017828.107135 mapwrite 0x00d6dc thru 0x010994 (0x32b9 bytes) 000846 1789017828.130554 fallocate 0x015c2b thru 0x0253b2 (0xf788 bytes) 000847 1789017828.138836 trunc from 0x032f6c to 0x00024e (0x32d1d bytes) 000848 1789017828.158630 read 0x000094 thru 0x00024e (0x1bb bytes) 000850 1789017828.159866 punch from 0x000139 to 0x00024e (0x116 bytes) 000851 1789017828.166721 punch from 0x0000bf to 0x00024e (0x190 bytes) 000852 1789017828.173442 write 0x00e7f5 thru 0x0102b7 (0x1ac3 bytes) 000853 1789017828.174868 write 0x01912b thru 0x01c757 (0x362d bytes) 000854 1789017828.177117 trunc from 0x01c758 to 0x0114ec (0xb26b bytes) 000855 1789017828.204002 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x114ed): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x114ed): Mirror 2: 0x57d3fc43 (stale) Mirror 3: 0xc20acc56 (stale) CRC-32 checksum value for chunk [0, 0x114ed): Mirror 1: 0x2dba657f Mirror 4: 0x2dba657f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x114ed: skipped 000856 1789017829.283468 mapwrite 0x02223d thru 0x02431c (0x20e0 bytes) 000857 1789017829.587836 punch from 0x01874d to 0x01acb9 (0x256d bytes) 000859 1789017829.601374 write 0x03d47f thru 0x03ffff (0x2b81 bytes) 000860 1789017829.605153 punch from 0x03701b to 0x03b7b1 (0x4797 bytes) 000861 1789017829.624384 read 0x014703 thru 0x01b1fa (0x6af8 bytes) 000862 1789017829.648196 trunc from 0x013aed to 0x01ced1 (0x93e5 bytes) 000863 1789017829.664449 mapread 0x00042b thru 0x0005ad (0x183 bytes) 000864 1789017829.698222 mapread 0x036c7b thru 0x03ffff (0x9385 bytes) 000865 1789017829.721405 write 0x01d8e8 thru 0x01f043 (0x175c bytes) 000866 1789017829.722488 mapwrite 0x0033d6 thru 0x00ce63 (0x9a8e bytes) 000867 1789017829.773066 trunc from 0x040000 to 0x035616 (0xa9e9 bytes) 000868 1789017829.820867 mapread 0x00860c thru 0x011744 (0x9139 bytes) 000869 1789017829.852167 punch from 0x03181e to 0x0340da (0x28bd bytes) 000871 1789017829.867434 trunc from 0x035617 to 0x025252 (0x103c4 bytes) 000872 1789017829.897994 mapwrite 0x0035d6 thru 0x008001 (0x4a2c bytes) 000873 1789017829.941704 read 0x01d139 thru 0x025252 (0x811a bytes) 000874 1789017829.974587 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25253): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25253): Mirror 2: 0x700e37b5 (stale) Mirror 3: 0xc3f08309 (stale) Mirror 4: 0x15252c46 (stale) CRC-32 checksum value for chunk [0, 0x25253): Mirror 1: 0x2b928ee6 Mirror 5: 0x2b928ee6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25253: skipped 000875 1789017831.632182 trunc from 0x025253 to 0x00a29f (0x1afb3 bytes) 000876 1789017831.727173 MIRROR_RESYNC 5 mirrors 000877 1789017832.882886 MIRROR_RESYNC 5 mirrors 000878 1789017833.421305 fallocate 0x010a1a thru 0x01ab7b (0xa162 bytes) 000880 1789017833.475126 write 0x03300a thru 0x033250 (0x247 bytes) 000882 1789017833.483449 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33251): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33251): Mirror 2: 0xe0364e41 (stale) Mirror 3: 0xe0364e41 (stale) Mirror 4: 0xe0364e41 (stale) Mirror 5: 0xe0364e41 (stale) CRC-32 checksum value for chunk [0, 0x33251): Mirror 1: 0x19c9b2ea Mirror 6: 0x19c9b2ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33251: skipped 000883 1789017835.166528 mapread 0x00f9e0 thru 0x0186f6 (0x8d17 bytes) 000884 1789017835.213975 write 0x011894 thru 0x0132e8 (0x1a55 bytes) 000885 1789017835.270479 trunc from 0x0280dd to 0x02e2ab (0x61cf bytes) 000886 1789017835.279921 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33251): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33251): Mirror 2: 0xe0364e41 (stale) Mirror 3: 0xe0364e41 (stale) Mirror 4: 0xe0364e41 (stale) Mirror 5: 0xe0364e41 (stale) Mirror 6: 0x19c9b2ea (stale) CRC-32 checksum value for chunk [0, 0x33251): Mirror 1: 0x56203d7e Mirror 7: 0x56203d7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33251: skipped 000888 1789017836.445982 mapread 0x013964 thru 0x01dae9 (0xa186 bytes) 000889 1789017836.487864 fallocate 0x021327 thru 0x026b44 (0x581e bytes) 000890 1789017836.539958 trunc from 0x01e71b to 0x022ce5 (0x45cb bytes) 000891 1789017836.550574 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33251): Mirror 2: 0xe0364e41 (stale) Mirror 3: 0xe0364e41 (stale) Mirror 4: 0xe0364e41 (stale) Mirror 5: 0xe0364e41 (stale) Mirror 7: 0x56203d7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33251): Mirror 2: 0xe0364e41 (stale) Mirror 3: 0xe0364e41 (stale) Mirror 4: 0xe0364e41 (stale) Mirror 5: 0xe0364e41 (stale) Mirror 7: 0x56203d7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33251: skipped 000892 1789017837.136466 trunc from 0x0295e7 to 0x03019b (0x6bb5 bytes) 000893 1789017837.142253 trunc from 0x033251 to 0x039b4e (0x68fe bytes) 000894 1789017837.168913 read 0x02474c thru 0x02c008 (0x78bd bytes) 000895 1789017837.181664 trunc from 0x039b4f to 0x03a249 (0x6fb bytes) 000896 1789017837.204568 write 0x02bff7 thru 0x035573 (0x957d bytes) 000897 1789017837.217277 mapwrite 0x01f905 thru 0x022317 (0x2a13 bytes) 000899 1789017837.252324 trunc from 0x03a24a to 0x025c9e (0x145ab bytes) 000900 1789017837.277909 read 0x0092d7 thru 0x00fbf1 (0x691b bytes) 000901 1789017837.290411 mapread 0x01ffd9 thru 0x01ffdc (0x4 bytes) 000902 1789017837.291978 mapread 0x021e39 thru 0x025c9e (0x3e66 bytes) 000903 1789017837.294110 mapwrite 0x03b453 thru 0x03ffff (0x4bad bytes) 000904 1789017837.345803 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7023a70a (stale) Mirror 3: 0x7023a70a (stale) Mirror 4: 0x7023a70a (stale) Mirror 5: 0x7023a70a (stale) Mirror 7: 0x20d55136 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3f5fc27d Mirror 8: 0x3f5fc27d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000905 1789017838.843999 fallocate 0x0095fb thru 0x015126 (0xbb2c bytes) 000906 1789017838.895975 read 0x010b77 thru 0x016b19 (0x5fa3 bytes) 000907 1789017838.906422 read 0x0333c5 thru 0x03e099 (0xacd5 bytes) 000908 1789017838.918410 punch from 0x0006d1 to 0x00fc41 (0xf571 bytes) 000909 1789017838.923584 write 0x02ad82 thru 0x031331 (0x65b0 bytes) 000910 1789017838.929054 punch from 0x03fd8c to 0x03ffff (0x274 bytes) 000911 1789017838.936343 trunc from 0x00772c to 0x01274c (0xb021 bytes) 000912 1789017838.941707 read 0x01ace5 thru 0x027bb6 (0xced2 bytes) 000914 1789017838.943813 trunc from 0x040000 to 0x008df0 (0x3720f bytes) 000915 1789017838.992702 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x8df1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8df1): Mirror 2: 0x13ac5d15 (stale) Mirror 3: 0x13ac5d15 (stale) Mirror 4: 0x13ac5d15 (stale) Mirror 5: 0x13ac5d15 (stale) Mirror 7: 0x13ac5d15 (stale) Mirror 8: 0x13ac5d15 (stale) CRC-32 checksum value for chunk [0, 0x8df1): Mirror 1: 0x60895b5b Mirror 9: 0x60895b5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8df1: skipped 000916 1789017840.191674 trunc from 0x0040b9 to 0x006bcd (0x2b15 bytes) 000917 1789017840.265881 fallocate 0x030658 thru 0x03527e (0x4c27 bytes) 000918 1789017840.273318 trunc from 0x008df1 to 0x021d78 (0x18f88 bytes) 000919 1789017840.327399 trunc from 0x021d79 to 0x033bb7 (0x11e3f bytes) 000920 1789017840.376569 trunc from 0x033bb8 to 0x03f90b (0xbd54 bytes) 000921 1789017840.396685 write 0x02a0ed thru 0x02a72c (0x640 bytes) 000922 1789017840.402193 fallocate 0x039410 thru 0x03fbc6 (0x67b7 bytes) 000923 1789017840.407491 punch from 0x005f09 to 0x00fd39 (0x9e31 bytes) 000924 1789017840.412759 trunc from 0x00649c to 0x009fe5 (0x3b4a bytes) 000925 1789017840.417917 write 0x024e2a thru 0x02a74b (0x5922 bytes) 000926 1789017840.427898 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3fbc7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbc7): Mirror 2: 0x6162d974 (stale) Mirror 3: 0x6162d974 (stale) Mirror 4: 0x6162d974 (stale) Mirror 5: 0x6162d974 (stale) Mirror 7: 0xf3ea351d (stale) Mirror 8: 0x643b2283 (stale) Mirror 9: 0xb5339f6 (stale) CRC-32 checksum value for chunk [0, 0x3fbc7): Mirror 1: 0x42bb23b Mirror 10: 0x42bb23b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbc7: skipped 000927 1789017841.628270 fallocate 0x00e240 thru 0x016b37 (0x88f8 bytes) 000928 1789017841.687156 trunc from 0x002327 to 0x005347 (0x3021 bytes) 000929 1789017841.697078 trunc from 0x03fbc7 to 0x005deb (0x39ddb bytes) 000931 1789017841.727479 mapwrite 0x021fe7 thru 0x02bb01 (0x9b1b bytes) 000932 1789017841.795340 mapread 0x00f06b thru 0x016e9b (0x7e31 bytes) 000933 1789017841.805278 fallocate 0x015990 thru 0x01af84 (0x55f5 bytes) 000934 1789017841.809867 read 0x02048f thru 0x02bb01 (0xb673 bytes) 000936 1789017841.819911 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bb02): Mirror 2: 0x755b9d91 (stale) Mirror 3: 0x755b9d91 (stale) Mirror 4: 0x755b9d91 (stale) Mirror 7: 0xba80a116 (stale) Mirror 8: 0x55bbfe58 (stale) Mirror 9: 0xb3b884bf (stale) Mirror 10: 0x8f7e6b6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bb02): Mirror 2: 0x755b9d91 (stale) Mirror 3: 0x755b9d91 (stale) Mirror 4: 0x755b9d91 (stale) Mirror 7: 0xba80a116 (stale) Mirror 8: 0x55bbfe58 (stale) Mirror 9: 0xb3b884bf (stale) Mirror 10: 0x8f7e6b6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bb02: skipped 000937 1789017842.285065 fallocate 0x018cad thru 0x027c21 (0xef75 bytes) 000938 1789017842.290387 fallocate 0x02d7bf thru 0x03287b (0x50bd bytes) 000939 1789017842.295781 read 0x015a63 thru 0x023a7f (0xe01d bytes) 000940 1789017842.305878 trunc from 0x03287c to 0x020891 (0x11fea bytes) 000941 1789017842.335062 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20892): Mirror 3: 0x70c8fb94 (stale) Mirror 4: 0x70c8fb94 (stale) Mirror 7: 0xefae825f (stale) Mirror 8: 0xa9275976 (stale) Mirror 9: 0x3fac98d6 (stale) Mirror 10: 0x3fac98d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20892): Mirror 3: 0x70c8fb94 (stale) Mirror 4: 0x70c8fb94 (stale) Mirror 7: 0xefae825f (stale) Mirror 8: 0xa9275976 (stale) Mirror 9: 0x3fac98d6 (stale) Mirror 10: 0x3fac98d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20892: skipped 000942 1789017842.745794 trunc from 0x020892 to 0x00bc2e (0x14c63 bytes) 000943 1789017842.771287 write 0x01f75b thru 0x0254b9 (0x5d5f bytes) 000944 1789017842.776773 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x254ba): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x254ba): Mirror 3: 0x1c03e1d2 (stale) Mirror 4: 0x1c03e1d2 (stale) Mirror 7: 0xe807a812 (stale) Mirror 8: 0x57041b70 (stale) Mirror 9: 0x1db95d0d (stale) Mirror 10: 0x2a4c015b (stale) CRC-32 checksum value for chunk [0, 0x254ba): Mirror 1: 0xf3341250 Mirror 11: 0xf3341250 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x254ba: skipped 000945 1789017843.712886 trunc from 0x0254ba to 0x00b96e (0x19b4b bytes) 000946 1789017843.814986 trunc from 0x0003c4 to 0x006219 (0x5e56 bytes) 000947 1789017843.847320 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb96f): Mirror 3: 0x66e54893 (stale) Mirror 4: 0x66e54893 (stale) Mirror 7: 0x66e54893 (stale) Mirror 8: 0x66e54893 (stale) Mirror 9: 0x38eaef74 (stale) Mirror 10: 0x38eaef74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb96f): Mirror 3: 0x66e54893 (stale) Mirror 4: 0x66e54893 (stale) Mirror 7: 0x66e54893 (stale) Mirror 8: 0x66e54893 (stale) Mirror 9: 0x38eaef74 (stale) Mirror 10: 0x38eaef74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb96f: skipped 000948 1789017844.415950 trunc from 0x00b96f to 0x0119af (0x6041 bytes) 000949 1789017844.441293 trunc from 0x005193 to 0x0119af (0xc81d bytes) 000950 1789017844.446994 trunc from 0x00a16a to 0x011497 (0x732e bytes) 000951 1789017844.453055 read 0x00cdeb thru 0x00ceed (0x103 bytes) 000952 1789017844.464029 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x119b0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x119b0): Mirror 3: 0xfbb82475 (stale) Mirror 4: 0xfbb82475 (stale) Mirror 7: 0xb4c17db7 (stale) Mirror 8: 0xb4c17db7 (stale) Mirror 9: 0x6de6b99 (stale) Mirror 10: 0x6de6b99 (stale) CRC-32 checksum value for chunk [0, 0x119b0): Mirror 1: 0x6de6b99 Mirror 11: 0x6de6b99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x119b0: skipped 000953 1789017845.314295 trunc from 0x00b7dc to 0x00d971 (0x2196 bytes) 000954 1789017845.395104 mapwrite 0x03075e thru 0x038795 (0x8038 bytes) 000955 1789017845.596723 write 0x000261 thru 0x00f5a2 (0xf342 bytes) 000956 1789017845.619626 fallocate 0x03b60f thru 0x03f086 (0x3a78 bytes) 000958 1789017845.636032 write 0x00d48e thru 0x014afc (0x766f bytes) 000959 1789017845.659326 mapwrite 0x00bafe thru 0x014d53 (0x9256 bytes) 000960 1789017845.702175 MIRROR_RESYNC 8 mirrors 000961 1789017847.507943 fallocate 0x018f09 thru 0x01df45 (0x503d bytes) 000962 1789017847.539556 trunc from 0x0155a4 to 0x01fb75 (0xa5d2 bytes) 000964 1789017847.544738 mapread 0x01cf37 thru 0x026dce (0x9e98 bytes) 000965 1789017847.564295 mapread 0x002c24 thru 0x00c88f (0x9c6c bytes) 000966 1789017847.574726 read 0x0376d8 thru 0x03bb12 (0x443b bytes) 000967 1789017847.583177 write 0x02c138 thru 0x02ffcf (0x3e98 bytes) 000968 1789017847.592650 mapwrite 0x03ae95 thru 0x03da3f (0x2bab bytes) 000969 1789017847.620172 trunc from 0x00606b to 0x0131c4 (0xd15a bytes) 000970 1789017847.627880 trunc from 0x01c964 to 0x022fcb (0x6668 bytes) 000971 1789017847.633328 punch from 0x014f4c to 0x022fb2 (0xe067 bytes) 000972 1789017847.645607 write 0x022645 thru 0x0237fa (0x11b6 bytes) 000973 1789017847.652690 mapread 0x036c1c thru 0x03963b (0x2a20 bytes) 000975 1789017847.660070 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f087): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f087): Mirror 3: 0x4effd9d0 (stale) Mirror 4: 0x4effd9d0 (stale) Mirror 7: 0x4effd9d0 (stale) Mirror 8: 0x4effd9d0 (stale) Mirror 9: 0x4effd9d0 (stale) Mirror 10: 0x4effd9d0 (stale) Mirror 11: 0x4effd9d0 (stale) CRC-32 checksum value for chunk [0, 0x3f087): Mirror 1: 0x7e2ef483 Mirror 12: 0x7e2ef483 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f087: skipped 000976 1789017848.747017 punch from 0x011cfc to 0x01c7a3 (0xaaa8 bytes) 000977 1789017848.795394 mapread 0x0397fc thru 0x03c3f4 (0x2bf9 bytes) 000978 1789017848.814036 mapwrite 0x009297 thru 0x00db6a (0x48d4 bytes) 000980 1789017848.845059 fallocate 0x02cb96 thru 0x02d54c (0x9b7 bytes) 000981 1789017848.850242 mapread 0x01a31c thru 0x028c92 (0xe977 bytes) 000982 1789017848.872890 write 0x01a8a8 thru 0x01c755 (0x1eae bytes) 000983 1789017848.878410 trunc from 0x03f087 to 0x03b2c7 (0x3dbf bytes) 000984 1789017848.901990 read 0x038a4a thru 0x03b2c7 (0x287e bytes) 000985 1789017848.902747 mapread 0x00339d thru 0x005d4c (0x29b0 bytes) 000986 1789017848.903818 punch from 0x0123e3 to 0x013125 (0xd43 bytes) 000987 1789017848.910311 mapread 0x00e279 thru 0x016132 (0x7eba bytes) 000988 1789017848.916422 mapwrite 0x02d6ee thru 0x02f8c4 (0x21d7 bytes) 000989 1789017848.939717 mapread 0x0256de thru 0x026748 (0x106b bytes) 000990 1789017848.940606 trunc from 0x03b2c8 to 0x004ddf (0x364e8 bytes) 000991 1789017848.960890 mapwrite 0x038fc6 thru 0x03afd9 (0x2014 bytes) 000992 1789017849.002288 trunc from 0x03afda to 0x01805a (0x22f7f bytes) 000993 1789017849.022736 read 0x0068ef thru 0x0083a0 (0x1ab2 bytes) 000994 1789017849.047340 mapwrite 0x026345 thru 0x02c536 (0x61f2 bytes) 000995 1789017849.109329 fallocate 0x0093aa thru 0x01186d (0x84c4 bytes) 000996 1789017849.124006 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c537): Mirror 3: 0xd25ced95 (stale) Mirror 4: 0xd25ced95 (stale) Mirror 8: 0xd25ced95 (stale) Mirror 9: 0xd25ced95 (stale) Mirror 10: 0xd25ced95 (stale) Mirror 11: 0xd25ced95 (stale) Mirror 12: 0xad70d5bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c537): Mirror 3: 0xd25ced95 (stale) Mirror 4: 0xd25ced95 (stale) Mirror 8: 0xd25ced95 (stale) Mirror 9: 0xd25ced95 (stale) Mirror 10: 0xd25ced95 (stale) Mirror 11: 0xd25ced95 (stale) Mirror 12: 0xad70d5bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c537: skipped 000997 1789017849.734497 punch from 0x0229ab to 0x022b10 (0x166 bytes) 000998 1789017849.739453 mapwrite 0x00923c thru 0x015f61 (0xcd26 bytes) 000999 1789017849.805916 mapwrite 0x0329cf thru 0x03ffff (0xd631 bytes) 001000 1789017849.926243 trunc from 0x023b3f to 0x026d75 (0x3237 bytes) All operations completed A-OK! PASS 70a (90s) == sanity-flr test 71: check mirror extend parent fid ==== 01:24:16 (1789017856) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x144:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 01:24:28 (1789017868) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165006 s, 12.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967663 s, 10.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 01:24:38 (1789017878) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3058688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3650400 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5627808 .. Extending file size to 6680800 .. Extending file size to 7029088 .. Extending file size to 6009920 .. Extending file size to 8258048 .. Extending file size to 8222112 .. Extending file size to 8666016 .. Extending file size to 8282688 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9008192 .. Extending file size to 8716128 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 61195 61196 61197 61200 61201 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:26:07 (1789017967) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1135520 .. Extending file size to 2345312 .. done Extending file size to 5511712 .. Extending file size to 5914944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9237024 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6510336 .. Extending file size to 7641440 .. Extending file size to 8465984 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8967680 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62636 62638 62639 62640 62641 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:27:31 (1789018051) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2340 1268964 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24792 3582228 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18952 3588068 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43744 7170296 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2340 1268964 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24792 3582228 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18952 3588068 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43744 7170296 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0769121 s, 27.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263641 s, 8.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x93a91f5 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:27:50 (1789018070) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0990303 s, 21.2 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 01:27:57 (1789018077) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:28:05 (1789018085) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0841356 s, 24.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:28:13 (1789018093) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175235 s, 12.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:28:24 (1789018104) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160715 s, 13.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:29:02 (1789018142) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114961 s, 9.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ad:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:29:58 (1789018198) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173284 s, 12.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1af:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1af:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:30:54 (1789018254) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg222-server: Pool lustre.test_204f created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179056 s, 11.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg222-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:32:15 (1789018335) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:32:26 (1789018346) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:32:36 (1789018356) oleg222-server: Pool lustre.test_206 created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg222-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:33:07 (1789018387) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132034 s, 7.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603942 s, 17.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd14073a7 Mirror 3: 0xd14073a7 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:33:22 (1789018402) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:33:28 (1789018408) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:33:31 (1789018411) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0726526 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x11c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xda0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xda0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xda0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f50000 Read this (4 bytes): BBBB 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:34:13 (1789018453) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0165931 s, 494 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:34:22 (1789018462) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846488 s, 12.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb7 lmm_fid: [0x200000405:0xb7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb7 lmm_fid: [0x200000405:0xb7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611319 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 01:34:24.000000000 -0400 Modify: 2026-09-10 01:34:24.000000000 -0400 Change: 2026-09-10 01:34:26.000000000 -0400 Birth: 2026-09-10 01:34:24.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:34:33 (1789018473) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640368 s, 16.4 MB/s oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg222-server: O/280000401/d26/15a: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:34:55 (1789018495) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.201526 s, 203 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0503309 s, 81.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 10 01:34 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 10 01:34 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:35:05 (1789018505) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:35:15 (1789018515) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2026 sec =========== 01:35:27 (1789018527) === sanity-flr: start cleanup 01:35:29 (1789018529) === === sanity-flr: finish cleanup 01:35:38 (1789018538) ===