-----============= acceptance-small: sanity-flr ============----- Wed Sep 16 19:33:17 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg234-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:33:41 (1789601621) === oleg234-client.virtnet: executing check_config_client /mnt/lustre oleg234-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg234-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e1487e7f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1487e7f800.idle_timeout=debug disable quota as required oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 19:34:31 (1789601671) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8256] == sanity-flr test 0a: lfs mirror create with -N option == 19:34:34 (1789601674) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:34:55 (1789601695) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:35:00 (1789601700) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:35:03 (1789601703) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:35:32 (1789601732) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:35:36 (1789601736) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:35:40 (1789601740) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e1487e7f800.stats=clear osc.lustre-OST0001-osc-ffff9e1487e7f800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:35:49 (1789601749) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:36:04 (1789601764) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:36:15 (1789601775) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:36:19 (1789601779) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:36:28 (1789601788) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 19:36:39 (1789601799) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:36:47 (1789601807) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:36:55 (1789601815) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:37:05 (1789601825) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:37:18 (1789601838) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:37:25 (1789601845) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.204411 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656488 s, 16.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:37:34 (1789601854) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.112703 s, 27.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:37:44 (1789601864) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:37:55 (1789601875) Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg234-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (147s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:40:21 (1789602021) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.757605 s, 13.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (50s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:41:11 (1789602071) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:42:43 (1789602163) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.46402 s, 21.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 16 19:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.984s user 0m0.000s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1487e7f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.355s user 0m0.001s sys 0m0.184s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.240s user 0m0.002s sys 0m0.188s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1487e7f800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:44:19 (1789602259) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (6s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:44:25 (1789602265) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:44:30 (1789602270) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 19:44:34 (1789602274) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 19:44:45 (1789602285) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.750078 s, 16.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:44:58 (1789602298) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056423 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234363 s, 44.7 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:45:07 (1789602307) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:45:16 (1789602316) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00150261 s, 681 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00430039 s, 238 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000817262 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00929394 s, 110 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 19:45:29 (1789602329) create test files with size 15744307 .. 53e8a667a8e05363bba622966b0fcb4f - 77d0520d9c6b9b94fa17523871adb99b - c6e6dfc96c4f5e23490975276085e786 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 19:45:56 (1789602356) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:46:12 (1789602372) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:46:21 (1789602381) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115849 s, 9.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637188 s, 16.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 19:46:33 (1789602393) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160934 s, 13.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.442899 s, 9.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254498 s, 8.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 19:46:54 (1789602414) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 19:46:58 (1789602418) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:47:02 (1789602422) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599852 s, 17.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (12s) == sanity-flr test 44a: lfs mirror split check =========== 19:47:14 (1789602434) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.288035 s, 10.9 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:47:26 (1789602446) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:47:39 (1789602459) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.35977 s, 29.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:47:54 (1789602474) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9e1485637800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.481744 s, 21.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9e148569e800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:48:13 (1789602493) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:49:03 (1789602543) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (52s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:49:55 (1789602595) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 19:50:00 (1789602600) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.290725 s, 10.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187717 s, 16.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xebad5e55 (nosync,stale) Mirror 2: 0xebad5e55 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3710502ab65d92077c82b1be126fe6b5 - mirror 1 checksum: 8f474e7569acb6059088030be66e2bbb - mirror 2 checksum: 3710502ab65d92077c82b1be126fe6b5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T19:50:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T19:50:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3710502ab65d92077c82b1be126fe6b5 - mirror 1 checksum: 8f474e7569acb6059088030be66e2bbb - mirror 2 checksum: 8f474e7569acb6059088030be66e2bbb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T19:50:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T19:50:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 19:50:12 (1789602612) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255577 s, 41.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:50:21 (1789602621) Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 19:50:35 (1789602635) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 19:51:08 (1789602668) targets are mounted 19:51:08 (1789602668) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313995 s, 33.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:51:29 (1789602689) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0683894 s, 299 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0348932 s, 587 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:51:42 (1789602702) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0823003 s, 249 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0118111 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:51:55 (1789602715) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0104363 s, 1.6 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:52:04 (1789602724) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.113589 s, 180 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (11s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:52:15 (1789602735) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181353 s, 226 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0798011 s, 51.3 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:52:25 (1789602745) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (34s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:53:00 (1789602780) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:53:33 (1789602813) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (28s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:54:01 (1789602841) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:54:28 (1789602868) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (12s) == sanity-flr test 70: mirror create and split race ====== 19:54:40 (1789602880) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57639 mirror split pid 57640 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7851: 57639 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7851: 57640 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 19:55:49 (1789602949) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 2952 truncating to largest ever: 0x2b59e 000001 1789602952.756180 trunc from 00000000 to 0x02b59d (0x2b59e bytes) fallocating to largest ever: 0x3466b 000002 1789602952.777969 fallocate 0x02a170 thru 0x03466a (0xa4fb bytes) 000003 1789602952.784684 fallocate 0x01e0be thru 0x02c5ad (0xe4f0 bytes) 000004 1789602952.792370 trunc from 0x005f51 to 0x012ff4 (0xd0a4 bytes) 000005 1789602952.797695 trunc from 0x03466b to 0x020c96 (0x139d4 bytes) 000007 1789602952.829888 punch from 0x0188ba to 0x01ccb4 (0x43fb bytes) 000008 1789602952.842171 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x399a4 000009 1789602953.702470 trunc from 0x020c97 to 0x0399a3 (0x18d0d bytes) 000010 1789602953.815134 mapread 0x00a2b9 thru 0x00d0c6 (0x2e0e bytes) 000011 1789602953.900187 write 0x03043b thru 0x0351c8 (0x4d8e bytes) 000012 1789602953.941153 mapread 0x0031b6 thru 0x0081a4 (0x4fef bytes) 000013 1789602953.956340 mapread 0x00384c thru 0x00c01f (0x87d4 bytes) 000014 1789602953.958228 fallocate 0x01c8fe thru 0x02c257 (0xf95a bytes) 000015 1789602953.972645 trunc from 0x006da8 to 0x015c62 (0xeebb bytes) 000016 1789602953.997466 read 0x025fe0 thru 0x02acfd (0x4d1e bytes) 000017 1789602954.062653 MIRROR_RESYNC 2 mirrors 000018 1789602955.080961 mapread 0x001ad4 thru 0x007d73 (0x62a0 bytes) 000020 1789602955.102985 write 0x03bfe3 thru 0x03ffff (0x401d bytes) 000021 1789602955.142381 read 0x03f348 thru 0x03ffff (0xcb8 bytes) 000022 1789602955.143469 trunc from 0x01e60c to 0x02a394 (0xbd89 bytes) 000023 1789602955.154563 punch from 0x027e84 to 0x031930 (0x9aad bytes) 000025 1789602955.163228 mapread 0x00541c thru 0x012c79 (0xd85e bytes) 000026 1789602955.194158 trunc from 0x040000 to 0x014e1c (0x2b1e3 bytes) 000027 1789602955.225173 mapread 0x008fce thru 0x01318c (0xa1bf bytes) 000028 1789602955.226578 fallocate 0x020d5f thru 0x0281ec (0x748e bytes) 000029 1789602955.234178 punch from 0x009927 to 0x00f9a7 (0x6081 bytes) 000030 1789602955.241010 fallocate 0x0053f8 thru 0x0129b9 (0xd5c2 bytes) 000031 1789602955.248542 punch from 0x0118bf to 0x014e1c (0x355e bytes) 000032 1789602955.256130 write 0x00b7d3 thru 0x01847f (0xccad bytes) 000034 1789602955.266978 write 0x01d542 thru 0x02a980 (0xd43f bytes) 000035 1789602955.271336 read 0x0207f3 thru 0x02a2be (0x9acc bytes) 000036 1789602955.272562 mapwrite 0x019792 thru 0x0203b6 (0x6c25 bytes) 000037 1789602955.334680 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000038 1789602955.603373 trunc from 0x02a981 to 0x01a0f3 (0x1088d bytes) 000039 1789602955.625517 write 0x01e8be thru 0x02a5ef (0xbd32 bytes) 000040 1789602955.627447 trunc from 0x004d7f to 0x0090aa (0x432c bytes) 000041 1789602955.632459 mapread 0x00bf49 thru 0x013649 (0x7701 bytes) 000043 1789602955.657885 trunc from 0x0182b5 to 0x01aea2 (0x2bee bytes) 000045 1789602955.665499 read 0x00c5e0 thru 0x017363 (0xad84 bytes) 000046 1789602955.672555 fallocate 0x01ce66 thru 0x024270 (0x740b bytes) 000047 1789602955.680468 trunc from 0x007d9a to 0x00ef12 (0x7179 bytes) 000048 1789602955.687215 trunc from 0x00c4ed to 0x01a2a9 (0xddbd bytes) 000049 1789602955.694980 fallocate 0x01c979 thru 0x027af0 (0xb178 bytes) 000051 1789602955.704930 mapread 0x0143d4 thru 0x024287 (0xfeb4 bytes) 000052 1789602955.727297 read 0x01f72e thru 0x024840 (0x5113 bytes) 000053 1789602955.728395 trunc from 0x003ccd to 0x00ac68 (0x6f9c bytes) 000054 1789602955.735864 trunc from 0x023c08 to 0x02a5ef (0x69e8 bytes) 000055 1789602955.761076 punch from 0x0208f1 to 0x027a54 (0x7164 bytes) 000056 1789602955.770353 read 0x02975a thru 0x02a5ef (0xe96 bytes) 000057 1789602955.798247 read 0x0118ca thru 0x017fc5 (0x66fc bytes) 000061 1789602955.807953 mapread 0x029b5b thru 0x02a5ef (0xa95 bytes) 000062 1789602955.809062 read 0x010ad8 thru 0x018830 (0x7d59 bytes) 000063 1789602955.827490 MIRROR_EXTEND to 2 mirrors 000064 1789602956.921336 fallocate 0x01f8e0 thru 0x029f9f (0xa6c0 bytes) 000065 1789602956.952434 punch from 0x00c3dd to 0x017094 (0xacb8 bytes) 000066 1789602956.958324 mapread 0x0033f0 thru 0x00fa77 (0xc688 bytes) 000067 1789602956.982003 punch from 0x000bb2 to 0x0038fb (0x2d4a bytes) 000068 1789602956.987267 punch from 0x0259d5 to 0x028a5e (0x308a bytes) 000069 1789602956.993304 mapwrite 0x00c7fb thru 0x0113c7 (0x4bcd bytes) 000070 1789602957.020272 trunc from 0x02a5f0 to 0x02e56a (0x3f7b bytes) 000071 1789602957.046232 mapwrite 0x001945 thru 0x007065 (0x5721 bytes) 000072 1789602957.071467 mapread 0x01e532 thru 0x02d956 (0xf425 bytes) 000073 1789602957.095008 read 0x002a77 thru 0x002b80 (0x10a bytes) 000074 1789602957.095841 mapwrite 0x011a85 thru 0x0167b3 (0x4d2f bytes) 000075 1789602957.114225 punch from 0x00f436 to 0x013111 (0x3cdc bytes) 000076 1789602957.120327 read 0x0025f0 thru 0x00fe40 (0xd851 bytes) 000077 1789602957.133702 punch from 0x0053e5 to 0x0086bd (0x32d9 bytes) 000078 1789602957.140406 trunc from 0x010913 to 0x01c01d (0xb70b bytes) 000079 1789602957.146318 fallocate 0x01b1b5 thru 0x01bb0d (0x959 bytes) 000080 1789602957.152027 mapread 0x01807e thru 0x01bb4f (0x3ad2 bytes) 000081 1789602957.174276 trunc from 0x00dbda to 0x019c2f (0xc056 bytes) 000082 1789602957.180280 trunc from 0x02e56b to 0x02527b (0x92ef bytes) 000083 1789602957.201292 write 0x037804 thru 0x03ffff (0x87fc bytes) 000084 1789602957.204413 read 0x0147d0 thru 0x0149ca (0x1fb bytes) 000085 1789602957.216973 punch from 0x012cae to 0x01bbaa (0x8efd bytes) 000086 1789602957.222309 punch from 0x0286ba to 0x02f219 (0x6b60 bytes) 000087 1789602957.227896 mapwrite 0x00fd95 thru 0x018347 (0x85b3 bytes) 000088 1789602957.318550 trunc from 0x00c88c to 0x00f36d (0x2ae2 bytes) 000089 1789602957.337684 mapread 0x028e39 thru 0x02d4e4 (0x46ac bytes) 000090 1789602957.435350 MIRROR_RESYNC 2 mirrors 000091 1789602958.408587 mapread 0x01cdbe thru 0x02951e (0xc761 bytes) 000092 1789602958.437128 trunc from 0x040000 to 0x02e9bc (0x11643 bytes) 000093 1789602958.472625 trunc from 0x02e9bd to 0x000df6 (0x2dbc6 bytes) 000094 1789602958.504193 trunc from 0x000df7 to 0x036dca (0x35fd4 bytes) 000095 1789602958.525288 MIRROR_RESYNC 2 mirrors 000096 1789602959.080241 mapread 0x02f206 thru 0x034af0 (0x58eb bytes) 000097 1789602959.104882 mapread 0x022c4a thru 0x028b30 (0x5ee7 bytes) 000098 1789602959.116934 fallocate 0x022a47 thru 0x02d10a (0xa6c4 bytes) 000099 1789602959.154945 read 0x031a18 thru 0x033222 (0x180b bytes) 000101 1789602959.169453 fallocate 0x014a69 thru 0x02034b (0xb8e3 bytes) 000102 1789602959.180045 punch from 0x02c048 to 0x02cde2 (0xd9b bytes) 000103 1789602959.185404 trunc from 0x036dcb to 0x021206 (0x15bc4 bytes) 000104 1789602959.225985 mapwrite 0x023270 thru 0x02ff0d (0xcc9e bytes) 000105 1789602959.303381 trunc from 0x02ff0e to 0x01bdf4 (0x14119 bytes) 000106 1789602959.325148 trunc from 0x011598 to 0x01bdf4 (0xa85d bytes) 000108 1789602959.332882 trunc from 0x01aa64 to 0x01bdf4 (0x1391 bytes) 000109 1789602959.337784 write 0x02c1d0 thru 0x03384b (0x767c bytes) 000110 1789602959.339385 punch from 0x022966 to 0x032929 (0xffc4 bytes) 000111 1789602959.357034 trunc from 0x03384c to 0x018777 (0x1b0d4 bytes) 000112 1789602959.379392 mapwrite 0x03e591 thru 0x03f821 (0x1291 bytes) 000113 1789602959.428931 mapwrite 0x00babc thru 0x00e653 (0x2b98 bytes) 000114 1789602959.454355 mapread 0x030c00 thru 0x03dee9 (0xd2ea bytes) 000115 1789602959.466626 read 0x030f1b thru 0x034923 (0x3a09 bytes) 000116 1789602959.468247 read 0x006159 thru 0x00729a (0x1142 bytes) 000117 1789602959.469483 punch from 0x001b12 to 0x00fbfe (0xe0ed bytes) 000118 1789602959.475900 punch from 0x0288bc to 0x02bca9 (0x33ee bytes) 000119 1789602959.486477 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000120 1789602959.652225 mapread 0x02069f thru 0x021ded (0x174f bytes) 000121 1789602959.667420 write 0x0370ce thru 0x037c1d (0xb50 bytes) 000122 1789602959.678260 mapwrite 0x01fcf6 thru 0x02ebdb (0xeee6 bytes) 000123 1789602959.718085 trunc from 0x03c0ed to 0x03f821 (0x3735 bytes) 000125 1789602959.726005 mapwrite 0x0121f0 thru 0x01e3b0 (0xc1c1 bytes) 000126 1789602959.787406 mapread 0x019c5f thru 0x029ac8 (0xfe6a bytes) 000127 1789602959.790201 read 0x01aca3 thru 0x0293a0 (0xe6fe bytes) 000128 1789602959.791892 trunc from 0x032dde to 0x03dd33 (0xaf56 bytes) 000129 1789602959.797318 punch from 0x00506a to 0x00edf2 (0x9d89 bytes) 000130 1789602959.803765 write 0x02f843 thru 0x0347fe (0x4fbc bytes) 000131 1789602959.812110 fallocate 0x011632 thru 0x01db80 (0xc54f bytes) 000132 1789602959.820365 trunc from 0x03f822 to 0x0326a9 (0xd178 bytes) 000133 1789602959.848953 punch from 0x008a5d to 0x011886 (0x8e2a bytes) 000135 1789602959.854364 mapread 0x0146d2 thru 0x0235bc (0xeeeb bytes) 000137 1789602959.857844 read 0x0298d4 thru 0x0326a9 (0x8dd6 bytes) 000138 1789602959.858670 punch from 0x001e96 to 0x00720d (0x5378 bytes) 000140 1789602959.866033 trunc from 0x0326aa to 0x01d3bd (0x152ec bytes) 000141 1789602959.894086 mapread 0x005cd3 thru 0x00b04b (0x5379 bytes) 000142 1789602959.916947 write 0x025877 thru 0x0269d8 (0x1162 bytes) 000143 1789602959.919303 mapwrite 0x019c64 thru 0x02519d (0xb53a bytes) 000145 1789602959.950135 write 0x01f722 thru 0x0288d7 (0x91b6 bytes) 000146 1789602959.953068 mapwrite 0x02dd05 thru 0x03d448 (0xf744 bytes) 000148 1789602960.037592 mapread 0x02418e thru 0x02e499 (0xa30c bytes) 000150 1789602960.045206 read 0x00aa62 thru 0x01142d (0x69cc bytes) 000151 1789602960.046523 trunc from 0x02100e to 0x02da27 (0xca1a bytes) 000152 1789602960.057482 mapwrite 0x0374b5 thru 0x03a613 (0x315f bytes) 000153 1789602960.098195 fallocate 0x01dcea thru 0x02706a (0x9381 bytes) 000154 1789602960.113359 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x40000 000155 1789602961.066524 fallocate 0x03f7e8 thru 0x03ffff (0x818 bytes) 000156 1789602961.132599 write 0x0372b3 thru 0x03ffff (0x8d4d bytes) 000157 1789602961.149906 write 0x01ed0e thru 0x02789c (0x8b8f bytes) 000158 1789602961.177990 trunc from 0x040000 to 0x015fdf (0x2a020 bytes) 000159 1789602961.268473 trunc from 0x006f2f to 0x00a466 (0x3538 bytes) 000160 1789602961.285605 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000161 1789602961.576639 trunc from 0x015fe0 to 0x00b6c5 (0xa91a bytes) 000162 1789602961.610445 fallocate 0x000cb0 thru 0x009edf (0x9230 bytes) 000164 1789602961.618401 trunc from 0x00b6c6 to 0x008318 (0x33ad bytes) 000165 1789602961.662315 MIRROR_EXTEND to 2 mirrors 000166 1789602962.963161 punch from 0x00086a to 0x0037f4 (0x2f8b bytes) 000167 1789602963.008302 mapwrite 0x024652 thru 0x030fe1 (0xc990 bytes) 000168 1789602963.095643 write 0x01e5f5 thru 0x02a5b5 (0xbfc1 bytes) 000169 1789602963.107311 mapwrite 0x028398 thru 0x03558c (0xd1f5 bytes) 000170 1789602963.225865 read 0x007a65 thru 0x00c9c0 (0x4f5c bytes) 000171 1789602963.242492 punch from 0x02c528 to 0x030148 (0x3c21 bytes) 000172 1789602963.253385 mapread 0x0036ae thru 0x010f6f (0xd8c2 bytes) 000173 1789602963.280885 punch from 0x02d044 to 0x02d48f (0x44c bytes) 000174 1789602963.305953 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000175 1789602963.558665 read 0x00e153 thru 0x013da8 (0x5c56 bytes) 000177 1789602963.591219 MIRROR_EXTEND to 2 mirrors 000178 1789602964.448576 punch from 0x02a9bd to 0x02bcf9 (0x133d bytes) 000179 1789602964.491202 trunc from 0x016bf6 to 0x01c395 (0x57a0 bytes) 000180 1789602964.496510 punch from 0x01c5f8 to 0x0266ad (0xa0b6 bytes) 000181 1789602964.502326 mapread 0x0070a9 thru 0x010039 (0x8f91 bytes) 000182 1789602964.521303 read 0x00c171 thru 0x00f8a3 (0x3733 bytes) 000183 1789602964.522202 write 0x03c5dc thru 0x03ffff (0x3a24 bytes) 000184 1789602964.528647 trunc from 0x040000 to 0x00088a (0x3f775 bytes) 000185 1789602964.556939 trunc from 0x0002af to 0x00088a (0x5dc bytes) 000186 1789602964.562270 fallocate 0x01b2a7 thru 0x02a83b (0xf595 bytes) 000187 1789602964.567427 fallocate 0x0314ce thru 0x039dbf (0x88f2 bytes) 000188 1789602964.572489 trunc from 0x02844f to 0x02a83b (0x23ed bytes) 000189 1789602964.581468 MIRROR_RESYNC 2 mirrors 000190 1789602965.279294 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000191 1789602965.446862 fallocate 0x002788 thru 0x00966c (0x6ee5 bytes) 000192 1789602965.474854 read 0x0048af thru 0x006e95 (0x25e7 bytes) 000194 1789602965.488615 read 0x0095a4 thru 0x00975c (0x1b9 bytes) 000195 1789602965.507822 trunc from 0x02a83c to 0x02bf0c (0x16d1 bytes) 000196 1789602965.539145 trunc from 0x02bf0d to 0x034d53 (0x8e47 bytes) 000197 1789602965.566129 punch from 0x02b409 to 0x034d53 (0x994b bytes) 000198 1789602965.571991 trunc from 0x004b16 to 0x004fa8 (0x493 bytes) 000199 1789602965.577690 trunc from 0x02d9d7 to 0x031544 (0x3b6e bytes) 000200 1789602965.582931 trunc from 0x034d54 to 0x002c82 (0x320d1 bytes) 000201 1789602965.608870 punch from 0x0007cf to 0x002c82 (0x24b4 bytes) 000202 1789602965.614086 fallocate 0x022244 thru 0x02fbbd (0xd97a bytes) 000203 1789602965.621202 mapread 0x000d76 thru 0x002c82 (0x1f0d bytes) 000204 1789602965.642002 punch from 0x0020fb to 0x002c82 (0xb88 bytes) 000206 1789602965.648463 trunc from 0x00028d to 0x002c82 (0x29f6 bytes) 000207 1789602965.656441 trunc from 0x002c83 to 0x03ae80 (0x381fe bytes) 000208 1789602965.688069 mapread 0x02d9fd thru 0x0325fb (0x4bff bytes) 000209 1789602965.718201 read 0x00d490 thru 0x0108b3 (0x3424 bytes) 000210 1789602965.731148 read 0x0337fa thru 0x03ae80 (0x7687 bytes) 000211 1789602965.731985 read 0x00a226 thru 0x011107 (0x6ee2 bytes) 000212 1789602965.732610 punch from 0x02f62d to 0x03305c (0x3a30 bytes) 000213 1789602965.737151 punch from 0x029224 to 0x0391ea (0xffc7 bytes) 000215 1789602965.742933 read 0x037ce7 thru 0x039ee0 (0x21fa bytes) 000216 1789602965.755902 mapread 0x024dfd thru 0x02f6de (0xa8e2 bytes) 000217 1789602965.763038 MIRROR_EXTEND to 2 mirrors 000218 1789602966.841081 mapwrite 0x03f4e6 thru 0x03ffff (0xb1a bytes) 000219 1789602966.899980 read 0x024a86 thru 0x0252c3 (0x83e bytes) 000220 1789602966.910238 mapwrite 0x036b5d thru 0x03ffff (0x94a3 bytes) 000221 1789602966.955642 mapwrite 0x01412e thru 0x01e092 (0x9f65 bytes) 000222 1789602967.039663 trunc from 0x040000 to 0x00d0d9 (0x32f26 bytes) 000223 1789602967.059222 write 0x032193 thru 0x03d44b (0xb2b9 bytes) 000224 1789602967.062074 mapwrite 0x01a9b7 thru 0x023a64 (0x90ae bytes) 000225 1789602967.106216 trunc from 0x00abac to 0x00c90b (0x1d60 bytes) 000226 1789602967.111962 read 0x014154 thru 0x021f46 (0xddf3 bytes) 000227 1789602967.112641 read 0x02a5d9 thru 0x033084 (0x8aac bytes) 000228 1789602967.124682 trunc from 0x0345bc to 0x03bcf9 (0x773e bytes) 000229 1789602967.131659 read 0x02683a thru 0x02c12d (0x58f4 bytes) 000230 1789602967.132371 trunc from 0x03d44c to 0x02064c (0x1cdff bytes) 000231 1789602967.160004 read 0x0001ed thru 0x001252 (0x1066 bytes) 000232 1789602967.162986 mapread 0x0199a6 thru 0x01ed9f (0x53fa bytes) 000233 1789602967.174838 MIRROR_RESYNC 2 mirrors 000234 1789602967.802449 MIRROR_EXTEND to 3 mirrors 000235 1789602968.567978 MIRROR_EXTEND to 4 mirrors 000236 1789602969.445896 trunc from 0x02064d to 0x003316 (0x1d336 bytes) 000237 1789602969.532995 fallocate 0x0263b3 thru 0x02bddc (0x5a2a bytes) 000238 1789602969.544279 mapread 0x00b3fa thru 0x019411 (0xe018 bytes) 000241 1789602969.657208 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2bddd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bddd): Mirror 2: 0x194b5261 (stale) Mirror 3: 0x194b5261 (stale) Mirror 4: 0x194b5261 (stale) CRC-32 checksum value for chunk [0, 0x2bddd): Mirror 1: 0x4acea85c Mirror 5: 0x4acea85c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bddd: skipped 000242 1789602971.041975 punch from 0x00be57 to 0x010d42 (0x4eec bytes) 000243 1789602971.119287 trunc from 0x02bddd to 0x0094d3 (0x22909 bytes) 000244 1789602971.208507 trunc from 0x0094d4 to 0x02d387 (0x23eb4 bytes) 000245 1789602971.266268 punch from 0x02908c to 0x02b50c (0x2481 bytes) 000246 1789602971.287305 mapwrite 0x01bfb3 thru 0x01d934 (0x1982 bytes) 000247 1789602971.383938 punch from 0x02d1ce to 0x02d387 (0x1ba bytes) 000248 1789602971.397130 punch from 0x015d63 to 0x01ff9a (0xa238 bytes) 000249 1789602971.413969 write 0x02ef5d thru 0x03ed14 (0xfdb8 bytes) 000250 1789602971.425607 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ed15): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed15): Mirror 2: 0x13012616 (stale) Mirror 3: 0x13012616 (stale) Mirror 4: 0x13012616 (stale) Mirror 5: 0x66268655 (stale) CRC-32 checksum value for chunk [0, 0x3ed15): Mirror 1: 0xd81c7a34 Mirror 6: 0xd81c7a34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed15: skipped 000251 1789602972.838289 read 0x01adec thru 0x01fd38 (0x4f4d bytes) 000254 1789602972.908979 punch from 0x0239d5 to 0x02775c (0x3d88 bytes) 000255 1789602972.989319 punch from 0x02de2a to 0x03a082 (0xc259 bytes) 000256 1789602973.024263 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed15): Mirror 2: 0x13012616 (stale) Mirror 3: 0x13012616 (stale) Mirror 4: 0x13012616 (stale) Mirror 6: 0xd81c7a34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed15): Mirror 2: 0x13012616 (stale) Mirror 3: 0x13012616 (stale) Mirror 4: 0x13012616 (stale) Mirror 6: 0xd81c7a34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed15: skipped 000258 1789602973.562955 trunc from 0x01c377 to 0x0274f2 (0xb17c bytes) 000259 1789602973.574261 read 0x01dc20 thru 0x02104a (0x342b bytes) 000260 1789602973.613149 mapwrite 0x02b3e3 thru 0x02fb78 (0x4796 bytes) 000261 1789602973.727180 read 0x0109c1 thru 0x015dd2 (0x5412 bytes) 000262 1789602973.750825 trunc from 0x03ed15 to 0x022f0b (0x1be09 bytes) 000263 1789602973.779354 punch from 0x01ba7d to 0x022f0b (0x748f bytes) 000264 1789602973.791167 read 0x00f1f0 thru 0x0101cc (0xfdd bytes) 000266 1789602973.800802 write 0x03d077 thru 0x03f249 (0x21d3 bytes) 000267 1789602973.805484 trunc from 0x02b274 to 0x036bb7 (0xb944 bytes) 000268 1789602973.816274 mapwrite 0x0303cf thru 0x034ab5 (0x46e7 bytes) 000269 1789602973.874918 read 0x0257e9 thru 0x0264ce (0xce6 bytes) 000270 1789602973.876038 read 0x028681 thru 0x032b7d (0xa4fd bytes) 000271 1789602973.877060 write 0x036e35 thru 0x03ffff (0x91cb bytes) 000272 1789602973.879320 punch from 0x012f14 to 0x022745 (0xf832 bytes) 000273 1789602973.892458 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xba8779fc (stale) Mirror 3: 0xba8779fc (stale) Mirror 4: 0xba8779fc (stale) Mirror 6: 0xdcddbae8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x700f6f92 Mirror 7: 0x700f6f92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000274 1789602974.897422 write 0x03720d thru 0x0383d7 (0x11cb bytes) 000275 1789602974.951939 punch from 0x005ee2 to 0x0087e7 (0x2906 bytes) 000276 1789602974.957767 mapwrite 0x006b99 thru 0x00a33c (0x37a4 bytes) 000277 1789602974.996786 trunc from 0x040000 to 0x003fcc (0x3c033 bytes) 000278 1789602975.020358 mapwrite 0x0395ad thru 0x03ffff (0x6a53 bytes) 000279 1789602975.066368 mapread 0x03661f thru 0x037fce (0x19b0 bytes) 000281 1789602975.068938 fallocate 0x0115c4 thru 0x01d14d (0xbb8a bytes) 000285 1789602975.076587 read 0x037065 thru 0x03ffff (0x8f9b bytes) 000286 1789602975.078290 trunc from 0x025320 to 0x02878e (0x346f bytes) 000287 1789602975.088103 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xba8779fc (stale) Mirror 3: 0xba8779fc (stale) Mirror 4: 0xba8779fc (stale) Mirror 6: 0xdcddbae8 (stale) Mirror 7: 0x700f6f92 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcf16d371 Mirror 8: 0xcf16d371 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000288 1789602976.051157 trunc from 0x040000 to 0x025ff6 (0x1a009 bytes) 000289 1789602976.149272 mapwrite 0x03fb40 thru 0x03ffff (0x4c0 bytes) 000290 1789602976.200023 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba8779fc (stale) Mirror 4: 0xba8779fc (stale) Mirror 6: 0xdcddbae8 (stale) Mirror 7: 0x700f6f92 (stale) Mirror 8: 0xcf16d371 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba8779fc (stale) Mirror 4: 0xba8779fc (stale) Mirror 6: 0xdcddbae8 (stale) Mirror 7: 0x700f6f92 (stale) Mirror 8: 0xcf16d371 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000292 1789602976.806777 trunc from 0x028204 to 0x02933b (0x1138 bytes) 000293 1789602976.821270 mapwrite 0x02a511 thru 0x03a4c5 (0xffb5 bytes) 000294 1789602976.988011 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba8779fc (stale) Mirror 4: 0xba8779fc (stale) Mirror 6: 0xdcddbae8 (stale) Mirror 7: 0x700f6f92 (stale) Mirror 8: 0xcf16d371 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1f8c635c Mirror 9: 0x1f8c635c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1789602978.577495 mapread 0x0012e5 thru 0x008366 (0x7082 bytes) 000296 1789602978.626284 trunc from 0x03d906 to 0x03e049 (0x744 bytes) 000297 1789602978.684615 write 0x03f72e thru 0x03ffff (0x8d2 bytes) 000298 1789602978.701067 fallocate 0x03eedf thru 0x03f20c (0x32e bytes) 000299 1789602978.708070 mapwrite 0x03bd27 thru 0x03ed24 (0x2ffe bytes) 000300 1789602978.751418 fallocate 0x01cd29 thru 0x0227de (0x5ab6 bytes) 000301 1789602978.763227 write 0x03629b thru 0x03ffff (0x9d65 bytes) 000302 1789602978.781298 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba8779fc (stale) Mirror 4: 0xba8779fc (stale) Mirror 6: 0xdcddbae8 (stale) Mirror 8: 0xcf16d371 (stale) Mirror 9: 0x1f8c635c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba8779fc (stale) Mirror 4: 0xba8779fc (stale) Mirror 6: 0xdcddbae8 (stale) Mirror 8: 0xcf16d371 (stale) Mirror 9: 0x1f8c635c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000303 1789602979.769540 trunc from 0x040000 to 0x0312aa (0xed55 bytes) 000304 1789602979.797179 punch from 0x02a0ce to 0x02e191 (0x40c4 bytes) 000305 1789602979.806850 fallocate 0x01fee5 thru 0x023244 (0x3360 bytes) 000306 1789602979.812428 trunc from 0x00ee17 to 0x01757a (0x8764 bytes) 000308 1789602979.821296 read 0x0304ae thru 0x0312aa (0xdfd bytes) 000309 1789602979.857961 mapread 0x023c96 thru 0x024e26 (0x1191 bytes) 000310 1789602979.880064 trunc from 0x02ae33 to 0x0303c6 (0x5594 bytes) 000311 1789602979.908654 fallocate 0x03832f thru 0x03ffff (0x7cd1 bytes) 000312 1789602979.924880 write 0x00c9a5 thru 0x016760 (0x9dbc bytes) 000313 1789602979.957265 read 0x006cef thru 0x00b09b (0x43ad bytes) 000314 1789602979.990357 mapwrite 0x02990e thru 0x033fd3 (0xa6c6 bytes) 000315 1789602980.045958 punch from 0x00a13e to 0x017cd6 (0xdb99 bytes) 000316 1789602980.052629 fallocate 0x033fb9 thru 0x03b858 (0x78a0 bytes) 000317 1789602980.059175 trunc from 0x040000 to 0x02cc82 (0x1337d bytes) 000318 1789602980.078987 trunc from 0x00b371 to 0x017e11 (0xcaa1 bytes) 000319 1789602980.088643 read 0x008ddb thru 0x010ed1 (0x80f7 bytes) 000320 1789602980.104627 write 0x030e1d thru 0x03b8c1 (0xaaa5 bytes) 000321 1789602980.106702 trunc from 0x00f739 to 0x01eaf7 (0xf3bf bytes) 000322 1789602980.116025 write 0x0242e7 thru 0x0268f5 (0x260f bytes) 000324 1789602980.120674 trunc from 0x004f81 to 0x00a23e (0x52be bytes) 000325 1789602980.128594 fallocate 0x02178e thru 0x028610 (0x6e83 bytes) 000326 1789602980.136273 trunc from 0x026709 to 0x0314af (0xada7 bytes) 000327 1789602980.164703 trunc from 0x03b8c2 to 0x0318f6 (0x9fcb bytes) 000328 1789602980.200778 write 0x01f108 thru 0x0264c6 (0x73bf bytes) 000329 1789602980.214312 trunc from 0x001379 to 0x00b723 (0xa3ab bytes) 000330 1789602980.221597 trunc from 0x008af8 to 0x017a64 (0xef6d bytes) 000331 1789602980.227510 fallocate 0x00af59 thru 0x017df1 (0xce99 bytes) 000332 1789602980.234624 punch from 0x02072f to 0x02d3a5 (0xcc77 bytes) 000334 1789602980.256047 mapwrite 0x016bd3 thru 0x0202b5 (0x96e3 bytes) 000335 1789602980.348322 write 0x037262 thru 0x03ffff (0x8d9e bytes) 000336 1789602980.354324 trunc from 0x040000 to 0x02b537 (0x14ac8 bytes) 000337 1789602980.375242 mapread 0x012d10 thru 0x014d06 (0x1ff7 bytes) 000338 1789602980.395568 trunc from 0x02b538 to 0x022f19 (0x861e bytes) 000339 1789602980.412472 trunc from 0x00d3a4 to 0x017e10 (0xaa6d bytes) 000340 1789602980.417047 write 0x03302c thru 0x03fb1c (0xcaf1 bytes) 000341 1789602980.419331 write 0x0142fb thru 0x0216e9 (0xd3ef bytes) 000342 1789602980.440546 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fb1d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb1d): Mirror 3: 0x5c3b17c4 (stale) Mirror 4: 0x5c3b17c4 (stale) Mirror 6: 0xf4978f8c (stale) Mirror 8: 0x618bc01a (stale) Mirror 9: 0x5873c6bb (stale) CRC-32 checksum value for chunk [0, 0x3fb1d): Mirror 1: 0xb9d2ddcb Mirror 10: 0xb9d2ddcb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb1d: skipped 000343 1789602982.037111 mapread 0x0169d4 thru 0x016cd9 (0x306 bytes) 000344 1789602982.096265 MIRROR_RESYNC 7 mirrors 000345 1789602984.621677 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000346 1789602985.252672 read 0x001e55 thru 0x00a8e9 (0x8a95 bytes) 000348 1789602985.264371 mapwrite 0x018281 thru 0x01cef1 (0x4c71 bytes) 000349 1789602985.426120 trunc from 0x03940f to 0x03eacc (0x56be bytes) 000350 1789602985.434661 write 0x0081f1 thru 0x00ecb6 (0x6ac6 bytes) 000351 1789602985.465445 MIRROR_RESYNC 6 mirrors 000354 1789602988.471014 read 0x0383f3 thru 0x03d40d (0x501b bytes) 000355 1789602988.491549 mapwrite 0x020195 thru 0x0224f8 (0x2364 bytes) 000356 1789602988.685267 punch from 0x036c0c to 0x03fb1c (0x8f11 bytes) 000357 1789602988.704790 punch from 0x00d923 to 0x00e7eb (0xec9 bytes) 000358 1789602988.732674 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb1d): Mirror 4: 0xa0abf514 (stale) Mirror 6: 0xa0abf514 (stale) Mirror 8: 0xa0abf514 (stale) Mirror 9: 0xa0abf514 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb1d): Mirror 4: 0xa0abf514 (stale) Mirror 6: 0xa0abf514 (stale) Mirror 8: 0xa0abf514 (stale) Mirror 9: 0xa0abf514 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb1d: skipped 000359 1789602989.354649 write 0x01f4b4 thru 0x01fe2b (0x978 bytes) 000360 1789602989.376563 trunc from 0x009869 to 0x012f1c (0x96b4 bytes) 000361 1789602989.387650 trunc from 0x03fb1d to 0x007a70 (0x380ac bytes) 000362 1789602989.408532 punch from 0x00545f to 0x007a70 (0x2612 bytes) 000363 1789602989.415103 fallocate 0x00862d thru 0x00b32b (0x2cff bytes) 000364 1789602989.422237 read 0x000ca8 thru 0x007a70 (0x6dc9 bytes) 000365 1789602989.434175 read 0x000bcc thru 0x007a70 (0x6ea5 bytes) 000366 1789602989.435810 trunc from 0x006226 to 0x007a70 (0x184b bytes) 000367 1789602989.440780 read 0x003128 thru 0x007a70 (0x4949 bytes) 000368 1789602989.447079 mapwrite 0x03b78b thru 0x03ffff (0x4875 bytes) 000369 1789602989.527713 fallocate 0x01e76e thru 0x023611 (0x4ea4 bytes) 000370 1789602989.542942 write 0x000cc4 thru 0x00b64d (0xa98a bytes) 000371 1789602989.546107 read 0x00fbd9 thru 0x01671e (0x6b46 bytes) 000372 1789602989.546925 mapread 0x013f74 thru 0x01b4bd (0x754a bytes) 000373 1789602989.548706 trunc from 0x02b89e to 0x037379 (0xbadc bytes) 000374 1789602989.555114 punch from 0x01a57f to 0x0233aa (0x8e2c bytes) 000375 1789602989.561237 trunc from 0x02806d to 0x02fa9a (0x7a2e bytes) 000376 1789602989.570093 mapwrite 0x0057df thru 0x009b94 (0x43b6 bytes) 000377 1789602989.589100 read 0x037e32 thru 0x03ffff (0x81ce bytes) 000378 1789602989.605474 trunc from 0x040000 to 0x032a99 (0xd566 bytes) 000379 1789602989.628320 trunc from 0x022567 to 0x022f46 (0x9e0 bytes) 000380 1789602989.639504 write 0x02b6d1 thru 0x02f17a (0x3aaa bytes) 000381 1789602989.640823 mapread 0x0238e6 thru 0x02bbfc (0x8317 bytes) 000382 1789602989.643805 read 0x008050 thru 0x0107e8 (0x8799 bytes) 000383 1789602989.645376 mapread 0x0112e3 thru 0x013fcb (0x2ce9 bytes) 000384 1789602989.653419 MIRROR_RESYNC 5 mirrors 000386 1789602991.138534 mapread 0x008f97 thru 0x01477a (0xb7e4 bytes) 000387 1789602991.179070 read 0x004d5b thru 0x01144c (0xc6f2 bytes) 000388 1789602991.180137 read 0x003e9a thru 0x009839 (0x59a0 bytes) 000389 1789602991.183894 mapwrite 0x016a4a thru 0x01b98b (0x4f42 bytes) 000391 1789602991.333127 trunc from 0x032a9a to 0x029dab (0x8cee bytes) 000392 1789602991.393079 mapwrite 0x01543c thru 0x01803d (0x2c02 bytes) 000393 1789602991.425942 mapread 0x01190b thru 0x020cec (0xf3e2 bytes) 000394 1789602991.427390 mapwrite 0x010982 thru 0x015a9e (0x511d bytes) 000396 1789602991.469207 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29dac): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29dac): Mirror 4: 0x5a63cf1c (stale) Mirror 6: 0x5a63cf1c (stale) Mirror 8: 0x5a63cf1c (stale) Mirror 9: 0x5a63cf1c (stale) CRC-32 checksum value for chunk [0, 0x29dac): Mirror 1: 0x8ab149da Mirror 10: 0x8ab149da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29dac: skipped 000397 1789602992.807473 trunc from 0x029dac to 0x03b0d8 (0x1132d bytes) 000398 1789602992.890148 trunc from 0x0301cf to 0x038e0e (0x8c40 bytes) 000399 1789602992.903215 mapwrite 0x0024a3 thru 0x00a098 (0x7bf6 bytes) 000400 1789602992.958161 mapwrite 0x00b8c5 thru 0x0195fc (0xdd38 bytes) 000401 1789602993.028164 fallocate 0x01f017 thru 0x023312 (0x42fc bytes) 000404 1789602993.040939 write 0x02a08b thru 0x02d9b8 (0x392e bytes) 000405 1789602993.060892 trunc from 0x02e856 to 0x03b0d8 (0xc883 bytes) 000406 1789602993.069051 write 0x0025fe thru 0x010669 (0xe06c bytes) 000407 1789602993.071090 fallocate 0x01d661 thru 0x028efd (0xb89d bytes) 000409 1789602993.078830 mapread 0x01c168 thru 0x01f2ac (0x3145 bytes) 000410 1789602993.108162 trunc from 0x03b0d9 to 0x037828 (0x38b0 bytes) 000411 1789602993.166004 trunc from 0x0141fe to 0x01d139 (0x8f3c bytes) 000415 1789602993.174432 fallocate 0x02914e thru 0x0311e1 (0x8094 bytes) 000416 1789602993.182615 trunc from 0x01932c to 0x0210a8 (0x7d7d bytes) 000417 1789602993.189234 write 0x036df5 thru 0x03ffff (0x920b bytes) 000418 1789602993.199549 write 0x00564f thru 0x007abd (0x246f bytes) 000419 1789602993.200694 mapread 0x014114 thru 0x01eba1 (0xaa8e bytes) 000420 1789602993.235534 read 0x00abea thru 0x015f2b (0xb342 bytes) 000421 1789602993.236373 trunc from 0x029a17 to 0x037e6c (0xe456 bytes) 000422 1789602993.279648 write 0x014ae5 thru 0x01b221 (0x673d bytes) 000423 1789602993.284210 trunc from 0x040000 to 0x0255cd (0x1aa32 bytes) 000424 1789602993.317906 trunc from 0x0255ce to 0x018793 (0xce3a bytes) 000425 1789602993.391211 fallocate 0x036cdd thru 0x037361 (0x685 bytes) 000426 1789602993.411621 fallocate 0x0095ac thru 0x013a60 (0xa4b5 bytes) 000427 1789602993.420937 write 0x0179e5 thru 0x022f65 (0xb581 bytes) 000428 1789602993.435611 mapwrite 0x0218d1 thru 0x030916 (0xf046 bytes) 000429 1789602993.573504 fallocate 0x0339ea thru 0x034cf3 (0x130a bytes) 000430 1789602993.580267 read 0x0232e6 thru 0x02d655 (0xa370 bytes) 000431 1789602993.580932 trunc from 0x015b91 to 0x01aa85 (0x4ef5 bytes) 000432 1789602993.588512 trunc from 0x027e4e to 0x034cf3 (0xcea6 bytes) 000433 1789602993.595871 write 0x01cf54 thru 0x027704 (0xa7b1 bytes) 000434 1789602993.612887 punch from 0x02a7e4 to 0x02bbd2 (0x13ef bytes) 000435 1789602993.632538 read 0x00a032 thru 0x01127f (0x724e bytes) 000436 1789602993.637272 trunc from 0x01c1ef to 0x026ad4 (0xa8e6 bytes) 000437 1789602993.673947 punch from 0x00477e to 0x0076a5 (0x2f28 bytes) 000438 1789602993.692896 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34cf4): Mirror 4: 0x4e47a06d (stale) Mirror 8: 0x4e47a06d (stale) Mirror 9: 0x4e47a06d (stale) Mirror 10: 0xe8c39a7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34cf4): Mirror 4: 0x4e47a06d (stale) Mirror 8: 0x4e47a06d (stale) Mirror 9: 0x4e47a06d (stale) Mirror 10: 0xe8c39a7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34cf4: skipped 000439 1789602994.192026 mapread 0x00ce98 thru 0x017286 (0xa3ef bytes) 000440 1789602994.232853 MIRROR_RESYNC 5 mirrors 000441 1789602997.003860 MIRROR_RESYNC 5 mirrors 000442 1789602997.639604 write 0x008769 thru 0x011350 (0x8be8 bytes) 000443 1789602997.693691 read 0x0295e4 thru 0x034cf3 (0xb710 bytes) 000444 1789602997.721505 fallocate 0x038da1 thru 0x03d517 (0x4777 bytes) 000445 1789602997.729906 mapwrite 0x0276b0 thru 0x028325 (0xc76 bytes) 000447 1789602997.760552 trunc from 0x002a6e to 0x0111e6 (0xe779 bytes) 000448 1789602997.769134 fallocate 0x005b23 thru 0x015815 (0xfcf3 bytes) 000449 1789602997.773935 fallocate 0x02d191 thru 0x037269 (0xa0d9 bytes) 000450 1789602997.782827 write 0x0379e2 thru 0x03c7eb (0x4e0a bytes) 000451 1789602997.798818 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d518): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d518): Mirror 4: 0x55b44369 (stale) Mirror 8: 0x55b44369 (stale) Mirror 9: 0x55b44369 (stale) Mirror 10: 0x55b44369 (stale) CRC-32 checksum value for chunk [0, 0x3d518): Mirror 1: 0xef95149b Mirror 11: 0xef95149b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d518: skipped 000452 1789602999.363437 punch from 0x00ae0b to 0x017393 (0xc589 bytes) 000453 1789602999.430043 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d518): Mirror 4: 0x55b44369 (stale) Mirror 8: 0x55b44369 (stale) Mirror 9: 0x55b44369 (stale) Mirror 10: 0x55b44369 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d518): Mirror 4: 0x55b44369 (stale) Mirror 8: 0x55b44369 (stale) Mirror 9: 0x55b44369 (stale) Mirror 10: 0x55b44369 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d518: skipped 000454 1789603000.235202 write 0x00207f thru 0x00d1dc (0xb15e bytes) 000455 1789603000.261003 mapread 0x003657 thru 0x01293d (0xf2e7 bytes) 000458 1789603000.314469 MIRROR_RESYNC 5 mirrors 000459 1789603002.733140 read 0x027c69 thru 0x02a8e2 (0x2c7a bytes) 000460 1789603002.745125 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000461 1789603003.175465 mapread 0x005dae thru 0x00de24 (0x8077 bytes) 000462 1789603003.206987 mapwrite 0x02aed6 thru 0x02f8d0 (0x49fb bytes) 000463 1789603003.336338 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d518): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d518): Mirror 8: 0x84637b0f (stale) Mirror 9: 0x84637b0f (stale) Mirror 10: 0x84637b0f (stale) CRC-32 checksum value for chunk [0, 0x3d518): Mirror 1: 0xdfc28e1b Mirror 11: 0xdfc28e1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d518: skipped 000465 1789603005.274859 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d518): Mirror 8: 0x84637b0f (stale) Mirror 10: 0x84637b0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d518): Mirror 8: 0x84637b0f (stale) Mirror 10: 0x84637b0f (stale) CRC-32 checksum value for chunk [0, 0x3d518): Mirror 1: 0xdfc28e1b Mirror 11: 0xdfc28e1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d518: skipped 000467 1789603006.325081 trunc from 0x03d518 to 0x009e48 (0x336cf bytes) 000468 1789603006.395197 write 0x01e35e thru 0x024a72 (0x6715 bytes) 000469 1789603006.396767 read 0x003538 thru 0x01249f (0xef68 bytes) 000472 1789603006.412870 trunc from 0x024a73 to 0x039df8 (0x15386 bytes) 000473 1789603006.448251 punch from 0x016870 to 0x0212f6 (0xaa87 bytes) 000474 1789603006.466359 trunc from 0x024d47 to 0x025c07 (0xec1 bytes) 000475 1789603006.475951 write 0x0358b2 thru 0x03ffff (0xa74e bytes) 000476 1789603006.486954 read 0x010de6 thru 0x0193ca (0x85e5 bytes) 000478 1789603006.513303 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd9bd80a0 (stale) Mirror 10: 0xd9bd80a0 (stale) Mirror 11: 0x6df65767 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x69b7df23 Mirror 12: 0x69b7df23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000479 1789603008.251284 trunc from 0x040000 to 0x037232 (0x8dcd bytes) 000480 1789603008.328625 trunc from 0x0202e6 to 0x021bf8 (0x1913 bytes) 000481 1789603008.346148 mapwrite 0x01e336 thru 0x0256c2 (0x738d bytes) 000482 1789603008.492657 mapread 0x00a188 thru 0x012cb6 (0x8b2f bytes) 000483 1789603008.520704 punch from 0x025d98 to 0x031b6d (0xbdd6 bytes) 000484 1789603008.541629 write 0x019828 thru 0x0234f7 (0x9cd0 bytes) 000485 1789603008.550502 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37233): Mirror 10: 0x8fcaedfb (stale) Mirror 11: 0xac582ad3 (stale) Mirror 12: 0x26b66613 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37233): Mirror 10: 0x8fcaedfb (stale) Mirror 11: 0xac582ad3 (stale) Mirror 12: 0x26b66613 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37233: skipped 000486 1789603009.267050 write 0x033cb8 thru 0x033dd7 (0x120 bytes) 000487 1789603009.282014 read 0x03138b thru 0x037232 (0x5ea8 bytes) 000488 1789603009.300148 trunc from 0x037233 to 0x00ddd0 (0x29462 bytes) 000490 1789603009.336601 punch from 0x000ec4 to 0x00924b (0x8388 bytes) 000491 1789603009.351497 read 0x0010df thru 0x0042d1 (0x31f3 bytes) 000492 1789603009.377384 write 0x03e4f5 thru 0x03ffff (0x1b0b bytes) 000493 1789603009.378948 mapread 0x0114b7 thru 0x015247 (0x3d91 bytes) 000494 1789603009.400560 mapwrite 0x027802 thru 0x027fca (0x7c9 bytes) 000495 1789603009.448823 MIRROR_RESYNC 4 mirrors 000496 1789603010.812051 mapread 0x031b68 thru 0x03ffff (0xe498 bytes) 000497 1789603010.842839 trunc from 0x0297f1 to 0x036d03 (0xd513 bytes) 000498 1789603010.895326 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x8a5f219b (stale) Mirror 12: 0x8a5f219b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x8a5f219b (stale) Mirror 12: 0x8a5f219b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000499 1789603011.283499 trunc from 0x00f0e7 to 0x019afb (0xaa15 bytes) 000500 1789603011.290985 trunc from 0x040000 to 0x03c4c2 (0x3b3d bytes) 000501 1789603011.319675 trunc from 0x0027ab to 0x01214a (0xf9a0 bytes) 000503 1789603011.326430 trunc from 0x00caf2 to 0x018c38 (0xc147 bytes) 000504 1789603011.331077 trunc from 0x0148d5 to 0x017c40 (0x336c bytes) 000506 1789603011.339655 mapread 0x024356 thru 0x02a5c0 (0x626b bytes) 000507 1789603011.365519 punch from 0x00f44d to 0x0121d7 (0x2d8b bytes) 000508 1789603011.371701 fallocate 0x001209 thru 0x00a2e9 (0x90e1 bytes) 000509 1789603011.377444 punch from 0x000276 to 0x00c3bf (0xc14a bytes) 000511 1789603011.384493 trunc from 0x03c4c3 to 0x03957a (0x2f48 bytes) 000512 1789603011.408198 punch from 0x018a4a to 0x01f7e7 (0x6d9e bytes) 000513 1789603011.414493 trunc from 0x01f35a to 0x02282f (0x34d6 bytes) 000514 1789603011.420254 mapread 0x0167a5 thru 0x0240cb (0xd927 bytes) 000515 1789603011.440350 fallocate 0x0191a3 thru 0x021ad8 (0x8936 bytes) 000516 1789603011.446396 read 0x028b23 thru 0x0320eb (0x95c9 bytes) 000517 1789603011.458393 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3957b): Mirror 12: 0x26662c60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3957b): Mirror 12: 0x26662c60 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3957b: skipped 000518 1789603011.778365 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000519 1789603011.938372 mapwrite 0x03a225 thru 0x03ffff (0x5ddb bytes) 000521 1789603012.024811 mapread 0x00dd58 thru 0x01d0b4 (0xf35d bytes) 000522 1789603012.058983 mapread 0x0011c6 thru 0x0053ff (0x423a bytes) 000523 1789603012.070010 punch from 0x0296b6 to 0x032f0d (0x9858 bytes) 000524 1789603012.076320 trunc from 0x040000 to 0x0197b1 (0x2684e bytes) 000525 1789603012.098880 punch from 0x0132ed to 0x014f9a (0x1cae bytes) 000527 1789603012.107271 read 0x000e7e thru 0x00a5f1 (0x9774 bytes) 000528 1789603012.113491 mapread 0x0026a7 thru 0x00c51d (0x9e77 bytes) 000529 1789603012.115903 punch from 0x00ea8c to 0x0109fa (0x1f6f bytes) 000530 1789603012.124874 fallocate 0x010138 thru 0x011fa1 (0x1e6a bytes) 000531 1789603012.130672 trunc from 0x0197b2 to 0x008e38 (0x10979 bytes) 000533 1789603012.155492 trunc from 0x008e39 to 0x030a44 (0x27c0c bytes) 000534 1789603012.176776 mapwrite 0x000719 thru 0x001bd7 (0x14bf bytes) 000535 1789603012.194067 fallocate 0x030b8c thru 0x03280f (0x1c84 bytes) 000536 1789603012.200208 trunc from 0x030a45 to 0x02703f (0x9a05 bytes) 000538 1789603012.223043 mapwrite 0x00eee7 thru 0x016b8b (0x7ca5 bytes) 000539 1789603012.260493 fallocate 0x02343d thru 0x030463 (0xd027 bytes) 000540 1789603012.265868 fallocate 0x023751 thru 0x02a85d (0x710d bytes) 000541 1789603012.271911 fallocate 0x00535a thru 0x00cc49 (0x78f0 bytes) 000542 1789603012.283019 MIRROR_EXTEND to 2 mirrors 000543 1789603012.992306 MIRROR_EXTEND to 3 mirrors 000544 1789603013.865844 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000545 1789603014.304863 trunc from 0x022c7a to 0x02848d (0x5814 bytes) 000546 1789603014.363018 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000547 1789603014.538059 punch from 0x003160 to 0x00a0c8 (0x6f69 bytes) 000548 1789603014.544810 punch from 0x011cb9 to 0x01805c (0x63a4 bytes) 000549 1789603014.553599 mapwrite 0x03350c thru 0x03ffff (0xcaf4 bytes) 000551 1789603014.664892 trunc from 0x040000 to 0x030e6a (0xf195 bytes) 000553 1789603014.690572 trunc from 0x030e6b to 0x023195 (0xdcd5 bytes) 000554 1789603014.707818 mapwrite 0x0387b6 thru 0x03af9a (0x27e5 bytes) 000556 1789603014.765865 trunc from 0x03af9b to 0x021e43 (0x19157 bytes) 000557 1789603014.799927 read 0x00f46b thru 0x00fe62 (0x9f8 bytes) 000558 1789603014.807374 mapread 0x015f06 thru 0x01ba1c (0x5b17 bytes) 000561 1789603014.832345 mapread 0x01c8c5 thru 0x021e43 (0x557f bytes) 000562 1789603014.841282 punch from 0x0103a1 to 0x01a01c (0x9c7c bytes) 000563 1789603014.848354 fallocate 0x029563 thru 0x037c9f (0xe73d bytes) 000565 1789603014.864003 mapwrite 0x02324f thru 0x02fe13 (0xcbc5 bytes) 000566 1789603014.926404 trunc from 0x037ca0 to 0x02966d (0xe632 bytes) 000567 1789603014.946014 fallocate 0x0096a8 thru 0x00f2f9 (0x5c52 bytes) 000568 1789603014.954495 write 0x007d7b thru 0x00b656 (0x38dc bytes) 000570 1789603014.967707 trunc from 0x02966e to 0x024e67 (0x4806 bytes) 000571 1789603014.985527 write 0x00e897 thru 0x0159bc (0x7126 bytes) 000573 1789603014.996040 trunc from 0x011380 to 0x020a62 (0xf6e3 bytes) 000575 1789603015.026199 write 0x023d51 thru 0x02eb5b (0xae0b bytes) 000576 1789603015.028989 mapwrite 0x01ef5d thru 0x0253a5 (0x6449 bytes) 000577 1789603015.068827 trunc from 0x022e81 to 0x02eb5b (0xbcdb bytes) 000578 1789603015.075665 mapread 0x018e9f thru 0x01ae8f (0x1ff1 bytes) 000579 1789603015.077801 write 0x019dc7 thru 0x02439a (0xa5d4 bytes) 000580 1789603015.084837 fallocate 0x014b03 thru 0x01750a (0x2a08 bytes) 000581 1789603015.090292 read 0x01e36d thru 0x021e64 (0x3af8 bytes) 000582 1789603015.091693 read 0x02aa96 thru 0x02eb5b (0x40c6 bytes) 000583 1789603015.103353 punch from 0x028b2e to 0x02eb5b (0x602e bytes) 000584 1789603015.109296 read 0x028da6 thru 0x02eb5b (0x5db6 bytes) 000585 1789603015.123920 read 0x014744 thru 0x017484 (0x2d41 bytes) 000586 1789603015.125477 trunc from 0x02eb5c to 0x03bcce (0xd173 bytes) 000587 1789603015.142416 mapwrite 0x00390e thru 0x0091b6 (0x58a9 bytes) 000588 1789603015.174951 fallocate 0x0326e6 thru 0x036d0c (0x4627 bytes) 000589 1789603015.186709 MIRROR_EXTEND to 2 mirrors 000590 1789603016.052490 mapread 0x00d36a thru 0x017311 (0x9fa8 bytes) 000591 1789603016.091698 write 0x002018 thru 0x00dc30 (0xbc19 bytes) 000592 1789603016.154444 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bccf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bccf): Mirror 2: 0xaa36825c (stale) CRC-32 checksum value for chunk [0, 0x3bccf): Mirror 1: 0x7c58ea20 Mirror 3: 0x7c58ea20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bccf: skipped 000593 1789603017.193108 punch from 0x0387e7 to 0x03bcce (0x34e8 bytes) 000595 1789603017.230364 fallocate 0x022180 thru 0x0262ce (0x414f bytes) 000596 1789603017.241507 fallocate 0x00ab85 thru 0x015978 (0xadf4 bytes) 000598 1789603017.248142 mapread 0x01fc73 thru 0x02f701 (0xfa8f bytes) 000599 1789603017.279967 mapread 0x02e05d thru 0x032c6a (0x4c0e bytes) 000600 1789603017.286482 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bccf): Mirror 2: 0xaa36825c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bccf): Mirror 2: 0xaa36825c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bccf: skipped 000601 1789603017.638555 mapwrite 0x00efdd thru 0x011b31 (0x2b55 bytes) 000602 1789603017.682953 write 0x0359a0 thru 0x038568 (0x2bc9 bytes) 000603 1789603017.705931 write 0x025d91 thru 0x02d384 (0x75f4 bytes) 000604 1789603017.730542 mapread 0x035b30 thru 0x03bcce (0x619f bytes) 000605 1789603017.746224 mapread 0x0278b8 thru 0x032998 (0xb0e1 bytes) 000606 1789603017.760223 trunc from 0x0263ec to 0x032123 (0xbd38 bytes) 000607 1789603017.781244 read 0x01d772 thru 0x02a9ef (0xd27e bytes) 000608 1789603017.792920 write 0x023473 thru 0x02cc53 (0x97e1 bytes) 000609 1789603017.800884 read 0x00c437 thru 0x018126 (0xbcf0 bytes) 000610 1789603017.805902 mapwrite 0x005f1b thru 0x0083cd (0x24b3 bytes) 000611 1789603017.853841 trunc from 0x018792 to 0x01fb7a (0x73e9 bytes) 000612 1789603017.860449 trunc from 0x02919e to 0x032bbc (0x9a1f bytes) 000613 1789603017.868694 fallocate 0x02d4a1 thru 0x03b96e (0xe4ce bytes) 000614 1789603017.888958 write 0x03f1f1 thru 0x03ffff (0xe0f bytes) 000615 1789603017.890374 trunc from 0x03e544 to 0x03ffff (0x1abc bytes) 000616 1789603017.899551 punch from 0x03f017 to 0x03ffff (0xfe9 bytes) 000618 1789603017.905324 punch from 0x02fec9 to 0x037ae6 (0x7c1e bytes) 000619 1789603017.910930 read 0x0272b9 thru 0x031e3f (0xab87 bytes) 000620 1789603017.927251 read 0x03c623 thru 0x03ffff (0x39dd bytes) 000621 1789603017.928342 write 0x03c1f2 thru 0x03c2d0 (0xdf bytes) 000622 1789603017.932022 trunc from 0x040000 to 0x03e97a (0x1685 bytes) 000623 1789603017.976634 write 0x0060f0 thru 0x0100be (0x9fcf bytes) 000624 1789603017.979774 fallocate 0x00d4ec thru 0x0152e3 (0x7df8 bytes) 000625 1789603017.984190 trunc from 0x03e97b to 0x02eeb6 (0xfac4 bytes) 000626 1789603018.025308 trunc from 0x015337 to 0x01882e (0x34f8 bytes) 000627 1789603018.034556 mapread 0x002884 thru 0x010396 (0xdb13 bytes) 000628 1789603018.035942 punch from 0x00cc6b to 0x012901 (0x5c97 bytes) 000629 1789603018.059376 trunc from 0x02eeb7 to 0x03acae (0xbdf8 bytes) 000630 1789603018.083717 trunc from 0x0171eb to 0x01da3e (0x6854 bytes) 000631 1789603018.091418 mapwrite 0x01356e thru 0x01eb4d (0xb5e0 bytes) 000632 1789603018.138299 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000633 1789603018.345010 MIRROR_EXTEND to 2 mirrors 000635 1789603019.305392 mapread 0x02e275 thru 0x038ca2 (0xaa2e bytes) 000636 1789603019.364431 trunc from 0x00e650 to 0x01d639 (0xefea bytes) 000638 1789603019.420328 mapread 0x0219c0 thru 0x0316e9 (0xfd2a bytes) 000639 1789603019.451670 trunc from 0x03acaf to 0x03704a (0x3c64 bytes) 000640 1789603019.478490 write 0x006d18 thru 0x00ce9f (0x6188 bytes) 000641 1789603019.490392 trunc from 0x03704b to 0x014cf5 (0x22355 bytes) 000642 1789603019.524647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000643 1789603019.712219 MIRROR_EXTEND to 2 mirrors 000644 1789603020.440959 punch from 0x005259 to 0x00ec34 (0x99dc bytes) 000645 1789603020.481930 read 0x0016bb thru 0x0017a5 (0xeb bytes) 000646 1789603020.492477 MIRROR_RESYNC 2 mirrors 000647 1789603021.144165 fallocate 0x034288 thru 0x0343f6 (0x16f bytes) 000648 1789603021.187333 write 0x025ad5 thru 0x0299a4 (0x3ed0 bytes) 000649 1789603021.191769 trunc from 0x0097b6 to 0x010391 (0x6bdc bytes) 000650 1789603021.199550 punch from 0x00d81f to 0x0115dd (0x3dbf bytes) 000651 1789603021.210259 read 0x00a953 thru 0x01178e (0x6e3c bytes) 000652 1789603021.247204 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000653 1789603021.578131 mapwrite 0x01ffbe thru 0x02aed1 (0xaf14 bytes) 000654 1789603021.722013 punch from 0x00309a to 0x005311 (0x2278 bytes) 000655 1789603021.728772 mapwrite 0x010607 thru 0x011714 (0x110e bytes) 000656 1789603021.755151 trunc from 0x02aed2 to 0x015821 (0x156b0 bytes) 000657 1789603021.779098 trunc from 0x001e1d to 0x011695 (0xf879 bytes) 000658 1789603021.787035 punch from 0x0038dc to 0x011f48 (0xe66d bytes) 000659 1789603021.795820 mapread 0x002c90 thru 0x00adae (0x811f bytes) 000660 1789603021.810268 mapread 0x006139 thru 0x00ec6f (0x8b37 bytes) 000661 1789603021.812315 mapread 0x00322b thru 0x00af7b (0x7d51 bytes) 000664 1789603021.816619 trunc from 0x007d3a to 0x00d499 (0x5760 bytes) 000667 1789603021.824660 write 0x001042 thru 0x004e30 (0x3def bytes) 000668 1789603021.826562 trunc from 0x0036e7 to 0x00e4bb (0xadd5 bytes) 000669 1789603021.840850 trunc from 0x015822 to 0x005f5b (0xf8c6 bytes) 000670 1789603021.864637 write 0x012bd9 thru 0x018b4e (0x5f76 bytes) 000671 1789603021.868489 mapwrite 0x019a53 thru 0x01a823 (0xdd1 bytes) 000673 1789603021.907787 mapread 0x00b100 thru 0x00cd31 (0x1c32 bytes) 000674 1789603021.919814 punch from 0x0118f7 to 0x014247 (0x2951 bytes) 000675 1789603021.925661 trunc from 0x009b0e to 0x018e5d (0xf350 bytes) 000676 1789603021.931185 mapread 0x007df9 thru 0x0171a7 (0xf3af bytes) 000677 1789603021.949209 trunc from 0x01a824 to 0x015ff9 (0x482a bytes) 000679 1789603021.970206 mapwrite 0x008f1c thru 0x017e8a (0xef6f bytes) 000680 1789603022.019935 mapread 0x00286b thru 0x011c59 (0xf3ef bytes) 000682 1789603022.021813 fallocate 0x016f95 thru 0x02375c (0xc7c8 bytes) 000683 1789603022.027345 mapwrite 0x005b56 thru 0x012377 (0xc822 bytes) 000684 1789603022.046513 trunc from 0x017e8b to 0x01d225 (0x539b bytes) 000686 1789603022.068206 mapwrite 0x01ed4e thru 0x02a9f3 (0xbca6 bytes) 000687 1789603022.134645 mapwrite 0x003a2f thru 0x0099c7 (0x5f99 bytes) 000688 1789603022.167680 read 0x00cd70 thru 0x01859d (0xb82e bytes) 000690 1789603022.171573 read 0x00e452 thru 0x01972d (0xb2dc bytes) 000691 1789603022.173652 fallocate 0x03bf57 thru 0x03ffff (0x40a9 bytes) 000692 1789603022.183542 trunc from 0x023bd8 to 0x02a9f3 (0x6e1c bytes) 000693 1789603022.187665 read 0x01fb5b thru 0x022251 (0x26f7 bytes) 000694 1789603022.189523 trunc from 0x02543a to 0x028bb8 (0x377f bytes) 000695 1789603022.196012 fallocate 0x03f57a thru 0x03ffff (0xa86 bytes) 000696 1789603022.205472 punch from 0x01588a to 0x01b15b (0x58d2 bytes) 000697 1789603022.210214 mapwrite 0x030f0e thru 0x03b05c (0xa14f bytes) 000698 1789603022.288661 punch from 0x035178 to 0x03b05c (0x5ee5 bytes) 000699 1789603022.295767 fallocate 0x02fbb7 thru 0x03b725 (0xbb6f bytes) 000700 1789603022.302349 write 0x032b77 thru 0x03b516 (0x89a0 bytes) 000701 1789603022.310537 trunc from 0x03b726 to 0x032adf (0x8c46 bytes) 000702 1789603022.329621 trunc from 0x032ae0 to 0x00550c (0x2d5d3 bytes) 000705 1789603022.349707 punch from 0x001681 to 0x0025f4 (0xf74 bytes) 000707 1789603022.356108 mapwrite 0x007854 thru 0x00b72c (0x3ed9 bytes) 000708 1789603022.414834 write 0x01157f thru 0x01d510 (0xbf92 bytes) 000710 1789603022.419116 trunc from 0x0105fa to 0x014d20 (0x4727 bytes) 000711 1789603022.431704 fallocate 0x037b33 thru 0x03dd6d (0x623b bytes) 000712 1789603022.437119 trunc from 0x01a3d2 to 0x01d510 (0x313f bytes) 000713 1789603022.442146 mapwrite 0x00cf29 thru 0x0128a0 (0x5978 bytes) 000714 1789603022.493158 trunc from 0x01d511 to 0x03201d (0x14b0d bytes) 000715 1789603022.511940 MIRROR_EXTEND to 2 mirrors 000716 1789603023.380864 trunc from 0x03201e to 0x0163af (0x1bc6e bytes) 000717 1789603023.423775 punch from 0x00afb0 to 0x0163af (0xb400 bytes) 000718 1789603023.429214 write 0x006c68 thru 0x00a473 (0x380c bytes) 000719 1789603023.439915 fallocate 0x0247f9 thru 0x025236 (0xa3e bytes) 000722 1789603023.449486 mapread 0x011ff8 thru 0x0163af (0x43b8 bytes) 000723 1789603023.474320 read 0x0101c8 thru 0x013130 (0x2f69 bytes) 000724 1789603023.475017 punch from 0x009dd9 to 0x0163af (0xc5d7 bytes) 000725 1789603023.508820 MIRROR_RESYNC 2 mirrors 000726 1789603024.048681 fallocate 0x02a954 thru 0x0338d8 (0x8f85 bytes) 000727 1789603024.083298 mapwrite 0x023dfd thru 0x032fc4 (0xf1c8 bytes) 000729 1789603024.178612 read 0x006ef2 thru 0x00a1e0 (0x32ef bytes) 000730 1789603024.187934 trunc from 0x032fc5 to 0x007cc3 (0x2b301 bytes) 000732 1789603024.216115 fallocate 0x0150d3 thru 0x018e44 (0x3d72 bytes) 000733 1789603024.222174 fallocate 0x0367da thru 0x039823 (0x304a bytes) 000734 1789603024.229645 fallocate 0x0057ee thru 0x013d01 (0xe514 bytes) 000735 1789603024.235215 punch from 0x003309 to 0x00823f (0x4f37 bytes) 000736 1789603024.241248 trunc from 0x008fea to 0x011b93 (0x8baa bytes) 000737 1789603024.246171 read 0x03546f thru 0x039823 (0x43b5 bytes) 000738 1789603024.261342 write 0x02e660 thru 0x03cf5a (0xe8fb bytes) 000739 1789603024.263657 trunc from 0x03cf5b to 0x009b22 (0x33438 bytes) 000740 1789603024.279166 fallocate 0x03f0e9 thru 0x03ffff (0xf17 bytes) 000741 1789603024.284925 trunc from 0x040000 to 0x00d496 (0x32b69 bytes) 000742 1789603024.304339 trunc from 0x009c36 to 0x00d496 (0x3861 bytes) 000743 1789603024.309420 mapread 0x001443 thru 0x00d496 (0xc054 bytes) 000744 1789603024.333146 mapread 0x007f1c thru 0x00bf12 (0x3ff7 bytes) 000745 1789603024.334607 write 0x0087cc thru 0x011e66 (0x969b bytes) 000746 1789603024.340343 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000747 1789603024.496372 write 0x004f6e thru 0x008b3a (0x3bcd bytes) 000748 1789603024.514064 mapread 0x000609 thru 0x00cf0f (0xc907 bytes) 000749 1789603024.539447 write 0x0210fc thru 0x02963c (0x8541 bytes) 000750 1789603024.542261 write 0x025983 thru 0x0297b8 (0x3e36 bytes) 000751 1789603024.548855 mapwrite 0x031bff thru 0x03870c (0x6b0e bytes) 000752 1789603024.589528 trunc from 0x03870d to 0x01286e (0x25e9e bytes) 000753 1789603024.605461 trunc from 0x00d2cd to 0x01286e (0x55a2 bytes) 000754 1789603024.613287 mapwrite 0x0027c8 thru 0x00fd05 (0xd53e bytes) 000755 1789603024.642442 punch from 0x00fefb to 0x01286e (0x2974 bytes) 000756 1789603024.648389 punch from 0x00dfbc to 0x01286e (0x48b3 bytes) 000757 1789603024.653449 read 0x00a90b thru 0x01286e (0x7f64 bytes) 000758 1789603024.665708 read 0x010f1d thru 0x01286e (0x1952 bytes) 000760 1789603024.673861 MIRROR_EXTEND to 2 mirrors 000762 1789603025.402814 fallocate 0x009709 thru 0x0159ea (0xc2e2 bytes) 000763 1789603025.432643 trunc from 0x01286f to 0x01f442 (0xcbd4 bytes) 000764 1789603025.464270 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000766 1789603025.615660 read 0x01e28d thru 0x01f442 (0x11b6 bytes) 000768 1789603025.627771 read 0x00a7f4 thru 0x016a1f (0xc22c bytes) 000769 1789603025.640649 read 0x01d592 thru 0x01f442 (0x1eb1 bytes) 000770 1789603025.641803 mapwrite 0x018ab1 thru 0x01ca28 (0x3f78 bytes) 000771 1789603025.668601 trunc from 0x00167e to 0x00be7d (0xa800 bytes) 000772 1789603025.674890 punch from 0x00eb35 to 0x01de60 (0xf32c bytes) 000773 1789603025.682931 read 0x0178c3 thru 0x01f442 (0x7b80 bytes) 000774 1789603025.695322 read 0x019f73 thru 0x01f442 (0x54d0 bytes) 000775 1789603025.696023 read 0x007305 thru 0x00ae58 (0x3b54 bytes) 000776 1789603025.696546 trunc from 0x0168e8 to 0x01ead2 (0x81eb bytes) 000778 1789603025.705303 trunc from 0x01f443 to 0x005d30 (0x19712 bytes) 000779 1789603025.728388 mapread 0x002992 thru 0x005d30 (0x339f bytes) 000780 1789603025.731641 fallocate 0x03521a thru 0x03c8b1 (0x7698 bytes) 000781 1789603025.738213 read 0x003c88 thru 0x005d30 (0x20a9 bytes) 000782 1789603025.744570 MIRROR_EXTEND to 2 mirrors 000783 1789603026.557850 mapwrite 0x018fae thru 0x0199c0 (0xa13 bytes) 000784 1789603026.649716 read 0x007b00 thru 0x016d96 (0xf297 bytes) 000786 1789603026.667205 trunc from 0x015aaf to 0x0199c0 (0x3f12 bytes) 000787 1789603026.672494 mapread 0x004489 thru 0x008d1e (0x4896 bytes) 000788 1789603026.704139 write 0x0150ab thru 0x016a33 (0x1989 bytes) 000789 1789603026.719473 punch from 0x013918 to 0x0199c0 (0x60a9 bytes) 000790 1789603026.774324 write 0x015467 thru 0x016a71 (0x160b bytes) 000791 1789603026.797927 mapwrite 0x018de2 thru 0x025a15 (0xcc34 bytes) 000792 1789603026.904319 fallocate 0x00e8a4 thru 0x012e09 (0x4566 bytes) 000793 1789603026.911685 trunc from 0x01341d to 0x01e61b (0xb1ff bytes) 000794 1789603026.923145 write 0x02e3df thru 0x039cfb (0xb91d bytes) 000795 1789603026.939972 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000796 1789603027.151453 mapwrite 0x028685 thru 0x03115c (0x8ad8 bytes) 000797 1789603027.192248 trunc from 0x039cfc to 0x01d7a4 (0x1c557 bytes) 000798 1789603027.224406 MIRROR_EXTEND to 2 mirrors 000799 1789603028.003127 trunc from 0x01d7a5 to 0x03e807 (0x21063 bytes) 000800 1789603028.077915 write 0x023251 thru 0x02dd94 (0xab44 bytes) 000801 1789603028.102109 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e808): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e808): Mirror 2: 0xdf9e4181 (stale) CRC-32 checksum value for chunk [0, 0x3e808): Mirror 1: 0xa550b54d Mirror 3: 0xa550b54d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e808: skipped 000802 1789603029.340866 write 0x01e954 thru 0x020313 (0x19c0 bytes) 000803 1789603029.393003 write 0x007b53 thru 0x01757e (0xfa2c bytes) 000804 1789603029.408321 mapwrite 0x00d3d5 thru 0x014ada (0x7706 bytes) 000805 1789603029.426617 write 0x02db10 thru 0x033373 (0x5864 bytes) 000806 1789603029.438609 mapwrite 0x02efe0 thru 0x031022 (0x2043 bytes) 000807 1789603029.454497 trunc from 0x03e808 to 0x006f87 (0x37880 bytes) 000808 1789603029.476656 trunc from 0x006bd9 to 0x006f87 (0x3af bytes) 000810 1789603029.482318 mapwrite 0x02b235 thru 0x02f3fd (0x41c9 bytes) 000811 1789603029.539743 trunc from 0x02f3fe to 0x00a3a2 (0x2505b bytes) 000812 1789603029.566076 write 0x02991c thru 0x02b529 (0x1c0e bytes) 000813 1789603029.567924 punch from 0x022fa9 to 0x02b529 (0x8581 bytes) 000814 1789603029.578628 mapread 0x021f14 thru 0x0231a2 (0x128f bytes) 000816 1789603029.592312 write 0x036b9f thru 0x03881d (0x1c7f bytes) 000817 1789603029.595250 read 0x009a8f thru 0x010337 (0x68a9 bytes) 000818 1789603029.606358 fallocate 0x01405d thru 0x015599 (0x153d bytes) 000819 1789603029.612144 fallocate 0x00d532 thru 0x00da5d (0x52c bytes) 000820 1789603029.618323 trunc from 0x03881e to 0x029476 (0xf3a7 bytes) 000821 1789603029.640657 mapread 0x00fed2 thru 0x01c761 (0xc890 bytes) 000822 1789603029.654387 mapread 0x014587 thru 0x019a32 (0x54ac bytes) 000823 1789603029.655925 mapread 0x002a66 thru 0x00994e (0x6ee9 bytes) 000824 1789603029.676615 trunc from 0x029477 to 0x01ec1d (0xa859 bytes) 000826 1789603029.696275 fallocate 0x012714 thru 0x0140a2 (0x198f bytes) 000827 1789603029.703278 trunc from 0x01605e to 0x017cfc (0x1c9f bytes) 000828 1789603029.709609 mapwrite 0x036cf2 thru 0x03e5d5 (0x78e4 bytes) 000829 1789603029.761621 punch from 0x02ff7d to 0x03596a (0x59ee bytes) 000830 1789603029.768404 punch from 0x00c00a to 0x00e69c (0x2693 bytes) 000831 1789603029.775433 punch from 0x0316f2 to 0x035280 (0x3b8f bytes) 000832 1789603029.781499 punch from 0x0145b3 to 0x024071 (0xfabf bytes) 000833 1789603029.787552 mapwrite 0x028765 thru 0x02f246 (0x6ae2 bytes) 000834 1789603029.818207 read 0x03b8a7 thru 0x03e5d5 (0x2d2f bytes) 000835 1789603029.819163 write 0x00c95a thru 0x01739b (0xaa42 bytes) 000836 1789603029.830328 fallocate 0x011946 thru 0x019403 (0x7abe bytes) 000837 1789603029.835808 fallocate 0x02cf1f thru 0x032ae9 (0x5bcb bytes) 000838 1789603029.841132 write 0x00481a thru 0x013334 (0xeb1b bytes) 000839 1789603029.843627 read 0x011ae5 thru 0x0173ab (0x58c7 bytes) 000841 1789603029.845664 read 0x011756 thru 0x01d742 (0xbfed bytes) 000842 1789603029.846552 fallocate 0x0005e7 thru 0x00f482 (0xee9c bytes) 000843 1789603029.851644 trunc from 0x031243 to 0x0319c2 (0x780 bytes) 000844 1789603029.857311 mapread 0x008e62 thru 0x00c671 (0x3810 bytes) 000846 1789603029.867268 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e5d6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e5d6): Mirror 2: 0xc2d4f96 (stale) Mirror 3: 0x7ffd580e (stale) CRC-32 checksum value for chunk [0, 0x3e5d6): Mirror 1: 0xd3662b8e Mirror 4: 0xd3662b8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e5d6: skipped 000847 1789603030.924337 mapwrite 0x02b13c thru 0x0389a0 (0xd865 bytes) 000848 1789603031.051590 write 0x03ee19 thru 0x03ffff (0x11e7 bytes) 000849 1789603031.064685 mapwrite 0x01a33c thru 0x0208f2 (0x65b7 bytes) 000850 1789603031.095708 fallocate 0x00431a thru 0x012b75 (0xe85c bytes) 000851 1789603031.101790 trunc from 0x040000 to 0x024d15 (0x1b2ea bytes) 000852 1789603031.129140 write 0x037d70 thru 0x03cda7 (0x5038 bytes) 000853 1789603031.135263 write 0x02089d thru 0x0214b9 (0xc1d bytes) 000854 1789603031.145146 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cda8): Mirror 3: 0x5ca5101a (stale) Mirror 4: 0x2bf4aec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cda8): Mirror 3: 0x5ca5101a (stale) Mirror 4: 0x2bf4aec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cda8: skipped 000855 1789603031.470861 trunc from 0x02260b to 0x0264af (0x3ea5 bytes) 000856 1789603031.477070 mapread 0x02975e thru 0x02b0b2 (0x1955 bytes) 000857 1789603031.503404 punch from 0x0381d1 to 0x03cda7 (0x4bd7 bytes) 000858 1789603031.509052 trunc from 0x017d42 to 0x02578e (0xda4d bytes) 000859 1789603031.514600 write 0x03f0cf thru 0x03ffff (0xf31 bytes) 000861 1789603031.521446 mapread 0x0160a9 thru 0x01ce76 (0x6dce bytes) 000863 1789603031.531678 fallocate 0x010310 thru 0x010fa8 (0xc99 bytes) 000864 1789603031.536789 mapwrite 0x0084ac thru 0x013d03 (0xb858 bytes) 000865 1789603031.570322 trunc from 0x02f668 to 0x02fa05 (0x39e bytes) 000866 1789603031.575468 mapwrite 0x03f06a thru 0x03ffff (0xf96 bytes) 000867 1789603031.587969 mapread 0x0241f6 thru 0x02cdf8 (0x8c03 bytes) 000868 1789603031.599576 read 0x0172a7 thru 0x01b982 (0x46dc bytes) 000869 1789603031.600512 mapread 0x022bf8 thru 0x0235a7 (0x9b0 bytes) 000870 1789603031.606417 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb4c4c856 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb4c4c856 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000871 1789603031.889774 mapwrite 0x021d85 thru 0x02e41f (0xc69b bytes) 000872 1789603031.957551 fallocate 0x03bc2e thru 0x03ffff (0x43d2 bytes) 000873 1789603031.963562 trunc from 0x040000 to 0x03df19 (0x20e6 bytes) 000874 1789603031.994371 punch from 0x010321 to 0x0152b4 (0x4f94 bytes) 000875 1789603032.001190 mapread 0x02fdc4 thru 0x03c9c7 (0xcc04 bytes) 000876 1789603032.012436 mapread 0x013c9c thru 0x01eae8 (0xae4d bytes) 000877 1789603032.024128 trunc from 0x029ce0 to 0x0388e0 (0xec01 bytes) 000878 1789603032.029778 punch from 0x009ffd to 0x010610 (0x6614 bytes) 000879 1789603032.036029 trunc from 0x01d617 to 0x0270e3 (0x9acd bytes) 000880 1789603032.042416 trunc from 0x03df1a to 0x00823c (0x35cdd bytes) 000882 1789603032.066394 mapwrite 0x02a066 thru 0x0362fa (0xc295 bytes) 000883 1789603032.131207 mapwrite 0x03ad7c thru 0x03de5a (0x30df bytes) 000885 1789603032.186045 trunc from 0x03de5b to 0x03e8a1 (0xa47 bytes) 000887 1789603032.204831 fallocate 0x012bbf thru 0x015eca (0x330c bytes) 000888 1789603032.212232 read 0x00dc7c thru 0x0194d5 (0xb85a bytes) 000889 1789603032.224697 trunc from 0x03e8a2 to 0x01173d (0x2d164 bytes) 000890 1789603032.243885 read 0x0091ed thru 0x00b631 (0x2445 bytes) 000891 1789603032.265667 write 0x000376 thru 0x00057d (0x208 bytes) 000892 1789603032.267602 punch from 0x00418f to 0x009835 (0x56a7 bytes) 000893 1789603032.274083 write 0x02fcaf thru 0x035546 (0x5898 bytes) 000894 1789603032.276180 write 0x01f70f thru 0x02117a (0x1a6c bytes) 000895 1789603032.285865 trunc from 0x035547 to 0x02bf42 (0x9604 bytes) 000896 1789603032.312596 fallocate 0x008b17 thru 0x008df1 (0x2db bytes) 000897 1789603032.319454 trunc from 0x02bf43 to 0x01f7e5 (0xc75d bytes) 000898 1789603032.337288 fallocate 0x013d5e thru 0x018890 (0x4b33 bytes) 000901 1789603032.346120 punch from 0x011825 to 0x014d2d (0x3509 bytes) 000902 1789603032.351414 trunc from 0x01e700 to 0x01f7e5 (0x10e6 bytes) 000903 1789603032.365827 mapread 0x01a526 thru 0x01f7e5 (0x52c0 bytes) 000904 1789603032.377623 read 0x003e11 thru 0x00b101 (0x72f1 bytes) 000905 1789603032.389434 read 0x01ef4a thru 0x01f7e5 (0x89c bytes) 000906 1789603032.392172 trunc from 0x00dbdc to 0x0124df (0x4904 bytes) 000907 1789603032.399443 punch from 0x002f5a to 0x012aa5 (0xfb4c bytes) 000909 1789603032.410163 trunc from 0x00a66b to 0x0104b2 (0x5e48 bytes) 000910 1789603032.418563 read 0x0011ad thru 0x0095b5 (0x8409 bytes) 000911 1789603032.436608 read 0x004039 thru 0x0086d8 (0x46a0 bytes) 000912 1789603032.439194 fallocate 0x00e2a4 thru 0x0191b5 (0xaf12 bytes) 000913 1789603032.448128 fallocate 0x028fa6 thru 0x02af7f (0x1fda bytes) 000914 1789603032.455177 punch from 0x017131 to 0x0197ff (0x26cf bytes) 000915 1789603032.463394 punch from 0x012d7d to 0x0213e0 (0xe664 bytes) 000916 1789603032.471805 trunc from 0x02af80 to 0x0345ad (0x962e bytes) 000917 1789603032.493347 write 0x01eb05 thru 0x02e00b (0xf507 bytes) 000918 1789603032.513667 read 0x0105a7 thru 0x0166e0 (0x613a bytes) 000919 1789603032.515115 mapread 0x017557 thru 0x0230b0 (0xbb5a bytes) 000920 1789603032.517501 trunc from 0x0345ae to 0x014174 (0x20439 bytes) 000921 1789603032.537596 write 0x00abed thru 0x01199e (0x6db2 bytes) 000922 1789603032.543435 read 0x010fed thru 0x014174 (0x3188 bytes) 000923 1789603032.545079 trunc from 0x014175 to 0x017a86 (0x3912 bytes) 000924 1789603032.569242 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) 000925 1789603032.751838 read 0x00c1f9 thru 0x017a86 (0xb88e bytes) 000926 1789603032.763687 write 0x003c9f thru 0x012186 (0xe4e8 bytes) 000928 1789603032.791740 mapread 0x0075d1 thru 0x01359c (0xbfcc bytes) 000929 1789603032.794595 write 0x019a79 thru 0x024fab (0xb533 bytes) 000930 1789603032.797818 mapread 0x00c38a thru 0x01b73c (0xf3b3 bytes) 000931 1789603032.807445 punch from 0x0228c5 to 0x024fab (0x26e7 bytes) 000932 1789603032.833553 punch from 0x00af2b to 0x01752c (0xc602 bytes) 000933 1789603032.846772 fallocate 0x0013e5 thru 0x00ddcd (0xc9e9 bytes) 000934 1789603032.852754 write 0x02c3fc thru 0x02d0c9 (0xcce bytes) 000935 1789603032.854151 fallocate 0x033336 thru 0x03ab00 (0x77cb bytes) 000936 1789603032.861496 mapwrite 0x014435 thru 0x02035d (0xbf29 bytes) 000938 1789603032.898180 punch from 0x02533c to 0x026228 (0xeed bytes) 000939 1789603032.903600 fallocate 0x018d41 thru 0x021870 (0x8b30 bytes) 000941 1789603032.910596 read 0x03002e thru 0x036e19 (0x6dec bytes) 000942 1789603032.931681 punch from 0x00ec55 to 0x01dd4f (0xf0fb bytes) 000943 1789603032.937505 write 0x02dec0 thru 0x0328fb (0x4a3c bytes) 000944 1789603032.939713 write 0x02fe81 thru 0x03d3b5 (0xd535 bytes) 000945 1789603032.943192 fallocate 0x012ce2 thru 0x019c8b (0x6faa bytes) 000946 1789603032.949204 fallocate 0x03bb5d thru 0x03fcb3 (0x4157 bytes) 000947 1789603032.955516 mapread 0x016019 thru 0x02140c (0xb3f4 bytes) 000949 1789603033.003465 read 0x018c10 thru 0x01914a (0x53b bytes) 000950 1789603033.004288 mapwrite 0x000749 thru 0x004338 (0x3bf0 bytes) 000951 1789603033.033120 trunc from 0x03fcb4 to 0x030cbe (0xeff5 bytes) 000952 1789603033.053664 fallocate 0x028cd8 thru 0x032869 (0x9b92 bytes) 000954 1789603033.059927 mapread 0x00998f thru 0x0144af (0xab21 bytes) 000956 1789603033.071405 mapread 0x00fccb thru 0x01cba8 (0xcede bytes) 000957 1789603033.072532 mapwrite 0x0123d0 thru 0x02080d (0xe43e bytes) 000959 1789603033.101440 punch from 0x022b48 to 0x024639 (0x1af2 bytes) 000960 1789603033.107257 read 0x02c1fc thru 0x030cbe (0x4ac3 bytes) 000962 1789603033.114622 MIRROR_EXTEND to 2 mirrors 000963 1789603034.242419 mapread 0x017a1b thru 0x01f21b (0x7801 bytes) 000964 1789603034.321020 fallocate 0x011258 thru 0x0117e7 (0x590 bytes) 000965 1789603034.413092 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30cbf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cbf): Mirror 2: 0x368f11c4 (stale) CRC-32 checksum value for chunk [0, 0x30cbf): Mirror 1: 0x368f11c4 Mirror 3: 0x368f11c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cbf: skipped 000966 1789603036.259850 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30cbf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cbf): Mirror 2: 0x368f11c4 (stale) CRC-32 checksum value for chunk [0, 0x30cbf): Mirror 1: 0x368f11c4 Mirror 3: 0x368f11c4 Mirror 4: 0x368f11c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cbf: skipped 000967 1789603038.456824 mapwrite 0x005438 thru 0x014958 (0xf521 bytes) 000970 1789603038.685637 fallocate 0x002948 thru 0x003e3e (0x14f7 bytes) 000971 1789603038.699160 trunc from 0x01c6ba to 0x01e040 (0x1987 bytes) 000972 1789603038.706428 read 0x012f61 thru 0x019c08 (0x6ca8 bytes) 000973 1789603038.707378 mapread 0x003fa2 thru 0x00e469 (0xa4c8 bytes) 000974 1789603038.709468 trunc from 0x0009b3 to 0x0057e1 (0x4e2f bytes) 000975 1789603038.715525 mapwrite 0x02a467 thru 0x039846 (0xf3e0 bytes) 000976 1789603038.796162 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39847): Mirror 2: 0x35385c2a (stale) Mirror 4: 0x35385c2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39847): Mirror 2: 0x35385c2a (stale) Mirror 4: 0x35385c2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39847: skipped 000977 1789603039.166130 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39847): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39847): Mirror 2: 0x35385c2a (stale) Mirror 4: 0x35385c2a (stale) CRC-32 checksum value for chunk [0, 0x39847): Mirror 1: 0x67d83aef Mirror 5: 0x67d83aef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39847: skipped 000978 1789603041.281377 read 0x00076f thru 0x00939c (0x8c2e bytes) 000979 1789603041.304046 mapread 0x00a435 thru 0x01546b (0xb037 bytes) 000981 1789603041.414959 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x178:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39847): Mirror 2: 0x35385c2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39847): Mirror 2: 0x35385c2a (stale) CRC-32 checksum value for chunk [0, 0x39847): Mirror 1: 0x67d83aef Mirror 5: 0x67d83aef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39847: skipped 000982 1789603042.338211 mapwrite 0x03108b thru 0x03686e (0x57e4 bytes) 000984 1789603042.522703 mapwrite 0x00b06f thru 0x018378 (0xd30a bytes) 000985 1789603042.577274 write 0x02b11c thru 0x036665 (0xb54a bytes) 000986 1789603042.596295 read 0x015801 thru 0x01a251 (0x4a51 bytes) 000987 1789603042.604227 trunc from 0x0078b1 to 0x0126c2 (0xae12 bytes) 000988 1789603042.619818 MIRROR_RESYNC 3 mirrors 000989 1789603043.886470 MIRROR_RESYNC 3 mirrors 000990 1789603044.584548 trunc from 0x00df51 to 0x017aa9 (0x9b59 bytes) 000991 1789603044.660510 read 0x0162bc thru 0x01b90b (0x5650 bytes) 000992 1789603044.693001 read 0x002caa thru 0x00acad (0x8004 bytes) 000993 1789603044.719488 trunc from 0x039847 to 0x01ea86 (0x1adc0 bytes) 000994 1789603044.792527 write 0x00f232 thru 0x01225b (0x302a bytes) 000995 1789603044.793854 mapread 0x009efd thru 0x017b33 (0xdc37 bytes) 000996 1789603044.801717 read 0x019144 thru 0x01ea86 (0x5943 bytes) 000997 1789603044.807217 mapwrite 0x00ea7e thru 0x00f47a (0x9fd bytes) 000998 1789603044.878274 MIRROR_RESYNC 3 mirrors 001000 1789603046.891237 mapwrite 0x0078a7 thru 0x01781f (0xff79 bytes) All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 19:57:34 (1789603054) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1bd:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (17s) == sanity-flr test 72: check append on FLR file ========== 19:57:51 (1789603071) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147383 s, 14.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110612 s, 9.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x29d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 19:58:03 (1789603083) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3624000 .. Extending file size to 414816 .. done Extending file size to 6024800 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7657312 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8351904 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6408224 .. Extending file size to 8473376 .. Extending file size to 8562048 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8782112 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61241 61242 61243 61246 61247 done 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (93s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:59:36 (1789603176) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2391936 .. Extending file size to 1863712 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7600608 .. Extending file size to 4619616 .. Extending file size to 8994688 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7963808 .. Extending file size to 8425408 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8546976 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9319360 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9032800 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62492 62493 62496 62497 62498 done 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:00:58 (1789603258) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1268800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18908 3588112 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24728 3582292 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43636 7170404 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1268800 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18908 3588112 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24728 3582292 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43636 7170404 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.354821 s, 5.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.29206 s, 7.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x75497ea6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:01:23 (1789603283) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178389 s, 11.8 MB/s PASS 202 (11s) == sanity-flr test 203: mirror file preserve mirror ID === 20:01:34 (1789603294) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1cc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b2:0x0] } PASS 203 (12s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:01:46 (1789603306) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153257 s, 13.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:01:57 (1789603317) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11838 s, 17.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b4:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:02:06 (1789603326) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159404 s, 13.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b6:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:02:41 (1789603361) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146495 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d4:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:03:37 (1789603417) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139376 s, 15.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:04:32 (1789603472) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg234-server: Pool lustre.test_204f created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174015 s, 12.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg234-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:05:52 (1789603552) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:06:01 (1789603561) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:06:10 (1789603570) oleg234-server: Pool lustre.test_206 created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg234-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:06:40 (1789603600) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826346 s, 12.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258318 s, 40.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x82a9e182 Mirror 3: 0x82a9e182 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:06:53 (1789603613) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:06:56 (1789603616) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:07:01 (1789603621) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0611887 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x790000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b50000 Read this (4 bytes): BBBB 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:07:44 (1789603664) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0333422 s, 246 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:07:54 (1789603674) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802798 s, 13.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ef lmm_fid: [0x200000405:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ef lmm_fid: [0x200000405:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c5:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611631 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 20:07:55.000000000 -0400 Modify: 2026-09-16 20:07:56.000000000 -0400 Change: 2026-09-16 20:07:58.000000000 -0400 Birth: 2026-09-16 20:07:55.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:08:04 (1789603684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547554 s, 19.2 MB/s oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg234-server: O/280000401/d1/2e1: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:08:27 (1789603707) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.224171 s, 183 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0259381 s, 158 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 16 20:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 16 20:08 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:08:35 (1789603715) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:08:43 (1789603723) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2134 sec =========== 20:08:53 (1789603733) === sanity-flr: start cleanup 20:08:55 (1789603735) === === sanity-flr: finish cleanup 20:09:04 (1789603744) ===