-----============= acceptance-small: sanity-flr ============----- Sun Aug 23 10:24:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg319-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:25:02 (1787495102) === oleg319-client.virtnet: executing check_config_client /mnt/lustre oleg319-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg319-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff955b48f34800.idle_timeout=debug osc.lustre-OST0001-osc-ffff955b48f34800.idle_timeout=debug disable quota as required oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:25:45 (1787495145) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8236] == sanity-flr test 0a: lfs mirror create with -N option == 10:25:48 (1787495148) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:26:07 (1787495167) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:26:11 (1787495171) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:26:14 (1787495174) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:26:40 (1787495200) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:26:43 (1787495203) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:26:46 (1787495206) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff955b48f34800.stats=clear osc.lustre-OST0001-osc-ffff955b48f34800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:26:54 (1787495214) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:27:05 (1787495225) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:27:14 (1787495234) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:27:18 (1787495238) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:27:26 (1787495246) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 10:27:35 (1787495255) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:27:44 (1787495264) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:27:53 (1787495273) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:28:01 (1787495281) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:28:12 (1787495292) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:28:19 (1787495299) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.67102 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116567 s, 9.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:28:28 (1787495308) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.435074 s, 31.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:28:36 (1787495316) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:28:46 (1787495326) Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg319-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:30:00 (1787495400) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.593816 s, 17.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:30:50 (1787495450) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg319-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:32:08 (1787495528) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.00017 s, 17.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 23 10:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.119s user 0m0.001s sys 0m0.183s Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff955b48f34800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.050s user 0m0.000s sys 0m0.178s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.233s user 0m0.001s sys 0m0.165s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg319-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff955b48f34800.ost_server_uuid in FULL state after 0 sec PASS 33b (87s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:33:35 (1787495615) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:33:38 (1787495618) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:33:41 (1787495621) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:33:44 (1787495624) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 10:33:51 (1787495631) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.405078 s, 31.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:34:02 (1787495642) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478618 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243191 s, 43.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:34:08 (1787495648) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:34:16 (1787495656) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000756 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00262676 s, 390 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00386758 s, 265 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00410831 s, 249 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 10:34:26 (1787495666) create test files with size 15745228 .. 5cd5985766fb3851922a6fbaccfa07f4 - 6846435f37e11370171ae463398e7b27 - 03ce45ef12052c50ba2a52f267411663 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg319-client.virtnet /mnt/lustre (opts:) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 10:34:49 (1787495689) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:35:03 (1787495703) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:35:09 (1787495709) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784222 s, 13.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499385 s, 21.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 10:35:18 (1787495718) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100235 s, 20.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.299808 s, 14.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203827 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 10:35:31 (1787495731) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 10:35:34 (1787495734) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:35:37 (1787495737) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242255 s, 43.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 10:35:44 (1787495744) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.278175 s, 11.3 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:35:54 (1787495754) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:36:03 (1787495763) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264451 s, 39.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg319-client.virtnet /mnt/lustre (opts:) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:36:12 (1787495772) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff955b59003000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.428782 s, 24.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg319-client.virtnet /mnt/lustre (opts:) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff955b44e30000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:36:28 (1787495788) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:37:07 (1787495827) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:37:37 (1787495857) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:37:40 (1787495860) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.214153 s, 14.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138178 s, 22.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x21610bf (nosync,stale) Mirror 2: 0x21610bf (nosync,stale) ** verify mirror 2 doesn't change original checksum: 86f9842147933953744434af939b962e - mirror 1 checksum: eb9eb28c8c8db59805153a758989560c - mirror 2 checksum: 86f9842147933953744434af939b962e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-23T10:37:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-23T10:37:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 86f9842147933953744434af939b962e - mirror 1 checksum: eb9eb28c8c8db59805153a758989560c - mirror 2 checksum: eb9eb28c8c8db59805153a758989560c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-23T10:37:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-23T10:37:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:37:49 (1787495869) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227172 s, 46.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:37:56 (1787495876) Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 10:38:08 (1787495888) shut down facet: mds1 facet_host: oleg319-server facet_failover_host: oleg319-server Failover mds1 to oleg319-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 10:38:41 (1787495921) targets are mounted 10:38:41 (1787495921) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307459 s, 34.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:38:57 (1787495937) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0372402 s, 550 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0245295 s, 835 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:39:07 (1787495947) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0690584 s, 297 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0121278 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:39:16 (1787495956) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00581445 s, 2.8 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:39:23 (1787495963) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0693538 s, 295 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:39:32 (1787495972) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00472568 s, 867 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0309164 s, 132 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:39:39 (1787495979) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:39:59 (1787495999) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:40:26 (1787496026) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:40:49 (1787496049) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:41:13 (1787496073) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 10:41:21 (1787496081) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57574 mirror split pid 57575 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57574 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57575 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 10:42:27 (1787496147) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6149 fallocating to largest ever: 0x24481 000001 1787496149.818289 fallocate 0x01f7f1 thru 0x024480 (0x4c90 bytes) 000002 1787496149.830864 trunc from 0x024481 to 0x00bfa1 (0x184df bytes) 000003 1787496149.848292 trunc from 0x0003e3 to 0x001f96 (0x1bb4 bytes) 000005 1787496149.855988 mapread 0x000ef6 thru 0x00bfa1 (0xb0ac bytes) 000007 1787496149.887329 mapread 0x0006c1 thru 0x000b87 (0x4c7 bytes) 000008 1787496149.891108 mapread 0x00b7d3 thru 0x00bfa1 (0x7cf bytes) 000011 1787496149.901732 MIRROR_EXTEND to 2 mirrors 000012 1787496150.588482 write 0x012c3d thru 0x016f56 (0x431a bytes) 000013 1787496150.653015 trunc from 0x00754d to 0x0112eb (0x9d9f bytes) 000014 1787496150.667293 punch from 0x005dd6 to 0x010703 (0xa92e bytes) 000015 1787496150.678427 trunc from 0x00bf57 to 0x011053 (0x50fd bytes) 000016 1787496150.689108 write 0x01d440 thru 0x02b26e (0xde2f bytes) 000017 1787496150.706508 trunc from 0x0240cb to 0x025030 (0xf66 bytes) 000018 1787496150.746287 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000019 1787496150.976792 read 0x008f8c thru 0x016032 (0xd0a7 bytes) 000021 1787496150.998857 trunc from 0x0116c0 to 0x01e9b3 (0xd2f4 bytes) 000022 1787496151.018637 write 0x0154d3 thru 0x01da53 (0x8581 bytes) 000023 1787496151.063755 read 0x001241 thru 0x004960 (0x3720 bytes) 000024 1787496151.081336 trunc from 0x02b1b3 to 0x02b26e (0xbc bytes) 000025 1787496151.089308 write 0x01f872 thru 0x02df3d (0xe6cc bytes) 000027 1787496151.091948 mapread 0x01a6bc thru 0x01cb3c (0x2481 bytes) 000028 1787496151.093335 mapread 0x0258d4 thru 0x0268e6 (0x1013 bytes) 000030 1787496151.095091 mapwrite 0x011048 thru 0x01d683 (0xc63c bytes) 000031 1787496151.142671 trunc from 0x01888e to 0x021a99 (0x920c bytes) 000032 1787496151.152009 read 0x01d7c8 thru 0x029401 (0xbc3a bytes) 000033 1787496151.171622 read 0x0151bf thru 0x022cc7 (0xdb09 bytes) 000035 1787496151.172716 trunc from 0x02df3e to 0x00e344 (0x1fbf9 bytes) truncating to largest ever: 0x2f049 000036 1787496151.194914 trunc from 0x00e345 to 0x02f048 (0x20d04 bytes) 000037 1787496151.227256 fallocate 0x01b084 thru 0x0266ee (0xb66b bytes) 000038 1787496151.235943 mapread 0x021869 thru 0x02567a (0x3e12 bytes) 000039 1787496151.253853 punch from 0x016fa6 to 0x023a22 (0xca7d bytes) 000040 1787496151.260246 mapread 0x01c991 thru 0x01e63a (0x1caa bytes) 000041 1787496151.282056 write 0x020838 thru 0x02b0b3 (0xa87c bytes) 000042 1787496151.289813 punch from 0x01546a to 0x01bf07 (0x6a9e bytes) truncating to largest ever: 0x38008 000044 1787496151.302940 trunc from 0x02f049 to 0x038007 (0x8fbf bytes) 000045 1787496151.342066 mapread 0x024abb thru 0x032319 (0xd85f bytes) 000046 1787496151.367098 mapwrite 0x0324fa thru 0x0365c6 (0x40cd bytes) 000047 1787496151.406376 fallocate 0x01a574 thru 0x023426 (0x8eb3 bytes) 000048 1787496151.411599 fallocate 0x011c76 thru 0x01e524 (0xc8af bytes) 000049 1787496151.421774 fallocate 0x001fb6 thru 0x00b7e8 (0x9833 bytes) 000050 1787496151.432405 write 0x00f823 thru 0x015bf9 (0x63d7 bytes) 000051 1787496151.441287 trunc from 0x007ff3 to 0x0171f1 (0xf1ff bytes) 000052 1787496151.471240 mapread 0x026589 thru 0x032c05 (0xc67d bytes) 000053 1787496151.473264 read 0x034f34 thru 0x038007 (0x30d4 bytes) 000054 1787496151.474875 read 0x01b428 thru 0x027099 (0xbc72 bytes) 000055 1787496151.500186 punch from 0x00699d to 0x016541 (0xfba5 bytes) 000058 1787496151.510902 write 0x020043 thru 0x025e6f (0x5e2d bytes) 000059 1787496151.518566 write 0x03ac1a thru 0x03ffff (0x53e6 bytes) 000060 1787496151.524519 mapread 0x021335 thru 0x025349 (0x4015 bytes) 000061 1787496151.526599 trunc from 0x040000 to 0x028286 (0x17d79 bytes) 000062 1787496151.547864 write 0x02972f thru 0x031621 (0x7ef3 bytes) 000065 1787496151.557300 mapread 0x027bf9 thru 0x02bee9 (0x42f1 bytes) 000066 1787496151.558937 fallocate 0x0260be thru 0x033d7c (0xdcbf bytes) 000067 1787496151.564535 mapwrite 0x0384df thru 0x03ffff (0x7b21 bytes) 000068 1787496151.622579 write 0x01b26b thru 0x0268eb (0xb681 bytes) 000070 1787496151.625800 punch from 0x034365 to 0x038b10 (0x47ac bytes) 000071 1787496151.634776 read 0x010e0e thru 0x019dda (0x8fcd bytes) fallocating to largest ever: 0x3fed7 000072 1787496151.660287 fallocate 0x03ab8d thru 0x03fed6 (0x534a bytes) 000075 1787496151.672823 mapread 0x00e8f7 thru 0x0106d7 (0x1de1 bytes) 000076 1787496151.674778 punch from 0x01c680 to 0x025059 (0x89da bytes) fallocating to largest ever: 0x40000 000077 1787496151.696608 fallocate 0x03c7f5 thru 0x03ffff (0x380b bytes) 000079 1787496151.703347 trunc from 0x01727c to 0x01d15b (0x5ee0 bytes) 000080 1787496151.708906 trunc from 0x040000 to 0x0073d0 (0x38c2f bytes) 000081 1787496151.730854 fallocate 0x02e45f thru 0x02f1ea (0xd8c bytes) 000082 1787496151.736578 trunc from 0x0035bd to 0x0073d0 (0x3e14 bytes) 000083 1787496151.741952 mapwrite 0x001669 thru 0x00c024 (0xa9bc bytes) 000084 1787496151.838799 mapwrite 0x0339f7 thru 0x03a07e (0x6688 bytes) 000086 1787496151.938114 mapwrite 0x03c5c2 thru 0x03fec9 (0x3908 bytes) 000087 1787496152.044780 mapwrite 0x001b89 thru 0x0067c9 (0x4c41 bytes) 000088 1787496152.067839 trunc from 0x03feca to 0x024bc3 (0x1b306 bytes) 000089 1787496152.088892 punch from 0x00b6ef to 0x015321 (0x9c33 bytes) 000091 1787496152.096176 trunc from 0x024bc4 to 0x026524 (0x1961 bytes) 000092 1787496152.108534 write 0x0073c6 thru 0x009172 (0x1dad bytes) 000093 1787496152.109729 mapread 0x006267 thru 0x00674d (0x4e7 bytes) 000094 1787496152.110613 fallocate 0x015e4d thru 0x021dbb (0xbf6f bytes) 000095 1787496152.117589 trunc from 0x026525 to 0x035b70 (0xf64c bytes) 000096 1787496152.138759 mapread 0x027a01 thru 0x02ff2b (0x852b bytes) 000097 1787496152.214247 MIRROR_EXTEND to 2 mirrors 000098 1787496152.988358 mapread 0x0248ab thru 0x026db8 (0x250e bytes) 000099 1787496153.015529 mapwrite 0x000568 thru 0x00ebb5 (0xe64e bytes) 000100 1787496153.119986 mapwrite 0x014f3b thru 0x018d6b (0x3e31 bytes) 000101 1787496153.177215 fallocate 0x00bb8d thru 0x017ec2 (0xc336 bytes) 000102 1787496153.193786 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35b71): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b71): Mirror 2: 0x2ba74ac1 (stale) CRC-32 checksum value for chunk [0, 0x35b71): Mirror 1: 0xdd60245c Mirror 3: 0xdd60245c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b71: skipped 000103 1787496154.236322 MIRROR_RESYNC 3 mirrors 000104 1787496155.101837 mapread 0x0205c3 thru 0x022499 (0x1ed7 bytes) 000105 1787496155.118939 write 0x009ee0 thru 0x00e638 (0x4759 bytes) 000106 1787496155.169791 punch from 0x021245 to 0x026ee0 (0x5c9c bytes) 000107 1787496155.185798 MIRROR_RESYNC 3 mirrors 000108 1787496156.256192 read 0x02ba9d thru 0x0327bd (0x6d21 bytes) 000109 1787496156.269008 fallocate 0x02a9e5 thru 0x03170c (0x6d28 bytes) 000110 1787496156.303272 trunc from 0x035b71 to 0x033a0b (0x2165 bytes) 000111 1787496156.344160 read 0x0050c3 thru 0x014254 (0xf192 bytes) 000112 1787496156.357964 trunc from 0x033a0c to 0x0172cb (0x1c740 bytes) 000113 1787496156.386351 mapread 0x00504e thru 0x00bfbb (0x6f6e bytes) 000114 1787496156.402113 MIRROR_RESYNC 3 mirrors 000116 1787496157.262913 read 0x00951f thru 0x0172cb (0xddad bytes) 000117 1787496157.283864 trunc from 0x0172cc to 0x029453 (0x12188 bytes) 000118 1787496157.382400 mapread 0x00eb5b thru 0x017c14 (0x90ba bytes) 000119 1787496157.428198 trunc from 0x029454 to 0x032148 (0x8cf5 bytes) 000121 1787496157.458976 fallocate 0x032124 thru 0x0342de (0x21bb bytes) 000122 1787496157.465864 mapread 0x006678 thru 0x008026 (0x19af bytes) 000123 1787496157.489278 read 0x025605 thru 0x032dd2 (0xd7ce bytes) 000124 1787496157.521535 punch from 0x030a1e to 0x0342de (0x38c1 bytes) 000125 1787496157.526680 read 0x01e7ae thru 0x026eca (0x871d bytes) 000126 1787496157.527303 mapread 0x017f8a thru 0x027234 (0xf2ab bytes) 000127 1787496157.528251 mapwrite 0x000212 thru 0x008f66 (0x8d55 bytes) 000128 1787496157.562009 punch from 0x022fa3 to 0x028f06 (0x5f64 bytes) 000129 1787496157.572208 trunc from 0x026f13 to 0x02f641 (0x872f bytes) 000130 1787496157.596780 punch from 0x00dacf to 0x01514a (0x767c bytes) 000132 1787496157.613250 mapwrite 0x00a1bf thru 0x00ebff (0x4a41 bytes) 000133 1787496157.710104 mapwrite 0x00bbe3 thru 0x00d1ff (0x161d bytes) 000134 1787496157.733266 trunc from 0x0342df to 0x024249 (0x10095 bytes) 000137 1787496157.749988 write 0x02512a thru 0x033929 (0xe800 bytes) 000138 1787496157.752609 mapwrite 0x02b864 thru 0x03281e (0x6fbb bytes) 000139 1787496157.780148 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3392a): Mirror 3: 0x694b41d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3392a): Mirror 3: 0x694b41d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3392a: skipped 000140 1787496158.260671 fallocate 0x01d898 thru 0x0255db (0x7d44 bytes) 000141 1787496158.280820 write 0x01c77b thru 0x01d3da (0xc60 bytes) 000142 1787496158.310682 write 0x02e6c1 thru 0x0355bc (0x6efc bytes) 000143 1787496158.322481 mapwrite 0x017bbb thru 0x021467 (0x98ad bytes) 000144 1787496158.379168 mapread 0x01b55a thru 0x01ce5f (0x1906 bytes) 000145 1787496158.381592 mapwrite 0x026d9f thru 0x035ef0 (0xf152 bytes) 000147 1787496158.500135 mapread 0x019b95 thru 0x01e40e (0x487a bytes) 000148 1787496158.502357 trunc from 0x008053 to 0x00a5a6 (0x2554 bytes) 000149 1787496158.515555 mapwrite 0x0155fd thru 0x01d9a0 (0x83a4 bytes) 000150 1787496158.551186 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000151 1787496158.707754 trunc from 0x01b844 to 0x01e8bc (0x3079 bytes) 000152 1787496158.714346 read 0x00b64c thru 0x00d8cb (0x2280 bytes) 000153 1787496158.725533 trunc from 0x01a70f to 0x022001 (0x78f3 bytes) 000154 1787496158.731426 write 0x03f669 thru 0x03ffff (0x997 bytes) 000155 1787496158.742882 fallocate 0x0032ad thru 0x009051 (0x5da5 bytes) 000156 1787496158.747096 punch from 0x0171ea to 0x022d11 (0xbb28 bytes) 000158 1787496158.753903 trunc from 0x040000 to 0x0011b7 (0x3ee48 bytes) 000159 1787496158.782434 write 0x0338a1 thru 0x038613 (0x4d73 bytes) 000160 1787496158.783902 punch from 0x005294 to 0x00b738 (0x64a5 bytes) 000161 1787496158.790809 punch from 0x00fa1c to 0x01731a (0x78ff bytes) 000162 1787496158.796554 trunc from 0x026b6b to 0x02be6f (0x5305 bytes) 000163 1787496158.804294 fallocate 0x028baa thru 0x032162 (0x95b9 bytes) 000164 1787496158.809887 mapread 0x006283 thru 0x00e264 (0x7fe2 bytes) 000165 1787496158.836789 fallocate 0x0399ea thru 0x03ffff (0x6616 bytes) 000166 1787496158.842151 trunc from 0x03d926 to 0x03e986 (0x1061 bytes) 000167 1787496158.846469 trunc from 0x040000 to 0x035cc7 (0xa338 bytes) 000168 1787496158.874618 punch from 0x009025 to 0x00e553 (0x552f bytes) 000170 1787496158.883318 mapread 0x03071a thru 0x0339fe (0x32e5 bytes) 000171 1787496158.902445 punch from 0x015e27 to 0x01d006 (0x71e0 bytes) 000172 1787496158.906961 punch from 0x011133 to 0x013b67 (0x2a35 bytes) 000173 1787496158.911948 fallocate 0x0136e3 thru 0x016951 (0x326f bytes) 000174 1787496158.927856 trunc from 0x035cc8 to 0x03b2b9 (0x55f2 bytes) 000175 1787496158.971559 mapread 0x008523 thru 0x01795c (0xf43a bytes) 000176 1787496159.015595 fallocate 0x0133fb thru 0x017986 (0x458c bytes) 000178 1787496159.036385 mapwrite 0x03208f thru 0x0395d9 (0x754b bytes) 000179 1787496159.071923 punch from 0x036aaf to 0x03b2b9 (0x480b bytes) 000181 1787496159.078156 punch from 0x01110c to 0x012c5d (0x1b52 bytes) 000182 1787496159.083325 trunc from 0x03b2ba to 0x022898 (0x18a21 bytes) 000183 1787496159.101489 trunc from 0x022899 to 0x01c5dc (0x62bc bytes) 000184 1787496159.129684 mapread 0x01a3f9 thru 0x01c5dc (0x21e4 bytes) 000185 1787496159.137930 trunc from 0x006f07 to 0x00ab79 (0x3c73 bytes) 000186 1787496159.144871 mapwrite 0x0193f7 thru 0x02892e (0xf538 bytes) 000187 1787496159.287151 trunc from 0x02892f to 0x00ec8f (0x19c9f bytes) 000188 1787496159.339567 mapread 0x0003d0 thru 0x008177 (0x7da8 bytes) 000189 1787496159.363347 mapwrite 0x03cd9d thru 0x03ffff (0x3263 bytes) 000190 1787496159.442806 trunc from 0x040000 to 0x00447a (0x3bb85 bytes) 000191 1787496159.483653 mapwrite 0x023c6a thru 0x02d070 (0x9407 bytes) 000192 1787496159.583657 write 0x011729 thru 0x0166da (0x4fb2 bytes) 000193 1787496159.614352 write 0x02b669 thru 0x0332c1 (0x7c59 bytes) 000194 1787496159.617757 trunc from 0x008dff to 0x00a769 (0x196b bytes) 000195 1787496159.638920 write 0x015e0f thru 0x01e3d5 (0x85c7 bytes) 000196 1787496159.646784 trunc from 0x031894 to 0x0332c1 (0x1a2e bytes) 000197 1787496159.692366 write 0x000123 thru 0x009dd7 (0x9cb5 bytes) 000199 1787496159.704608 punch from 0x00adf3 to 0x01421e (0x942c bytes) 000200 1787496159.726735 MIRROR_EXTEND to 2 mirrors 000201 1787496160.740656 mapwrite 0x009b0e thru 0x013d5b (0xa24e bytes) 000202 1787496160.854959 read 0x00f809 thru 0x010c6e (0x1466 bytes) 000203 1787496160.856815 fallocate 0x01777a thru 0x018bc1 (0x1448 bytes) 000204 1787496160.864662 write 0x015ba9 thru 0x01ed15 (0x916d bytes) 000205 1787496160.908385 read 0x009bbe thru 0x00db1d (0x3f60 bytes) 000206 1787496160.917742 mapwrite 0x01de1d thru 0x01f89f (0x1a83 bytes) 000207 1787496160.976352 punch from 0x00c4be to 0x01033c (0x3e7f bytes) 000208 1787496160.989060 write 0x02eaa2 thru 0x033d8f (0x52ee bytes) 000209 1787496161.025184 MIRROR_RESYNC 2 mirrors 000210 1787496161.829262 mapread 0x023779 thru 0x025d43 (0x25cb bytes) 000211 1787496161.857426 fallocate 0x03db4e thru 0x03ffff (0x24b2 bytes) 000212 1787496161.899905 punch from 0x000107 to 0x006342 (0x623c bytes) 000213 1787496161.905612 punch from 0x00eba6 to 0x01dad6 (0xef31 bytes) 000215 1787496161.920253 read 0x036cd1 thru 0x038af1 (0x1e21 bytes) 000216 1787496161.928750 read 0x022514 thru 0x031163 (0xec50 bytes) 000217 1787496161.946222 write 0x005cde thru 0x009846 (0x3b69 bytes) 000218 1787496161.953909 MIRROR_RESYNC 2 mirrors 000219 1787496162.940222 read 0x031a5d thru 0x03bc4a (0xa1ee bytes) 000220 1787496162.960024 write 0x024bf6 thru 0x02e099 (0x94a4 bytes) 000221 1787496163.039607 punch from 0x031ae1 to 0x03b162 (0x9682 bytes) 000222 1787496163.063647 write 0x029c2c thru 0x02ae68 (0x123d bytes) 000224 1787496163.067301 trunc from 0x040000 to 0x00b778 (0x34887 bytes) 000225 1787496163.143553 mapwrite 0x015ba2 thru 0x01869b (0x2afa bytes) 000226 1787496163.251395 trunc from 0x01869c to 0x000ce5 (0x179b6 bytes) 000227 1787496163.293302 punch from 0x000201 to 0x000ce5 (0xae5 bytes) 000228 1787496163.308895 trunc from 0x000ce6 to 0x02abc2 (0x29edd bytes) 000230 1787496163.349818 fallocate 0x0076b7 thru 0x008697 (0xfe1 bytes) 000231 1787496163.377012 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2abc3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2abc3): Mirror 2: 0x7e75de91 (stale) CRC-32 checksum value for chunk [0, 0x2abc3): Mirror 1: 0xda3849b3 Mirror 3: 0xda3849b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2abc3: skipped 000232 1787496164.366860 trunc from 0x007efe to 0x00956a (0x166d bytes) 000233 1787496164.411927 punch from 0x0241a3 to 0x02abc2 (0x6a20 bytes) 000234 1787496164.417553 trunc from 0x02abc3 to 0x011bc3 (0x18fff bytes) 000235 1787496164.448462 punch from 0x00eaa3 to 0x011bc3 (0x3121 bytes) 000236 1787496164.459952 mapwrite 0x03e6af thru 0x03ffff (0x1951 bytes) 000237 1787496164.498013 mapread 0x02c869 thru 0x02ff84 (0x371c bytes) 000238 1787496164.508759 read 0x01a2dc thru 0x026219 (0xbf3e bytes) 000239 1787496164.517015 trunc from 0x040000 to 0x014a69 (0x2b596 bytes) 000240 1787496164.532096 trunc from 0x007262 to 0x00b891 (0x4630 bytes) 000241 1787496164.538969 mapread 0x0063cd thru 0x00b5ca (0x51fe bytes) 000242 1787496164.549734 read 0x0011e9 thru 0x007a05 (0x681d bytes) 000243 1787496164.550440 punch from 0x00f94f to 0x014a69 (0x511b bytes) 000244 1787496164.556948 fallocate 0x03c19d thru 0x03ffff (0x3e63 bytes) 000245 1787496164.562887 mapread 0x012e10 thru 0x01b2eb (0x84dc bytes) 000247 1787496164.581535 trunc from 0x02c910 to 0x032233 (0x5924 bytes) 000248 1787496164.587822 mapwrite 0x037346 thru 0x03ffff (0x8cba bytes) 000249 1787496164.614293 write 0x018080 thru 0x026ac6 (0xea47 bytes) 000250 1787496164.620804 trunc from 0x02b906 to 0x02c7ac (0xea7 bytes) 000251 1787496164.627897 read 0x00a0a6 thru 0x00cbce (0x2b29 bytes) 000253 1787496164.629873 trunc from 0x001af1 to 0x00c792 (0xaca2 bytes) 000254 1787496164.641822 read 0x022afb thru 0x03203d (0xf543 bytes) 000255 1787496164.669818 read 0x0168e5 thru 0x018211 (0x192d bytes) 000256 1787496164.672830 write 0x015c6e thru 0x0235e6 (0xd979 bytes) 000257 1787496164.675089 mapwrite 0x01860a thru 0x01888f (0x286 bytes) 000258 1787496164.732151 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000259 1787496165.112999 read 0x017d3e thru 0x01991b (0x1bde bytes) 000260 1787496165.127958 read 0x004c7b thru 0x00ee83 (0xa209 bytes) 000261 1787496165.149841 mapwrite 0x000b4a thru 0x00dc75 (0xd12c bytes) 000263 1787496165.189487 trunc from 0x040000 to 0x02d8cf (0x12730 bytes) 000264 1787496165.232587 trunc from 0x02d8d0 to 0x0001e4 (0x2d6eb bytes) 000265 1787496165.276682 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000266 1787496165.476844 trunc from 0x000109 to 0x0001e4 (0xdc bytes) 000267 1787496165.486895 mapwrite 0x031811 thru 0x031fd9 (0x7c9 bytes) 000268 1787496165.552790 read 0x01cc74 thru 0x0293b2 (0xc73f bytes) 000269 1787496165.563843 punch from 0x01ae60 to 0x027ae3 (0xcc84 bytes) 000271 1787496165.568861 trunc from 0x01cfbe to 0x024abf (0x7b02 bytes) 000272 1787496165.578490 read 0x01e119 thru 0x0298c7 (0xb7af bytes) 000273 1787496165.597990 write 0x037619 thru 0x03ffff (0x89e7 bytes) 000274 1787496165.599944 trunc from 0x040000 to 0x0300ef (0xff10 bytes) 000276 1787496165.648596 mapwrite 0x03649a thru 0x03ffff (0x9b66 bytes) 000277 1787496165.709431 punch from 0x002d2e to 0x00ebe1 (0xbeb4 bytes) 000278 1787496165.717889 mapread 0x00c19f thru 0x01239a (0x61fc bytes) 000279 1787496165.746171 mapwrite 0x00a850 thru 0x015b87 (0xb338 bytes) 000280 1787496165.790543 punch from 0x01e1d2 to 0x022036 (0x3e65 bytes) 000281 1787496165.795997 trunc from 0x02f6a6 to 0x030032 (0x98d bytes) 000283 1787496165.803829 mapread 0x03905d thru 0x03e1d2 (0x5176 bytes) 000284 1787496165.805023 punch from 0x01a9e2 to 0x025f5e (0xb57d bytes) 000285 1787496165.810753 trunc from 0x040000 to 0x03477c (0xb883 bytes) 000287 1787496165.827449 read 0x01b56a thru 0x02aeb8 (0xf94f bytes) 000288 1787496165.838888 read 0x017446 thru 0x020aea (0x96a5 bytes) 000289 1787496165.840303 write 0x00c298 thru 0x0191de (0xcf47 bytes) 000290 1787496165.843513 trunc from 0x03477d to 0x01c236 (0x18546 bytes) 000292 1787496165.862948 mapread 0x01b625 thru 0x01c236 (0xc12 bytes) 000294 1787496165.865204 write 0x03d9cd thru 0x03ffff (0x2633 bytes) 000295 1787496165.867236 mapwrite 0x032508 thru 0x0395b3 (0x70ac bytes) 000296 1787496165.911818 MIRROR_EXTEND to 2 mirrors 000297 1787496166.901495 punch from 0x0231d3 to 0x02d4c3 (0xa2f1 bytes) 000298 1787496166.959474 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000300 1787496167.149859 punch from 0x02849b to 0x02dc89 (0x57ef bytes) 000301 1787496167.157352 write 0x03673b thru 0x0399e5 (0x32ab bytes) 000302 1787496167.184847 trunc from 0x019b43 to 0x022d92 (0x9250 bytes) 000303 1787496167.194222 punch from 0x03d7b2 to 0x03ffff (0x284e bytes) 000304 1787496167.205548 trunc from 0x040000 to 0x028d9d (0x17262 bytes) 000305 1787496167.236442 mapread 0x00a6a9 thru 0x00f848 (0x51a0 bytes) 000306 1787496167.276386 mapwrite 0x0349cb thru 0x0395fa (0x4c30 bytes) 000308 1787496167.335377 punch from 0x02ba0d to 0x0395fa (0xdbee bytes) 000309 1787496167.341840 fallocate 0x018be5 thru 0x02635a (0xd776 bytes) 000311 1787496167.348369 trunc from 0x008e26 to 0x00af22 (0x20fd bytes) 000312 1787496167.354857 write 0x02e926 thru 0x03b710 (0xcdeb bytes) 000313 1787496167.362778 mapwrite 0x03fc99 thru 0x03ffff (0x367 bytes) 000314 1787496167.402961 read 0x01ab16 thru 0x026a91 (0xbf7c bytes) 000315 1787496167.419918 write 0x020986 thru 0x02ec95 (0xe310 bytes) 000316 1787496167.422908 trunc from 0x040000 to 0x03f186 (0xe79 bytes) 000317 1787496167.439288 write 0x034a0d thru 0x0386b6 (0x3caa bytes) 000318 1787496167.440684 read 0x02ce8f thru 0x032ad7 (0x5c49 bytes) 000319 1787496167.441316 write 0x018b87 thru 0x024956 (0xbdd0 bytes) 000320 1787496167.453537 punch from 0x03d75b to 0x03f186 (0x1a2c bytes) 000321 1787496167.458897 fallocate 0x02ac87 thru 0x0379cd (0xcd47 bytes) 000322 1787496167.464230 mapwrite 0x0226eb thru 0x02cbfe (0xa514 bytes) 000323 1787496167.485703 read 0x038e82 thru 0x03e3a4 (0x5523 bytes) 000326 1787496167.496499 trunc from 0x03f187 to 0x020f38 (0x1e24e bytes) 000327 1787496167.515277 mapread 0x011901 thru 0x013b54 (0x2254 bytes) 000329 1787496167.517469 write 0x01eba6 thru 0x02b572 (0xc9cd bytes) 000330 1787496167.519686 write 0x01a304 thru 0x027ce3 (0xd9e0 bytes) 000331 1787496167.521811 read 0x0194cd thru 0x022ac1 (0x95f5 bytes) 000332 1787496167.522590 trunc from 0x02b573 to 0x037879 (0xc307 bytes) 000333 1787496167.551882 punch from 0x00424b to 0x011e6f (0xdc25 bytes) 000334 1787496167.568787 trunc from 0x03647c to 0x037879 (0x13fe bytes) 000335 1787496167.578762 mapwrite 0x00ef3c thru 0x01c692 (0xd757 bytes) 000336 1787496167.640712 mapread 0x00d2d7 thru 0x0112e4 (0x400e bytes) 000338 1787496167.646008 read 0x0168e5 thru 0x023782 (0xce9e bytes) 000339 1787496167.651690 punch from 0x019970 to 0x01b012 (0x16a3 bytes) 000340 1787496167.668629 trunc from 0x0354bd to 0x037879 (0x23bd bytes) 000341 1787496167.678409 write 0x013c47 thru 0x02344e (0xf808 bytes) 000342 1787496167.686867 trunc from 0x00e749 to 0x014fbb (0x6873 bytes) 000343 1787496167.720898 trunc from 0x03787a to 0x0096fc (0x2e17d bytes) 000344 1787496167.761686 mapread 0x00093d thru 0x0096fc (0x8dc0 bytes) 000345 1787496167.762649 write 0x038c82 thru 0x03ffff (0x737e bytes) 000347 1787496167.765818 fallocate 0x03225e thru 0x03c1d1 (0x9f74 bytes) 000348 1787496167.772856 mapwrite 0x03b341 thru 0x03fa94 (0x4754 bytes) 000350 1787496167.794084 mapread 0x02aa40 thru 0x039b6c (0xf12d bytes) 000351 1787496167.817133 fallocate 0x0038b7 thru 0x0046b6 (0xe00 bytes) 000352 1787496167.822702 punch from 0x0153b0 to 0x01da3f (0x8690 bytes) 000353 1787496167.827730 read 0x030921 thru 0x03be53 (0xb533 bytes) 000354 1787496167.828655 mapwrite 0x03dd36 thru 0x03ffff (0x22ca bytes) 000355 1787496167.845805 mapread 0x032376 thru 0x033efd (0x1b88 bytes) 000356 1787496167.847280 fallocate 0x0109e2 thru 0x01c44c (0xba6b bytes) 000357 1787496167.854163 punch from 0x016e20 to 0x025d88 (0xef69 bytes) 000358 1787496167.861830 mapread 0x039545 thru 0x03ffff (0x6abb bytes) 000359 1787496167.863534 trunc from 0x01aaeb to 0x02780a (0xcd20 bytes) 000360 1787496167.870829 read 0x0246fe thru 0x02985c (0x515f bytes) 000362 1787496167.884393 fallocate 0x0107d1 thru 0x019381 (0x8bb1 bytes) 000363 1787496167.894441 trunc from 0x040000 to 0x02563f (0x1a9c0 bytes) 000364 1787496167.913499 trunc from 0x025640 to 0x019cc6 (0xb979 bytes) 000365 1787496167.935192 trunc from 0x00b8c4 to 0x016baa (0xb2e7 bytes) 000366 1787496167.944427 punch from 0x00fd30 to 0x013a3d (0x3d0e bytes) 000367 1787496167.954235 trunc from 0x019cc7 to 0x0113c1 (0x8905 bytes) 000368 1787496167.977563 trunc from 0x00eb9d to 0x0113c1 (0x2825 bytes) 000370 1787496167.985268 trunc from 0x0113c2 to 0x011ef5 (0xb34 bytes) 000371 1787496168.019804 fallocate 0x002075 thru 0x00d7ff (0xb78b bytes) 000372 1787496168.033225 punch from 0x00924b to 0x011ef5 (0x8cab bytes) 000373 1787496168.053747 write 0x03e498 thru 0x03ffff (0x1b68 bytes) 000374 1787496168.056112 mapread 0x03e5c1 thru 0x03ffff (0x1a3f bytes) 000375 1787496168.068774 write 0x025556 thru 0x02c938 (0x73e3 bytes) 000376 1787496168.092248 trunc from 0x040000 to 0x039c07 (0x63f8 bytes) 000377 1787496168.115004 write 0x02a978 thru 0x036882 (0xbf0b bytes) 000378 1787496168.123609 mapwrite 0x03e2d4 thru 0x03ffff (0x1d2c bytes) 000379 1787496168.189197 fallocate 0x01d2ed thru 0x022874 (0x5588 bytes) 000380 1787496168.194203 trunc from 0x032902 to 0x034c61 (0x2360 bytes) 000381 1787496168.205113 read 0x02c6a2 thru 0x03ab13 (0xe472 bytes) 000384 1787496168.219620 trunc from 0x0396cf to 0x03ffff (0x6931 bytes) 000385 1787496168.226224 trunc from 0x0042bb to 0x012947 (0xe68d bytes) 000386 1787496168.231323 trunc from 0x040000 to 0x024273 (0x1bd8c bytes) 000387 1787496168.257009 MIRROR_EXTEND to 2 mirrors 000388 1787496168.797318 trunc from 0x012133 to 0x01dc2f (0xbafd bytes) 000389 1787496168.851642 trunc from 0x024274 to 0x024f41 (0xcce bytes) 000390 1787496168.905511 trunc from 0x0043c0 to 0x00513d (0xd7e bytes) 000392 1787496168.912112 punch from 0x021059 to 0x02320d (0x21b5 bytes) 000393 1787496168.917613 trunc from 0x024f42 to 0x010bff (0x14342 bytes) 000395 1787496168.942267 MIRROR_RESYNC 2 mirrors 000396 1787496169.411291 trunc from 0x00ee0e to 0x0101e2 (0x13d5 bytes) 000397 1787496169.449117 write 0x039f53 thru 0x03ffff (0x60ad bytes) 000399 1787496169.452712 mapread 0x00c5cb thru 0x018fa2 (0xc9d8 bytes) 000400 1787496169.482846 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa375a41 Mirror 3: 0xa375a41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000401 1787496170.203517 trunc from 0x017b30 to 0x018a59 (0xf2a bytes) 000402 1787496170.261467 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000403 1787496170.785404 trunc from 0x02cb6c to 0x034f7d (0x8412 bytes) 000404 1787496170.792224 trunc from 0x0120db to 0x012885 (0x7ab bytes) 000405 1787496170.800343 fallocate 0x011d8c thru 0x014bb6 (0x2e2b bytes) 000406 1787496170.807142 mapread 0x03a109 thru 0x03ffff (0x5ef7 bytes) 000407 1787496170.844665 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa375a41 Mirror 3: 0xa375a41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000408 1787496172.000651 trunc from 0x0121e1 to 0x016ac4 (0x48e4 bytes) 000409 1787496172.053852 trunc from 0x0398a0 to 0x03ffff (0x6760 bytes) 000410 1787496172.063019 read 0x0324d0 thru 0x03288b (0x3bc bytes) 000411 1787496172.074185 trunc from 0x02057e to 0x02bf64 (0xb9e7 bytes) 000412 1787496172.079245 trunc from 0x02c881 to 0x037e8e (0xb60e bytes) 000413 1787496172.085708 mapwrite 0x01556c thru 0x01586a (0x2ff bytes) 000414 1787496172.104302 trunc from 0x040000 to 0x023126 (0x1ced9 bytes) 000415 1787496172.130088 read 0x022e87 thru 0x023126 (0x2a0 bytes) 000416 1787496172.163303 trunc from 0x023127 to 0x021eb1 (0x1275 bytes) 000417 1787496172.190304 punch from 0x00429e to 0x010cb0 (0xca13 bytes) 000418 1787496172.200622 write 0x019a78 thru 0x01cf15 (0x349e bytes) 000419 1787496172.203172 trunc from 0x021eb2 to 0x019ccd (0x81e4 bytes) 000420 1787496172.225244 trunc from 0x00fe4e to 0x010ad2 (0xc85 bytes) 000421 1787496172.230661 trunc from 0x0097fe to 0x00c5dd (0x2de0 bytes) 000422 1787496172.235965 write 0x00dc29 thru 0x01b078 (0xd450 bytes) 000423 1787496172.243159 trunc from 0x00e3cb to 0x00fffe (0x1c34 bytes) 000424 1787496172.259819 fallocate 0x00f0be thru 0x01265b (0x359e bytes) 000425 1787496172.271636 punch from 0x00c416 to 0x012891 (0x647c bytes) 000426 1787496172.279799 trunc from 0x018d05 to 0x01b078 (0x2374 bytes) 000427 1787496172.287698 trunc from 0x00bdf0 to 0x01819f (0xc3b0 bytes) 000429 1787496172.300337 read 0x00b66d thru 0x00ca38 (0x13cc bytes) 000430 1787496172.323110 write 0x001d20 thru 0x00eed0 (0xd1b1 bytes) 000431 1787496172.327465 trunc from 0x007a7d to 0x00f71c (0x7ca0 bytes) 000433 1787496172.345992 write 0x026ae1 thru 0x029224 (0x2744 bytes) 000434 1787496172.348821 read 0x01c6c7 thru 0x029224 (0xcb5e bytes) 000435 1787496172.359806 mapread 0x01fea8 thru 0x029224 (0x937d bytes) 000436 1787496172.360845 read 0x019043 thru 0x01d517 (0x44d5 bytes) 000437 1787496172.363584 trunc from 0x029225 to 0x013aa7 (0x1577d bytes) 000438 1787496172.383136 punch from 0x00063f to 0x002087 (0x1a49 bytes) 000439 1787496172.392877 punch from 0x000602 to 0x00bcb0 (0xb6af bytes) 000440 1787496172.398461 read 0x005ba2 thru 0x006f2c (0x138b bytes) 000441 1787496172.408455 fallocate 0x0110e0 thru 0x01a9b2 (0x98d3 bytes) 000442 1787496172.417154 mapwrite 0x007396 thru 0x0076d4 (0x33f bytes) 000443 1787496172.439092 fallocate 0x01dcd8 thru 0x0201af (0x24d8 bytes) 000444 1787496172.448798 mapread 0x0024ed thru 0x00601a (0x3b2e bytes) 000445 1787496172.451498 write 0x01d991 thru 0x028eb5 (0xb525 bytes) 000446 1787496172.482708 fallocate 0x00d5ab thru 0x011756 (0x41ac bytes) 000448 1787496172.492706 trunc from 0x028eb6 to 0x02d6a7 (0x47f2 bytes) 000449 1787496172.526522 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d6a8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d6a8): Mirror 2: 0x1ea9968 (stale) Mirror 3: 0x1ea9968 (stale) CRC-32 checksum value for chunk [0, 0x2d6a8): Mirror 1: 0x527a295b Mirror 4: 0x527a295b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d6a8: skipped 000450 1787496173.636630 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d6a8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d6a8): Mirror 2: 0x1ea9968 (stale) Mirror 3: 0x1ea9968 (stale) CRC-32 checksum value for chunk [0, 0x2d6a8): Mirror 1: 0x527a295b Mirror 4: 0x527a295b Mirror 5: 0x527a295b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d6a8: skipped 000452 1787496174.801375 punch from 0x0015b0 to 0x00a2b4 (0x8d05 bytes) 000453 1787496174.855237 mapwrite 0x010e0d thru 0x01efc3 (0xe1b7 bytes) 000454 1787496174.934601 trunc from 0x02d6a8 to 0x032422 (0x4d7b bytes) 000455 1787496174.969583 trunc from 0x032423 to 0x0302e2 (0x2140 bytes) 000457 1787496174.984743 fallocate 0x02ec63 thru 0x030528 (0x18c6 bytes) 000458 1787496174.989790 mapwrite 0x03324f thru 0x03a36e (0x7120 bytes) 000459 1787496175.032092 punch from 0x02edb0 to 0x039f87 (0xb1d8 bytes) 000462 1787496175.042826 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a36f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a36f): Mirror 2: 0xb6baa7a0 (stale) Mirror 3: 0x2ff52204 (stale) Mirror 4: 0xbcf5f57b (stale) Mirror 5: 0xbcf5f57b (stale) CRC-32 checksum value for chunk [0, 0x3a36f): Mirror 1: 0x6fba5d11 Mirror 6: 0x6fba5d11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a36f: skipped 000463 1787496176.013494 mapread 0x034e14 thru 0x03a36e (0x555b bytes) 000464 1787496176.073670 punch from 0x0105b1 to 0x017863 (0x72b3 bytes) 000465 1787496176.109873 write 0x02f3bd thru 0x03717b (0x7dbf bytes) 000466 1787496176.123783 fallocate 0x000e23 thru 0x0087e3 (0x79c1 bytes) 000467 1787496176.129131 trunc from 0x03a36f to 0x002f0c (0x37462 bytes) 000468 1787496176.159194 mapread 0x00188c thru 0x002f0c (0x1681 bytes) 000470 1787496176.166269 trunc from 0x002f0d to 0x013252 (0x10346 bytes) 000471 1787496176.194449 read 0x0018fa thru 0x011309 (0xfa10 bytes) 000473 1787496176.223941 MIRROR_RESYNC 6 mirrors 000474 1787496177.598491 mapread 0x0084ae thru 0x00e242 (0x5d95 bytes) 000475 1787496177.643337 mapread 0x00c837 thru 0x013252 (0x6a1c bytes) 000476 1787496177.661157 read 0x0048a1 thru 0x0120bd (0xd81d bytes) 000477 1787496177.661774 fallocate 0x013596 thru 0x013a4f (0x4ba bytes) 000478 1787496177.746877 fallocate 0x009bcd thru 0x00b104 (0x1538 bytes) 000479 1787496177.752950 punch from 0x00a5f5 to 0x013252 (0x8c5e bytes) 000480 1787496177.762478 read 0x000746 thru 0x00a8de (0xa199 bytes) 000481 1787496177.792734 read 0x000c72 thru 0x00244d (0x17dc bytes) 000482 1787496177.798983 mapread 0x008a3b thru 0x00dfe9 (0x55af bytes) 000483 1787496177.805605 MIRROR_RESYNC 6 mirrors 000484 1787496179.417901 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000485 1787496180.045208 MIRROR_RESYNC 5 mirrors 000486 1787496180.706572 read 0x012408 thru 0x013252 (0xe4b bytes) 000487 1787496180.713331 fallocate 0x02eedc thru 0x037057 (0x817c bytes) 000488 1787496180.754871 fallocate 0x00ade7 thru 0x015d2d (0xaf47 bytes) 000489 1787496180.759584 trunc from 0x015d2e to 0x03662f (0x20902 bytes) 000490 1787496180.790365 trunc from 0x036630 to 0x00aa41 (0x2bbee bytes) 000491 1787496180.806718 trunc from 0x00aa42 to 0x00dbb9 (0x3178 bytes) 000493 1787496180.824529 fallocate 0x010221 thru 0x016f84 (0x6d64 bytes) 000494 1787496180.829791 write 0x03cca6 thru 0x03e595 (0x18f0 bytes) 000495 1787496180.835826 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e596): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e596): Mirror 2: 0x6767a66 (stale) Mirror 3: 0x6767a66 (stale) Mirror 5: 0x6767a66 (stale) Mirror 6: 0x6767a66 (stale) CRC-32 checksum value for chunk [0, 0x3e596): Mirror 1: 0x669c5c7c Mirror 7: 0x669c5c7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e596: skipped 000496 1787496182.172366 read 0x01070c thru 0x01f67a (0xef6f bytes) 000498 1787496182.212937 fallocate 0x00495b thru 0x00722e (0x28d4 bytes) 000499 1787496182.292712 write 0x023d84 thru 0x030425 (0xc6a2 bytes) 000500 1787496182.313564 mapwrite 0x001f68 thru 0x010e85 (0xef1e bytes) 000501 1787496182.382328 mapread 0x038aca thru 0x03e595 (0x5acc bytes) 000503 1787496182.395425 read 0x018cb1 thru 0x01d8df (0x4c2f bytes) 000504 1787496182.416323 write 0x02abbb thru 0x037df1 (0xd237 bytes) 000505 1787496182.419640 punch from 0x0042d4 to 0x0128ae (0xe5db bytes) 000506 1787496182.428349 mapwrite 0x008818 thru 0x0173ca (0xebb3 bytes) 000507 1787496182.509133 trunc from 0x013705 to 0x01f475 (0xbd71 bytes) 000509 1787496182.515848 write 0x01bbd5 thru 0x01c8bf (0xceb bytes) 000510 1787496182.524905 trunc from 0x032adf to 0x03e595 (0xbab7 bytes) 000511 1787496182.530677 mapwrite 0x03aa9a thru 0x03ffff (0x5566 bytes) 000512 1787496182.590008 fallocate 0x02bcb3 thru 0x034f57 (0x92a5 bytes) 000513 1787496182.595555 trunc from 0x040000 to 0x0255dc (0x1aa23 bytes) 000514 1787496182.612145 read 0x01ccd3 thru 0x01d102 (0x430 bytes) 000515 1787496182.622012 write 0x00da73 thru 0x0144d2 (0x6a60 bytes) 000516 1787496182.623268 write 0x01dfe1 thru 0x02b065 (0xd085 bytes) 000517 1787496182.624799 punch from 0x01716d to 0x023a88 (0xc91c bytes) 000518 1787496182.640553 trunc from 0x014361 to 0x020134 (0xbdd4 bytes) 000519 1787496182.648710 mapwrite 0x02edb0 thru 0x03c366 (0xd5b7 bytes) 000520 1787496182.750470 trunc from 0x032a51 to 0x0394d1 (0x6a81 bytes) 000522 1787496182.779966 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c367): Mirror 3: 0xb4297c45 (stale) Mirror 5: 0xb4297c45 (stale) Mirror 6: 0xb4297c45 (stale) Mirror 7: 0xb4297c45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c367): Mirror 3: 0xb4297c45 (stale) Mirror 5: 0xb4297c45 (stale) Mirror 6: 0xb4297c45 (stale) Mirror 7: 0xb4297c45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c367: skipped 000523 1787496183.259231 fallocate 0x02c529 thru 0x037f28 (0xba00 bytes) 000524 1787496183.266196 mapread 0x00f9a7 thru 0x01177b (0x1dd5 bytes) 000525 1787496183.288898 trunc from 0x02da21 to 0x030546 (0x2b26 bytes) 000526 1787496183.300735 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c367): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c367): Mirror 3: 0xb4297c45 (stale) Mirror 5: 0xb4297c45 (stale) Mirror 6: 0xb4297c45 (stale) Mirror 7: 0xb4297c45 (stale) CRC-32 checksum value for chunk [0, 0x3c367): Mirror 1: 0x633b9a1b Mirror 8: 0x633b9a1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c367: skipped 000527 1787496184.590362 trunc from 0x03c367 to 0x000bde (0x3b788 bytes) 000528 1787496184.646493 punch from 0x000581 to 0x000bde (0x65e bytes) 000529 1787496184.651998 fallocate 0x0350e5 thru 0x0383ed (0x3309 bytes) 000530 1787496184.660563 punch from 0x00c9fc to 0x01b9d2 (0xefd7 bytes) 000531 1787496184.668707 write 0x029b49 thru 0x036446 (0xc8fe bytes) 000532 1787496184.684684 fallocate 0x01eed3 thru 0x027235 (0x8363 bytes) 000533 1787496184.691165 write 0x01c632 thru 0x026db3 (0xa782 bytes) 000534 1787496184.710536 mapread 0x0044eb thru 0x01331e (0xee34 bytes) 000536 1787496184.737207 mapread 0x020ad2 thru 0x0293ed (0x891c bytes) 000537 1787496184.742972 punch from 0x00de11 to 0x016917 (0x8b07 bytes) 000538 1787496184.748604 write 0x03fbc0 thru 0x03ffff (0x440 bytes) 000539 1787496184.749802 mapread 0x001252 thru 0x007778 (0x6527 bytes) 000540 1787496184.750816 trunc from 0x040000 to 0x009e32 (0x361cd bytes) 000541 1787496184.772722 mapwrite 0x00c1d2 thru 0x00f2c6 (0x30f5 bytes) 000542 1787496184.857701 read 0x006e25 thru 0x00f2c6 (0x84a2 bytes) 000543 1787496184.858925 read 0x00b5cb thru 0x00e0b0 (0x2ae6 bytes) 000544 1787496184.867648 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf2c7): Mirror 3: 0xa9d0b816 (stale) Mirror 5: 0xa9d0b816 (stale) Mirror 7: 0xa9d0b816 (stale) Mirror 8: 0x78c41612 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf2c7): Mirror 3: 0xa9d0b816 (stale) Mirror 5: 0xa9d0b816 (stale) Mirror 7: 0xa9d0b816 (stale) Mirror 8: 0x78c41612 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf2c7: skipped 000545 1787496185.498634 punch from 0x00991d to 0x00f2c6 (0x59aa bytes) 000546 1787496185.508528 read 0x00e43b thru 0x00f2c6 (0xe8c bytes) 000547 1787496185.526352 MIRROR_RESYNC 5 mirrors 000548 1787496186.755824 MIRROR_EXTEND to 6 mirrors 000549 1787496187.675363 trunc from 0x00f2c7 to 0x013e95 (0x4bcf bytes) 000550 1787496187.744858 fallocate 0x00cb05 thru 0x00e5be (0x1aba bytes) 000551 1787496187.751572 trunc from 0x0041af to 0x011ba1 (0xd9f3 bytes) 000552 1787496187.757658 trunc from 0x005485 to 0x00bff9 (0x6b75 bytes) 000553 1787496187.766623 trunc from 0x013e96 to 0x036975 (0x22ae0 bytes) 000554 1787496187.801064 trunc from 0x036976 to 0x000f52 (0x35a23 bytes) 000555 1787496187.825472 fallocate 0x0161f4 thru 0x016353 (0x160 bytes) 000556 1787496187.831859 fallocate 0x030b3d thru 0x038235 (0x76f9 bytes) 000557 1787496187.839285 trunc from 0x000e5d to 0x000f52 (0xf6 bytes) 000558 1787496187.849312 trunc from 0x000f53 to 0x008b79 (0x7c27 bytes) 000559 1787496187.875693 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x8b7a): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b7a): Mirror 3: 0x784fe530 (stale) Mirror 5: 0x784fe530 (stale) Mirror 7: 0x784fe530 (stale) Mirror 8: 0x784fe530 (stale) Mirror 9: 0x784fe530 (stale) CRC-32 checksum value for chunk [0, 0x8b7a): Mirror 1: 0x784fe530 Mirror 10: 0x784fe530 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b7a: skipped 000560 1787496188.979500 trunc from 0x008b7a to 0x030b72 (0x27ff9 bytes) 000561 1787496189.051895 punch from 0x002150 to 0x008bfd (0x6aae bytes) 000562 1787496189.061501 trunc from 0x00c057 to 0x019304 (0xd2ae bytes) 000564 1787496189.069621 trunc from 0x030b73 to 0x03bf4b (0xb3d9 bytes) 000565 1787496189.089369 mapread 0x0258f7 thru 0x02b962 (0x606c bytes) 000566 1787496189.107957 mapread 0x0088da thru 0x00c2ff (0x3a26 bytes) 000567 1787496189.117792 read 0x036611 thru 0x03b811 (0x5201 bytes) 000568 1787496189.127729 trunc from 0x03bf4c to 0x000af2 (0x3b459 bytes) 000569 1787496189.148282 trunc from 0x000af3 to 0x008c5a (0x8168 bytes) 000570 1787496189.180321 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x8c5b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8c5b): Mirror 3: 0x7d96f5f6 (stale) Mirror 5: 0x7d96f5f6 (stale) Mirror 7: 0x7d96f5f6 (stale) Mirror 8: 0x7d96f5f6 (stale) Mirror 9: 0x7d96f5f6 (stale) Mirror 10: 0x7d96f5f6 (stale) CRC-32 checksum value for chunk [0, 0x8c5b): Mirror 1: 0x7d96f5f6 Mirror 11: 0x7d96f5f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8c5b: skipped 000571 1787496190.102542 mapwrite 0x0001b9 thru 0x006119 (0x5f61 bytes) 000572 1787496190.275961 trunc from 0x00392a to 0x0058da (0x1fb1 bytes) 000573 1787496190.287598 trunc from 0x004234 to 0x008c5a (0x4a27 bytes) 000574 1787496190.297344 trunc from 0x0024fc to 0x008c5a (0x675f bytes) 000576 1787496190.308975 read 0x004c70 thru 0x006df4 (0x2185 bytes) 000577 1787496190.322427 trunc from 0x0074dd to 0x008c5a (0x177e bytes) 000579 1787496190.329567 mapread 0x0030c0 thru 0x008c5a (0x5b9b bytes) 000580 1787496190.347240 fallocate 0x01368a thru 0x020e15 (0xd78c bytes) 000581 1787496190.352955 fallocate 0x012f49 thru 0x01e234 (0xb2ec bytes) 000582 1787496190.358082 mapread 0x007d60 thru 0x008c5a (0xefb bytes) 000584 1787496190.359998 read 0x0052f3 thru 0x0053bc (0xca bytes) 000585 1787496190.361005 punch from 0x000eca to 0x008c5a (0x7d91 bytes) 000586 1787496190.367020 read 0x0072fe thru 0x008c5a (0x195d bytes) 000587 1787496190.379222 trunc from 0x00773f to 0x008c5a (0x151c bytes) 000588 1787496190.384605 mapread 0x0083e1 thru 0x008c5a (0x87a bytes) 000589 1787496190.390607 write 0x03d052 thru 0x03ffff (0x2fae bytes) 000590 1787496190.396867 fallocate 0x017f47 thru 0x02707d (0xf137 bytes) 000591 1787496190.402696 fallocate 0x02e034 thru 0x03ce26 (0xedf3 bytes) 000592 1787496190.411621 mapread 0x000e9c thru 0x00c6b9 (0xb81e bytes) 000593 1787496190.424874 read 0x0175cd thru 0x01c304 (0x4d38 bytes) 000594 1787496190.425645 fallocate 0x0274fe thru 0x02b8b4 (0x43b7 bytes) 000595 1787496190.430840 punch from 0x039f6e to 0x03cee9 (0x2f7c bytes) 000596 1787496190.435179 write 0x013ed3 thru 0x01b5d6 (0x7704 bytes) 000597 1787496190.437159 trunc from 0x013121 to 0x013a9e (0x97e bytes) 000598 1787496190.457474 write 0x001b58 thru 0x00f02a (0xd4d3 bytes) 000599 1787496190.460337 trunc from 0x040000 to 0x01a9c3 (0x2563c bytes) 000600 1787496190.513747 mapwrite 0x02cf84 thru 0x036807 (0x9884 bytes) 000601 1787496190.573693 punch from 0x00a3a9 to 0x00bdfc (0x1a54 bytes) 000602 1787496190.580818 read 0x01d11b thru 0x021022 (0x3f08 bytes) 000603 1787496190.581797 read 0x017a93 thru 0x01a7bf (0x2d2d bytes) 000604 1787496190.583712 read 0x023aeb thru 0x0304ae (0xc9c4 bytes) 000605 1787496190.584620 write 0x01ac72 thru 0x02487d (0x9c0c bytes) 000606 1787496190.589332 mapwrite 0x02fc80 thru 0x03600b (0x638c bytes) 000607 1787496190.639000 write 0x00aec1 thru 0x01085f (0x599f bytes) 000608 1787496190.645451 mapwrite 0x01203d thru 0x01b433 (0x93f7 bytes) 000609 1787496190.670158 mapwrite 0x00fa3d thru 0x0133fa (0x39be bytes) 000610 1787496190.698222 fallocate 0x03c6af thru 0x03ffff (0x3951 bytes) 000611 1787496190.705754 trunc from 0x036808 to 0x010414 (0x263f3 bytes) 000612 1787496190.735290 mapwrite 0x013653 thru 0x01bab8 (0x8466 bytes) 000613 1787496190.822918 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1bab9): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bab9): Mirror 3: 0x851c08a0 (stale) Mirror 5: 0x851c08a0 (stale) Mirror 7: 0x851c08a0 (stale) Mirror 8: 0x851c08a0 (stale) Mirror 9: 0x851c08a0 (stale) Mirror 10: 0x851c08a0 (stale) Mirror 11: 0x851c08a0 (stale) CRC-32 checksum value for chunk [0, 0x1bab9): Mirror 1: 0x6b6b8cb7 Mirror 12: 0x6b6b8cb7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bab9: skipped 000615 1787496192.060417 trunc from 0x018fa9 to 0x01bab8 (0x2b10 bytes) 000616 1787496192.130942 write 0x00c614 thru 0x01adfa (0xe7e7 bytes) 000617 1787496192.148014 fallocate 0x0283df thru 0x02cf5f (0x4b81 bytes) 000618 1787496192.152775 mapread 0x01d2bf thru 0x01faeb (0x282d bytes) 000619 1787496192.169765 trunc from 0x02cf60 to 0x02d2c0 (0x361 bytes) 000620 1787496192.207839 mapwrite 0x036224 thru 0x03a95d (0x473a bytes) 000621 1787496192.311917 fallocate 0x03d9c1 thru 0x03ffff (0x263f bytes) 000622 1787496192.320283 mapread 0x006911 thru 0x013129 (0xc819 bytes) 000623 1787496192.338695 mapwrite 0x021a38 thru 0x02aee9 (0x94b2 bytes) 000624 1787496192.374101 trunc from 0x03a95e to 0x02496e (0x15fef bytes) 000625 1787496192.395952 punch from 0x015a07 to 0x01fd45 (0xa33f bytes) 000626 1787496192.409902 mapread 0x000267 thru 0x001da4 (0x1b3e bytes) 000627 1787496192.417475 mapread 0x021827 thru 0x02496e (0x3148 bytes) 000628 1787496192.418341 fallocate 0x031340 thru 0x03e1c5 (0xce86 bytes) 000630 1787496192.426571 write 0x0044d2 thru 0x00f555 (0xb084 bytes) 000631 1787496192.430642 write 0x03f1cb thru 0x03ffff (0xe35 bytes) 000632 1787496192.431939 trunc from 0x040000 to 0x00c6b9 (0x33946 bytes) 000633 1787496192.457168 punch from 0x00592c to 0x007a7b (0x2150 bytes) 000634 1787496192.487699 trunc from 0x00c6ba to 0x01d30e (0x10c55 bytes) 000635 1787496192.505490 read 0x00c57d thru 0x00ec06 (0x268a bytes) 000636 1787496192.515284 fallocate 0x0350c2 thru 0x03ffff (0xaf3e bytes) 000637 1787496192.530357 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1d30f): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d30f): Mirror 3: 0xda8f5f2f (stale) Mirror 5: 0xda8f5f2f (stale) Mirror 7: 0xda8f5f2f (stale) Mirror 8: 0xda8f5f2f (stale) Mirror 9: 0xda8f5f2f (stale) Mirror 10: 0xda8f5f2f (stale) Mirror 11: 0xda8f5f2f (stale) Mirror 12: 0x65300016 (stale) CRC-32 checksum value for chunk [0, 0x1d30f): Mirror 1: 0x809c01a8 Mirror 13: 0x809c01a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d30f: skipped 000639 1787496193.648442 mapread 0x0104d9 thru 0x01a1d5 (0x9cfd bytes) 000640 1787496193.694017 write 0x0004d6 thru 0x0080f3 (0x7c1e bytes) 000641 1787496193.769502 punch from 0x00290c to 0x0039a1 (0x1096 bytes) 000642 1787496193.807934 trunc from 0x01d30f to 0x030774 (0x13466 bytes) 000643 1787496193.890933 punch from 0x02ec09 to 0x030774 (0x1b6c bytes) 000644 1787496193.896536 write 0x020d57 thru 0x02d536 (0xc7e0 bytes) 000645 1787496193.911114 read 0x0206c5 thru 0x02ff8d (0xf8c9 bytes) 000646 1787496193.917489 punch from 0x01d5e3 to 0x0205ca (0x2fe8 bytes) 000647 1787496193.932911 write 0x032645 thru 0x03ffff (0xd9bb bytes) 000649 1787496193.935744 punch from 0x01bcc5 to 0x029c73 (0xdfaf bytes) 000650 1787496193.940954 fallocate 0x03569a thru 0x035f14 (0x87b bytes) 000651 1787496193.946709 mapwrite 0x017080 thru 0x023184 (0xc105 bytes) 000653 1787496194.041728 punch from 0x02ceac to 0x0347e8 (0x793d bytes) 000654 1787496194.048737 mapwrite 0x00a4e9 thru 0x00aef2 (0xa0a bytes) 000656 1787496194.076074 read 0x024989 thru 0x02ccac (0x8324 bytes) 000657 1787496194.091210 punch from 0x024ab7 to 0x026473 (0x19bd bytes) 000658 1787496194.096641 mapread 0x02ad19 thru 0x02fa12 (0x4cfa bytes) 000659 1787496194.097936 mapread 0x008e82 thru 0x009199 (0x318 bytes) 000660 1787496194.099737 read 0x00f5a7 thru 0x018a10 (0x946a bytes) 000661 1787496194.100437 mapread 0x01b1e2 thru 0x02569c (0xa4bb bytes) 000662 1787496194.108288 mapwrite 0x02d9c3 thru 0x0336e9 (0x5d27 bytes) 000663 1787496194.127330 punch from 0x0017f5 to 0x003261 (0x1a6d bytes) 000664 1787496194.137176 MIRROR_RESYNC 10 mirrors 000665 1787496197.674294 trunc from 0x01ab23 to 0x023e9e (0x937c bytes) 000666 1787496197.745511 fallocate 0x006c48 thru 0x01595b (0xed14 bytes) 000667 1787496197.758826 write 0x022740 thru 0x025fdb (0x389c bytes) 000668 1787496197.775915 trunc from 0x007026 to 0x013315 (0xc2f0 bytes) 000669 1787496197.782885 read 0x00bb7b thru 0x0154a9 (0x992f bytes) 000670 1787496197.793946 punch from 0x00c36d to 0x01bf48 (0xfbdc bytes) 000672 1787496197.804749 mapread 0x023136 thru 0x02c4a2 (0x936d bytes) 000673 1787496197.837635 punch from 0x015d40 to 0x021e79 (0xc13a bytes) 000674 1787496197.846062 read 0x00120f thru 0x00d94b (0xc73d bytes) 000675 1787496197.864726 mapread 0x03ec2a thru 0x03f0cc (0x4a3 bytes) 000676 1787496197.866311 punch from 0x01138f to 0x014cd8 (0x394a bytes) 000677 1787496197.870998 read 0x031cd6 thru 0x03e36e (0xc699 bytes) 000678 1787496197.871946 fallocate 0x01f6e2 thru 0x01ff43 (0x862 bytes) 000679 1787496197.881712 mapwrite 0x00aa11 thru 0x00ee67 (0x4457 bytes) 000680 1787496197.900020 punch from 0x0271fc to 0x03377f (0xc584 bytes) 000681 1787496197.905025 mapwrite 0x014762 thru 0x021677 (0xcf16 bytes) 000682 1787496197.956802 read 0x03bc3a thru 0x03ffff (0x43c6 bytes) 000683 1787496197.957993 read 0x0258ea thru 0x027674 (0x1d8b bytes) 000684 1787496197.962234 fallocate 0x03a7f1 thru 0x03e4c0 (0x3cd0 bytes) 000685 1787496197.972708 mapwrite 0x017906 thru 0x01bfc6 (0x46c1 bytes) 000686 1787496198.031392 MIRROR_RESYNC 10 mirrors 000687 1787496201.281623 fallocate 0x02d0cc thru 0x039f58 (0xce8d bytes) 000688 1787496201.334860 punch from 0x031578 to 0x037a2a (0x64b3 bytes) 000689 1787496201.342938 punch from 0x028d96 to 0x02fc2a (0x6e95 bytes) 000690 1787496201.351604 fallocate 0x021391 thru 0x02eca9 (0xd919 bytes) 000691 1787496201.362638 read 0x020728 thru 0x02d386 (0xcc5f bytes) 000692 1787496201.387444 mapwrite 0x02fcb7 thru 0x0303a9 (0x6f3 bytes) 000693 1787496201.423682 write 0x015d13 thru 0x018e64 (0x3152 bytes) 000694 1787496201.442235 punch from 0x028e24 to 0x02ce2b (0x4008 bytes) 000695 1787496201.452928 trunc from 0x02da45 to 0x03ad11 (0xd2cd bytes) 000696 1787496201.458392 write 0x0006a7 thru 0x007256 (0x6bb0 bytes) 000697 1787496201.495927 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe753d9b2 (stale) Mirror 5: 0xe753d9b2 (stale) Mirror 7: 0xe753d9b2 (stale) Mirror 8: 0xe753d9b2 (stale) Mirror 9: 0xe753d9b2 (stale) Mirror 10: 0xe753d9b2 (stale) Mirror 11: 0xe753d9b2 (stale) Mirror 12: 0xe753d9b2 (stale) Mirror 13: 0xe753d9b2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96f0b31c Mirror 14: 0x96f0b31c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000698 1787496202.994572 mapwrite 0x036ac4 thru 0x03ffff (0x953c bytes) 000699 1787496203.150079 mapwrite 0x0216af thru 0x02f44b (0xdd9d bytes) 000700 1787496203.205250 trunc from 0x038629 to 0x03bca3 (0x367b bytes) 000701 1787496203.213726 write 0x01e373 thru 0x01ee17 (0xaa5 bytes) 000702 1787496203.226189 trunc from 0x00dc84 to 0x019245 (0xb5c2 bytes) 000703 1787496203.238573 trunc from 0x012ea1 to 0x01e5ca (0xb72a bytes) 000704 1787496203.250393 write 0x03cd7f thru 0x03ffff (0x3281 bytes) 000705 1787496203.251929 punch from 0x0177cd to 0x02021e (0x8a52 bytes) 000706 1787496203.257560 mapread 0x027dc1 thru 0x032086 (0xa2c6 bytes) 000708 1787496203.262591 mapwrite 0x02492f thru 0x032c6c (0xe33e bytes) 000709 1787496203.284655 punch from 0x00f051 to 0x0179d9 (0x8989 bytes) 000710 1787496203.289003 mapwrite 0x01fa1f thru 0x02f31e (0xf900 bytes) 000711 1787496203.315252 fallocate 0x026cda thru 0x0288c2 (0x1be9 bytes) 000713 1787496203.320823 mapwrite 0x02173a thru 0x0257dd (0x40a4 bytes) 000714 1787496203.336648 mapwrite 0x00f152 thru 0x015397 (0x6246 bytes) 000715 1787496203.373603 fallocate 0x030f6e thru 0x03ee98 (0xdf2b bytes) 000716 1787496203.379815 read 0x035f7f thru 0x03ffff (0xa081 bytes) 000717 1787496203.392604 punch from 0x030c0d to 0x0326bf (0x1ab3 bytes) 000718 1787496203.402641 trunc from 0x032600 to 0x035db2 (0x37b3 bytes) 000719 1787496203.416856 mapread 0x013c2f thru 0x01b738 (0x7b0a bytes) 000720 1787496203.418088 write 0x0145ca thru 0x022144 (0xdb7b bytes) 000721 1787496203.421746 fallocate 0x01d142 thru 0x0240d6 (0x6f95 bytes) 000722 1787496203.429769 mapread 0x01070b thru 0x011cd1 (0x15c7 bytes) 000723 1787496203.430710 fallocate 0x018f98 thru 0x01b69c (0x2705 bytes) 000724 1787496203.438381 trunc from 0x025fea to 0x03116d (0xb184 bytes) 000725 1787496203.455660 trunc from 0x025fd0 to 0x02adeb (0x4e1c bytes) 000727 1787496203.468882 write 0x03beba thru 0x03ffff (0x4146 bytes) 000728 1787496203.477744 fallocate 0x036840 thru 0x03ffff (0x97c0 bytes) 000729 1787496203.503858 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe753d9b2 (stale) Mirror 5: 0xe753d9b2 (stale) Mirror 7: 0xe753d9b2 (stale) Mirror 8: 0xe753d9b2 (stale) Mirror 9: 0xe753d9b2 (stale) Mirror 11: 0xe753d9b2 (stale) Mirror 12: 0xe753d9b2 (stale) Mirror 13: 0xe753d9b2 (stale) Mirror 14: 0x96f0b31c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe753d9b2 (stale) Mirror 5: 0xe753d9b2 (stale) Mirror 7: 0xe753d9b2 (stale) Mirror 8: 0xe753d9b2 (stale) Mirror 9: 0xe753d9b2 (stale) Mirror 11: 0xe753d9b2 (stale) Mirror 12: 0xe753d9b2 (stale) Mirror 13: 0xe753d9b2 (stale) Mirror 14: 0x96f0b31c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000730 1787496204.281535 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe753d9b2 (stale) Mirror 5: 0xe753d9b2 (stale) Mirror 7: 0xe753d9b2 (stale) Mirror 8: 0xe753d9b2 (stale) Mirror 9: 0xe753d9b2 (stale) Mirror 11: 0xe753d9b2 (stale) Mirror 12: 0xe753d9b2 (stale) Mirror 13: 0xe753d9b2 (stale) Mirror 14: 0x96f0b31c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb743fbaa Mirror 15: 0xb743fbaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000731 1787496205.959455 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe753d9b2 (stale) Mirror 7: 0xe753d9b2 (stale) Mirror 8: 0xe753d9b2 (stale) Mirror 9: 0xe753d9b2 (stale) Mirror 11: 0xe753d9b2 (stale) Mirror 12: 0xe753d9b2 (stale) Mirror 13: 0xe753d9b2 (stale) Mirror 14: 0x96f0b31c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe753d9b2 (stale) Mirror 7: 0xe753d9b2 (stale) Mirror 8: 0xe753d9b2 (stale) Mirror 9: 0xe753d9b2 (stale) Mirror 11: 0xe753d9b2 (stale) Mirror 12: 0xe753d9b2 (stale) Mirror 13: 0xe753d9b2 (stale) Mirror 14: 0x96f0b31c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb743fbaa Mirror 15: 0xb743fbaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000732 1787496207.021229 read 0x01732c thru 0x01c399 (0x506e bytes) 000733 1787496207.056819 trunc from 0x009395 to 0x014785 (0xb3f1 bytes) 000734 1787496207.122246 mapread 0x00317d thru 0x006908 (0x378c bytes) 000735 1787496207.144516 mapread 0x014f9c thru 0x01cf39 (0x7f9e bytes) 000736 1787496207.154262 fallocate 0x03b991 thru 0x03ffff (0x466f bytes) 000737 1787496207.159063 write 0x025590 thru 0x02cc9f (0x7710 bytes) 000738 1787496207.172390 mapread 0x02a0db thru 0x02c9b3 (0x28d9 bytes) 000739 1787496207.174438 write 0x02d9a9 thru 0x03b361 (0xd9b9 bytes) 000740 1787496207.184466 mapwrite 0x013c81 thru 0x018f8b (0x530b bytes) 000741 1787496207.204226 mapread 0x038377 thru 0x03ffff (0x7c89 bytes) 000742 1787496207.211610 mapread 0x03a0ed thru 0x03ffff (0x5f13 bytes) 000743 1787496207.212680 mapwrite 0x02574a thru 0x02f541 (0x9df8 bytes) 000744 1787496207.231704 trunc from 0x040000 to 0x029349 (0x16cb6 bytes) 000747 1787496207.255059 fallocate 0x01387e thru 0x0190e0 (0x5863 bytes) 000748 1787496207.259866 write 0x030485 thru 0x03ee65 (0xe9e1 bytes) 000749 1787496207.262877 mapwrite 0x03977b thru 0x03c594 (0x2e1a bytes) 000750 1787496207.286005 punch from 0x0375ee to 0x03de04 (0x6817 bytes) 000751 1787496207.291058 read 0x00c120 thru 0x018fc8 (0xcea9 bytes) 000752 1787496207.291817 fallocate 0x03cbed thru 0x03f78d (0x2ba1 bytes) 000753 1787496207.296778 fallocate 0x0299f2 thru 0x032fc8 (0x95d7 bytes) 000754 1787496207.300800 mapread 0x00b94c thru 0x0193c5 (0xda7a bytes) 000755 1787496207.301742 write 0x02a23e thru 0x038357 (0xe11a bytes) 000757 1787496207.311390 punch from 0x00d0f3 to 0x01999e (0xc8ac bytes) 000758 1787496207.317405 fallocate 0x02ce07 thru 0x0346fd (0x78f7 bytes) 000759 1787496207.323723 trunc from 0x02f121 to 0x03649d (0x737d bytes) 000760 1787496207.349218 fallocate 0x034020 thru 0x03ac40 (0x6c21 bytes) 000761 1787496207.360967 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ee66): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee66): Mirror 3: 0xc3d884d5 (stale) Mirror 7: 0xc3d884d5 (stale) Mirror 8: 0xc3d884d5 (stale) Mirror 9: 0xc3d884d5 (stale) Mirror 11: 0xc3d884d5 (stale) Mirror 12: 0xc3d884d5 (stale) Mirror 13: 0xc3d884d5 (stale) Mirror 14: 0x6a852796 (stale) Mirror 15: 0x3f33f41a (stale) CRC-32 checksum value for chunk [0, 0x3ee66): Mirror 1: 0x65082332 Mirror 16: 0x65082332 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee66: skipped 000762 1787496208.408606 write 0x00aaa7 thru 0x00ba7f (0xfd9 bytes) 000763 1787496208.476637 fallocate 0x005340 thru 0x008549 (0x320a bytes) 000764 1787496208.483933 trunc from 0x00e6a2 to 0x010855 (0x21b4 bytes) 000765 1787496208.490385 fallocate 0x0028e0 thru 0x01009e (0xd7bf bytes) 000766 1787496208.495425 punch from 0x00c3ed to 0x0158ce (0x94e2 bytes) 000767 1787496208.505622 write 0x01de2f thru 0x02986e (0xba40 bytes) 000768 1787496208.521631 read 0x014a8b thru 0x023d1d (0xf293 bytes) 000769 1787496208.537884 MIRROR_RESYNC 11 mirrors 000771 1787496212.756093 trunc from 0x03ee66 to 0x03e50a (0x95b bytes) 000772 1787496212.798970 mapwrite 0x034104 thru 0x03ffff (0xbefc bytes) 000773 1787496212.882685 mapread 0x00945d thru 0x012bd6 (0x977a bytes) 000774 1787496212.929478 read 0x036c8b thru 0x03be00 (0x5176 bytes) 000775 1787496212.930267 mapwrite 0x004634 thru 0x01403b (0xfa08 bytes) 000776 1787496212.971574 mapwrite 0x03722a thru 0x03ffff (0x8dd6 bytes) 000777 1787496213.001124 trunc from 0x01ed09 to 0x01f92c (0xc24 bytes) 000778 1787496213.005913 mapwrite 0x0247e2 thru 0x02c49f (0x7cbe bytes) 000779 1787496213.072172 trunc from 0x040000 to 0x02e09f (0x11f60 bytes) 000780 1787496213.105770 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2e0a0): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0a0): Mirror 3: 0x33e7f9f0 (stale) Mirror 7: 0x33e7f9f0 (stale) Mirror 8: 0x33e7f9f0 (stale) Mirror 9: 0x33e7f9f0 (stale) Mirror 11: 0x33e7f9f0 (stale) Mirror 12: 0x33e7f9f0 (stale) Mirror 13: 0x33e7f9f0 (stale) Mirror 14: 0x33e7f9f0 (stale) Mirror 15: 0x33e7f9f0 (stale) Mirror 16: 0x33e7f9f0 (stale) CRC-32 checksum value for chunk [0, 0x2e0a0): Mirror 1: 0xb51ab718 Mirror 17: 0xb51ab718 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0a0: skipped 000781 1787496214.611706 mapwrite 0x018157 thru 0x01d988 (0x5832 bytes) 000782 1787496214.766851 fallocate 0x009c1d thru 0x00b39c (0x1780 bytes) 000783 1787496214.776885 write 0x017e0d thru 0x027854 (0xfa48 bytes) 000784 1787496214.793487 write 0x037460 thru 0x03ffff (0x8ba0 bytes) 000785 1787496214.802083 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2ac0ff31 (stale) Mirror 7: 0x2ac0ff31 (stale) Mirror 8: 0x2ac0ff31 (stale) Mirror 9: 0x2ac0ff31 (stale) Mirror 11: 0x2ac0ff31 (stale) Mirror 12: 0x2ac0ff31 (stale) Mirror 13: 0x2ac0ff31 (stale) Mirror 14: 0x2ac0ff31 (stale) Mirror 15: 0x2ac0ff31 (stale) Mirror 16: 0x2ac0ff31 (stale) Mirror 17: 0x4a5a784d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x92abc9be Mirror 18: 0x92abc9be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1787496216.389689 fallocate 0x02e06b thru 0x0381ad (0xa143 bytes) 000787 1787496216.479509 mapwrite 0x0285ec thru 0x02be21 (0x3836 bytes) 000788 1787496216.542530 mapread 0x014c57 thru 0x020ac6 (0xbe70 bytes) 000789 1787496216.583664 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2ac0ff31 (stale) Mirror 7: 0x2ac0ff31 (stale) Mirror 9: 0x2ac0ff31 (stale) Mirror 11: 0x2ac0ff31 (stale) Mirror 12: 0x2ac0ff31 (stale) Mirror 13: 0x2ac0ff31 (stale) Mirror 14: 0x2ac0ff31 (stale) Mirror 15: 0x2ac0ff31 (stale) Mirror 16: 0x2ac0ff31 (stale) Mirror 17: 0x4a5a784d (stale) Mirror 18: 0x92abc9be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2ac0ff31 (stale) Mirror 7: 0x2ac0ff31 (stale) Mirror 9: 0x2ac0ff31 (stale) Mirror 11: 0x2ac0ff31 (stale) Mirror 12: 0x2ac0ff31 (stale) Mirror 13: 0x2ac0ff31 (stale) Mirror 14: 0x2ac0ff31 (stale) Mirror 15: 0x2ac0ff31 (stale) Mirror 16: 0x2ac0ff31 (stale) Mirror 17: 0x4a5a784d (stale) Mirror 18: 0x92abc9be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000790 1787496217.667319 trunc from 0x003196 to 0x009614 (0x647f bytes) 000792 1787496217.675558 mapread 0x03eaf5 thru 0x03ffff (0x150b bytes) 000793 1787496217.685839 fallocate 0x000f8b thru 0x00e116 (0xd18c bytes) 000794 1787496217.694820 trunc from 0x040000 to 0x03dc76 (0x2389 bytes) 000795 1787496217.737557 write 0x004c1f thru 0x0114a0 (0xc882 bytes) 000796 1787496217.755568 punch from 0x0272ef to 0x0327c7 (0xb4d9 bytes) 000797 1787496217.762565 fallocate 0x023465 thru 0x02db2d (0xa6c9 bytes) 000799 1787496217.783662 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc77): Mirror 3: 0x8e78b165 (stale) Mirror 7: 0x8e78b165 (stale) Mirror 11: 0x8e78b165 (stale) Mirror 12: 0x8e78b165 (stale) Mirror 13: 0x8e78b165 (stale) Mirror 14: 0x8e78b165 (stale) Mirror 15: 0x8e78b165 (stale) Mirror 16: 0x8e78b165 (stale) Mirror 17: 0xf972cbc7 (stale) Mirror 18: 0xd5026d98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc77): Mirror 3: 0x8e78b165 (stale) Mirror 7: 0x8e78b165 (stale) Mirror 11: 0x8e78b165 (stale) Mirror 12: 0x8e78b165 (stale) Mirror 13: 0x8e78b165 (stale) Mirror 14: 0x8e78b165 (stale) Mirror 15: 0x8e78b165 (stale) Mirror 16: 0x8e78b165 (stale) Mirror 17: 0xf972cbc7 (stale) Mirror 18: 0xd5026d98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc77: skipped 000800 1787496218.554783 read 0x003e46 thru 0x00daf3 (0x9cae bytes) 000801 1787496218.567472 mapread 0x01b7ac thru 0x029d76 (0xe5cb bytes) 000803 1787496218.603335 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3dc77): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc77): Mirror 3: 0x8e78b165 (stale) Mirror 7: 0x8e78b165 (stale) Mirror 11: 0x8e78b165 (stale) Mirror 12: 0x8e78b165 (stale) Mirror 13: 0x8e78b165 (stale) Mirror 14: 0x8e78b165 (stale) Mirror 15: 0x8e78b165 (stale) Mirror 16: 0x8e78b165 (stale) Mirror 17: 0xf972cbc7 (stale) Mirror 18: 0xd5026d98 (stale) CRC-32 checksum value for chunk [0, 0x3dc77): Mirror 1: 0x3f3c75c5 Mirror 19: 0x3f3c75c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc77: skipped 000804 1787496219.868121 write 0x00d35f thru 0x00f6b6 (0x2358 bytes) 000805 1787496219.988762 fallocate 0x01f143 thru 0x024aa5 (0x5963 bytes) 000806 1787496219.992931 trunc from 0x03dc77 to 0x005921 (0x38355 bytes) 000807 1787496220.023676 write 0x03d8a8 thru 0x03ffff (0x2758 bytes) 000808 1787496220.040526 MIRROR_RESYNC 12 mirrors 000809 1787496222.515186 write 0x01c932 thru 0x026f5e (0xa62d bytes) 000811 1787496222.561761 punch from 0x01da35 to 0x02d4df (0xfaab bytes) 000812 1787496222.584577 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3906ab09 (stale) Mirror 7: 0x3906ab09 (stale) Mirror 11: 0x3906ab09 (stale) Mirror 12: 0x3906ab09 (stale) Mirror 13: 0x3906ab09 (stale) Mirror 14: 0x3906ab09 (stale) Mirror 15: 0x3906ab09 (stale) Mirror 16: 0x3906ab09 (stale) Mirror 17: 0x3906ab09 (stale) Mirror 18: 0x3906ab09 (stale) Mirror 19: 0x3906ab09 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9d6edc1c Mirror 20: 0x9d6edc1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1787496224.030432 read 0x02dbad thru 0x032cf5 (0x5149 bytes) 000815 1787496224.052777 punch from 0x01d859 to 0x02bb23 (0xe2cb bytes) 000816 1787496224.116369 mapread 0x0137de thru 0x019bd0 (0x63f3 bytes) 000818 1787496224.144944 trunc from 0x012b68 to 0x01daf2 (0xaf8b bytes) 000819 1787496224.154210 trunc from 0x040000 to 0x00866e (0x37991 bytes) 000820 1787496224.196522 read 0x001d31 thru 0x00866e (0x693e bytes) 000821 1787496224.212542 trunc from 0x0002cd to 0x000b21 (0x855 bytes) 000822 1787496224.232443 write 0x00875a thru 0x00b53e (0x2de5 bytes) 000823 1787496224.235327 mapread 0x00959f thru 0x00b53e (0x1fa0 bytes) 000824 1787496224.242370 read 0x007ecd thru 0x00b53e (0x3672 bytes) 000825 1787496224.248597 trunc from 0x00b53f to 0x01d0ed (0x11baf bytes) 000826 1787496224.297588 fallocate 0x03a817 thru 0x03cb4e (0x2338 bytes) 000827 1787496224.311342 trunc from 0x03cb4f to 0x01751c (0x25632 bytes) 000828 1787496224.344457 MIRROR_RESYNC 13 mirrors 000829 1787496226.836534 trunc from 0x0134d4 to 0x01751c (0x4049 bytes) 000830 1787496226.878703 write 0x0004dc thru 0x0044bb (0x3fe0 bytes) 000831 1787496226.891475 trunc from 0x00023e to 0x0007ae (0x571 bytes) 000832 1787496226.909364 punch from 0x010854 to 0x01751c (0x6cc9 bytes) 000833 1787496226.918481 punch from 0x00a5e7 to 0x00c952 (0x236c bytes) 000834 1787496226.930598 fallocate 0x03a550 thru 0x03ffff (0x5ab0 bytes) 000835 1787496226.942079 write 0x01af5e thru 0x028801 (0xd8a4 bytes) 000836 1787496226.951237 write 0x017a31 thru 0x02694f (0xef1f bytes) 000837 1787496226.975427 MIRROR_RESYNC 13 mirrors 000838 1787496230.739884 punch from 0x020847 to 0x0279f4 (0x71ae bytes) 000839 1787496230.796804 trunc from 0x028802 to 0x034a1b (0xc21a bytes) 000840 1787496230.825698 write 0x01742d thru 0x022cc6 (0xb89a bytes) 000841 1787496230.842705 mapwrite 0x0370b7 thru 0x03a225 (0x316f bytes) 000842 1787496230.924003 MIRROR_RESYNC 13 mirrors 000843 1787496235.592424 read 0x0218ca thru 0x030948 (0xf07f bytes) 000844 1787496235.603187 mapread 0x0190db thru 0x022f05 (0x9e2b bytes) 000845 1787496235.635266 trunc from 0x006ac1 to 0x0123b7 (0xb8f7 bytes) 000847 1787496235.703259 punch from 0x02bef7 to 0x02dfa8 (0x20b2 bytes) 000849 1787496235.714629 write 0x02e5ab thru 0x030afe (0x2554 bytes) 000850 1787496235.728542 write 0x00025f thru 0x00f923 (0xf6c5 bytes) 000853 1787496235.760641 mapwrite 0x009b48 thru 0x017f4c (0xe405 bytes) 000854 1787496235.860535 fallocate 0x001453 thru 0x00fb37 (0xe6e5 bytes) 000855 1787496235.870387 trunc from 0x03a226 to 0x01da01 (0x1c824 bytes) 000856 1787496235.905833 fallocate 0x009540 thru 0x014214 (0xacd5 bytes) 000857 1787496235.916543 mapwrite 0x00ef03 thru 0x014972 (0x5a70 bytes) 000859 1787496235.950343 mapwrite 0x01364b thru 0x019320 (0x5cd6 bytes) 000860 1787496236.063929 mapwrite 0x014b69 thru 0x01e5e0 (0x9a78 bytes) 000861 1787496236.134998 fallocate 0x0181ec thru 0x018d19 (0xb2e bytes) 000862 1787496236.152554 mapread 0x017d79 thru 0x01e5e0 (0x6868 bytes) 000864 1787496236.156191 write 0x01d0d0 thru 0x023a36 (0x6967 bytes) 000865 1787496236.158702 trunc from 0x0135ef to 0x018238 (0x4c4a bytes) 000866 1787496236.164646 trunc from 0x0100b8 to 0x01db8d (0xdad6 bytes) 000867 1787496236.224921 mapwrite 0x039c58 thru 0x03ffff (0x63a8 bytes) 000868 1787496236.297072 trunc from 0x040000 to 0x027be1 (0x1841e bytes) 000869 1787496236.322723 MIRROR_RESYNC 13 mirrors 000870 1787496239.012136 punch from 0x023fae to 0x02433c (0x38f bytes) 000871 1787496239.066217 trunc from 0x027be2 to 0x002f9f (0x24c42 bytes) 000872 1787496239.094531 mapread 0x002c49 thru 0x002f9f (0x357 bytes) 000873 1787496239.100287 mapread 0x00296f thru 0x002f9f (0x631 bytes) 000874 1787496239.101396 punch from 0x00266f to 0x002f9f (0x931 bytes) 000877 1787496239.114412 trunc from 0x00017a to 0x002f9f (0x2e26 bytes) 000878 1787496239.128595 punch from 0x000ad1 to 0x002f9f (0x24cf bytes) 000879 1787496239.140863 fallocate 0x02f3e8 thru 0x02fc3b (0x854 bytes) 000881 1787496239.150358 trunc from 0x00142b to 0x002f9f (0x1b75 bytes) 000883 1787496239.162534 trunc from 0x002fa0 to 0x00f74d (0xc7ae bytes) 000884 1787496239.194803 mapread 0x00726b thru 0x00f74d (0x84e3 bytes) 000885 1787496239.221189 MIRROR_RESYNC 13 mirrors 000886 1787496242.183490 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000887 1787496242.798424 fallocate 0x018921 thru 0x0284a9 (0xfb89 bytes) 000888 1787496242.904368 fallocate 0x014865 thru 0x01bb2f (0x72cb bytes) 000889 1787496242.915123 mapread 0x00b9dc thru 0x00f74d (0x3d72 bytes) 000890 1787496242.961341 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf74e): Mirror 7: 0xfff65e81 (stale) Mirror 11: 0xfff65e81 (stale) Mirror 12: 0xfff65e81 (stale) Mirror 14: 0xfff65e81 (stale) Mirror 15: 0xfff65e81 (stale) Mirror 16: 0xfff65e81 (stale) Mirror 17: 0xfff65e81 (stale) Mirror 18: 0xfff65e81 (stale) Mirror 19: 0xfff65e81 (stale) Mirror 20: 0xfff65e81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf74e): Mirror 7: 0xfff65e81 (stale) Mirror 11: 0xfff65e81 (stale) Mirror 12: 0xfff65e81 (stale) Mirror 14: 0xfff65e81 (stale) Mirror 15: 0xfff65e81 (stale) Mirror 16: 0xfff65e81 (stale) Mirror 17: 0xfff65e81 (stale) Mirror 18: 0xfff65e81 (stale) Mirror 19: 0xfff65e81 (stale) Mirror 20: 0xfff65e81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf74e: skipped 000891 1787496244.044998 mapread 0x00af78 thru 0x00f74d (0x47d6 bytes) 000892 1787496244.102259 mapread 0x008ef8 thru 0x00f74d (0x6856 bytes) 000893 1787496244.103161 read 0x0062a2 thru 0x00f74d (0x94ac bytes) 000894 1787496244.103714 trunc from 0x00f74e to 0x016ac9 (0x737c bytes) 000895 1787496244.160796 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16aca): Mirror 11: 0x3acb91f0 (stale) Mirror 12: 0x3acb91f0 (stale) Mirror 14: 0x3acb91f0 (stale) Mirror 15: 0x3acb91f0 (stale) Mirror 16: 0x3acb91f0 (stale) Mirror 17: 0x3acb91f0 (stale) Mirror 18: 0x3acb91f0 (stale) Mirror 19: 0x3acb91f0 (stale) Mirror 20: 0x3acb91f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16aca): Mirror 11: 0x3acb91f0 (stale) Mirror 12: 0x3acb91f0 (stale) Mirror 14: 0x3acb91f0 (stale) Mirror 15: 0x3acb91f0 (stale) Mirror 16: 0x3acb91f0 (stale) Mirror 17: 0x3acb91f0 (stale) Mirror 18: 0x3acb91f0 (stale) Mirror 19: 0x3acb91f0 (stale) Mirror 20: 0x3acb91f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16aca: skipped 000896 1787496245.090891 trunc from 0x0122de to 0x0149f6 (0x2719 bytes) 000897 1787496245.097463 trunc from 0x016aca to 0x011178 (0x5951 bytes) 000898 1787496245.138755 mapwrite 0x020ad5 thru 0x030256 (0xf782 bytes) 000899 1787496245.301961 fallocate 0x021c25 thru 0x02bbb3 (0x9f8f bytes) 000900 1787496245.316947 write 0x006de7 thru 0x00ce24 (0x603e bytes) 000901 1787496245.367780 fallocate 0x01e620 thru 0x02989f (0xb280 bytes) 000902 1787496245.384717 punch from 0x026df0 to 0x02f78b (0x899c bytes) 000903 1787496245.397905 read 0x02f4fb thru 0x030256 (0xd5c bytes) 000904 1787496245.404782 trunc from 0x030257 to 0x02a8a8 (0x59ae bytes) 000905 1787496245.431121 read 0x029148 thru 0x02a8a8 (0x1761 bytes) 000906 1787496245.448988 mapwrite 0x011b22 thru 0x01e2be (0xc79d bytes) 000907 1787496245.476702 fallocate 0x033a38 thru 0x036b9a (0x3163 bytes) 000908 1787496245.483497 write 0x0282fb thru 0x02c386 (0x408c bytes) 000909 1787496245.484703 read 0x02a10b thru 0x02c386 (0x227c bytes) 000911 1787496245.487334 write 0x0371ad thru 0x03b66c (0x44c0 bytes) 000912 1787496245.491289 write 0x01c241 thru 0x025a25 (0x97e5 bytes) 000913 1787496245.494378 mapwrite 0x024c16 thru 0x025881 (0xc6c bytes) 000914 1787496245.513383 mapwrite 0x03e1a7 thru 0x03ffff (0x1e59 bytes) 000915 1787496245.556574 mapwrite 0x017cdf thru 0x01b441 (0x3763 bytes) 000916 1787496245.585790 fallocate 0x03957d thru 0x03ffff (0x6a83 bytes) 000917 1787496245.594992 mapread 0x038ec5 thru 0x03ff2d (0x7069 bytes) 000919 1787496245.597795 write 0x01d9b6 thru 0x01f8a7 (0x1ef2 bytes) 000920 1787496245.598907 read 0x0240f0 thru 0x02cf0e (0x8e1f bytes) 000922 1787496245.605751 MIRROR_RESYNC 10 mirrors 000923 1787496250.650066 mapread 0x029e57 thru 0x032532 (0x86dc bytes) 000924 1787496250.668382 punch from 0x035314 to 0x03f35f (0xa04c bytes) 000925 1787496250.704201 mapread 0x007cf3 thru 0x01783e (0xfb4c bytes) 000926 1787496250.737105 read 0x00de1c thru 0x01cc1f (0xee04 bytes) 000927 1787496250.742788 write 0x004f07 thru 0x00b089 (0x6183 bytes) 000928 1787496250.744208 mapread 0x03c1a7 thru 0x03ffff (0x3e59 bytes) 000929 1787496250.755522 mapread 0x0313dc thru 0x03a593 (0x91b8 bytes) 000930 1787496250.772539 write 0x003b15 thru 0x00e996 (0xae82 bytes) 000931 1787496250.775454 read 0x0293e5 thru 0x033ee5 (0xab01 bytes) 000932 1787496250.791281 write 0x017da5 thru 0x025038 (0xd294 bytes) 000933 1787496250.798986 mapwrite 0x02569a thru 0x02bf31 (0x6898 bytes) 000934 1787496250.848930 mapwrite 0x0362be thru 0x03ffff (0x9d42 bytes) 000935 1787496250.875542 trunc from 0x029290 to 0x03489f (0xb610 bytes) 000936 1787496250.883837 read 0x01168c thru 0x01a891 (0x9206 bytes) 000937 1787496250.886059 mapread 0x006d0a thru 0x014c31 (0xdf28 bytes) 000938 1787496250.890782 punch from 0x008cc6 to 0x018900 (0xfc3b bytes) 000939 1787496250.896316 punch from 0x033fa3 to 0x03ffff (0xc05d bytes) 000940 1787496250.905890 read 0x0027de thru 0x003046 (0x869 bytes) 000941 1787496250.906781 mapwrite 0x00474f thru 0x006c85 (0x2537 bytes) 000942 1787496250.958437 write 0x03131a thru 0x035e30 (0x4b17 bytes) 000943 1787496250.986007 fallocate 0x03dd14 thru 0x03ffff (0x22ec bytes) 000944 1787496250.991980 trunc from 0x03f3b1 to 0x03ffff (0xc4f bytes) 000945 1787496250.997574 trunc from 0x040000 to 0x03c4bb (0x3b44 bytes) 000946 1787496251.041018 fallocate 0x00a1e4 thru 0x014291 (0xa0ae bytes) 000947 1787496251.054148 mapread 0x0123f8 thru 0x01afa1 (0x8baa bytes) 000949 1787496251.060305 mapread 0x020081 thru 0x02b4be (0xb43e bytes) 000951 1787496251.062297 mapwrite 0x021a9f thru 0x026797 (0x4cf9 bytes) 000952 1787496251.086980 mapread 0x038d69 thru 0x03bb36 (0x2dce bytes) 000953 1787496251.088500 mapwrite 0x00b846 thru 0x01a5bf (0xed7a bytes) 000955 1787496251.124633 read 0x009c39 thru 0x0144cf (0xa897 bytes) 000956 1787496251.125963 mapread 0x03025b thru 0x0371f3 (0x6f99 bytes) 000958 1787496251.131648 trunc from 0x03c4bc to 0x00356f (0x38f4c bytes) 000959 1787496251.172218 mapwrite 0x03b871 thru 0x03ffff (0x478f bytes) 000961 1787496251.274464 fallocate 0x01adbc thru 0x0220f1 (0x7336 bytes) 000962 1787496251.282261 mapread 0x004bd0 thru 0x006b3e (0x1f6f bytes) 000963 1787496251.284851 mapwrite 0x02f714 thru 0x03d6c6 (0xdfb3 bytes) 000964 1787496251.327517 punch from 0x03740d to 0x03ffff (0x8bf3 bytes) 000965 1787496251.351600 fallocate 0x004898 thru 0x00fa00 (0xb169 bytes) 000966 1787496251.369871 trunc from 0x040000 to 0x039e45 (0x61ba bytes) 000967 1787496251.385021 trunc from 0x021831 to 0x023dee (0x25be bytes) 000968 1787496251.392500 mapread 0x014bb6 thru 0x0248ce (0xfd19 bytes) 000970 1787496251.432486 trunc from 0x0028cb to 0x002f2f (0x665 bytes) 000971 1787496251.448091 mapread 0x028379 thru 0x032e31 (0xaab9 bytes) 000972 1787496251.456632 trunc from 0x029b48 to 0x02d548 (0x3a01 bytes) 000973 1787496251.479713 MIRROR_RESYNC 10 mirrors 000974 1787496254.035713 MIRROR_EXTEND to 11 mirrors 000975 1787496255.032803 write 0x003bff thru 0x00469b (0xa9d bytes) 000976 1787496255.093862 read 0x01628f thru 0x025a3f (0xf7b1 bytes) 000978 1787496255.104004 fallocate 0x0378f4 thru 0x03ffff (0x870c bytes) 000979 1787496255.109076 write 0x00f305 thru 0x01410a (0x4e06 bytes) 000980 1787496255.119018 trunc from 0x040000 to 0x02b707 (0x148f8 bytes) 000981 1787496255.140703 mapwrite 0x006db2 thru 0x016182 (0xf3d1 bytes) 000983 1787496255.197445 trunc from 0x02b708 to 0x03e9f5 (0x132ee bytes) 000985 1787496255.218500 mapread 0x01cd5d thru 0x0260cd (0x9371 bytes) 000986 1787496255.234924 trunc from 0x03e9f6 to 0x021c78 (0x1cd7d bytes) 000987 1787496255.264360 write 0x030e1f thru 0x03e414 (0xd5f6 bytes) 000988 1787496255.267820 trunc from 0x03e415 to 0x00057e (0x3de96 bytes) 000989 1787496255.304870 fallocate 0x00ae81 thru 0x00f41b (0x459b bytes) 000990 1787496255.311833 trunc from 0x0046b7 to 0x007eae (0x37f8 bytes) 000991 1787496255.322925 mapwrite 0x034d2d thru 0x03cc37 (0x7f0b bytes) 000992 1787496255.419622 fallocate 0x025f22 thru 0x034ac7 (0xeba6 bytes) 000994 1787496255.437331 mapread 0x0370ea thru 0x03cc37 (0x5b4e bytes) 000995 1787496255.438611 trunc from 0x02b699 to 0x02f086 (0x39ee bytes) 000996 1787496255.450025 trunc from 0x03cc38 to 0x01e86d (0x1e3ca bytes) 000997 1787496255.480829 write 0x0220bd thru 0x02c191 (0xa0d5 bytes) 000998 1787496255.483827 trunc from 0x02c192 to 0x0106c4 (0x1bacd bytes) 000999 1787496255.508476 punch from 0x00662f to 0x0106c4 (0xa096 bytes) 001000 1787496255.514174 mapread 0x00d988 thru 0x00dc6b (0x2e4 bytes) All operations completed A-OK! PASS 70a (114s) == sanity-flr test 71: check mirror extend parent fid ==== 10:44:21 (1787496261) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x19b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 10:44:31 (1787496271) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28581 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184677 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 10:44:40 (1787496280) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3794144 .. Extending file size to 4726976 .. Extending file size to 5117536 .. Extending file size to 4062976 .. Extending file size to 7840160 .. Extending file size to 8138976 .. Extending file size to 8604736 .. Extending file size to 8503776 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9339520 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996832 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61332 61333 61336 61337 61339 done 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) 192.168.203.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:46:01 (1787496361) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6657984 .. Extending file size to 3643936 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8991744 .. Extending file size to 6165600 .. Extending file size to 8996544 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9006624 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9063136 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 63033 63034 63035 63038 63040 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) 192.168.203.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:47:18 (1787496438) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18592 3588428 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24288 3582732 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42880 7171160 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18592 3588428 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24288 3582732 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42880 7171160 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134679 s, 15.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244883 s, 8.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7a6c41fb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:47:40 (1787496460) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128584 s, 16.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 10:47:46 (1787496466) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:47:54 (1787496474) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131285 s, 16.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:48:03 (1787496483) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121002 s, 17.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:48:12 (1787496492) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0912503 s, 23.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:48:47 (1787496527) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110305 s, 9.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:49:39 (1787496579) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140706 s, 14.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:50:32 (1787496632) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg319-server: Pool lustre.test_204f created Adding targets to pool oleg319-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg319-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176813 s, 11.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg319-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg319-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg319-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:51:47 (1787496707) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:51:55 (1787496715) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:52:04 (1787496724) oleg319-server: Pool lustre.test_206 created Adding targets to pool oleg319-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg319-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg319-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg319-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg319-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:52:33 (1787496753) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673148 s, 15.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621849 s, 16.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1f7f0c5d Mirror 3: 0x1f7f0c5d PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:52:44 (1787496764) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:52:48 (1787496768) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:52:51 (1787496771) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0406804 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xcb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1900000 Read this (4 bytes): BBBB 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:53:25 (1787496805) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0239826 s, 342 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:53:32 (1787496812) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330566 s, 31.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xaf lmm_fid: [0x200000405:0xaf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xaf lmm_fid: [0x200000405:0xaf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611311 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-23 10:53:34.000000000 -0400 Modify: 2026-08-23 10:53:34.000000000 -0400 Change: 2026-08-23 10:53:35.000000000 -0400 Birth: 2026-08-23 10:53:34.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:53:40 (1787496820) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275601 s, 38.0 MB/s oleg319-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg319-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg319-server: O/280000401/d26/15a: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:54:05 (1787496845) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.273836 s, 150 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.030377 s, 135 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 23 10:54 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 23 10:54 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:54:12 (1787496852) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:54:19 (1787496859) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1785 sec =========== 10:54:27 (1787496867) === sanity-flr: start cleanup 10:54:29 (1787496869) === === sanity-flr: finish cleanup 10:54:37 (1787496877) ===