-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 23:26:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg236-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:26:34 (1788837994) === oleg236-client.virtnet: executing check_config_client /mnt/lustre oleg236-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg236-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff932dd8313000.idle_timeout=debug osc.lustre-OST0001-osc-ffff932dd8313000.idle_timeout=debug disable quota as required oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:27:10 (1788838030) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7628] == sanity-flr test 0a: lfs mirror create with -N option == 23:27:12 (1788838032) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:27:30 (1788838050) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:27:33 (1788838053) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:27:36 (1788838056) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:27:59 (1788838079) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:28:02 (1788838082) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:28:05 (1788838085) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff932dd8313000.stats=clear osc.lustre-OST0001-osc-ffff932dd8313000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:28:13 (1788838093) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:28:24 (1788838104) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:28:31 (1788838111) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:28:34 (1788838114) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:28:41 (1788838121) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 23:28:50 (1788838130) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:28:57 (1788838137) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:29:01 (1788838141) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:29:08 (1788838148) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:29:18 (1788838158) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:29:24 (1788838164) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.64228 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175327 s, 6.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:29:34 (1788838174) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304117 s, 34.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:29:43 (1788838183) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:29:53 (1788838193) Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg236-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:31:03 (1788838263) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.616047 s, 18.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:31:47 (1788838307) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg236-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:33:04 (1788838384) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.64674 s, 14.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 23:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.912s user 0m0.000s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff932dd8313000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.275s user 0m0.000s sys 0m0.172s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.282s user 0m0.001s sys 0m0.168s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg236-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff932dd8313000.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:34:43 (1788838483) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:34:47 (1788838487) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:34:51 (1788838491) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:34:54 (1788838494) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 23:35:01 (1788838501) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.480981 s, 26.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:35:15 (1788838515) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261167 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024312 s, 43.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:35:22 (1788838522) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:35:30 (1788838530) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163251 s, 627 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00145003 s, 706 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000970905 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00171332 s, 598 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 23:35:40 (1788838540) create test files with size 15754121 .. b2530a85bd2ef80cfaace211a5101986 - 0ee54550ed9a8ec468dc616b66e716b4 - 1af9dd127b2b5b148c3d5694fda82df3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg236-client.virtnet /mnt/lustre (opts:) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 23:36:05 (1788838565) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:36:19 (1788838579) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:36:26 (1788838586) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12943 s, 8.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407347 s, 25.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 23:36:36 (1788838596) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198163 s, 10.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.609817 s, 6.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140109 s, 15.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 23:36:55 (1788838615) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 23:36:58 (1788838618) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:37:01 (1788838621) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463934 s, 22.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 23:37:09 (1788838629) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:37:13 (1788838633) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xbf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:37:25 (1788838645) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.468477 s, 22.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg236-client.virtnet /mnt/lustre (opts:) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:37:39 (1788838659) llite.lustre-ffff932dd9b65000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.269213 s, 38.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg236-client.virtnet /mnt/lustre (opts:) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff932dd9a45800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:37:52 (1788838672) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:38:30 (1788838710) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:39:16 (1788838756) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 23:39:20 (1788838760) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.308994 s, 10.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.119154 s, 26.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4330deaf (nosync,stale) Mirror 2: 0x4330deaf (nosync,stale) ** verify mirror 2 doesn't change original checksum: b44d9c40991ed2a9b022ab79bb52ad25 - mirror 1 checksum: 2874d0518bb204b6e118926bec3adaac - mirror 2 checksum: b44d9c40991ed2a9b022ab79bb52ad25 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:39:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:39:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b44d9c40991ed2a9b022ab79bb52ad25 - mirror 1 checksum: 2874d0518bb204b6e118926bec3adaac - mirror 2 checksum: 2874d0518bb204b6e118926bec3adaac - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:39:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:39:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:39:34 (1788838774) Failing mds1 on oleg236-server Stopping /mnt/lustre-mds1 (opts:) on oleg236-server 23:39:40 (1788838780) shut down facet: mds1 facet_host: oleg236-server facet_failover_host: oleg236-server Failover mds1 to oleg236-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0000 23:40:06 (1788838806) targets are mounted 23:40:06 (1788838806) facet_failover done oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252069 s, 41.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:40:23 (1788838823) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:40:27 (1788838827) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:40:31 (1788838831) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0199138 s, 823 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:40:39 (1788838839) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:40:42 (1788838842) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.01833 s, 223 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0583017 s, 70.3 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:40:50 (1788838850) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:41:11 (1788838871) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:41:38 (1788838898) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:42:00 (1788838920) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:42:24 (1788838944) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 23:42:32 (1788838952) mirror create pid 54332 mirror split pid 54333 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54332 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54333 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 23:43:39 (1788839019) Seed set to 9021 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1788839021.541898 mapwrite 0x018b37 thru 0x01ab2b (0x1ff5 bytes) 000002 1788839021.628081 read 0x018c99 thru 0x01ab2b (0x1e93 bytes) 000003 1788839021.629149 read 0x00587b thru 0x00cdfd (0x7583 bytes) 000004 1788839021.638566 write 0x017f17 thru 0x01cf5b (0x5045 bytes) 000005 1788839021.640057 mapread 0x01cde4 thru 0x01cf5b (0x178 bytes) 000007 1788839021.641548 mapread 0x00db82 thru 0x00eba2 (0x1021 bytes) truncating to largest ever: 0xa8cd 000008 1788839021.643115 trunc from 0x01cf5c to 0x00a8cc (0x1268f bytes) 000013 1788839021.660307 read 0x0069b9 thru 0x00a8cc (0x3f14 bytes) 000014 1788839021.661162 mapread 0x004bfa thru 0x00a8cc (0x5cd3 bytes) 000015 1788839021.662319 write 0x023d6b thru 0x030e2c (0xd0c2 bytes) 000017 1788839021.666824 write 0x00ab16 thru 0x014cd5 (0xa1c0 bytes) 000018 1788839021.674223 write 0x0008a8 thru 0x00eebb (0xe614 bytes) 000019 1788839021.675738 read 0x01f151 thru 0x0289a7 (0x9857 bytes) 000020 1788839021.683738 trunc from 0x030e2d to 0x00222f (0x2ebfd bytes) 000021 1788839021.705079 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x173f5 000022 1788839022.136865 trunc from 0x002230 to 0x0173f4 (0x151c5 bytes) 000023 1788839022.305751 mapread 0x0020e0 thru 0x007a83 (0x59a4 bytes) 000024 1788839022.320430 punch from 0x00602f to 0x0070c0 (0x1092 bytes) 000025 1788839022.324514 mapwrite 0x027ac3 thru 0x02d343 (0x5881 bytes) 000026 1788839022.390911 trunc from 0x02d344 to 0x013001 (0x1a342 bytes) 000027 1788839022.408434 mapwrite 0x01414b thru 0x01ca0d (0x88c3 bytes) 000028 1788839022.491539 write 0x02fd68 thru 0x03381f (0x3ab8 bytes) 000029 1788839022.492599 trunc from 0x033820 to 0x014add (0x1ed42 bytes) 000031 1788839022.515087 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x14ade): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14ade): Mirror 2: 0x8276acc4 (stale) CRC-32 checksum value for chunk [0, 0x14ade): Mirror 1: 0xbcf55e59 Mirror 3: 0xbcf55e59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14ade: skipped truncating to largest ever: 0x2c3d6 000032 1788839023.475455 trunc from 0x014ade to 0x02c3d5 (0x178f8 bytes) 000034 1788839023.568116 mapread 0x008e3d thru 0x00b326 (0x24ea bytes) 000035 1788839023.588468 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c3d6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3d6): Mirror 2: 0x318dc7e3 (stale) Mirror 3: 0x4226dde4 (stale) CRC-32 checksum value for chunk [0, 0x2c3d6): Mirror 1: 0x4226dde4 Mirror 4: 0x4226dde4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3d6: skipped 000036 1788839024.697861 MIRROR_RESYNC 4 mirrors 000037 1788839025.923440 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000038 1788839026.275602 read 0x02968c thru 0x02c3d5 (0x2d4a bytes) 000040 1788839026.283350 punch from 0x01efe1 to 0x025d50 (0x6d70 bytes) 000041 1788839026.373703 mapwrite 0x01e53a thru 0x02d309 (0xedd0 bytes) 000042 1788839026.490087 trunc from 0x02d30a to 0x0233ed (0x9f1c bytes) 000044 1788839026.518762 write 0x0388c8 thru 0x03ffff (0x7738 bytes) truncating to largest ever: 0x36ff7 000046 1788839026.522178 trunc from 0x040000 to 0x036ff6 (0x9009 bytes) 000047 1788839026.542015 write 0x013898 thru 0x01ae5b (0x75c4 bytes) 000048 1788839026.549746 read 0x0335fc thru 0x036ff6 (0x39fb bytes) 000049 1788839026.562760 read 0x006880 thru 0x013c77 (0xd3f8 bytes) 000050 1788839026.564169 write 0x03a153 thru 0x03f4b4 (0x5362 bytes) 000051 1788839026.569615 write 0x03a18d thru 0x03ffff (0x5e73 bytes) 000052 1788839026.573103 read 0x03c70a thru 0x03ffff (0x38f6 bytes) 000053 1788839026.573829 trunc from 0x040000 to 0x00275f (0x3d8a0 bytes) 000054 1788839026.598959 MIRROR_RESYNC 3 mirrors 000058 1788839027.324748 write 0x0205e4 thru 0x029a71 (0x948e bytes) 000059 1788839027.386712 trunc from 0x029a72 to 0x01e500 (0xb571 bytes) 000060 1788839027.408103 mapread 0x011cad thru 0x015887 (0x3bdb bytes) 000062 1788839027.424591 read 0x018ebe thru 0x01e500 (0x5643 bytes) 000063 1788839027.429972 read 0x01d413 thru 0x01e500 (0x10ee bytes) 000064 1788839027.430886 trunc from 0x01e501 to 0x0020cd (0x1c433 bytes) 000065 1788839027.449872 mapread 0x001a8e thru 0x0020cd (0x640 bytes) 000068 1788839027.457256 write 0x00439d thru 0x011684 (0xd2e8 bytes) 000069 1788839027.459217 write 0x00f70d thru 0x01a836 (0xb12a bytes) 000071 1788839027.466169 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1a837): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a837): Mirror 2: 0xf042cbd0 (stale) Mirror 3: 0xf042cbd0 (stale) CRC-32 checksum value for chunk [0, 0x1a837): Mirror 1: 0xc1be6b67 Mirror 4: 0xc1be6b67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a837: skipped 000072 1788839028.075884 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a837): Mirror 2: 0xf042cbd0 (stale) Mirror 3: 0xf042cbd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a837): Mirror 2: 0xf042cbd0 (stale) Mirror 3: 0xf042cbd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a837: skipped 000073 1788839028.325769 mapread 0x0037cf thru 0x00c999 (0x91cb bytes) 000075 1788839028.342688 write 0x025cc5 thru 0x02f19c (0x94d8 bytes) 000076 1788839028.353916 mapwrite 0x03c5ce thru 0x03ffff (0x3a32 bytes) 000078 1788839028.458002 trunc from 0x040000 to 0x02924f (0x16db0 bytes) 000080 1788839028.474913 mapwrite 0x007b20 thru 0x016d10 (0xf1f1 bytes) 000084 1788839028.534462 punch from 0x024c34 to 0x02924f (0x461c bytes) 000085 1788839028.538912 punch from 0x003d7d to 0x00e21e (0xa4a2 bytes) 000086 1788839028.545310 MIRROR_RESYNC 3 mirrors 000087 1788839029.537938 punch from 0x0006c1 to 0x008a45 (0x8385 bytes) 000088 1788839029.601926 punch from 0x016f8e to 0x01f31a (0x838d bytes) 000091 1788839029.609995 mapwrite 0x001e9d thru 0x00d6f4 (0xb858 bytes) 000092 1788839029.659784 read 0x0199df thru 0x01b1e6 (0x1808 bytes) 000093 1788839029.663200 punch from 0x01fd46 to 0x02924f (0x950a bytes) 000094 1788839029.666198 punch from 0x018335 to 0x01854a (0x216 bytes) 000095 1788839029.669881 read 0x011145 thru 0x019ff9 (0x8eb5 bytes) 000096 1788839029.679600 write 0x0293d8 thru 0x02d22b (0x3e54 bytes) 000097 1788839029.681333 trunc from 0x02d22c to 0x01f661 (0xdbca bytes) truncating to largest ever: 0x3929e 000098 1788839029.706987 trunc from 0x01f662 to 0x03929d (0x19c3c bytes) 000099 1788839029.728014 read 0x0231ca thru 0x0235da (0x411 bytes) 000102 1788839029.739992 write 0x02fede thru 0x03be38 (0xbf5b bytes) 000105 1788839029.749210 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3be39): Mirror 3: 0xeaf8ea85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be39): Mirror 3: 0xeaf8ea85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be39: skipped 000107 1788839030.073697 mapwrite 0x002b4a thru 0x00b687 (0x8b3e bytes) 000111 1788839030.160965 trunc from 0x03be39 to 0x033710 (0x8728 bytes) 000112 1788839030.180736 MIRROR_RESYNC 2 mirrors 000113 1788839030.824005 mapwrite 0x037796 thru 0x03e3f3 (0x6c5e bytes) 000115 1788839030.979448 trunc from 0x03e3f4 to 0x00bf06 (0x324ed bytes) 000116 1788839031.006094 mapwrite 0x033a35 thru 0x03ffff (0xc5cb bytes) 000117 1788839031.080802 mapwrite 0x00090d thru 0x00d333 (0xca27 bytes) 000118 1788839031.132642 trunc from 0x040000 to 0x00dda0 (0x3225f bytes) 000120 1788839031.148023 mapread 0x00322c thru 0x007ae2 (0x48b7 bytes) 000121 1788839031.148894 write 0x029a97 thru 0x030549 (0x6ab3 bytes) 000122 1788839031.150047 trunc from 0x03054a to 0x00dc33 (0x22916 bytes) 000123 1788839031.194507 trunc from 0x00dc34 to 0x027f7b (0x1a348 bytes) 000125 1788839031.222464 write 0x03d6b6 thru 0x03ffff (0x294a bytes) 000126 1788839031.223549 trunc from 0x040000 to 0x02baf8 (0x14507 bytes) 000128 1788839031.267882 trunc from 0x02baf9 to 0x00a362 (0x21796 bytes) 000129 1788839031.292970 read 0x008f34 thru 0x00a362 (0x142f bytes) 000130 1788839031.293880 mapwrite 0x022e7f thru 0x02778e (0x4910 bytes) 000132 1788839031.451563 read 0x00816a thru 0x01755b (0xf3f2 bytes) 000134 1788839031.477010 mapread 0x001bed thru 0x00987e (0x7c92 bytes) 000135 1788839031.484228 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3a4bc 000137 1788839032.094239 trunc from 0x02778f to 0x03a4bb (0x12d2d bytes) 000138 1788839032.183833 trunc from 0x03a4bc to 0x024038 (0x16483 bytes) 000141 1788839032.206264 trunc from 0x024039 to 0x00dc97 (0x163a1 bytes) 000142 1788839032.223422 mapwrite 0x03eb39 thru 0x03ffff (0x14c7 bytes) 000143 1788839032.286776 trunc from 0x040000 to 0x019fa7 (0x26058 bytes) 000145 1788839032.312066 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000146 1788839032.458179 read 0x00967c thru 0x00e7a6 (0x512b bytes) 000147 1788839032.468274 mapwrite 0x02a1f5 thru 0x0307a2 (0x65ae bytes) 000148 1788839032.555893 read 0x012112 thru 0x021bec (0xfadb bytes) 000149 1788839032.567004 write 0x00ff83 thru 0x01bfaf (0xc02d bytes) 000151 1788839032.568687 mapwrite 0x012bbf thru 0x013975 (0xdb7 bytes) 000152 1788839032.673065 write 0x00852b thru 0x0177b4 (0xf28a bytes) 000153 1788839032.675006 read 0x00838a thru 0x0169b0 (0xe627 bytes) 000154 1788839032.684319 punch from 0x0295ff to 0x02b94f (0x2351 bytes) 000160 1788839032.710056 write 0x004739 thru 0x010e81 (0xc749 bytes) 000161 1788839032.711528 write 0x01eacb thru 0x02e42d (0xf963 bytes) 000165 1788839032.715475 read 0x02a221 thru 0x0307a2 (0x6582 bytes) 000166 1788839032.716136 write 0x01ff5d thru 0x02e10f (0xe1b3 bytes) 000167 1788839032.717574 mapread 0x023368 thru 0x0307a2 (0xd43b bytes) 000168 1788839032.718986 mapread 0x00844c thru 0x016c12 (0xe7c7 bytes) 000170 1788839032.720984 punch from 0x0272fb to 0x028447 (0x114d bytes) 000171 1788839032.739241 mapwrite 0x01dc4f thru 0x027241 (0x95f3 bytes) 000172 1788839032.803621 write 0x007104 thru 0x012eef (0xbdec bytes) 000173 1788839032.806035 mapread 0x019ade thru 0x02016e (0x6691 bytes) 000174 1788839032.807601 read 0x015918 thru 0x0177af (0x1e98 bytes) 000175 1788839032.808742 read 0x01c7ca thru 0x01e416 (0x1c4d bytes) 000177 1788839032.810148 punch from 0x029f46 to 0x02e5a7 (0x4662 bytes) 000178 1788839032.822260 write 0x0183b3 thru 0x0237f1 (0xb43f bytes) 000179 1788839032.824563 read 0x022cc1 thru 0x02b2f8 (0x8638 bytes) 000180 1788839032.833087 trunc from 0x0307a3 to 0x02b456 (0x534c bytes) 000181 1788839032.852654 read 0x012bbe thru 0x014f4d (0x2390 bytes) 000188 1788839032.859280 mapwrite 0x00e676 thru 0x01160a (0x2f95 bytes) 000189 1788839032.910760 punch from 0x027b7d to 0x02b456 (0x38da bytes) 000191 1788839032.918271 mapwrite 0x0146a0 thru 0x023eda (0xf83b bytes) truncating to largest ever: 0x3df41 000192 1788839032.979986 trunc from 0x02b457 to 0x03df40 (0x12aea bytes) 000194 1788839032.999958 write 0x03654b thru 0x03ffff (0x9ab5 bytes) 000197 1788839033.020549 mapread 0x0016c5 thru 0x00aee7 (0x9823 bytes) 000198 1788839033.021798 read 0x02a1e0 thru 0x035ae0 (0xb901 bytes) 000199 1788839033.022985 mapread 0x027a5c thru 0x0360cc (0xe671 bytes) 000200 1788839033.024223 write 0x014994 thru 0x017448 (0x2ab5 bytes) 000201 1788839033.025865 mapwrite 0x03c3b1 thru 0x03ffff (0x3c4f bytes) 000204 1788839033.091576 mapwrite 0x0155d7 thru 0x020f94 (0xb9be bytes) 000205 1788839033.158948 mapread 0x005f87 thru 0x00670b (0x785 bytes) 000206 1788839033.160102 read 0x033da2 thru 0x036032 (0x2291 bytes) 000209 1788839033.164557 mapread 0x02fa20 thru 0x03cf46 (0xd527 bytes) 000210 1788839033.166527 punch from 0x0354e3 to 0x03ffff (0xab1d bytes) 000211 1788839033.171557 punch from 0x00b59e to 0x015c51 (0xa6b4 bytes) 000212 1788839033.176877 read 0x03b2ee thru 0x03ffff (0x4d12 bytes) 000217 1788839033.194602 read 0x025f27 thru 0x029936 (0x3a10 bytes) 000218 1788839033.195274 punch from 0x0321c6 to 0x038ae1 (0x691c bytes) 000219 1788839033.205382 punch from 0x006b55 to 0x008dcd (0x2279 bytes) 000220 1788839033.213990 punch from 0x01e3c1 to 0x01fc92 (0x18d2 bytes) 000221 1788839033.224848 mapread 0x0207b7 thru 0x0267ee (0x6038 bytes) 000222 1788839033.227525 read 0x033d97 thru 0x0384b9 (0x4723 bytes) 000224 1788839033.241101 MIRROR_EXTEND to 2 mirrors 000226 1788839034.142228 punch from 0x000f44 to 0x001406 (0x4c3 bytes) 000227 1788839034.217661 mapwrite 0x03bd52 thru 0x03f801 (0x3ab0 bytes) 000228 1788839034.281946 write 0x0158f4 thru 0x01c987 (0x7094 bytes) 000229 1788839034.291963 punch from 0x008aa5 to 0x014e9e (0xc3fa bytes) 000230 1788839034.303152 punch from 0x01047a to 0x01952b (0x90b2 bytes) 000231 1788839034.307046 mapwrite 0x023998 thru 0x02c03c (0x86a5 bytes) 000234 1788839034.369371 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb239771f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xceb1c1e1 Mirror 3: 0xceb1c1e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000235 1788839035.259363 read 0x0314cd thru 0x035f36 (0x4a6a bytes) 000237 1788839035.304835 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000239 1788839035.756162 read 0x02e0ec thru 0x036b78 (0x8a8d bytes) 000240 1788839035.774913 read 0x031464 thru 0x0394d5 (0x8072 bytes) 000241 1788839035.793231 mapread 0x0063b7 thru 0x00a85e (0x44a8 bytes) 000242 1788839035.800019 write 0x024e4d thru 0x02f6f9 (0xa8ad bytes) 000246 1788839035.923946 mapread 0x039e76 thru 0x03ffff (0x618a bytes) 000248 1788839035.983308 read 0x00ac85 thru 0x00d3b4 (0x2730 bytes) 000249 1788839036.017236 write 0x009ba2 thru 0x00b22d (0x168c bytes) 000250 1788839036.020063 mapread 0x006438 thru 0x012b11 (0xc6da bytes) 000252 1788839036.028168 write 0x002766 thru 0x00f58f (0xce2a bytes) 000253 1788839036.035324 write 0x01b626 thru 0x01c43c (0xe17 bytes) 000254 1788839036.036279 trunc from 0x040000 to 0x00c33b (0x33cc4 bytes) 000255 1788839036.103583 read 0x00576b thru 0x006596 (0xe2c bytes) 000258 1788839036.105251 mapwrite 0x018690 thru 0x02852f (0xfea0 bytes) 000259 1788839036.209195 read 0x00acd5 thru 0x011cdc (0x7008 bytes) 000260 1788839036.210292 read 0x01bd37 thru 0x023353 (0x761d bytes) 000262 1788839036.218695 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28530): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28530): Mirror 3: 0x8d17ac35 (stale) CRC-32 checksum value for chunk [0, 0x28530): Mirror 1: 0xbf2dda92 Mirror 4: 0xbf2dda92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28530: skipped 000265 1788839037.135724 mapread 0x010f7b thru 0x019e93 (0x8f19 bytes) 000269 1788839037.211070 mapwrite 0x00dcec thru 0x010a61 (0x2d76 bytes) 000270 1788839037.417908 MIRROR_RESYNC 3 mirrors 000273 1788839038.670159 punch from 0x000cdb to 0x004f8c (0x42b2 bytes) 000275 1788839038.755066 punch from 0x02758c to 0x02841b (0xe90 bytes) 000279 1788839038.762209 mapwrite 0x03b913 thru 0x03ffff (0x46ed bytes) 000280 1788839038.860240 mapwrite 0x02cfd1 thru 0x03c341 (0xf371 bytes) 000281 1788839038.947741 trunc from 0x040000 to 0x01e355 (0x21caa bytes) 000282 1788839038.984329 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1e356): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e356): Mirror 3: 0x1c6587d2 (stale) Mirror 4: 0x1c6587d2 (stale) CRC-32 checksum value for chunk [0, 0x1e356): Mirror 1: 0x6ae78fde Mirror 5: 0x6ae78fde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e356: skipped 000284 1788839039.822010 write 0x00d99d thru 0x0180ee (0xa752 bytes) 000286 1788839039.916996 read 0x01b1fd thru 0x01e355 (0x3159 bytes) 000287 1788839039.922622 mapread 0x019c57 thru 0x01e1ac (0x4556 bytes) 000288 1788839039.945729 mapwrite 0x02adbf thru 0x037f62 (0xd1a4 bytes) 000289 1788839040.040818 mapread 0x02ee67 thru 0x02fa8d (0xc27 bytes) 000290 1788839040.042189 mapread 0x0175c7 thru 0x017ba5 (0x5df bytes) 000291 1788839040.048737 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f63): Mirror 3: 0x8008daca (stale) Mirror 5: 0xb5902ead (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f63): Mirror 3: 0x8008daca (stale) Mirror 5: 0xb5902ead (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f63: skipped 000292 1788839040.382063 mapread 0x01d0f6 thru 0x02b576 (0xe481 bytes) 000294 1788839040.405363 MIRROR_RESYNC 3 mirrors 000295 1788839041.506451 write 0x033024 thru 0x03c128 (0x9105 bytes) 000296 1788839041.621018 mapwrite 0x02e49e thru 0x03dd87 (0xf8ea bytes) 000298 1788839041.735760 read 0x03aa7f thru 0x03dd87 (0x3309 bytes) 000299 1788839041.736768 mapwrite 0x01945a thru 0x01cc9d (0x3844 bytes) 000300 1788839041.795545 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 3: 0x53f0be09 (stale) Mirror 5: 0x53f0be09 (stale) CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 1: 0x99b7e472 Mirror 6: 0x99b7e472 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd88: skipped 000301 1788839042.946912 write 0x02d01c thru 0x03b9d8 (0xe9bd bytes) 000303 1788839043.069777 mapread 0x03c753 thru 0x03dd87 (0x1635 bytes) 000304 1788839043.088346 MIRROR_RESYNC 4 mirrors 000306 1788839044.575833 punch from 0x020210 to 0x02a32c (0xa11d bytes) 000307 1788839044.639764 read 0x039655 thru 0x03b5b9 (0x1f65 bytes) 000308 1788839044.645890 mapwrite 0x0319b1 thru 0x036d39 (0x5389 bytes) 000309 1788839044.720633 MIRROR_RESYNC 4 mirrors 000311 1788839045.745804 MIRROR_RESYNC 4 mirrors 000312 1788839046.196821 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000313 1788839046.610681 mapread 0x02f37e thru 0x0362e0 (0x6f63 bytes) 000314 1788839046.654045 read 0x020955 thru 0x02cba3 (0xc24f bytes) 000315 1788839046.683883 write 0x014e50 thru 0x01699f (0x1b50 bytes) 000316 1788839046.802056 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 5: 0x7f639146 (stale) Mirror 6: 0x7f639146 (stale) CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 1: 0xa94b9e0c Mirror 7: 0xa94b9e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd88: skipped 000317 1788839047.972906 punch from 0x024bea to 0x0279f5 (0x2e0c bytes) 000319 1788839048.070594 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 5: 0x7f639146 (stale) Mirror 6: 0x7f639146 (stale) Mirror 7: 0xa94b9e0c (stale) CRC-32 checksum value for chunk [0, 0x3dd88): Mirror 1: 0xa94b9e0c Mirror 8: 0xa94b9e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd88: skipped 000320 1788839048.945578 trunc from 0x03dd88 to 0x026cb3 (0x170d4 bytes) 000321 1788839049.043651 mapread 0x014f60 thru 0x0167e9 (0x188a bytes) 000322 1788839049.075143 write 0x0066f5 thru 0x00a749 (0x4055 bytes) 000323 1788839049.084653 write 0x009cbd thru 0x0130e0 (0x9424 bytes) 000324 1788839049.086868 mapread 0x003b64 thru 0x01110e (0xd5ab bytes) 000326 1788839049.097131 mapread 0x0021fb thru 0x00e58a (0xc390 bytes) 000327 1788839049.098650 mapread 0x00bb44 thru 0x0126e1 (0x6b9e bytes) 000328 1788839049.100095 punch from 0x01d26f to 0x01f353 (0x20e5 bytes) 000329 1788839049.115659 mapwrite 0x002da0 thru 0x004b47 (0x1da8 bytes) 000330 1788839049.157887 read 0x014547 thru 0x021e24 (0xd8de bytes) 000331 1788839049.173234 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26cb4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26cb4): Mirror 5: 0x74d0b118 (stale) Mirror 6: 0x74d0b118 (stale) Mirror 7: 0x310750f5 (stale) Mirror 8: 0x310750f5 (stale) CRC-32 checksum value for chunk [0, 0x26cb4): Mirror 1: 0x1c36296d Mirror 9: 0x1c36296d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26cb4: skipped 000333 1788839050.269447 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26cb4): Mirror 6: 0x74d0b118 (stale) Mirror 7: 0x310750f5 (stale) Mirror 8: 0x310750f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26cb4): Mirror 6: 0x74d0b118 (stale) Mirror 7: 0x310750f5 (stale) Mirror 8: 0x310750f5 (stale) CRC-32 checksum value for chunk [0, 0x26cb4): Mirror 1: 0x1c36296d Mirror 9: 0x1c36296d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26cb4: skipped 000334 1788839050.881583 mapread 0x0098b8 thru 0x013e0b (0xa554 bytes) 000335 1788839050.929708 punch from 0x007246 to 0x00f617 (0x83d2 bytes) 000336 1788839051.022092 trunc from 0x026cb4 to 0x00bade (0x1b1d5 bytes) 000337 1788839051.089213 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xbadf): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbadf): Mirror 6: 0xd024232 (stale) Mirror 7: 0xd024232 (stale) Mirror 8: 0xd024232 (stale) Mirror 9: 0x411739f8 (stale) CRC-32 checksum value for chunk [0, 0xbadf): Mirror 1: 0x8aa4251d Mirror 10: 0x8aa4251d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbadf: skipped 000338 1788839052.058307 write 0x02df9f thru 0x031362 (0x33c4 bytes) 000340 1788839052.189264 MIRROR_RESYNC 6 mirrors 000342 1788839053.852008 mapwrite 0x0114e0 thru 0x01c5e6 (0xb107 bytes) 000343 1788839054.041688 mapwrite 0x0342ab thru 0x03ffff (0xbd55 bytes) 000344 1788839054.114714 punch from 0x010dd9 to 0x012ee7 (0x210f bytes) 000345 1788839054.123723 MIRROR_RESYNC 6 mirrors 000347 1788839055.962915 trunc from 0x040000 to 0x00e52e (0x31ad1 bytes) 000349 1788839056.138521 punch from 0x006001 to 0x00e52e (0x852e bytes) 000351 1788839056.147454 write 0x00c59c thru 0x01a21b (0xdc80 bytes) 000352 1788839056.165824 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a21c): Mirror 6: 0x127b942c (stale) Mirror 7: 0x127b942c (stale) Mirror 9: 0x127b942c (stale) Mirror 10: 0x127b942c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a21c): Mirror 6: 0x127b942c (stale) Mirror 7: 0x127b942c (stale) Mirror 9: 0x127b942c (stale) Mirror 10: 0x127b942c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a21c: skipped 000353 1788839056.607229 write 0x022ee6 thru 0x02dd6e (0xae89 bytes) 000354 1788839056.610424 mapwrite 0x00b556 thru 0x0168af (0xb35a bytes) 000356 1788839056.693846 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd6f): Mirror 6: 0x224ba014 (stale) Mirror 7: 0x224ba014 (stale) Mirror 10: 0x224ba014 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd6f): Mirror 6: 0x224ba014 (stale) Mirror 7: 0x224ba014 (stale) Mirror 10: 0x224ba014 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd6f: skipped 000357 1788839057.006018 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd6f): Mirror 7: 0x224ba014 (stale) Mirror 10: 0x224ba014 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd6f): Mirror 7: 0x224ba014 (stale) Mirror 10: 0x224ba014 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd6f: skipped 000360 1788839057.344746 trunc from 0x02dd6f to 0x00fcc8 (0x1e0a6 bytes) 000362 1788839057.367871 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 7: 0x33ccfef0 (stale) Mirror 10: 0x33ccfef0 (stale) CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 1: 0x50f48489 Mirror 11: 0x50f48489 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfcc9: skipped 000363 1788839058.198366 write 0x014924 thru 0x01ed18 (0xa3f5 bytes) 000364 1788839058.285087 mapwrite 0x02b45b thru 0x0356d2 (0xa278 bytes) 000365 1788839058.368875 punch from 0x0287f5 to 0x030830 (0x803c bytes) 000366 1788839058.378502 MIRROR_RESYNC 4 mirrors 000368 1788839060.028650 trunc from 0x0356d3 to 0x0188a8 (0x1ce2a bytes) 000372 1788839060.109649 read 0x018878 thru 0x0188a8 (0x31 bytes) 000374 1788839060.116009 mapread 0x0056c3 thru 0x014cb5 (0xf5f3 bytes) 000375 1788839060.141699 trunc from 0x0188a9 to 0x03778f (0x1eee7 bytes) 000377 1788839060.172739 trunc from 0x037790 to 0x03a46a (0x2cdb bytes) 000378 1788839060.208035 write 0x03bba9 thru 0x03fb6f (0x3fc7 bytes) 000380 1788839060.215521 punch from 0x005adf to 0x0067eb (0xd0d bytes) 000381 1788839060.258777 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb70): Mirror 7: 0x9d8dac87 (stale) Mirror 11: 0x9d8dac87 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb70): Mirror 7: 0x9d8dac87 (stale) Mirror 11: 0x9d8dac87 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb70: skipped 000383 1788839060.649109 mapwrite 0x03d56c thru 0x03ffff (0x2a94 bytes) 000385 1788839060.757606 write 0x02db0f thru 0x03704c (0x953e bytes) 000386 1788839060.791631 trunc from 0x040000 to 0x036e7b (0x9184 bytes) 000387 1788839060.822785 read 0x000e59 thru 0x00109c (0x244 bytes) 000388 1788839060.844518 MIRROR_RESYNC 3 mirrors 000389 1788839061.945008 mapwrite 0x02d538 thru 0x037edf (0xa9a8 bytes) 000391 1788839062.087683 mapread 0x02ec27 thru 0x02f844 (0xc1e bytes) 000392 1788839062.095561 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 7: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 7: 0xce4692d6 (stale) Mirror 11: 0xce4692d6 (stale) CRC-32 checksum value for chunk [0, 0x37ee0): Mirror 1: 0x73cbf164 Mirror 12: 0x73cbf164 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ee0: skipped 000393 1788839063.303968 trunc from 0x037ee0 to 0x009d87 (0x2e158 bytes) 000394 1788839063.473886 trunc from 0x009d88 to 0x001347 (0x8a40 bytes) 000396 1788839063.503436 mapwrite 0x00d224 thru 0x0119a1 (0x477e bytes) 000397 1788839063.641107 MIRROR_RESYNC 4 mirrors 000398 1788839064.693545 mapread 0x0053ea thru 0x00fe5f (0xaa76 bytes) 000399 1788839064.725785 trunc from 0x0119a2 to 0x0151b5 (0x3814 bytes) 000400 1788839064.829855 trunc from 0x0151b6 to 0x017393 (0x21de bytes) 000401 1788839064.862840 mapread 0x0118c1 thru 0x012f14 (0x1654 bytes) 000405 1788839064.876876 trunc from 0x017394 to 0x02fd49 (0x189b6 bytes) 000406 1788839064.911818 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fd4a): Mirror 11: 0x87acc942 (stale) Mirror 12: 0x87acc942 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd4a): Mirror 11: 0x87acc942 (stale) Mirror 12: 0x87acc942 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd4a: skipped 000408 1788839065.206763 write 0x01b9bc thru 0x02301e (0x7663 bytes) 000409 1788839065.224167 punch from 0x027871 to 0x02b0ed (0x387d bytes) 000410 1788839065.238522 punch from 0x0195bd to 0x0259f2 (0xc436 bytes) 000413 1788839065.243767 mapwrite 0x026b0c thru 0x02dbbd (0x70b2 bytes) 000415 1788839065.369242 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2fd4a): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd4a): Mirror 11: 0x87acc942 (stale) Mirror 12: 0x87acc942 (stale) CRC-32 checksum value for chunk [0, 0x2fd4a): Mirror 1: 0x3e0e965b Mirror 13: 0x3e0e965b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd4a: skipped 000417 1788839066.431370 write 0x020b9b thru 0x02e6b2 (0xdb18 bytes) 000419 1788839066.516358 write 0x0182cb thru 0x025a14 (0xd74a bytes) 000420 1788839066.523834 trunc from 0x02fd4a to 0x02c5a0 (0x37a9 bytes) 000422 1788839066.592911 MIRROR_RESYNC 4 mirrors 000423 1788839068.373166 MIRROR_RESYNC 4 mirrors 000426 1788839068.912426 mapwrite 0x03360c thru 0x039b69 (0x655e bytes) 000427 1788839069.223158 read 0x0217d6 thru 0x023860 (0x208b bytes) 000428 1788839069.236852 MIRROR_RESYNC 4 mirrors 000429 1788839070.651272 punch from 0x0182fc to 0x01fa01 (0x7706 bytes) 000431 1788839070.716815 mapread 0x032e8c thru 0x039b69 (0x6cde bytes) 000432 1788839070.736302 read 0x031cb6 thru 0x034ace (0x2e19 bytes) 000433 1788839070.736924 punch from 0x01da3a to 0x024266 (0x682d bytes) 000434 1788839070.745799 MIRROR_RESYNC 4 mirrors 000435 1788839072.625173 trunc from 0x039b6a to 0x014c5b (0x24f0e bytes) 000436 1788839072.699063 write 0x01e1ae thru 0x025a4d (0x78a0 bytes) 000437 1788839072.700852 mapread 0x0057b5 thru 0x009de9 (0x4635 bytes) 000438 1788839072.716200 mapwrite 0x01411b thru 0x01535d (0x1243 bytes) 000439 1788839072.761248 mapwrite 0x00f7aa thru 0x011019 (0x1870 bytes) 000440 1788839072.840262 trunc from 0x025a4e to 0x016c76 (0xedd7 bytes) 000444 1788839072.897288 read 0x00bf4d thru 0x011da5 (0x5e59 bytes) 000445 1788839072.897967 punch from 0x009471 to 0x00a4dc (0x106c bytes) 000446 1788839072.905213 mapwrite 0x011ec3 thru 0x01be90 (0x9fce bytes) 000448 1788839073.030303 write 0x021847 thru 0x03028a (0xea44 bytes) 000449 1788839073.034514 write 0x015732 thru 0x021d68 (0xc637 bytes) 000450 1788839073.037258 write 0x0146ee thru 0x016205 (0x1b18 bytes) 000451 1788839073.039183 write 0x01f569 thru 0x027578 (0x8010 bytes) 000452 1788839073.047265 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3028b): Mirror 12: 0xc639a6c1 (stale) Mirror 13: 0xc639a6c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3028b): Mirror 12: 0xc639a6c1 (stale) Mirror 13: 0xc639a6c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3028b: skipped 000453 1788839073.474927 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3028b): Mirror 13: 0xc639a6c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3028b): Mirror 13: 0xc639a6c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3028b: skipped 000454 1788839073.783960 mapwrite 0x01f034 thru 0x026c41 (0x7c0e bytes) 000455 1788839073.853559 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000456 1788839074.045647 mapread 0x02453f thru 0x03028a (0xbd4c bytes) 000458 1788839074.072581 trunc from 0x03028b to 0x01fe70 (0x1041a bytes) 000459 1788839074.087112 mapread 0x01990c thru 0x01ab49 (0x123e bytes) 000460 1788839074.095451 mapread 0x01f39b thru 0x01fe70 (0xad6 bytes) 000461 1788839074.096152 write 0x03b0e1 thru 0x03ffff (0x4f1f bytes) 000462 1788839074.097201 punch from 0x01ca07 to 0x01d8b8 (0xeb2 bytes) 000463 1788839074.108897 trunc from 0x040000 to 0x0190b6 (0x26f49 bytes) 000464 1788839074.128070 read 0x01501a thru 0x018d61 (0x3d48 bytes) 000467 1788839074.130019 mapread 0x008974 thru 0x01092f (0x7fbc bytes) 000468 1788839074.138748 punch from 0x00259d to 0x0105b3 (0xe017 bytes) 000469 1788839074.143830 trunc from 0x0190b7 to 0x015691 (0x3a25 bytes) 000471 1788839074.161184 read 0x003389 thru 0x00a1c0 (0x6e38 bytes) 000472 1788839074.170610 write 0x01123c thru 0x020053 (0xee18 bytes) 000474 1788839074.172579 mapread 0x002ee5 thru 0x00c282 (0x939e bytes) 000476 1788839074.175750 mapwrite 0x01e5c9 thru 0x020e92 (0x28ca bytes) 000478 1788839074.244579 read 0x011b63 thru 0x01b245 (0x96e3 bytes) 000479 1788839074.245624 mapread 0x0030e6 thru 0x00bd2b (0x8c46 bytes) 000480 1788839074.247110 mapwrite 0x001bc4 thru 0x00e4ea (0xc927 bytes) 000481 1788839074.289321 punch from 0x01bf71 to 0x020e92 (0x4f22 bytes) 000482 1788839074.294617 mapread 0x00b791 thru 0x00ff49 (0x47b9 bytes) 000483 1788839074.296371 mapwrite 0x03659c thru 0x03f00f (0x8a74 bytes) 000485 1788839074.372517 punch from 0x00cd2d to 0x00dbdd (0xeb1 bytes) 000490 1788839074.383014 read 0x015d05 thru 0x0239b8 (0xdcb4 bytes) 000491 1788839074.385843 read 0x00f7a5 thru 0x013821 (0x407d bytes) 000492 1788839074.386577 trunc from 0x03f010 to 0x01c046 (0x22fc9 bytes) 000494 1788839074.417113 MIRROR_EXTEND to 2 mirrors 000496 1788839075.149735 punch from 0x005f09 to 0x00cc30 (0x6d28 bytes) 000497 1788839075.231138 mapwrite 0x0075d3 thru 0x007b55 (0x583 bytes) 000498 1788839075.301877 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c047): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c047): Mirror 2: 0x147cf5e5 (stale) CRC-32 checksum value for chunk [0, 0x1c047): Mirror 1: 0xbb4c4c76 Mirror 3: 0xbb4c4c76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c047: skipped 000499 1788839076.364739 punch from 0x008ae3 to 0x00bda2 (0x32c0 bytes) 000500 1788839076.473884 mapread 0x01a98e thru 0x01c046 (0x16b9 bytes) 000501 1788839076.490090 trunc from 0x01c047 to 0x00ad4a (0x112fc bytes) 000502 1788839076.547915 write 0x02ac97 thru 0x03112e (0x6498 bytes) 000503 1788839076.549234 punch from 0x026c68 to 0x03112e (0xa4c7 bytes) 000504 1788839076.563266 write 0x012200 thru 0x012397 (0x198 bytes) 000506 1788839076.574387 mapread 0x02ed8a thru 0x03112e (0x23a5 bytes) 000507 1788839076.583807 trunc from 0x03112f to 0x0094a1 (0x27c8d bytes) 000512 1788839076.603118 mapwrite 0x03a2af thru 0x03ffff (0x5d51 bytes) 000514 1788839076.725373 mapread 0x03e60b thru 0x03ffff (0x19f5 bytes) 000515 1788839076.727076 punch from 0x012968 to 0x01acb9 (0x8352 bytes) 000516 1788839076.738354 read 0x0362b0 thru 0x0398dc (0x362d bytes) 000517 1788839076.739133 trunc from 0x040000 to 0x0175c9 (0x28a36 bytes) 000519 1788839076.771008 write 0x020e39 thru 0x0280cb (0x7293 bytes) 000520 1788839076.772309 mapread 0x0206b8 thru 0x0259e0 (0x5329 bytes) 000521 1788839076.775016 mapread 0x01680a thru 0x0210e1 (0xa8d8 bytes) 000523 1788839076.806854 mapread 0x021380 thru 0x0280cb (0x6d4c bytes) 000524 1788839076.810422 write 0x022259 thru 0x02e71f (0xc4c7 bytes) 000525 1788839076.829840 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e720): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e720): Mirror 2: 0xc5568d43 (stale) Mirror 3: 0xd14b0623 (stale) CRC-32 checksum value for chunk [0, 0x2e720): Mirror 1: 0xab712fd0 Mirror 4: 0xab712fd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e720: skipped 000526 1788839078.320899 mapread 0x0032a1 thru 0x004a29 (0x1789 bytes) 000528 1788839078.379442 trunc from 0x02e720 to 0x03513b (0x6a1c bytes) 000530 1788839078.571372 mapread 0x03336c thru 0x03513b (0x1dd0 bytes) 000531 1788839078.596001 mapwrite 0x009533 thru 0x01351c (0x9fea bytes) truncating to largest ever: 0x3ed6c 000532 1788839078.689059 trunc from 0x03513c to 0x03ed6b (0x9c30 bytes) 000533 1788839078.719997 mapread 0x027d78 thru 0x02c82b (0x4ab4 bytes) 000535 1788839078.731784 mapwrite 0x035205 thru 0x03bc18 (0x6a14 bytes) 000537 1788839078.833681 read 0x034dd7 thru 0x03e4bb (0x96e5 bytes) 000542 1788839078.840750 punch from 0x02c1f5 to 0x034b4a (0x8956 bytes) 000543 1788839078.859431 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ed6c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed6c): Mirror 2: 0x25ba06c6 (stale) Mirror 3: 0x87cb1f27 (stale) Mirror 4: 0xc441123c (stale) CRC-32 checksum value for chunk [0, 0x3ed6c): Mirror 1: 0xad7bb8fe Mirror 5: 0xad7bb8fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed6c: skipped 000544 1788839080.302025 punch from 0x00cb64 to 0x013b73 (0x7010 bytes) 000546 1788839080.408530 punch from 0x01e3b3 to 0x02d22a (0xee78 bytes) 000547 1788839080.422495 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed6c): Mirror 2: 0x25ba06c6 (stale) Mirror 4: 0xc441123c (stale) Mirror 5: 0xad7bb8fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed6c): Mirror 2: 0x25ba06c6 (stale) Mirror 4: 0xc441123c (stale) Mirror 5: 0xad7bb8fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed6c: skipped 000549 1788839081.037208 read 0x03924d thru 0x03ed6b (0x5b1f bytes) 000550 1788839081.055123 trunc from 0x03ed6c to 0x01595c (0x2940f bytes) 000551 1788839081.097807 write 0x0179e5 thru 0x02311b (0xb737 bytes) 000553 1788839081.099714 punch from 0x00da47 to 0x011046 (0x3600 bytes) 000555 1788839081.132466 mapwrite 0x0059f5 thru 0x008e7a (0x3486 bytes) 000556 1788839081.229777 trunc from 0x02311c to 0x026cf6 (0x3bdb bytes) 000557 1788839081.252989 write 0x010acc thru 0x02058e (0xfac3 bytes) 000558 1788839081.261852 read 0x0165d2 thru 0x020175 (0x9ba4 bytes) 000560 1788839081.273007 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26cf7): Mirror 2: 0x11bcd808 (stale) Mirror 4: 0xc4d37b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26cf7): Mirror 2: 0x11bcd808 (stale) Mirror 4: 0xc4d37b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26cf7: skipped 000561 1788839081.604193 punch from 0x00402b to 0x00fefd (0xbed3 bytes) 000562 1788839081.609682 mapwrite 0x027381 thru 0x02a777 (0x33f7 bytes) 000563 1788839081.704525 MIRROR_RESYNC 3 mirrors 000564 1788839082.593713 mapwrite 0x01e519 thru 0x026b00 (0x85e8 bytes) 000565 1788839082.933906 mapwrite 0x02de03 thru 0x032b8e (0x4d8c bytes) 000567 1788839083.022123 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32b8f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b8f): Mirror 2: 0xa2ee2418 (stale) Mirror 4: 0xa2ee2418 (stale) CRC-32 checksum value for chunk [0, 0x32b8f): Mirror 1: 0x10552ded Mirror 5: 0x10552ded lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b8f: skipped 000568 1788839084.088900 punch from 0x029db1 to 0x02b842 (0x1a92 bytes) 000569 1788839084.163330 write 0x024698 thru 0x03448b (0xfdf4 bytes) 000570 1788839084.173495 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3448c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3448c): Mirror 2: 0xf8e95b02 (stale) Mirror 4: 0xf8e95b02 (stale) Mirror 5: 0x5315d08f (stale) CRC-32 checksum value for chunk [0, 0x3448c): Mirror 1: 0x12a2ce06 Mirror 6: 0x12a2ce06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3448c: skipped 000571 1788839085.496958 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3448c): Mirror 4: 0xf8e95b02 (stale) Mirror 5: 0x5315d08f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3448c): Mirror 4: 0xf8e95b02 (stale) Mirror 5: 0x5315d08f (stale) CRC-32 checksum value for chunk [0, 0x3448c): Mirror 1: 0x12a2ce06 Mirror 6: 0x12a2ce06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3448c: skipped 000572 1788839086.146525 mapread 0x0340ff thru 0x03448b (0x38d bytes) 000573 1788839086.189175 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3448c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3448c): Mirror 4: 0xf8e95b02 (stale) Mirror 5: 0x5315d08f (stale) CRC-32 checksum value for chunk [0, 0x3448c): Mirror 1: 0x12a2ce06 Mirror 6: 0x12a2ce06 Mirror 7: 0x12a2ce06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3448c: skipped 000575 1788839087.048348 trunc from 0x03448c to 0x01f967 (0x14b24 bytes) 000576 1788839087.193139 trunc from 0x01f968 to 0x015638 (0xa32f bytes) 000577 1788839087.263469 punch from 0x00d611 to 0x015638 (0x8028 bytes) 000580 1788839087.274733 write 0x00877d thru 0x0162bc (0xdb40 bytes) 000581 1788839087.283683 mapwrite 0x03839b thru 0x03ffff (0x7c65 bytes) 000584 1788839087.404928 mapwrite 0x000ecd thru 0x0077a5 (0x68d9 bytes) 000585 1788839087.446991 mapread 0x0020bd thru 0x010da1 (0xece5 bytes) 000588 1788839087.449575 mapread 0x000476 thru 0x00cb67 (0xc6f2 bytes) 000590 1788839087.451453 read 0x02e933 thru 0x038079 (0x9747 bytes) 000595 1788839087.466551 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc9bb8343 (stale) Mirror 6: 0xf63f8989 (stale) Mirror 7: 0xf63f8989 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc9bb8343 (stale) Mirror 6: 0xf63f8989 (stale) Mirror 7: 0xf63f8989 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000596 1788839087.911788 write 0x0155c0 thru 0x022b5f (0xd5a0 bytes) 000599 1788839087.921766 trunc from 0x040000 to 0x0251db (0x1ae24 bytes) 000600 1788839087.941983 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x251dc): Mirror 4: 0xfed88aee (stale) Mirror 6: 0x5ed78ba2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x251dc): Mirror 4: 0xfed88aee (stale) Mirror 6: 0x5ed78ba2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x251dc: skipped 000601 1788839088.244319 mapread 0x0087f1 thru 0x014ea7 (0xc6b7 bytes) 000602 1788839088.268359 write 0x004e97 thru 0x006501 (0x166b bytes) 000603 1788839088.269608 trunc from 0x0251dc to 0x005b9a (0x1f641 bytes) 000604 1788839088.294642 MIRROR_RESYNC 3 mirrors 000605 1788839089.017906 trunc from 0x005b9b to 0x021fbb (0x1c421 bytes) 000606 1788839089.087560 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21fbc): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21fbc): Mirror 4: 0x37e44582 (stale) Mirror 6: 0x37e44582 (stale) CRC-32 checksum value for chunk [0, 0x21fbc): Mirror 1: 0x37e44582 Mirror 7: 0x37e44582 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21fbc: skipped 000607 1788839089.881499 read 0x00ce73 thru 0x00f192 (0x2320 bytes) 000608 1788839089.910858 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21fbc): Mirror 6: 0x37e44582 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21fbc): Mirror 6: 0x37e44582 (stale) CRC-32 checksum value for chunk [0, 0x21fbc): Mirror 1: 0x37e44582 Mirror 7: 0x37e44582 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21fbc: skipped 000609 1788839090.444564 write 0x039a18 thru 0x03ffff (0x65e8 bytes) 000611 1788839090.503463 trunc from 0x040000 to 0x035231 (0xadce bytes) 000613 1788839090.533290 read 0x01bc9b thru 0x02b5f4 (0xf95a bytes) 000614 1788839090.547073 MIRROR_RESYNC 3 mirrors 000615 1788839091.430133 read 0x00bbc3 thru 0x00e88e (0x2ccc bytes) 000616 1788839091.438207 write 0x015eb6 thru 0x02210b (0xc256 bytes) 000618 1788839091.530341 mapread 0x01f825 thru 0x02c924 (0xd100 bytes) 000619 1788839091.549567 write 0x008a05 thru 0x0118c3 (0x8ebf bytes) 000620 1788839091.559072 mapwrite 0x039103 thru 0x03ffff (0x6efd bytes) 000621 1788839091.669596 punch from 0x035058 to 0x03fd96 (0xad3f bytes) 000622 1788839091.680732 write 0x02edb2 thru 0x03a251 (0xb4a0 bytes) 000623 1788839091.695819 read 0x03c72e thru 0x03ffff (0x38d2 bytes) 000625 1788839091.708971 mapread 0x0312f9 thru 0x03a396 (0x909e bytes) 000627 1788839091.710325 mapwrite 0x03b2b2 thru 0x03ffff (0x4d4e bytes) 000628 1788839091.754847 trunc from 0x040000 to 0x004ff2 (0x3b00d bytes) 000631 1788839091.774771 trunc from 0x004ff3 to 0x01f096 (0x1a0a4 bytes) 000632 1788839091.793770 punch from 0x01440a to 0x0192ea (0x4ee1 bytes) 000634 1788839091.799950 mapwrite 0x035031 thru 0x03c1d6 (0x71a6 bytes) 000636 1788839091.909740 write 0x014d87 thru 0x01ded1 (0x914b bytes) 000637 1788839091.925353 write 0x006ad3 thru 0x0098de (0x2e0c bytes) 000640 1788839091.942290 mapwrite 0x037006 thru 0x03ffff (0x8ffa bytes) 000642 1788839092.040733 trunc from 0x040000 to 0x03e966 (0x1699 bytes) 000643 1788839092.089457 MIRROR_RESYNC 3 mirrors 000645 1788839093.247367 punch from 0x03a8d0 to 0x03e966 (0x4097 bytes) 000646 1788839093.309371 mapwrite 0x01b5f5 thru 0x02b243 (0xfc4f bytes) 000648 1788839093.398715 mapwrite 0x019874 thru 0x027001 (0xd78e bytes) 000650 1788839093.447473 write 0x01fb0a thru 0x02d3d4 (0xd8cb bytes) 000652 1788839093.450679 trunc from 0x03e967 to 0x008756 (0x36210 bytes) 000653 1788839093.472334 read 0x0002d9 thru 0x0060e6 (0x5e0e bytes) 000654 1788839093.480722 write 0x01c09c thru 0x022ea8 (0x6e0d bytes) 000657 1788839093.485347 trunc from 0x022ea9 to 0x02c089 (0x91e1 bytes) 000658 1788839093.505687 write 0x009978 thru 0x00bd5d (0x23e6 bytes) 000660 1788839093.521229 punch from 0x0016bd to 0x00e313 (0xcc57 bytes) 000661 1788839093.542346 read 0x0278ee thru 0x02c089 (0x479c bytes) 000662 1788839093.561091 trunc from 0x02c08a to 0x02d433 (0x13aa bytes) 000663 1788839093.583322 punch from 0x0246a7 to 0x02d433 (0x8d8d bytes) 000665 1788839093.593651 punch from 0x010fec to 0x018dab (0x7dc0 bytes) 000666 1788839093.603269 punch from 0x0052e1 to 0x00902b (0x3d4b bytes) 000669 1788839093.612912 MIRROR_RESYNC 3 mirrors 000670 1788839094.667314 trunc from 0x02d434 to 0x02a31d (0x3116 bytes) 000671 1788839094.742025 mapwrite 0x004957 thru 0x006a8b (0x2135 bytes) 000673 1788839094.824322 read 0x012421 thru 0x0127f8 (0x3d8 bytes) 000674 1788839094.829705 mapwrite 0x03e38b thru 0x03ffff (0x1c75 bytes) 000676 1788839094.928336 mapread 0x01d185 thru 0x0287aa (0xb626 bytes) 000679 1788839094.951331 write 0x02ce35 thru 0x02f9c3 (0x2b8f bytes) 000681 1788839094.958846 trunc from 0x040000 to 0x01023d (0x2fdc2 bytes) 000683 1788839094.974096 punch from 0x000d5e to 0x00abd6 (0x9e79 bytes) 000684 1788839094.978565 mapwrite 0x01ee47 thru 0x021fef (0x31a9 bytes) 000685 1788839095.060320 read 0x01a024 thru 0x01dd44 (0x3d21 bytes) 000686 1788839095.060946 read 0x01e1ac thru 0x021fef (0x3e44 bytes) 000687 1788839095.061451 mapwrite 0x02cf06 thru 0x03b737 (0xe832 bytes) 000689 1788839095.191646 punch from 0x033785 to 0x0373bf (0x3c3b bytes) 000690 1788839095.199173 trunc from 0x03b738 to 0x01467c (0x270bb bytes) 000693 1788839095.227270 read 0x008204 thru 0x00f6ad (0x74aa bytes) 000695 1788839095.229167 trunc from 0x01467d to 0x013244 (0x1438 bytes) 000696 1788839095.254727 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13245): Mirror 6: 0x94203890 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13245): Mirror 6: 0x94203890 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13245: skipped 000698 1788839095.515316 read 0x011605 thru 0x013244 (0x1c40 bytes) 000699 1788839095.521148 punch from 0x00bd8b to 0x013244 (0x74ba bytes) 000700 1788839095.530083 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13245): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13245): Mirror 6: 0x94203890 (stale) CRC-32 checksum value for chunk [0, 0x13245): Mirror 1: 0xd19e61fd Mirror 7: 0xd19e61fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13245: skipped 000701 1788839096.550900 mapread 0x010702 thru 0x013244 (0x2b43 bytes) 000702 1788839096.585269 mapwrite 0x013683 thru 0x01a4e0 (0x6e5e bytes) 000703 1788839096.679072 read 0x012164 thru 0x018547 (0x63e4 bytes) 000704 1788839096.680202 read 0x00ea74 thru 0x01a4e0 (0xba6d bytes) 000705 1788839096.685376 MIRROR_RESYNC 3 mirrors 000706 1788839097.618238 read 0x013449 thru 0x015faa (0x2b62 bytes) 000707 1788839097.622428 mapwrite 0x03193d thru 0x03a20a (0x88ce bytes) 000708 1788839097.833718 mapread 0x001472 thru 0x009b70 (0x86ff bytes) 000709 1788839097.843883 write 0x01a8dc thru 0x01cf85 (0x26aa bytes) 000710 1788839097.853223 punch from 0x00ae70 to 0x00b01b (0x1ac bytes) 000711 1788839097.862308 punch from 0x010c5b to 0x013fc6 (0x336c bytes) 000712 1788839097.866575 punch from 0x000a95 to 0x007e67 (0x73d3 bytes) 000713 1788839097.875067 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a20b): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a20b): Mirror 6: 0x87bdee46 (stale) Mirror 7: 0x87bdee46 (stale) CRC-32 checksum value for chunk [0, 0x3a20b): Mirror 1: 0x676ade9f Mirror 8: 0x676ade9f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a20b: skipped 000714 1788839098.862269 MIRROR_RESYNC 4 mirrors 000715 1788839100.070311 trunc from 0x03a20b to 0x01bc53 (0x1e5b7 bytes) 000717 1788839100.159362 trunc from 0x01bc54 to 0x0235d8 (0x7985 bytes) 000724 1788839100.189454 trunc from 0x0235d9 to 0x03dbf9 (0x1a621 bytes) 000725 1788839100.222921 punch from 0x00aad0 to 0x017b6e (0xd09f bytes) 000726 1788839100.230442 mapwrite 0x019a2c thru 0x01de94 (0x4469 bytes) 000727 1788839100.320157 mapread 0x027238 thru 0x030230 (0x8ff9 bytes) 000729 1788839100.346875 punch from 0x02e98d to 0x030d95 (0x2409 bytes) 000730 1788839100.360652 punch from 0x00c36e to 0x00f233 (0x2ec6 bytes) 000731 1788839100.371801 punch from 0x03d7f4 to 0x03dbf9 (0x406 bytes) 000732 1788839100.378701 mapread 0x033ddb thru 0x03dbf9 (0x9e1f bytes) 000733 1788839100.389798 write 0x03cfb4 thru 0x03ffff (0x304c bytes) 000738 1788839100.393956 mapwrite 0x02e43e thru 0x031ce4 (0x38a7 bytes) 000739 1788839100.455833 mapread 0x02918a thru 0x03203f (0x8eb6 bytes) 000740 1788839100.469671 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2a48f843 (stale) Mirror 7: 0x2a48f843 (stale) Mirror 8: 0x2a48f843 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xae89c0b9 Mirror 9: 0xae89c0b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000741 1788839101.370861 read 0x0110f9 thru 0x01bf39 (0xae41 bytes) 000742 1788839101.378893 punch from 0x02d629 to 0x039836 (0xc20e bytes) 000744 1788839101.455032 MIRROR_RESYNC 5 mirrors 000746 1788839103.443892 write 0x004b1e thru 0x00f092 (0xa575 bytes) 000747 1788839103.512481 write 0x0034ff thru 0x013147 (0xfc49 bytes) 000748 1788839103.529446 mapread 0x029c41 thru 0x03820e (0xe5ce bytes) 000749 1788839103.567167 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x726f2d0e (stale) Mirror 8: 0x726f2d0e (stale) Mirror 9: 0x726f2d0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x726f2d0e (stale) Mirror 8: 0x726f2d0e (stale) Mirror 9: 0x726f2d0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000750 1788839103.844012 mapread 0x01c4e7 thru 0x02311e (0x6c38 bytes) 000752 1788839103.859844 mapwrite 0x0114b0 thru 0x013deb (0x293c bytes) 000754 1788839103.911412 read 0x01d2b3 thru 0x01e1ba (0xf08 bytes) 000755 1788839103.913209 mapwrite 0x00b399 thru 0x01a1ee (0xee56 bytes) 000756 1788839103.962604 punch from 0x01ce62 to 0x022743 (0x58e2 bytes) 000757 1788839103.968199 read 0x004d23 thru 0x01287e (0xdb5c bytes) 000758 1788839103.977376 mapread 0x011b62 thru 0x01f8db (0xdd7a bytes) 000759 1788839103.982261 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x726f2d0e (stale) Mirror 9: 0x726f2d0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x726f2d0e (stale) Mirror 9: 0x726f2d0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000761 1788839104.316217 MIRROR_RESYNC 3 mirrors 000762 1788839105.483599 MIRROR_RESYNC 3 mirrors 000764 1788839105.934040 punch from 0x034119 to 0x034501 (0x3e9 bytes) 000766 1788839105.996721 write 0x012e89 thru 0x018590 (0x5708 bytes) 000767 1788839106.003963 write 0x0245ac thru 0x02b1f7 (0x6c4c bytes) 000770 1788839106.017944 mapread 0x009d65 thru 0x018e9b (0xf137 bytes) 000771 1788839106.037913 read 0x00a67a thru 0x014fc8 (0xa94f bytes) 000773 1788839106.040842 mapread 0x01c9ab thru 0x023422 (0x6a78 bytes) 000774 1788839106.055610 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x357c36b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x357c36b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000775 1788839106.292189 mapread 0x02c33c thru 0x02f202 (0x2ec7 bytes) 000777 1788839106.312965 mapwrite 0x02dab4 thru 0x03a3e6 (0xc933 bytes) 000779 1788839106.369352 write 0x02c223 thru 0x02f635 (0x3413 bytes) 000781 1788839106.372169 mapread 0x02265d thru 0x028a5d (0x6401 bytes) 000782 1788839106.380322 mapwrite 0x021c79 thru 0x02a105 (0x848d bytes) 000783 1788839106.422770 read 0x008eb0 thru 0x00b0ae (0x21ff bytes) 000784 1788839106.428467 trunc from 0x040000 to 0x01a2f0 (0x25d0f bytes) 000785 1788839106.444928 write 0x024eeb thru 0x02bfff (0x7115 bytes) 000786 1788839106.446436 write 0x02d7d4 thru 0x031e5c (0x4689 bytes) 000787 1788839106.451626 read 0x00bba1 thru 0x01583a (0x9c9a bytes) 000788 1788839106.460353 mapread 0x021ae7 thru 0x028d86 (0x72a0 bytes) 000789 1788839106.465960 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31e5d): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e5d): Mirror 8: 0x75619aba (stale) CRC-32 checksum value for chunk [0, 0x31e5d): Mirror 1: 0x4e7e60c8 Mirror 9: 0x4e7e60c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e5d: skipped 000792 1788839107.369265 mapwrite 0x03d6c1 thru 0x03ffff (0x293f bytes) 000794 1788839107.527620 write 0x02f544 thru 0x0341c9 (0x4c86 bytes) 000795 1788839107.541348 trunc from 0x040000 to 0x003d06 (0x3c2f9 bytes) 000796 1788839107.579943 MIRROR_RESYNC 3 mirrors 000798 1788839108.220270 mapwrite 0x029a71 thru 0x031829 (0x7db9 bytes) 000800 1788839108.403908 punch from 0x02a60e to 0x031829 (0x721c bytes) 000801 1788839108.409538 read 0x005706 thru 0x00c409 (0x6d04 bytes) 000802 1788839108.418572 mapread 0x02a991 thru 0x02fcc8 (0x5338 bytes) 000803 1788839108.431598 write 0x00ce7e thru 0x012b93 (0x5d16 bytes) 000805 1788839108.440971 write 0x01188d thru 0x01ad1d (0x9491 bytes) 000806 1788839108.455517 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3182a): Mirror 8: 0x348838f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3182a): Mirror 8: 0x348838f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3182a: skipped 000807 1788839108.735618 punch from 0x02b212 to 0x02b69c (0x48b bytes) 000808 1788839108.745083 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000810 1788839108.876318 mapwrite 0x023811 thru 0x024382 (0xb72 bytes) 000811 1788839108.933462 trunc from 0x03182a to 0x008a71 (0x28db8 bytes) 000812 1788839108.951563 write 0x03e532 thru 0x03ffff (0x1ace bytes) 000813 1788839108.953187 mapwrite 0x017898 thru 0x020176 (0x88df bytes) 000816 1788839109.016656 write 0x03b5e6 thru 0x03c9ce (0x13e9 bytes) 000817 1788839109.036138 trunc from 0x040000 to 0x001238 (0x3edc7 bytes) 000818 1788839109.080600 mapwrite 0x013e01 thru 0x0142ad (0x4ad bytes) 000819 1788839109.202640 read 0x007d0d thru 0x010ea2 (0x9196 bytes) 000820 1788839109.226159 write 0x0378b4 thru 0x03b192 (0x38df bytes) 000822 1788839109.229351 mapread 0x02b101 thru 0x036885 (0xb785 bytes) 000823 1788839109.248118 trunc from 0x03b193 to 0x010a35 (0x2a75d bytes) 000826 1788839109.273610 mapread 0x008f3f thru 0x009ce4 (0xda6 bytes) 000827 1788839109.278810 MIRROR_EXTEND to 2 mirrors 000828 1788839110.191529 mapwrite 0x019372 thru 0x021613 (0x82a2 bytes) 000829 1788839110.369167 trunc from 0x021614 to 0x03bfed (0x1a9da bytes) 000830 1788839110.397006 mapread 0x02776b thru 0x030608 (0x8e9e bytes) 000831 1788839110.425623 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000832 1788839110.587724 read 0x022b9e thru 0x023b15 (0xf78 bytes) 000833 1788839110.598184 mapwrite 0x018ce5 thru 0x025abe (0xcdda bytes) 000835 1788839110.712573 trunc from 0x03bfee to 0x01fc8a (0x1c363 bytes) 000837 1788839110.733791 mapread 0x01f8b0 thru 0x01fc8a (0x3db bytes) 000838 1788839110.734706 read 0x008b94 thru 0x00a9a1 (0x1e0e bytes) 000844 1788839110.759895 trunc from 0x01fc8b to 0x027972 (0x7ce8 bytes) 000845 1788839110.823074 trunc from 0x027973 to 0x0240e7 (0x388b bytes) 000847 1788839110.893589 MIRROR_EXTEND to 2 mirrors 000848 1788839111.756452 MIRROR_EXTEND to 3 mirrors 000849 1788839112.606371 mapwrite 0x01ff23 thru 0x029da6 (0x9e84 bytes) 000850 1788839112.761681 punch from 0x00ef8c to 0x01c126 (0xd19b bytes) 000851 1788839112.767148 mapwrite 0x03404b thru 0x03ffff (0xbfb5 bytes) 000852 1788839112.865848 read 0x003987 thru 0x0046ca (0xd44 bytes) 000853 1788839112.873744 punch from 0x026bc7 to 0x02e49e (0x78d8 bytes) 000855 1788839112.885156 MIRROR_RESYNC 3 mirrors 000860 1788839114.124331 mapwrite 0x006a31 thru 0x016472 (0xfa42 bytes) 000861 1788839114.308336 mapwrite 0x03ed7f thru 0x03ffff (0x1281 bytes) 000862 1788839114.354310 mapwrite 0x02ae92 thru 0x035b4b (0xacba bytes) 000863 1788839114.434929 MIRROR_RESYNC 3 mirrors 000864 1788839115.260288 mapread 0x021433 thru 0x02cf19 (0xbae7 bytes) 000866 1788839115.288435 mapwrite 0x01bdac thru 0x0262b3 (0xa508 bytes) 000867 1788839115.445554 read 0x021383 thru 0x02eea2 (0xdb20 bytes) 000868 1788839115.452387 trunc from 0x040000 to 0x01fb47 (0x204b8 bytes) 000871 1788839115.480068 mapread 0x01d1a1 thru 0x01d7fa (0x65a bytes) 000874 1788839115.483786 write 0x017e33 thru 0x01e24e (0x641c bytes) 000876 1788839115.492030 mapwrite 0x02a1c5 thru 0x02b50a (0x1346 bytes) 000879 1788839115.594261 mapread 0x01de4f thru 0x024f31 (0x70e3 bytes) 000880 1788839115.606286 mapwrite 0x02f054 thru 0x02f64d (0x5fa bytes) 000881 1788839115.746593 mapwrite 0x025868 thru 0x0350a6 (0xf83f bytes) 000884 1788839115.886521 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x350a7): Mirror 3: 0xaf0644a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x350a7): Mirror 3: 0xaf0644a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x350a7: skipped 000885 1788839116.207150 trunc from 0x0350a7 to 0x006141 (0x2ef65 bytes) 000887 1788839116.228112 trunc from 0x006142 to 0x030b67 (0x2aa26 bytes) 000889 1788839116.250691 mapread 0x0215e4 thru 0x0285fd (0x701a bytes) 000891 1788839116.266197 read 0x025211 thru 0x02d4b1 (0x82a1 bytes) 000892 1788839116.270680 mapwrite 0x010488 thru 0x017d37 (0x78b0 bytes) 000893 1788839116.330837 mapwrite 0x0135df thru 0x018b06 (0x5528 bytes) 000894 1788839116.367197 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000895 1788839116.480751 mapwrite 0x0345fe thru 0x038fef (0x49f2 bytes) 000896 1788839116.559349 read 0x0248ff thru 0x0282ca (0x39cc bytes) 000897 1788839116.572858 mapread 0x0031fd thru 0x007ab5 (0x48b9 bytes) 000899 1788839116.584246 mapwrite 0x01c6d5 thru 0x01d9cd (0x12f9 bytes) 000901 1788839116.632299 mapwrite 0x0202fb thru 0x02b91c (0xb622 bytes) 000902 1788839116.729032 punch from 0x02d53b to 0x03210a (0x4bd0 bytes) 000906 1788839116.738687 trunc from 0x038ff0 to 0x0028c3 (0x3672c bytes) 000907 1788839116.760472 punch from 0x000c3b to 0x0028c3 (0x1c89 bytes) 000908 1788839116.765075 read 0x00002d thru 0x0028c3 (0x2897 bytes) 000909 1788839116.773340 punch from 0x002379 to 0x0028c3 (0x54b bytes) 000910 1788839116.779971 read 0x0021c8 thru 0x0028c3 (0x6fc bytes) 000912 1788839116.800352 trunc from 0x0028c4 to 0x0181e7 (0x15924 bytes) 000913 1788839116.825129 mapwrite 0x03646b thru 0x03e370 (0x7f06 bytes) 000914 1788839116.918426 trunc from 0x03e371 to 0x039b05 (0x486b bytes) 000916 1788839116.935763 punch from 0x0009f0 to 0x0092de (0x88ef bytes) 000917 1788839116.940773 write 0x026a90 thru 0x02b101 (0x4672 bytes) 000921 1788839116.946162 trunc from 0x039b06 to 0x03d31f (0x381a bytes) 000922 1788839116.971993 mapread 0x03d112 thru 0x03d31f (0x20e bytes) 000923 1788839116.980480 write 0x024946 thru 0x030dda (0xc495 bytes) 000924 1788839116.982300 write 0x02cbab thru 0x02d0fc (0x552 bytes) 000929 1788839116.991559 MIRROR_EXTEND to 2 mirrors 000930 1788839117.730144 mapread 0x02611a thru 0x033f1a (0xde01 bytes) 000933 1788839117.755270 mapwrite 0x00e051 thru 0x00f45f (0x140f bytes) 000934 1788839117.905162 punch from 0x03c354 to 0x03d31f (0xfcc bytes) 000937 1788839117.911130 mapwrite 0x01f930 thru 0x02dc43 (0xe314 bytes) 000940 1788839117.959486 write 0x030be8 thru 0x035d46 (0x515f bytes) 000941 1788839117.964534 read 0x020e46 thru 0x028239 (0x73f4 bytes) 000942 1788839117.965906 trunc from 0x03d320 to 0x02fc1f (0xd700 bytes) 000943 1788839117.983884 mapread 0x003f6e thru 0x0048eb (0x97e bytes) 000944 1788839117.990272 mapwrite 0x024593 thru 0x0314c3 (0xcf31 bytes) 000947 1788839118.047308 mapwrite 0x03c334 thru 0x03d10f (0xddc bytes) 000948 1788839118.118607 trunc from 0x03d110 to 0x02c204 (0x10f0b bytes) 000949 1788839118.135327 mapread 0x00c61f thru 0x016881 (0xa263 bytes) 000951 1788839118.137341 punch from 0x02058d to 0x0227ee (0x2262 bytes) 000954 1788839118.143350 trunc from 0x02c205 to 0x004c0a (0x275fa bytes) 000955 1788839118.160108 punch from 0x003cf3 to 0x004c0a (0xf18 bytes) 000957 1788839118.168756 read 0x0028ec thru 0x004c0a (0x231f bytes) 000958 1788839118.178442 punch from 0x000ca4 to 0x0045c5 (0x3922 bytes) 000959 1788839118.190362 write 0x020159 thru 0x0222f9 (0x21a1 bytes) 000960 1788839118.193242 write 0x00f133 thru 0x01001f (0xeed bytes) 000961 1788839118.210238 read 0x0128ee thru 0x015748 (0x2e5b bytes) 000962 1788839118.211855 punch from 0x012306 to 0x01afc8 (0x8cc3 bytes) 000964 1788839118.252363 mapread 0x00ba19 thru 0x00d2b2 (0x189a bytes) 000966 1788839118.254805 read 0x0191bf thru 0x0222f9 (0x913b bytes) 000967 1788839118.270813 read 0x016ffd thru 0x01cc6d (0x5c71 bytes) 000968 1788839118.271375 trunc from 0x0222fa to 0x0112e3 (0x11016 bytes) 000969 1788839118.307963 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b7:0x0]: Invalid argument (22) 000971 1788839118.463298 read 0x010d74 thru 0x0112e3 (0x570 bytes) 000972 1788839118.470229 trunc from 0x0112e4 to 0x032468 (0x21185 bytes) 000973 1788839118.487344 read 0x01bef0 thru 0x01c2a6 (0x3b7 bytes) 000976 1788839118.501019 write 0x01bfa4 thru 0x01f988 (0x39e5 bytes) 000977 1788839118.502136 mapwrite 0x02b45a thru 0x037350 (0xbef7 bytes) 000978 1788839118.588234 punch from 0x00a801 to 0x014ce0 (0xa4e0 bytes) 000979 1788839118.595151 mapread 0x016503 thru 0x025de5 (0xf8e3 bytes) 000980 1788839118.596209 write 0x019913 thru 0x01fcf8 (0x63e6 bytes) 000981 1788839118.598031 mapwrite 0x0380b3 thru 0x03c350 (0x429e bytes) 000983 1788839118.687997 write 0x02dab5 thru 0x03a7d6 (0xcd22 bytes) 000985 1788839118.689940 mapread 0x0257b5 thru 0x02ef0f (0x975b bytes) 000987 1788839118.692148 mapread 0x007902 thru 0x00b709 (0x3e08 bytes) 000989 1788839118.705243 mapread 0x035b47 thru 0x03c350 (0x680a bytes) 000991 1788839118.707884 mapread 0x001b6a thru 0x003e29 (0x22c0 bytes) 000992 1788839118.709151 write 0x0298f9 thru 0x030748 (0x6e50 bytes) 000995 1788839118.711227 mapwrite 0x02024e thru 0x022ea0 (0x2c53 bytes) 000996 1788839118.782073 mapread 0x009b99 thru 0x0108bc (0x6d24 bytes) 000998 1788839118.807240 mapread 0x021730 thru 0x023efe (0x27cf bytes) 001000 1788839118.811161 read 0x00b7ee thru 0x017f28 (0xc73b bytes) All operations completed A-OK! PASS 70a (104s) == sanity-flr test 71: check mirror extend parent fid ==== 23:45:23 (1788839123) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 23:45:26 (1788839126) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168083 s, 12.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132326 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 23:45:33 (1788839133) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3605728 .. Extending file size to 8375232 .. Extending file size to 8374112 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8962752 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8484192 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9116160 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57469 57470 57471 57473 57476 done 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) 192.168.202.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:46:56 (1788839216) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5896000 .. Extending file size to 6127744 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7506912 .. Extending file size to 7196672 .. Extending file size to 8300672 .. Extending file size to 8715872 .. Extending file size to 8453184 .. Extending file size to 9099648 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8637440 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8907456 .. done Waiting 59029 59030 59031 59032 59034 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) 192.168.202.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:48:22 (1788839302) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154247 s, 13.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.366593 s, 5.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd6aabea8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:48:45 (1788839325) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0865552 s, 24.2 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 23:48:55 (1788839335) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x208:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:49:03 (1788839343) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.419426 s, 5.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:49:13 (1788839353) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255266 s, 8.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } PASS 204b (12s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:49:25 (1788839365) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.422788 s, 5.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:50:02 (1788839402) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg236-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146283 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:51:05 (1788839465) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg236-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202617 s, 10.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:52:04 (1788839524) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg236-server mds-ost sync done. Creating new pool test_204f oleg236-server: Pool lustre.test_204f created Adding targets to pool oleg236-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg236-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.472388 s, 4.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg236-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg236-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg236-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:53:26 (1788839606) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:53:33 (1788839613) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:53:41 (1788839621) oleg236-server: Pool lustre.test_206 created Adding targets to pool oleg236-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg236-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg236-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg236-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg236-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:54:07 (1788839647) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08071 s, 13.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377289 s, 27.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3bef0b32 Mirror 3: 0x3bef0b32 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:54:17 (1788839657) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:54:21 (1788839661) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:54:24 (1788839664) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.128026 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x18b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x21d0000 Read this (4 bytes): BBBB 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:55:00 (1788839700) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0209927 s, 390 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:55:07 (1788839707) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401856 s, 26.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22d lmm_fid: [0x200000404:0x22d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22d lmm_fid: [0x200000404:0x22d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834477 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:55:08.000000000 -0400 Modify: 2026-09-07 23:55:08.000000000 -0400 Change: 2026-09-07 23:55:09.000000000 -0400 Birth: 2026-09-07 23:55:08.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:55:14 (1788839714) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:55:17 (1788839717) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.621605 s, 65.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0399459 s, 103 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 23:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 23:55 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:55:26 (1788839726) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:55:33 (1788839733) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1765 sec =========== 23:55:41 (1788839741) === sanity-flr: start cleanup 23:55:42 (1788839742) === === sanity-flr: finish cleanup 23:55:48 (1788839748) ===