-----============= acceptance-small: sanity-flr ============----- Fri Jul 3 23:22:57 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg213-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:23:16 (1783135396) === oleg213-client.virtnet: executing check_config_client /mnt/lustre oleg213-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg213-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff958e50001800.idle_timeout=debug osc.lustre-OST0001-osc-ffff958e50001800.idle_timeout=debug disable quota as required oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:23:51 (1783135431) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7606] == sanity-flr test 0a: lfs mirror create with -N option == 23:23:53 (1783135433) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:24:08 (1783135448) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:24:11 (1783135451) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:24:14 (1783135454) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:24:37 (1783135477) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:24:40 (1783135480) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:24:43 (1783135483) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff958e50001800.stats=clear osc.lustre-OST0001-osc-ffff958e50001800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:24:50 (1783135490) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:25:01 (1783135501) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:25:10 (1783135510) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:25:13 (1783135513) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:25:21 (1783135521) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 23:25:29 (1783135529) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:25:37 (1783135537) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:25:41 (1783135541) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:25:49 (1783135549) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:25:59 (1783135559) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:26:07 (1783135567) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.778303 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310611 s, 3.4 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:26:18 (1783135578) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.571644 s, 25.7 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:26:28 (1783135588) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:26:39 (1783135599) Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg213-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:27:52 (1783135672) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.801164 s, 14.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (50s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:28:42 (1783135722) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg213-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:30:06 (1783135806) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.81543 s, 13.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 3 23:30 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.027s user 0m0.000s sys 0m0.162s Stopping /mnt/lustre-ost1 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958e50001800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.230s user 0m0.001s sys 0m0.190s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg213-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg213-server oleg213-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.281s user 0m0.001s sys 0m0.196s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg213-server: oleg213-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg213-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958e50001800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:31:36 (1783135896) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:31:39 (1783135899) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:31:43 (1783135903) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:31:46 (1783135906) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:31:54 (1783135914) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.833545 s, 15.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:32:08 (1783135928) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500848 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418306 s, 25.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:32:15 (1783135935) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:32:23 (1783135943) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125272 s, 817 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000980464 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000784231 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00131025 s, 782 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 23:32:33 (1783135953) create test files with size 15731870 .. 0b94ef1f3f86b2ad97ae7634b421cc30 - 0e50de0b86f18c837c8cde79ab5cb090 - 91d37a0df3c8e9643b03a274bce422a8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 23:32:55 (1783135975) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:33:08 (1783135988) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:33:14 (1783135994) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982393 s, 10.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848406 s, 12.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 23:33:23 (1783136003) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15844 s, 13.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.313313 s, 13.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164631 s, 12.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 23:33:39 (1783136019) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 23:33:43 (1783136023) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:33:47 (1783136027) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284438 s, 36.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 23:33:54 (1783136034) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:33:58 (1783136038) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:34:08 (1783136048) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.341243 s, 30.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:34:19 (1783136059) llite.lustre-ffff958e44f6a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.237308 s, 44.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg213-client.virtnet /mnt/lustre (opts:) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff958e43e76800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:34:32 (1783136072) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:35:21 (1783136121) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:35:57 (1783136157) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 23:36:01 (1783136161) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.17835 s, 17.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.144935 s, 21.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5fbdb1b2 (nosync,stale) Mirror 2: 0x5fbdb1b2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6c9d8571d719f842afc484502936c785 - mirror 1 checksum: 66f64b11cd08db81e7b7dd7b3606a34a - mirror 2 checksum: 6c9d8571d719f842afc484502936c785 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-03T23:36:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-03T23:36:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6c9d8571d719f842afc484502936c785 - mirror 1 checksum: 66f64b11cd08db81e7b7dd7b3606a34a - mirror 2 checksum: 66f64b11cd08db81e7b7dd7b3606a34a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-03T23:36:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-03T23:36:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:36:13 (1783136173) Failing mds1 on oleg213-server Stopping /mnt/lustre-mds1 (opts:) on oleg213-server 23:36:19 (1783136179) shut down facet: mds1 facet_host: oleg213-server facet_failover_host: oleg213-server Failover mds1 to oleg213-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 23:36:42 (1783136202) targets are mounted 23:36:42 (1783136202) facet_failover done oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336059 s, 31.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:37:00 (1783136220) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:37:03 (1783136223) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:37:06 (1783136226) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0133451 s, 1.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:37:14 (1783136234) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:37:17 (1783136237) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:37:21 (1783136241) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:37:43 (1783136263) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:38:15 (1783136295) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:38:37 (1783136317) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:39:00 (1783136340) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 23:39:07 (1783136347) mirror create pid 53749 mirror split pid 53750 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53749 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53750 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 23:40:13 (1783136413) Seed set to 6415 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1783136415.994966 mapwrite 0x003a94 thru 0x010f61 (0xd4ce bytes) 000003 1783136416.089898 mapwrite 0x0051c4 thru 0x013d5e (0xeb9b bytes) truncating to largest ever: 0x2d034 000007 1783136416.167534 trunc from 0x013d5f to 0x02d033 (0x192d5 bytes) 000009 1783136416.187052 write 0x03e9eb thru 0x03fda6 (0x13bc bytes) 000011 1783136416.189705 mapwrite 0x03eb76 thru 0x03ffff (0x148a bytes) 000012 1783136416.249901 write 0x03e4c7 thru 0x03ffff (0x1b39 bytes) 000014 1783136416.256759 trunc from 0x040000 to 0x02bb1f (0x144e0 bytes) 000015 1783136416.277969 write 0x005f4b thru 0x00d046 (0x70fc bytes) 000016 1783136416.279224 mapread 0x02662e thru 0x02bb1f (0x54f2 bytes) 000019 1783136416.292204 read 0x000147 thru 0x00e285 (0xe13f bytes) 000021 1783136416.299945 write 0x032d6d thru 0x038ae9 (0x5d7d bytes) 000023 1783136416.307959 MIRROR_EXTEND to 2 mirrors 000024 1783136416.970169 mapwrite 0x000f5c thru 0x00e6ec (0xd791 bytes) 000026 1783136417.233555 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) 000028 1783136417.468781 write 0x01374e thru 0x01df06 (0xa7b9 bytes) 000029 1783136417.485101 mapwrite 0x015bef thru 0x01bfa2 (0x63b4 bytes) 000031 1783136417.535589 mapread 0x018ff9 thru 0x028994 (0xf99c bytes) 000033 1783136417.557905 mapwrite 0x01dc1b thru 0x02377f (0x5b65 bytes) 000034 1783136417.616124 read 0x014954 thru 0x017cab (0x3358 bytes) 000035 1783136417.616752 trunc from 0x038aea to 0x0192bd (0x1f82c bytes) 000037 1783136417.663983 write 0x00d1bf thru 0x013d8f (0x6bd1 bytes) 000039 1783136417.673373 mapwrite 0x0008aa thru 0x0021c2 (0x1919 bytes) 000040 1783136417.742053 mapread 0x001957 thru 0x001c83 (0x32d bytes) 000045 1783136417.748909 mapwrite 0x02d0e7 thru 0x035a61 (0x897b bytes) 000051 1783136417.834176 mapread 0x032735 thru 0x035a61 (0x332d bytes) 000052 1783136417.836112 write 0x00db86 thru 0x01a99c (0xce17 bytes) 000053 1783136417.844014 read 0x001b21 thru 0x00d19f (0xb67f bytes) 000055 1783136417.846367 write 0x010a3a thru 0x0171c8 (0x678f bytes) 000057 1783136417.849415 mapwrite 0x025406 thru 0x032855 (0xd450 bytes) 000058 1783136417.915980 trunc from 0x035a62 to 0x020c9b (0x14dc6 bytes) 000059 1783136417.932924 write 0x00421b thru 0x0136ab (0xf491 bytes) 000061 1783136417.940674 mapwrite 0x02f776 thru 0x035c16 (0x64a1 bytes) 000063 1783136418.053843 mapwrite 0x02ad62 thru 0x037d40 (0xcfdf bytes) 000064 1783136418.187118 mapwrite 0x0208ea thru 0x0237ba (0x2ed1 bytes) 000065 1783136418.265738 write 0x034b6e thru 0x03ffff (0xb492 bytes) 000067 1783136418.272023 mapread 0x003032 thru 0x00c0db (0x90aa bytes) 000073 1783136418.282399 write 0x03077d thru 0x03ffff (0xf883 bytes) 000076 1783136418.287586 mapread 0x00c75a thru 0x013dbb (0x7662 bytes) 000078 1783136418.292631 mapwrite 0x00e209 thru 0x01cd4c (0xeb44 bytes) 000079 1783136418.356775 read 0x004e73 thru 0x0096af (0x483d bytes) 000082 1783136418.360169 read 0x01fd12 thru 0x01fed6 (0x1c5 bytes) 000083 1783136418.366044 MIRROR_EXTEND to 2 mirrors 000087 1783136419.284143 read 0x0366fd thru 0x03ffff (0x9903 bytes) 000088 1783136419.293022 trunc from 0x040000 to 0x00d50c (0x32af3 bytes) 000091 1783136419.434863 mapread 0x00a895 thru 0x00d50c (0x2c78 bytes) 000092 1783136419.476040 write 0x000258 thru 0x004d2d (0x4ad6 bytes) truncating to largest ever: 0x2dd19 000093 1783136419.484787 trunc from 0x00d50d to 0x02dd18 (0x2080c bytes) 000094 1783136419.508383 read 0x01de4c thru 0x026cc6 (0x8e7b bytes) 000095 1783136419.528984 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dd19): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd19): Mirror 2: 0x7c5b1eb6 (stale) CRC-32 checksum value for chunk [0, 0x2dd19): Mirror 1: 0x2c880a4a Mirror 3: 0x2c880a4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd19: skipped truncating to largest ever: 0x3189e 000097 1783136420.505401 trunc from 0x02dd19 to 0x03189d (0x3b85 bytes) 000099 1783136420.686374 write 0x00a035 thru 0x00b1ba (0x1186 bytes) 000100 1783136420.703547 write 0x00b72a thru 0x013ab3 (0x838a bytes) 000101 1783136420.720514 trunc from 0x03189e to 0x00c8a7 (0x24ff6 bytes) 000102 1783136420.767009 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc8a8): Mirror 2: 0xa6077b1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc8a8): Mirror 2: 0xa6077b1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc8a8: skipped 000104 1783136421.119324 mapwrite 0x02eed0 thru 0x0303ec (0x151d bytes) 000105 1783136421.324824 mapread 0x001fdc thru 0x0022d3 (0x2f8 bytes) 000107 1783136421.336392 read 0x02f30f thru 0x0303ec (0x10de bytes) 000108 1783136421.354678 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x303ed): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303ed): Mirror 2: 0x3a783900 (stale) CRC-32 checksum value for chunk [0, 0x303ed): Mirror 1: 0x2a671d7d Mirror 3: 0x2a671d7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303ed: skipped 000115 1783136422.018349 trunc from 0x0303ed to 0x01e364 (0x12088 bytes) 000116 1783136422.073671 mapread 0x0175ad thru 0x01e364 (0x6db8 bytes) 000117 1783136422.093276 mapwrite 0x0242c1 thru 0x02b0f1 (0x6e31 bytes) 000121 1783136422.169958 mapwrite 0x03181c thru 0x03d41f (0xbc04 bytes) 000123 1783136422.230117 trunc from 0x03d420 to 0x01c717 (0x20d08 bytes) 000124 1783136422.249898 write 0x025d94 thru 0x028a7a (0x2ce7 bytes) 000125 1783136422.251239 write 0x02f3c0 thru 0x03091c (0x155d bytes) 000126 1783136422.252111 mapwrite 0x004f10 thru 0x0068e4 (0x19d5 bytes) 000128 1783136422.292175 mapwrite 0x021f13 thru 0x030191 (0xe27f bytes) 000129 1783136422.353769 mapwrite 0x000602 thru 0x00f7a4 (0xf1a3 bytes) 000131 1783136422.412037 trunc from 0x03091d to 0x0183f1 (0x1852b bytes) 000132 1783136422.431690 read 0x003bc0 thru 0x004e1e (0x125f bytes) 000135 1783136422.436204 trunc from 0x0183f2 to 0x02d44a (0x15059 bytes) 000136 1783136422.458050 write 0x00fb3f thru 0x01d90b (0xddcd bytes) 000137 1783136422.468554 MIRROR_RESYNC 3 mirrors 000138 1783136423.303857 write 0x03e4f5 thru 0x03ffff (0x1b0b bytes) 000139 1783136423.402110 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8c7e782d (stale) Mirror 3: 0x8c7e782d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x70e66b51 Mirror 4: 0x70e66b51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000140 1783136424.838140 mapwrite 0x002e40 thru 0x00448c (0x164d bytes) 000143 1783136425.350778 MIRROR_RESYNC 4 mirrors 000144 1783136426.862800 trunc from 0x040000 to 0x02ae34 (0x151cb bytes) 000146 1783136426.943703 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ae35): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ae35): Mirror 2: 0x27c13ceb (stale) Mirror 3: 0x27c13ceb (stale) Mirror 4: 0x27c13ceb (stale) CRC-32 checksum value for chunk [0, 0x2ae35): Mirror 1: 0x27c13ceb Mirror 5: 0x27c13ceb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ae35: skipped 000147 1783136427.683331 write 0x03b924 thru 0x03ffff (0x46dc bytes) 000149 1783136427.789230 mapread 0x03fdee thru 0x03ffff (0x212 bytes) 000150 1783136427.792193 read 0x01c284 thru 0x01d28a (0x1007 bytes) 000154 1783136427.800452 mapread 0x03e323 thru 0x03ffff (0x1cdd bytes) 000155 1783136427.801418 mapread 0x01df86 thru 0x02698c (0x8a07 bytes) 000156 1783136427.818476 mapread 0x02d9cb thru 0x03ba88 (0xe0be bytes) 000158 1783136427.839395 mapwrite 0x02c714 thru 0x02d6c8 (0xfb5 bytes) 000160 1783136427.961942 trunc from 0x040000 to 0x02ffc0 (0x1003f bytes) 000161 1783136428.000430 mapwrite 0x02475b thru 0x030afd (0xc3a3 bytes) 000163 1783136428.131461 trunc from 0x030afe to 0x02bc94 (0x4e69 bytes) 000164 1783136428.153449 write 0x005c61 thru 0x0103bc (0xa75c bytes) 000167 1783136428.177815 mapwrite 0x00913d thru 0x0096fd (0x5c1 bytes) 000169 1783136428.398485 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bc95): Mirror 3: 0xeabbc68f (stale) Mirror 4: 0xeabbc68f (stale) Mirror 5: 0xeabbc68f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc95): Mirror 3: 0xeabbc68f (stale) Mirror 4: 0xeabbc68f (stale) Mirror 5: 0xeabbc68f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc95: skipped 000171 1783136428.870311 read 0x0147bb thru 0x01cbba (0x8400 bytes) 000172 1783136428.877596 mapread 0x0275a3 thru 0x029439 (0x1e97 bytes) 000173 1783136428.894939 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2bc95): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc95): Mirror 3: 0xeabbc68f (stale) Mirror 4: 0xeabbc68f (stale) Mirror 5: 0xeabbc68f (stale) CRC-32 checksum value for chunk [0, 0x2bc95): Mirror 1: 0x3617e09c Mirror 6: 0x3617e09c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc95: skipped 000175 1783136429.675865 mapwrite 0x01f3ff thru 0x0255f1 (0x61f3 bytes) 000178 1783136429.901841 write 0x031676 thru 0x03ffff (0xe98a bytes) 000179 1783136429.903853 mapread 0x03c21d thru 0x03e1f2 (0x1fd6 bytes) 000182 1783136429.915537 write 0x03476a thru 0x037343 (0x2bda bytes) 000184 1783136429.928035 trunc from 0x040000 to 0x02e8a9 (0x11756 bytes) 000185 1783136429.970763 mapwrite 0x029e40 thru 0x03165a (0x781b bytes) 000186 1783136430.296926 mapwrite 0x00228c thru 0x011b25 (0xf89a bytes) 000188 1783136430.577551 MIRROR_RESYNC 5 mirrors truncating to largest ever: 0x3b791 000190 1783136431.952461 trunc from 0x03165b to 0x03b790 (0xa136 bytes) 000195 1783136432.039955 mapwrite 0x0203f0 thru 0x028d28 (0x8939 bytes) 000196 1783136432.118604 write 0x01f588 thru 0x0210ca (0x1b43 bytes) 000199 1783136432.120835 mapread 0x01730c thru 0x01a4ae (0x31a3 bytes) 000203 1783136432.138870 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b791): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b791): Mirror 3: 0x94e49b80 (stale) Mirror 4: 0x94e49b80 (stale) Mirror 5: 0x94e49b80 (stale) Mirror 6: 0x94e49b80 (stale) CRC-32 checksum value for chunk [0, 0x3b791): Mirror 1: 0x846994e6 Mirror 7: 0x846994e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b791: skipped 000207 1783136432.836766 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b791): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b791): Mirror 3: 0x94e49b80 (stale) Mirror 4: 0x94e49b80 (stale) Mirror 5: 0x94e49b80 (stale) Mirror 6: 0x94e49b80 (stale) CRC-32 checksum value for chunk [0, 0x3b791): Mirror 1: 0x846994e6 Mirror 7: 0x846994e6 Mirror 8: 0x846994e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b791: skipped 000208 1783136433.628041 trunc from 0x03b791 to 0x010c59 (0x2ab37 bytes) 000210 1783136433.727420 trunc from 0x010c5a to 0x0305b2 (0x1f959 bytes) 000211 1783136433.745851 read 0x0260fb thru 0x0305b2 (0xa4b8 bytes) 000214 1783136433.754735 mapread 0x00c3c6 thru 0x012142 (0x5d7d bytes) 000216 1783136433.776221 mapwrite 0x018852 thru 0x0280d3 (0xf882 bytes) 000218 1783136433.841406 write 0x036a16 thru 0x03ffff (0x95ea bytes) 000219 1783136433.843424 mapwrite 0x005562 thru 0x00abea (0x5689 bytes) 000220 1783136433.902426 mapread 0x035966 thru 0x03f719 (0x9db4 bytes) 000221 1783136433.910121 trunc from 0x040000 to 0x03499d (0xb662 bytes) 000223 1783136433.930170 write 0x02fc7d thru 0x031fc2 (0x2346 bytes) 000225 1783136433.932855 mapwrite 0x008385 thru 0x00a7c8 (0x2444 bytes) 000226 1783136433.977950 write 0x00697c thru 0x00cee9 (0x656e bytes) 000227 1783136433.979311 read 0x002978 thru 0x006a10 (0x4099 bytes) 000229 1783136433.981465 mapread 0x0054ea thru 0x00e294 (0x8dab bytes) truncating to largest ever: 0x3fed2 000232 1783136433.987505 trunc from 0x03499e to 0x03fed1 (0xb534 bytes) 000237 1783136434.008873 MIRROR_RESYNC 7 mirrors 000238 1783136435.397375 mapread 0x030106 thru 0x0308d1 (0x7cc bytes) 000239 1783136435.404136 read 0x02f417 thru 0x032fb4 (0x3b9e bytes) 000241 1783136435.428807 mapwrite 0x010e46 thru 0x0130be (0x2279 bytes) 000245 1783136435.748385 read 0x008888 thru 0x016387 (0xdb00 bytes) 000246 1783136435.758947 mapread 0x0365cb thru 0x036ea4 (0x8da bytes) 000248 1783136435.764684 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 3: 0x5f08deeb (stale) Mirror 4: 0x5f08deeb (stale) Mirror 5: 0x5f08deeb (stale) Mirror 6: 0x5f08deeb (stale) Mirror 7: 0x5f08deeb (stale) Mirror 8: 0x5f08deeb (stale) CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 1: 0x5343afd6 Mirror 9: 0x5343afd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fed2: skipped 000249 1783136436.788276 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 3: 0x5f08deeb (stale) Mirror 4: 0x5f08deeb (stale) Mirror 5: 0x5f08deeb (stale) Mirror 6: 0x5f08deeb (stale) Mirror 8: 0x5f08deeb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 3: 0x5f08deeb (stale) Mirror 4: 0x5f08deeb (stale) Mirror 5: 0x5f08deeb (stale) Mirror 6: 0x5f08deeb (stale) Mirror 8: 0x5f08deeb (stale) CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 1: 0x5343afd6 Mirror 9: 0x5343afd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fed2: skipped 000250 1783136437.815701 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 3: 0x5f08deeb (stale) Mirror 4: 0x5f08deeb (stale) Mirror 5: 0x5f08deeb (stale) Mirror 6: 0x5f08deeb (stale) Mirror 8: 0x5f08deeb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 3: 0x5f08deeb (stale) Mirror 4: 0x5f08deeb (stale) Mirror 5: 0x5f08deeb (stale) Mirror 6: 0x5f08deeb (stale) Mirror 8: 0x5f08deeb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fed2: skipped 000251 1783136438.499817 mapwrite 0x0376f2 thru 0x03ffff (0x890e bytes) 000252 1783136438.705903 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1df16c47 (stale) Mirror 4: 0x1df16c47 (stale) Mirror 5: 0x1df16c47 (stale) Mirror 6: 0x1df16c47 (stale) Mirror 8: 0x1df16c47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7a9c8ea0 Mirror 9: 0x7a9c8ea0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000253 1783136440.066956 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1df16c47 (stale) Mirror 4: 0x1df16c47 (stale) Mirror 5: 0x1df16c47 (stale) Mirror 6: 0x1df16c47 (stale) Mirror 8: 0x1df16c47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7a9c8ea0 Mirror 9: 0x7a9c8ea0 Mirror 10: 0x7a9c8ea0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1783136441.132911 write 0x010692 thru 0x01737d (0x6cec bytes) 000258 1783136441.258721 mapwrite 0x00b6a5 thru 0x00c22c (0xb88 bytes) 000259 1783136441.374739 mapwrite 0x00fd94 thru 0x01dfec (0xe259 bytes) 000260 1783136441.514693 write 0x02a9bd thru 0x02bddb (0x141f bytes) 000263 1783136441.538666 mapwrite 0x02951e thru 0x02f8dd (0x63c0 bytes) 000264 1783136441.693892 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1df16c47 (stale) Mirror 4: 0x1df16c47 (stale) Mirror 6: 0x1df16c47 (stale) Mirror 8: 0x1df16c47 (stale) Mirror 9: 0x7a9c8ea0 (stale) Mirror 10: 0x7a9c8ea0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1df16c47 (stale) Mirror 4: 0x1df16c47 (stale) Mirror 6: 0x1df16c47 (stale) Mirror 8: 0x1df16c47 (stale) Mirror 9: 0x7a9c8ea0 (stale) Mirror 10: 0x7a9c8ea0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000265 1783136442.402746 read 0x01594c thru 0x024c58 (0xf30d bytes) 000266 1783136442.425200 write 0x03cf09 thru 0x03ffff (0x30f7 bytes) 000267 1783136442.457487 read 0x03d7d3 thru 0x03ffff (0x282d bytes) 000268 1783136442.463762 MIRROR_RESYNC 7 mirrors 000270 1783136444.084129 trunc from 0x040000 to 0x02402a (0x1bfd5 bytes) 000272 1783136444.168725 read 0x019775 thru 0x02402a (0xa8b6 bytes) 000273 1783136444.178368 read 0x00d079 thru 0x01a440 (0xd3c8 bytes) 000278 1783136444.191034 write 0x023b59 thru 0x027185 (0x362d bytes) 000281 1783136444.198609 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27186): Mirror 3: 0x93428164 (stale) Mirror 4: 0x93428164 (stale) Mirror 8: 0x93428164 (stale) Mirror 9: 0x93428164 (stale) Mirror 10: 0x93428164 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27186): Mirror 3: 0x93428164 (stale) Mirror 4: 0x93428164 (stale) Mirror 8: 0x93428164 (stale) Mirror 9: 0x93428164 (stale) Mirror 10: 0x93428164 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27186: skipped 000284 1783136444.676957 mapwrite 0x028cc2 thru 0x02ed7e (0x60bd bytes) 000286 1783136444.901535 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 4: 0xb5154999 (stale) Mirror 8: 0xb5154999 (stale) Mirror 9: 0xb5154999 (stale) Mirror 10: 0xb5154999 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 4: 0xb5154999 (stale) Mirror 8: 0xb5154999 (stale) Mirror 9: 0xb5154999 (stale) Mirror 10: 0xb5154999 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed7f: skipped 000287 1783136445.408123 mapread 0x027f20 thru 0x02abb8 (0x2c99 bytes) 000290 1783136445.460492 MIRROR_RESYNC 5 mirrors 000292 1783136446.489383 mapwrite 0x018c21 thru 0x01902f (0x40f bytes) 000293 1783136446.694254 read 0x00dc98 thru 0x011514 (0x387d bytes) 000295 1783136446.704995 write 0x00d810 thru 0x00d822 (0x13 bytes) 000297 1783136446.717188 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 4: 0x2e0989a6 (stale) Mirror 8: 0x2e0989a6 (stale) Mirror 9: 0x2e0989a6 (stale) Mirror 10: 0x2e0989a6 (stale) CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 1: 0xc653d7b1 Mirror 11: 0xc653d7b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed7f: skipped 000298 1783136447.614581 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 4: 0x2e0989a6 (stale) Mirror 8: 0x2e0989a6 (stale) Mirror 9: 0x2e0989a6 (stale) Mirror 10: 0x2e0989a6 (stale) CRC-32 checksum value for chunk [0, 0x2ed7f): Mirror 1: 0xc653d7b1 Mirror 11: 0xc653d7b1 Mirror 12: 0xc653d7b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed7f: skipped 000300 1783136448.380481 MIRROR_RESYNC 7 mirrors 000301 1783136449.315031 MIRROR_EXTEND to 8 mirrors 000302 1783136449.912449 write 0x00c808 thru 0x017ff2 (0xb7eb bytes) 000303 1783136449.996374 MIRROR_RESYNC 8 mirrors 000306 1783136451.228769 write 0x03222f thru 0x0385e3 (0x63b5 bytes) 000307 1783136451.292972 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x385e4): Mirror 4: 0x89d19cea (stale) Mirror 8: 0x89d19cea (stale) Mirror 10: 0x89d19cea (stale) Mirror 11: 0x89d19cea (stale) Mirror 12: 0x89d19cea (stale) Mirror 13: 0x89d19cea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385e4): Mirror 4: 0x89d19cea (stale) Mirror 8: 0x89d19cea (stale) Mirror 10: 0x89d19cea (stale) Mirror 11: 0x89d19cea (stale) Mirror 12: 0x89d19cea (stale) Mirror 13: 0x89d19cea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385e4: skipped 000312 1783136451.844380 trunc from 0x0385e4 to 0x03d4ea (0x4f07 bytes) 000313 1783136451.862295 write 0x020eac thru 0x028149 (0x729e bytes) 000314 1783136451.879306 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d4eb): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4eb): Mirror 4: 0x8d79b34a (stale) Mirror 8: 0x8d79b34a (stale) Mirror 10: 0x8d79b34a (stale) Mirror 11: 0x8d79b34a (stale) Mirror 12: 0x8d79b34a (stale) Mirror 13: 0x8d79b34a (stale) CRC-32 checksum value for chunk [0, 0x3d4eb): Mirror 1: 0x493abfe9 Mirror 14: 0x493abfe9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4eb: skipped 000319 1783136452.993377 mapwrite 0x008269 thru 0x00ac55 (0x29ed bytes) 000320 1783136453.168984 read 0x032918 thru 0x03d4ea (0xabd3 bytes) 000322 1783136453.180603 MIRROR_RESYNC 8 mirrors 000324 1783136454.657566 trunc from 0x03d4eb to 0x036498 (0x7052 bytes) 000326 1783136454.715824 read 0x026cd6 thru 0x031649 (0xa974 bytes) 000329 1783136454.725274 mapwrite 0x0030e8 thru 0x005e3d (0x2d56 bytes) 000330 1783136454.786772 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x36499): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36499): Mirror 4: 0x625f211a (stale) Mirror 8: 0x625f211a (stale) Mirror 10: 0x625f211a (stale) Mirror 11: 0x625f211a (stale) Mirror 12: 0x625f211a (stale) Mirror 13: 0x625f211a (stale) Mirror 14: 0x625f211a (stale) CRC-32 checksum value for chunk [0, 0x36499): Mirror 1: 0xdb066db Mirror 15: 0xdb066db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36499: skipped 000332 1783136455.713631 read 0x0361ca thru 0x036498 (0x2cf bytes) 000333 1783136455.732139 mapwrite 0x00f3f4 thru 0x01b50f (0xc11c bytes) 000334 1783136456.022461 read 0x021a43 thru 0x02326d (0x182b bytes) 000335 1783136456.050342 MIRROR_RESYNC 9 mirrors 000336 1783136457.592916 mapread 0x00a8f1 thru 0x014604 (0x9d14 bytes) 000338 1783136457.615129 mapwrite 0x01d9a6 thru 0x01fbe3 (0x223e bytes) 000339 1783136457.767207 mapread 0x01e297 thru 0x0293cc (0xb136 bytes) 000340 1783136457.774939 trunc from 0x036499 to 0x02c996 (0x9b02 bytes) 000341 1783136457.815098 write 0x008356 thru 0x0115c1 (0x926c bytes) 000342 1783136457.838389 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c997): Mirror 4: 0x887a3ee1 (stale) Mirror 8: 0x887a3ee1 (stale) Mirror 11: 0x887a3ee1 (stale) Mirror 12: 0x887a3ee1 (stale) Mirror 13: 0x887a3ee1 (stale) Mirror 14: 0x887a3ee1 (stale) Mirror 15: 0x887a3ee1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c997): Mirror 4: 0x887a3ee1 (stale) Mirror 8: 0x887a3ee1 (stale) Mirror 11: 0x887a3ee1 (stale) Mirror 12: 0x887a3ee1 (stale) Mirror 13: 0x887a3ee1 (stale) Mirror 14: 0x887a3ee1 (stale) Mirror 15: 0x887a3ee1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c997: skipped 000346 1783136458.373264 mapwrite 0x03c6e5 thru 0x03ffff (0x391b bytes) 000349 1783136458.449355 MIRROR_RESYNC 8 mirrors 000351 1783136459.712013 write 0x005ec8 thru 0x011386 (0xb4bf bytes) 000352 1783136459.801996 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1024b9bf (stale) Mirror 8: 0x1024b9bf (stale) Mirror 11: 0x1024b9bf (stale) Mirror 12: 0x1024b9bf (stale) Mirror 13: 0x1024b9bf (stale) Mirror 14: 0x1024b9bf (stale) Mirror 15: 0x1024b9bf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf1e53a7f Mirror 16: 0xf1e53a7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000355 1783136460.731465 write 0x03b345 thru 0x03ffff (0x4cbb bytes) 000356 1783136460.797913 mapread 0x00b060 thru 0x00f740 (0x46e1 bytes) 000359 1783136460.817743 mapread 0x0014d0 thru 0x00dacb (0xc5fc bytes) 000360 1783136460.823440 trunc from 0x040000 to 0x00dab2 (0x3254d bytes) 000361 1783136460.838035 trunc from 0x00dab3 to 0x014526 (0x6a74 bytes) 000363 1783136460.854430 mapwrite 0x02967a thru 0x02a066 (0x9ed bytes) 000364 1783136460.986181 mapwrite 0x02481e thru 0x02f8af (0xb092 bytes) 000365 1783136461.073522 mapwrite 0x02b23a thru 0x02f0ab (0x3e72 bytes) 000366 1783136461.111787 write 0x00e116 thru 0x012c37 (0x4b22 bytes) 000367 1783136461.122641 read 0x00252f thru 0x00ab80 (0x8652 bytes) 000368 1783136461.123417 read 0x02d00a thru 0x02f8af (0x28a6 bytes) 000371 1783136461.126522 mapwrite 0x030bd7 thru 0x03ba05 (0xae2f bytes) 000374 1783136461.211394 MIRROR_RESYNC 9 mirrors 000376 1783136462.285816 mapread 0x01887a thru 0x025d19 (0xd4a0 bytes) 000377 1783136462.315383 read 0x005837 thru 0x00e30c (0x8ad6 bytes) 000379 1783136462.322973 read 0x0230c8 thru 0x027b3e (0x4a77 bytes) 000380 1783136462.326251 read 0x003602 thru 0x00b659 (0x8058 bytes) 000382 1783136462.336328 read 0x016f57 thru 0x01754b (0x5f5 bytes) 000385 1783136462.337900 mapwrite 0x0084ee thru 0x00bad9 (0x35ec bytes) 000386 1783136462.464551 mapread 0x02e9b6 thru 0x031252 (0x289d bytes) 000388 1783136462.467187 read 0x0072df thru 0x011bda (0xa8fc bytes) 000393 1783136462.469967 write 0x004e6e thru 0x008d44 (0x3ed7 bytes) 000395 1783136462.471606 write 0x018fa5 thru 0x0256b7 (0xc713 bytes) 000396 1783136462.473434 mapwrite 0x031f56 thru 0x034cdd (0x2d88 bytes) 000397 1783136462.516592 mapread 0x034f25 thru 0x03ba05 (0x6ae1 bytes) 000400 1783136462.519519 mapwrite 0x011acd thru 0x0179c7 (0x5efb bytes) 000401 1783136462.565337 read 0x014a2e thru 0x0246b0 (0xfc83 bytes) 000402 1783136462.569595 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba06): Mirror 4: 0x49517834 (stale) Mirror 11: 0x49517834 (stale) Mirror 12: 0x49517834 (stale) Mirror 13: 0x49517834 (stale) Mirror 14: 0x49517834 (stale) Mirror 15: 0x49517834 (stale) Mirror 16: 0x49517834 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba06): Mirror 4: 0x49517834 (stale) Mirror 11: 0x49517834 (stale) Mirror 12: 0x49517834 (stale) Mirror 13: 0x49517834 (stale) Mirror 14: 0x49517834 (stale) Mirror 15: 0x49517834 (stale) Mirror 16: 0x49517834 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba06: skipped 000405 1783136463.015563 mapread 0x00eeef thru 0x01b113 (0xc225 bytes) 000406 1783136463.062363 mapwrite 0x008c88 thru 0x009787 (0xb00 bytes) 000408 1783136463.112400 read 0x036132 thru 0x037f8e (0x1e5d bytes) 000409 1783136463.120420 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ba06): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba06): Mirror 4: 0x49517834 (stale) Mirror 11: 0x49517834 (stale) Mirror 12: 0x49517834 (stale) Mirror 13: 0x49517834 (stale) Mirror 14: 0x49517834 (stale) Mirror 15: 0x49517834 (stale) Mirror 16: 0x49517834 (stale) CRC-32 checksum value for chunk [0, 0x3ba06): Mirror 1: 0x4c835f77 Mirror 17: 0x4c835f77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba06: skipped 000411 1783136463.747985 write 0x02267a thru 0x029f81 (0x7908 bytes) 000413 1783136463.821943 mapwrite 0x034e56 thru 0x03ffff (0xb1aa bytes) 000415 1783136463.937950 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf1a3fbf7 (stale) Mirror 11: 0xf1a3fbf7 (stale) Mirror 12: 0xf1a3fbf7 (stale) Mirror 13: 0xf1a3fbf7 (stale) Mirror 14: 0xf1a3fbf7 (stale) Mirror 15: 0xf1a3fbf7 (stale) Mirror 17: 0xd964e1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf1a3fbf7 (stale) Mirror 11: 0xf1a3fbf7 (stale) Mirror 12: 0xf1a3fbf7 (stale) Mirror 13: 0xf1a3fbf7 (stale) Mirror 14: 0xf1a3fbf7 (stale) Mirror 15: 0xf1a3fbf7 (stale) Mirror 17: 0xd964e1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000416 1783136464.325403 mapread 0x033e60 thru 0x03adf3 (0x6f94 bytes) 000417 1783136464.348696 trunc from 0x040000 to 0x03e651 (0x19ae bytes) 000418 1783136464.379489 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e652): Mirror 11: 0xc6c6a467 (stale) Mirror 12: 0xc6c6a467 (stale) Mirror 13: 0xc6c6a467 (stale) Mirror 14: 0xc6c6a467 (stale) Mirror 15: 0xc6c6a467 (stale) Mirror 17: 0x3d4a0cfc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e652): Mirror 11: 0xc6c6a467 (stale) Mirror 12: 0xc6c6a467 (stale) Mirror 13: 0xc6c6a467 (stale) Mirror 14: 0xc6c6a467 (stale) Mirror 15: 0xc6c6a467 (stale) Mirror 17: 0x3d4a0cfc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e652: skipped 000421 1783136464.846014 write 0x007ebf thru 0x00c3ee (0x4530 bytes) 000423 1783136464.864181 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e652): Mirror 11: 0xc6c6a467 (stale) Mirror 12: 0xc6c6a467 (stale) Mirror 14: 0xc6c6a467 (stale) Mirror 15: 0xc6c6a467 (stale) Mirror 17: 0x3d4a0cfc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e652): Mirror 11: 0xc6c6a467 (stale) Mirror 12: 0xc6c6a467 (stale) Mirror 14: 0xc6c6a467 (stale) Mirror 15: 0xc6c6a467 (stale) Mirror 17: 0x3d4a0cfc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e652: skipped 000425 1783136465.276111 trunc from 0x03e652 to 0x036074 (0x85dd bytes) 000426 1783136465.299878 read 0x019ea4 thru 0x01af92 (0x10ef bytes) 000427 1783136465.304928 mapwrite 0x01c81b thru 0x026a6e (0xa254 bytes) 000430 1783136465.377948 read 0x026d81 thru 0x02b859 (0x4ad9 bytes) 000433 1783136465.387924 write 0x012882 thru 0x013a89 (0x1208 bytes) 000434 1783136465.389355 read 0x01bd5b thru 0x021b48 (0x5dee bytes) 000435 1783136465.390087 mapread 0x024bc0 thru 0x02dc8c (0x90cd bytes) 000436 1783136465.391361 trunc from 0x036075 to 0x00fdac (0x262c8 bytes) 000438 1783136465.433634 write 0x02fe76 thru 0x03964a (0x97d5 bytes) 000439 1783136465.437996 mapread 0x021bfe thru 0x029c4a (0x804d bytes) 000442 1783136465.456159 trunc from 0x03964b to 0x024c57 (0x149f3 bytes) 000443 1783136465.518342 read 0x0109e0 thru 0x01abed (0xa20e bytes) 000446 1783136465.524164 mapread 0x00d94a thru 0x01377f (0x5e36 bytes) 000447 1783136465.539932 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24c58): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c58): Mirror 11: 0x227cd867 (stale) Mirror 12: 0x227cd867 (stale) Mirror 14: 0x227cd867 (stale) Mirror 15: 0x227cd867 (stale) Mirror 17: 0xbc40bae9 (stale) CRC-32 checksum value for chunk [0, 0x24c58): Mirror 1: 0x373865a6 Mirror 18: 0x373865a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c58: skipped 000448 1783136466.414977 mapwrite 0x01340c thru 0x01f05e (0xbc53 bytes) 000449 1783136466.620038 write 0x007608 thru 0x015651 (0xe04a bytes) 000451 1783136466.627893 mapread 0x02296e thru 0x022c09 (0x29c bytes) 000452 1783136466.629372 trunc from 0x024c58 to 0x01e789 (0x64ce bytes) 000455 1783136466.662775 mapwrite 0x000f47 thru 0x00cd09 (0xbdc3 bytes) 000457 1783136466.742357 mapwrite 0x00ac50 thru 0x0116a2 (0x6a53 bytes) 000460 1783136466.788181 mapread 0x002cc5 thru 0x010d85 (0xe0c1 bytes) 000461 1783136466.789416 write 0x00e8ee thru 0x011fb7 (0x36ca bytes) 000462 1783136466.790669 trunc from 0x01e78a to 0x02d1cb (0xea42 bytes) 000464 1783136466.812860 mapwrite 0x0334fa thru 0x03d4f2 (0x9ff9 bytes) 000465 1783136466.892136 mapread 0x03aec8 thru 0x03d4f2 (0x262b bytes) 000467 1783136466.893817 write 0x00addb thru 0x01956a (0xe790 bytes) 000468 1783136466.896759 read 0x002be5 thru 0x00e957 (0xbd73 bytes) 000471 1783136466.899797 write 0x02b2b7 thru 0x03108e (0x5dd8 bytes) 000474 1783136466.907574 trunc from 0x03d4f3 to 0x02c914 (0x10bde bytes) 000477 1783136466.924544 read 0x00b19b thru 0x012618 (0x747e bytes) 000478 1783136466.931196 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c915): Mirror 11: 0x9cedff0 (stale) Mirror 12: 0x9cedff0 (stale) Mirror 14: 0x9cedff0 (stale) Mirror 17: 0xcd666928 (stale) Mirror 18: 0x157fc388 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c915): Mirror 11: 0x9cedff0 (stale) Mirror 12: 0x9cedff0 (stale) Mirror 14: 0x9cedff0 (stale) Mirror 17: 0xcd666928 (stale) Mirror 18: 0x157fc388 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c915: skipped 000480 1783136467.416094 write 0x031801 thru 0x039e1e (0x861e bytes) 000482 1783136467.420252 read 0x0117ab thru 0x014d7f (0x35d5 bytes) 000484 1783136467.437560 MIRROR_RESYNC 6 mirrors 000488 1783136468.430747 MIRROR_RESYNC 6 mirrors 000489 1783136468.837905 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) 000490 1783136469.229867 mapwrite 0x0331ab thru 0x03ffff (0xce55 bytes) 000495 1783136469.420801 read 0x0032a7 thru 0x0117cb (0xe525 bytes) 000498 1783136469.442313 trunc from 0x040000 to 0x038389 (0x7c76 bytes) 000501 1783136469.507554 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3838a): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3838a): Mirror 11: 0xe56a3e64 (stale) Mirror 12: 0xe56a3e64 (stale) Mirror 14: 0xe56a3e64 (stale) Mirror 18: 0xe56a3e64 (stale) CRC-32 checksum value for chunk [0, 0x3838a): Mirror 1: 0x3e0d0332 Mirror 19: 0x3e0d0332 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3838a: skipped 000503 1783136470.246546 trunc from 0x03838a to 0x011215 (0x27174 bytes) 000504 1783136470.371368 write 0x025391 thru 0x02bd10 (0x6980 bytes) 000505 1783136470.372352 mapread 0x00bac7 thru 0x010cc3 (0x51fd bytes) 000509 1783136470.403589 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2bd11): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd11): Mirror 11: 0x73cdd9fb (stale) Mirror 12: 0x73cdd9fb (stale) Mirror 14: 0x73cdd9fb (stale) Mirror 18: 0x73cdd9fb (stale) Mirror 19: 0x73cdd9fb (stale) CRC-32 checksum value for chunk [0, 0x2bd11): Mirror 1: 0x4a47f506 Mirror 20: 0x4a47f506 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd11: skipped 000514 1783136471.331780 mapread 0x0244db thru 0x02bd10 (0x7836 bytes) 000515 1783136471.360702 mapread 0x011013 thru 0x016d79 (0x5d67 bytes) 000516 1783136471.378216 write 0x00c213 thru 0x00d820 (0x160e bytes) 000518 1783136471.489644 write 0x01c3a9 thru 0x02af5f (0xebb7 bytes) 000519 1783136471.501562 write 0x02a92f thru 0x037892 (0xcf64 bytes) 000520 1783136471.504551 trunc from 0x037893 to 0x02bc46 (0xbc4c bytes) 000521 1783136471.558134 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bc47): Mirror 12: 0x33045cda (stale) Mirror 14: 0x33045cda (stale) Mirror 18: 0x33045cda (stale) Mirror 19: 0x33045cda (stale) Mirror 20: 0x556d80c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc47): Mirror 12: 0x33045cda (stale) Mirror 14: 0x33045cda (stale) Mirror 18: 0x33045cda (stale) Mirror 19: 0x33045cda (stale) Mirror 20: 0x556d80c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc47: skipped 000522 1783136472.008448 mapwrite 0x01299b thru 0x01c982 (0x9fe8 bytes) 000524 1783136472.077860 mapwrite 0x03baf1 thru 0x03ffff (0x450f bytes) 000526 1783136472.137265 write 0x02dc56 thru 0x038de6 (0xb191 bytes) 000533 1783136472.150317 trunc from 0x040000 to 0x011f6d (0x2e092 bytes) 000537 1783136472.173739 write 0x01b5ae thru 0x01c1d3 (0xc26 bytes) 000542 1783136472.178506 trunc from 0x01c1d4 to 0x009fa8 (0x1222b bytes) 000543 1783136472.192827 trunc from 0x009fa9 to 0x02451d (0x1a575 bytes) 000546 1783136472.211318 mapread 0x00ef5f thru 0x01c6df (0xd781 bytes) 000548 1783136472.228537 trunc from 0x02451e to 0x012f9c (0x11581 bytes) 000549 1783136472.244036 mapwrite 0x001080 thru 0x00e7b3 (0xd734 bytes) 000550 1783136472.305896 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12f9d): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f9d): Mirror 12: 0x26ed45ac (stale) Mirror 14: 0x26ed45ac (stale) Mirror 18: 0x26ed45ac (stale) Mirror 19: 0x26ed45ac (stale) Mirror 20: 0xa6119fa1 (stale) CRC-32 checksum value for chunk [0, 0x12f9d): Mirror 1: 0x75a41ef8 Mirror 21: 0x75a41ef8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f9d: skipped 000551 1783136473.026404 write 0x01b569 thru 0x020da5 (0x583d bytes) 000554 1783136473.102508 trunc from 0x020da6 to 0x03e396 (0x1d5f1 bytes) 000556 1783136473.178394 trunc from 0x03e397 to 0x02fc14 (0xe782 bytes) 000558 1783136473.206369 mapwrite 0x023b78 thru 0x028e1f (0x52a8 bytes) 000559 1783136473.296573 read 0x00240c thru 0x0086e0 (0x62d5 bytes) 000561 1783136473.306676 mapwrite 0x034a07 thru 0x039141 (0x473b bytes) 000562 1783136473.394765 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39142): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39142): Mirror 12: 0xd47051f9 (stale) Mirror 14: 0xd47051f9 (stale) Mirror 18: 0xd47051f9 (stale) Mirror 19: 0xa80dfb15 (stale) Mirror 20: 0x720b8809 (stale) Mirror 21: 0x619d27d0 (stale) CRC-32 checksum value for chunk [0, 0x39142): Mirror 1: 0x309d910e Mirror 22: 0x309d910e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39142: skipped 000563 1783136474.414673 write 0x0153bb thru 0x024e19 (0xfa5f bytes) 000565 1783136474.518953 MIRROR_RESYNC 8 mirrors 000567 1783136475.825448 mapwrite 0x026974 thru 0x034104 (0xd791 bytes) 000568 1783136476.008207 read 0x027027 thru 0x03543b (0xe415 bytes) 000571 1783136476.012726 mapread 0x012856 thru 0x0219f0 (0xf19b bytes) 000572 1783136476.042725 mapread 0x032259 thru 0x039141 (0x6ee9 bytes) 000577 1783136476.055592 mapread 0x01fd45 thru 0x02442d (0x46e9 bytes) 000578 1783136476.057724 mapread 0x017059 thru 0x023a50 (0xc9f8 bytes) 000582 1783136476.068227 write 0x015856 thru 0x023bbf (0xe36a bytes) 000584 1783136476.071740 write 0x02e197 thru 0x03a3fc (0xc266 bytes) 000587 1783136476.079585 mapwrite 0x000dbd thru 0x00111c (0x360 bytes) 000588 1783136476.170139 mapread 0x004263 thru 0x00cad0 (0x886e bytes) 000589 1783136476.172708 read 0x035ac9 thru 0x037f25 (0x245d bytes) 000590 1783136476.173845 trunc from 0x03a3fd to 0x0070d6 (0x33326 bytes) 000591 1783136476.201944 trunc from 0x0070d7 to 0x005b5d (0x1579 bytes) 000593 1783136476.220636 write 0x02ba24 thru 0x0382d3 (0xc8b0 bytes) 000594 1783136476.228817 MIRROR_RESYNC 8 mirrors 000595 1783136477.538548 mapwrite 0x026678 thru 0x02fc91 (0x961a bytes) 000597 1783136477.741530 mapread 0x019b89 thru 0x01f2db (0x5753 bytes) 000598 1783136477.748820 read 0x02484a thru 0x027a10 (0x31c7 bytes) 000601 1783136477.750821 read 0x00e508 thru 0x01d7f9 (0xf2f2 bytes) 000602 1783136477.761416 trunc from 0x0382d4 to 0x02ff65 (0x836e bytes) 000603 1783136477.785268 mapwrite 0x018cc8 thru 0x01c3b8 (0x36f1 bytes) 000604 1783136477.831878 write 0x0005d7 thru 0x0043bc (0x3de6 bytes) 000605 1783136477.833780 mapwrite 0x01bb43 thru 0x0216bb (0x5b79 bytes) 000606 1783136477.886051 mapwrite 0x034d21 thru 0x03a8cf (0x5baf bytes) 000607 1783136477.991027 mapread 0x0353f0 thru 0x03a8cf (0x54e0 bytes) 000609 1783136477.992759 write 0x012921 thru 0x0140cd (0x17ad bytes) 000610 1783136477.999791 read 0x027e4a thru 0x028adc (0xc93 bytes) 000611 1783136478.000924 read 0x0269f3 thru 0x029590 (0x2b9e bytes) 000612 1783136478.020886 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a8d0): Mirror 14: 0x25ee4bc6 (stale) Mirror 18: 0x25ee4bc6 (stale) Mirror 19: 0x25ee4bc6 (stale) Mirror 20: 0x25ee4bc6 (stale) Mirror 21: 0x25ee4bc6 (stale) Mirror 22: 0x25ee4bc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8d0): Mirror 14: 0x25ee4bc6 (stale) Mirror 18: 0x25ee4bc6 (stale) Mirror 19: 0x25ee4bc6 (stale) Mirror 20: 0x25ee4bc6 (stale) Mirror 21: 0x25ee4bc6 (stale) Mirror 22: 0x25ee4bc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8d0: skipped 000613 1783136478.717351 trunc from 0x03a8d0 to 0x03fd1e (0x544f bytes) 000617 1783136478.737424 mapread 0x03b16b thru 0x03fd1e (0x4bb4 bytes) 000618 1783136478.758253 mapwrite 0x02bab3 thru 0x02d921 (0x1e6f bytes) 000620 1783136478.808151 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd1f): Mirror 14: 0x6c84b425 (stale) Mirror 19: 0x6c84b425 (stale) Mirror 20: 0x6c84b425 (stale) Mirror 21: 0x6c84b425 (stale) Mirror 22: 0x6c84b425 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd1f): Mirror 14: 0x6c84b425 (stale) Mirror 19: 0x6c84b425 (stale) Mirror 20: 0x6c84b425 (stale) Mirror 21: 0x6c84b425 (stale) Mirror 22: 0x6c84b425 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd1f: skipped 000621 1783136479.320798 mapwrite 0x02cf3f thru 0x03557c (0x863e bytes) 000622 1783136479.444164 read 0x004939 thru 0x0068e5 (0x1fad bytes) 000623 1783136479.453249 write 0x02b20a thru 0x033181 (0x7f78 bytes) 000627 1783136479.463528 mapread 0x01c222 thru 0x02b7c6 (0xf5a5 bytes) 000630 1783136479.482909 write 0x022419 thru 0x0268a5 (0x448d bytes) 000634 1783136479.487245 write 0x03de83 thru 0x03ffff (0x217d bytes) 000636 1783136479.494045 mapread 0x012be1 thru 0x0174d5 (0x48f5 bytes) 000640 1783136479.524202 mapread 0x01f43a thru 0x028f91 (0x9b58 bytes) 000642 1783136479.528655 mapread 0x02aada thru 0x02b8ea (0xe11 bytes) 000643 1783136479.533785 mapread 0x009220 thru 0x012934 (0x9715 bytes) 000644 1783136479.548036 mapread 0x001dc2 thru 0x00d4c3 (0xb702 bytes) 000645 1783136479.555222 write 0x00a4a7 thru 0x016096 (0xbbf0 bytes) 000648 1783136479.560856 read 0x0082e1 thru 0x00e2c5 (0x5fe5 bytes) 000649 1783136479.561673 write 0x00eda2 thru 0x0107f8 (0x1a57 bytes) 000650 1783136479.573577 write 0x0112fa thru 0x01f036 (0xdd3d bytes) 000652 1783136479.581559 mapread 0x00c63d thru 0x01024e (0x3c12 bytes) 000653 1783136479.603286 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x177a5132 (stale) Mirror 20: 0x177a5132 (stale) Mirror 21: 0x177a5132 (stale) Mirror 22: 0x177a5132 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x177a5132 (stale) Mirror 20: 0x177a5132 (stale) Mirror 21: 0x177a5132 (stale) Mirror 22: 0x177a5132 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000658 1783136480.028341 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x177a5132 (stale) Mirror 20: 0x177a5132 (stale) Mirror 21: 0x177a5132 (stale) Mirror 22: 0x177a5132 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2b4e3564 Mirror 23: 0x2b4e3564 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1783136480.985533 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x177a5132 (stale) Mirror 20: 0x177a5132 (stale) Mirror 21: 0x177a5132 (stale) Mirror 22: 0x177a5132 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x177a5132 (stale) Mirror 20: 0x177a5132 (stale) Mirror 21: 0x177a5132 (stale) Mirror 22: 0x177a5132 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000661 1783136481.323174 MIRROR_RESYNC 5 mirrors 000662 1783136482.374569 read 0x0131c8 thru 0x01f1d0 (0xc009 bytes) 000664 1783136482.392755 write 0x003bcc thru 0x010bef (0xd024 bytes) 000665 1783136482.490540 mapwrite 0x016866 thru 0x02293b (0xc0d6 bytes) 000666 1783136482.625124 mapwrite 0x03400c thru 0x03811f (0x4114 bytes) 000667 1783136482.719693 write 0x030b0e thru 0x039986 (0x8e79 bytes) 000669 1783136482.723166 trunc from 0x040000 to 0x03e10d (0x1ef2 bytes) 000671 1783136482.762289 mapwrite 0x01c69b thru 0x024982 (0x82e8 bytes) 000675 1783136482.849560 mapread 0x03d27d thru 0x03e10d (0xe91 bytes) 000676 1783136482.856797 trunc from 0x03e10e to 0x02e468 (0xfca5 bytes) 000678 1783136482.880010 write 0x018fd1 thru 0x026ed1 (0xdf01 bytes) 000679 1783136482.885911 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2e469): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e469): Mirror 19: 0xf5a2cf68 (stale) Mirror 20: 0xf5a2cf68 (stale) Mirror 21: 0xf5a2cf68 (stale) Mirror 22: 0xf5a2cf68 (stale) CRC-32 checksum value for chunk [0, 0x2e469): Mirror 1: 0x3e525f65 Mirror 23: 0x3e525f65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e469: skipped 000681 1783136483.755264 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e469): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e469): Mirror 19: 0xf5a2cf68 (stale) Mirror 20: 0xf5a2cf68 (stale) Mirror 21: 0xf5a2cf68 (stale) Mirror 22: 0xf5a2cf68 (stale) CRC-32 checksum value for chunk [0, 0x2e469): Mirror 1: 0x3e525f65 Mirror 23: 0x3e525f65 Mirror 24: 0x3e525f65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e469: skipped 000683 1783136484.729437 read 0x017b6f thru 0x01c48e (0x4920 bytes) 000685 1783136484.787008 mapwrite 0x010686 thru 0x01ae90 (0xa80b bytes) 000687 1783136485.084563 write 0x03c62d thru 0x03ffff (0x39d3 bytes) 000688 1783136485.085835 mapwrite 0x03c3a8 thru 0x03ffff (0x3c58 bytes) 000689 1783136485.165580 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x2b4e3564 (stale) Mirror 20: 0x2b4e3564 (stale) Mirror 21: 0x2b4e3564 (stale) Mirror 22: 0x2b4e3564 (stale) Mirror 23: 0x961778b (stale) Mirror 24: 0x961778b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1bcfd8d Mirror 25: 0x1bcfd8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000690 1783136485.921989 read 0x015d0c thru 0x01d9a1 (0x7c96 bytes) 000692 1783136485.944980 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x2b4e3564 (stale) Mirror 20: 0x2b4e3564 (stale) Mirror 22: 0x2b4e3564 (stale) Mirror 23: 0x961778b (stale) Mirror 24: 0x961778b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x2b4e3564 (stale) Mirror 20: 0x2b4e3564 (stale) Mirror 22: 0x2b4e3564 (stale) Mirror 23: 0x961778b (stale) Mirror 24: 0x961778b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1bcfd8d Mirror 25: 0x1bcfd8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1783136486.528197 read 0x02fd47 thru 0x032e5d (0x3117 bytes) 000695 1783136486.552014 MIRROR_RESYNC 7 mirrors 000696 1783136487.504953 read 0x00a623 thru 0x00bfd3 (0x19b1 bytes) 000697 1783136487.509410 read 0x03cd17 thru 0x03ffff (0x32e9 bytes) 000698 1783136487.522525 MIRROR_RESYNC 7 mirrors 000699 1783136487.968587 mapwrite 0x025425 thru 0x028fb1 (0x3b8d bytes) 000700 1783136488.106456 mapread 0x03b40a thru 0x03ffff (0x4bf6 bytes) 000701 1783136488.115885 trunc from 0x040000 to 0x034e3e (0xb1c1 bytes) 000706 1783136488.160485 mapwrite 0x002e82 thru 0x00bc10 (0x8d8f bytes) 000709 1783136488.218540 trunc from 0x034e3f to 0x01a90e (0x1a530 bytes) 000710 1783136488.232564 mapread 0x005f08 thru 0x013f30 (0xe029 bytes) 000711 1783136488.240525 mapread 0x008f99 thru 0x00b4d9 (0x2541 bytes) 000712 1783136488.242287 mapwrite 0x00389c thru 0x00c5da (0x8d3f bytes) 000713 1783136488.288269 mapread 0x01a5f2 thru 0x01a90e (0x31d bytes) 000714 1783136488.293941 write 0x0014e2 thru 0x002675 (0x1194 bytes) 000718 1783136488.301842 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a90f): Mirror 19: 0x6fd7cac4 (stale) Mirror 20: 0x6fd7cac4 (stale) Mirror 23: 0x6fd7cac4 (stale) Mirror 24: 0x6fd7cac4 (stale) Mirror 25: 0x6fd7cac4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a90f): Mirror 19: 0x6fd7cac4 (stale) Mirror 20: 0x6fd7cac4 (stale) Mirror 23: 0x6fd7cac4 (stale) Mirror 24: 0x6fd7cac4 (stale) Mirror 25: 0x6fd7cac4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a90f: skipped 000719 1783136488.611500 mapwrite 0x0217ac thru 0x02ccd1 (0xb526 bytes) 000720 1783136488.746487 trunc from 0x02ccd2 to 0x0264ff (0x67d2 bytes) 000721 1783136488.788495 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26500): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26500): Mirror 19: 0x897050bc (stale) Mirror 20: 0x897050bc (stale) Mirror 23: 0x897050bc (stale) Mirror 24: 0x897050bc (stale) Mirror 25: 0x897050bc (stale) CRC-32 checksum value for chunk [0, 0x26500): Mirror 1: 0x8e79a66e Mirror 26: 0x8e79a66e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26500: skipped 000724 1783136489.672045 write 0x02b56a thru 0x03a8d8 (0xf36f bytes) 000725 1783136489.748036 trunc from 0x03a8d9 to 0x00d43c (0x2d49c bytes) 000727 1783136489.779140 read 0x00cb4c thru 0x00d43c (0x8f1 bytes) 000729 1783136489.790957 write 0x02e533 thru 0x03c95f (0xe42d bytes) 000731 1783136489.798024 trunc from 0x03c960 to 0x038f57 (0x3a08 bytes) 000732 1783136489.823451 trunc from 0x038f58 to 0x029442 (0xfb15 bytes) 000733 1783136489.840101 write 0x03f1d2 thru 0x03ffff (0xe2e bytes) 000734 1783136489.841217 read 0x02371c thru 0x02a3a9 (0x6c8e bytes) 000738 1783136489.858341 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x1bcfd8d (stale) Mirror 20: 0x1bcfd8d (stale) Mirror 23: 0x1bcfd8d (stale) Mirror 24: 0x1bcfd8d (stale) Mirror 25: 0x1bcfd8d (stale) Mirror 26: 0x299c955 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9168eaeb Mirror 27: 0x9168eaeb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000739 1783136490.707462 write 0x03996c thru 0x03ffff (0x6694 bytes) 000741 1783136490.780275 mapread 0x00f268 thru 0x01c3eb (0xd184 bytes) 000743 1783136490.807314 write 0x0015d5 thru 0x006e3d (0x5869 bytes) 000744 1783136490.819327 trunc from 0x040000 to 0x0044f2 (0x3bb0d bytes) 000745 1783136490.838163 mapwrite 0x004f44 thru 0x00ddb8 (0x8e75 bytes) 000747 1783136490.923930 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xddb9): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xddb9): Mirror 19: 0xf13fc9c4 (stale) Mirror 20: 0xf13fc9c4 (stale) Mirror 23: 0xf13fc9c4 (stale) Mirror 24: 0xf13fc9c4 (stale) Mirror 25: 0xf13fc9c4 (stale) Mirror 26: 0xbc7497d6 (stale) Mirror 27: 0x28499e0e (stale) CRC-32 checksum value for chunk [0, 0xddb9): Mirror 1: 0x26f83264 Mirror 28: 0x26f83264 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xddb9: skipped 000749 1783136491.756961 read 0x008de4 thru 0x00ddb8 (0x4fd5 bytes) 000750 1783136491.774403 mapwrite 0x0072bc thru 0x0118be (0xa603 bytes) 000751 1783136491.951470 write 0x0179eb thru 0x01caee (0x5104 bytes) 000754 1783136491.962516 mapwrite 0x01a26f thru 0x021a66 (0x77f8 bytes) 000755 1783136492.066928 mapwrite 0x00b4d8 thru 0x00eecf (0x39f8 bytes) 000759 1783136492.113177 read 0x0182f5 thru 0x01d6ab (0x53b7 bytes) 000764 1783136492.122731 read 0x01fcf2 thru 0x0215c0 (0x18cf bytes) 000766 1783136492.124499 mapwrite 0x026e68 thru 0x035daa (0xef43 bytes) 000768 1783136492.208921 write 0x02fa48 thru 0x03f656 (0xfc0f bytes) 000769 1783136492.212735 trunc from 0x03f657 to 0x02d03c (0x1261a bytes) 000771 1783136492.230769 mapwrite 0x02f565 thru 0x03b6bc (0xc158 bytes) 000772 1783136492.347896 write 0x0123b9 thru 0x0180e2 (0x5d2a bytes) 000773 1783136492.349174 write 0x02a38d thru 0x02e73d (0x43b1 bytes) 000774 1783136492.362886 mapread 0x01446d thru 0x017ecb (0x3a5f bytes) 000775 1783136492.364108 mapread 0x005c2a thru 0x0061d5 (0x5ac bytes) 000777 1783136492.365748 trunc from 0x03b6bd to 0x0385a8 (0x3114 bytes) 000778 1783136492.386790 trunc from 0x0385a9 to 0x03daf6 (0x554e bytes) 000780 1783136492.408283 trunc from 0x03daf7 to 0x03a18d (0x3969 bytes) 000781 1783136492.430582 mapwrite 0x0268d3 thru 0x027619 (0xd47 bytes) 000782 1783136492.470253 write 0x03a627 thru 0x03e2b0 (0x3c8a bytes) 000783 1783136492.478896 write 0x02ce1f thru 0x039a64 (0xcc46 bytes) 000784 1783136492.485497 mapwrite 0x0031fc thru 0x00855a (0x535f bytes) 000787 1783136492.537402 trunc from 0x03e2b1 to 0x012bfa (0x2b6b6 bytes) 000788 1783136492.558757 read 0x002805 thru 0x0069be (0x41ba bytes) 000790 1783136492.561359 trunc from 0x012bfb to 0x02e953 (0x1bd59 bytes) 000791 1783136492.600658 trunc from 0x02e954 to 0x01f23c (0xf717 bytes) 000792 1783136492.620539 trunc from 0x01f23d to 0x02fa32 (0x107f6 bytes) 000794 1783136492.636369 read 0x0209c2 thru 0x02cd32 (0xc371 bytes) 000796 1783136492.648945 write 0x0017e3 thru 0x007b0c (0x632a bytes) 000797 1783136492.654162 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2fa33): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa33): Mirror 19: 0x6e1b6361 (stale) Mirror 20: 0x6e1b6361 (stale) Mirror 23: 0x6e1b6361 (stale) Mirror 24: 0x6e1b6361 (stale) Mirror 25: 0x6e1b6361 (stale) Mirror 26: 0x170b2afd (stale) Mirror 27: 0x4f3b03fc (stale) Mirror 28: 0x3c84c4c8 (stale) CRC-32 checksum value for chunk [0, 0x2fa33): Mirror 1: 0x95699e1b Mirror 29: 0x95699e1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa33: skipped 000798 1783136493.406783 write 0x00fd80 thru 0x017144 (0x73c5 bytes) 000799 1783136493.499468 mapwrite 0x02a9ba thru 0x032be0 (0x8227 bytes) 000800 1783136493.602468 mapread 0x02a554 thru 0x02abc4 (0x671 bytes) 000802 1783136493.604103 mapwrite 0x00b208 thru 0x01162b (0x6424 bytes) 000803 1783136493.655932 read 0x022a36 thru 0x02b960 (0x8f2b bytes) 000804 1783136493.661455 write 0x031550 thru 0x03b076 (0x9b27 bytes) 000805 1783136493.663530 write 0x01c11e thru 0x029005 (0xcee8 bytes) 000807 1783136493.669150 trunc from 0x03b077 to 0x009107 (0x31f6f bytes) 000808 1783136493.681057 read 0x008cf5 thru 0x009107 (0x413 bytes) 000809 1783136493.682010 read 0x007fce thru 0x009107 (0x113a bytes) 000810 1783136493.682897 mapread 0x005d7e thru 0x008550 (0x27d3 bytes) 000812 1783136493.684806 read 0x004dd0 thru 0x009107 (0x4338 bytes) 000813 1783136493.689170 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x9108): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9108): Mirror 19: 0x65687111 (stale) Mirror 20: 0x65687111 (stale) Mirror 23: 0x65687111 (stale) Mirror 24: 0x65687111 (stale) Mirror 25: 0x65687111 (stale) Mirror 26: 0x69777d2e (stale) Mirror 27: 0x69777d2e (stale) Mirror 28: 0xcc3eedaa (stale) Mirror 29: 0x6deeea73 (stale) CRC-32 checksum value for chunk [0, 0x9108): Mirror 1: 0x6deeea73 Mirror 30: 0x6deeea73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9108: skipped 000815 1783136494.458057 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x9108): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9108): Mirror 19: 0x65687111 (stale) Mirror 20: 0x65687111 (stale) Mirror 23: 0x65687111 (stale) Mirror 24: 0x65687111 (stale) Mirror 25: 0x65687111 (stale) Mirror 26: 0x69777d2e (stale) Mirror 27: 0x69777d2e (stale) Mirror 28: 0xcc3eedaa (stale) Mirror 29: 0x6deeea73 (stale) CRC-32 checksum value for chunk [0, 0x9108): Mirror 1: 0x6deeea73 Mirror 30: 0x6deeea73 Mirror 31: 0x6deeea73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9108: skipped 000816 1783136495.319654 mapwrite 0x00aa62 thru 0x00d221 (0x27c0 bytes) 000817 1783136495.471051 trunc from 0x00d222 to 0x02c047 (0x1ee26 bytes) 000818 1783136495.515983 read 0x021c15 thru 0x02a8d0 (0x8cbc bytes) 000820 1783136495.547730 MIRROR_RESYNC 12 mirrors 000822 1783136497.313028 read 0x01a189 thru 0x01d192 (0x300a bytes) 000824 1783136497.327775 MIRROR_RESYNC 12 mirrors 000827 1783136497.991944 write 0x01b57e thru 0x01ea93 (0x3516 bytes) 000828 1783136498.098057 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2c048): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c048): Mirror 19: 0xb33c2d8a (stale) Mirror 20: 0xb33c2d8a (stale) Mirror 23: 0xb33c2d8a (stale) Mirror 24: 0xb33c2d8a (stale) Mirror 25: 0xb33c2d8a (stale) Mirror 26: 0xb33c2d8a (stale) Mirror 27: 0xb33c2d8a (stale) Mirror 28: 0xb33c2d8a (stale) Mirror 29: 0xb33c2d8a (stale) Mirror 30: 0xb33c2d8a (stale) Mirror 31: 0xb33c2d8a (stale) CRC-32 checksum value for chunk [0, 0x2c048): Mirror 1: 0x2e7da2f5 Mirror 32: 0x2e7da2f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c048: skipped 000830 1783136499.981256 trunc from 0x02c048 to 0x004b3e (0x27509 bytes) 000833 1783136500.100807 read 0x001230 thru 0x001bff (0x9d0 bytes) 000834 1783136500.108357 trunc from 0x004b3f to 0x027a03 (0x22ec5 bytes) 000835 1783136500.154422 write 0x00313b thru 0x01307f (0xff45 bytes) 000836 1783136500.204477 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a04): Mirror 19: 0x874e3528 (stale) Mirror 20: 0x874e3528 (stale) Mirror 23: 0x874e3528 (stale) Mirror 24: 0x874e3528 (stale) Mirror 25: 0x874e3528 (stale) Mirror 26: 0x874e3528 (stale) Mirror 27: 0x874e3528 (stale) Mirror 28: 0x874e3528 (stale) Mirror 29: 0x874e3528 (stale) Mirror 30: 0x874e3528 (stale) Mirror 32: 0x8da67b17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a04): Mirror 19: 0x874e3528 (stale) Mirror 20: 0x874e3528 (stale) Mirror 23: 0x874e3528 (stale) Mirror 24: 0x874e3528 (stale) Mirror 25: 0x874e3528 (stale) Mirror 26: 0x874e3528 (stale) Mirror 27: 0x874e3528 (stale) Mirror 28: 0x874e3528 (stale) Mirror 29: 0x874e3528 (stale) Mirror 30: 0x874e3528 (stale) Mirror 32: 0x8da67b17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a04: skipped 000839 1783136501.258327 MIRROR_RESYNC 12 mirrors 000840 1783136502.878238 mapwrite 0x034b84 thru 0x03bec3 (0x7340 bytes) 000842 1783136503.053960 mapwrite 0x0321ce thru 0x037b0e (0x5941 bytes) 000843 1783136503.104687 trunc from 0x03bec4 to 0x01e1a2 (0x1dd21 bytes) 000844 1783136503.133240 read 0x0132fe thru 0x015410 (0x2113 bytes) 000845 1783136503.140417 mapread 0x00a020 thru 0x012ef1 (0x8ed2 bytes) 000846 1783136503.157291 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1e1a3): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e1a3): Mirror 19: 0xccddfb37 (stale) Mirror 20: 0xccddfb37 (stale) Mirror 23: 0xccddfb37 (stale) Mirror 24: 0xccddfb37 (stale) Mirror 25: 0xccddfb37 (stale) Mirror 26: 0xccddfb37 (stale) Mirror 27: 0xccddfb37 (stale) Mirror 28: 0xccddfb37 (stale) Mirror 29: 0xccddfb37 (stale) Mirror 30: 0xccddfb37 (stale) Mirror 32: 0xccddfb37 (stale) CRC-32 checksum value for chunk [0, 0x1e1a3): Mirror 1: 0xccddfb37 Mirror 33: 0xccddfb37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e1a3: skipped 000847 1783136504.115650 read 0x00d41d thru 0x018575 (0xb159 bytes) 000848 1783136504.137862 mapread 0x0073c0 thru 0x00a970 (0x35b1 bytes) 000850 1783136504.212881 trunc from 0x01e1a3 to 0x03d70a (0x1f568 bytes) 000853 1783136504.440794 mapwrite 0x009888 thru 0x01353d (0x9cb6 bytes) 000854 1783136504.568296 trunc from 0x03d70b to 0x0145ca (0x29140 bytes) 000856 1783136504.603902 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x145cb): Mirror 19: 0x326b91e (stale) Mirror 20: 0x326b91e (stale) Mirror 23: 0x326b91e (stale) Mirror 24: 0x326b91e (stale) Mirror 25: 0x326b91e (stale) Mirror 26: 0x326b91e (stale) Mirror 27: 0x326b91e (stale) Mirror 29: 0x326b91e (stale) Mirror 30: 0x326b91e (stale) Mirror 32: 0x326b91e (stale) Mirror 33: 0x326b91e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x145cb): Mirror 19: 0x326b91e (stale) Mirror 20: 0x326b91e (stale) Mirror 23: 0x326b91e (stale) Mirror 24: 0x326b91e (stale) Mirror 25: 0x326b91e (stale) Mirror 26: 0x326b91e (stale) Mirror 27: 0x326b91e (stale) Mirror 29: 0x326b91e (stale) Mirror 30: 0x326b91e (stale) Mirror 32: 0x326b91e (stale) Mirror 33: 0x326b91e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x145cb: skipped 000857 1783136505.457960 write 0x002aa0 thru 0x004659 (0x1bba bytes) 000858 1783136505.468887 mapwrite 0x00cbb8 thru 0x00e39b (0x17e4 bytes) 000860 1783136505.530679 read 0x0061fa thru 0x014202 (0xe009 bytes) 000861 1783136505.537839 mapwrite 0x02c691 thru 0x03b03c (0xe9ac bytes) 000862 1783136505.652357 mapwrite 0x00eb6f thru 0x00eba2 (0x34 bytes) 000863 1783136505.698051 read 0x02b5fc thru 0x0341ec (0x8bf1 bytes) 000864 1783136505.700470 mapread 0x00a8c9 thru 0x00b471 (0xba9 bytes) 000865 1783136505.702474 mapwrite 0x022342 thru 0x02d1e5 (0xaea4 bytes) 000866 1783136505.830266 read 0x02693e thru 0x02dba8 (0x726b bytes) 000868 1783136505.844128 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b03d): Mirror 19: 0x16e5abe (stale) Mirror 20: 0x16e5abe (stale) Mirror 23: 0x16e5abe (stale) Mirror 24: 0x16e5abe (stale) Mirror 25: 0x16e5abe (stale) Mirror 26: 0x16e5abe (stale) Mirror 27: 0x16e5abe (stale) Mirror 29: 0x16e5abe (stale) Mirror 32: 0x16e5abe (stale) Mirror 33: 0x16e5abe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b03d): Mirror 19: 0x16e5abe (stale) Mirror 20: 0x16e5abe (stale) Mirror 23: 0x16e5abe (stale) Mirror 24: 0x16e5abe (stale) Mirror 25: 0x16e5abe (stale) Mirror 26: 0x16e5abe (stale) Mirror 27: 0x16e5abe (stale) Mirror 29: 0x16e5abe (stale) Mirror 32: 0x16e5abe (stale) Mirror 33: 0x16e5abe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b03d: skipped 000869 1783136506.587459 write 0x038a85 thru 0x03ffff (0x757b bytes) 000870 1783136506.595501 mapread 0x02bef7 thru 0x02fd76 (0x3e80 bytes) 000871 1783136506.614956 trunc from 0x040000 to 0x010d8b (0x2f274 bytes) 000872 1783136506.637816 mapwrite 0x00b989 thru 0x01173b (0x5db3 bytes) 000876 1783136506.718666 write 0x00b759 thru 0x012e45 (0x76ed bytes) 000877 1783136506.720411 mapwrite 0x025197 thru 0x026a09 (0x1873 bytes) 000879 1783136506.796348 mapread 0x023188 thru 0x026a09 (0x3882 bytes) 000881 1783136506.801566 mapwrite 0x03a022 thru 0x03ffff (0x5fde bytes) 000882 1783136506.921809 mapread 0x0220c1 thru 0x02836f (0x62af bytes) 000883 1783136506.936491 write 0x03a7ef thru 0x03ce6a (0x267c bytes) 000889 1783136506.945126 mapread 0x00c172 thru 0x011617 (0x54a6 bytes) 000891 1783136506.947250 write 0x015f53 thru 0x0220fe (0xc1ac bytes) 000894 1783136506.964468 trunc from 0x040000 to 0x016896 (0x29769 bytes) 000895 1783136506.994295 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x16897): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16897): Mirror 19: 0x78fb678a (stale) Mirror 20: 0x78fb678a (stale) Mirror 23: 0x78fb678a (stale) Mirror 24: 0x78fb678a (stale) Mirror 25: 0x78fb678a (stale) Mirror 26: 0x78fb678a (stale) Mirror 27: 0x78fb678a (stale) Mirror 29: 0x78fb678a (stale) Mirror 32: 0x78fb678a (stale) Mirror 33: 0x78fb678a (stale) CRC-32 checksum value for chunk [0, 0x16897): Mirror 1: 0xd2192d1c Mirror 34: 0xd2192d1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16897: skipped 000896 1783136507.904666 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16897): Mirror 19: 0x78fb678a (stale) Mirror 20: 0x78fb678a (stale) Mirror 23: 0x78fb678a (stale) Mirror 24: 0x78fb678a (stale) Mirror 26: 0x78fb678a (stale) Mirror 27: 0x78fb678a (stale) Mirror 29: 0x78fb678a (stale) Mirror 32: 0x78fb678a (stale) Mirror 33: 0x78fb678a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16897): Mirror 19: 0x78fb678a (stale) Mirror 20: 0x78fb678a (stale) Mirror 23: 0x78fb678a (stale) Mirror 24: 0x78fb678a (stale) Mirror 26: 0x78fb678a (stale) Mirror 27: 0x78fb678a (stale) Mirror 29: 0x78fb678a (stale) Mirror 32: 0x78fb678a (stale) Mirror 33: 0x78fb678a (stale) CRC-32 checksum value for chunk [0, 0x16897): Mirror 1: 0xd2192d1c Mirror 34: 0xd2192d1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16897: skipped 000897 1783136508.749716 write 0x034b4b thru 0x037334 (0x27ea bytes) 000899 1783136508.854798 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x37335): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37335): Mirror 19: 0x5de8c56a (stale) Mirror 20: 0x5de8c56a (stale) Mirror 23: 0x5de8c56a (stale) Mirror 24: 0x5de8c56a (stale) Mirror 26: 0x5de8c56a (stale) Mirror 27: 0x5de8c56a (stale) Mirror 29: 0x5de8c56a (stale) Mirror 32: 0x5de8c56a (stale) Mirror 33: 0x5de8c56a (stale) Mirror 34: 0xf3b26885 (stale) CRC-32 checksum value for chunk [0, 0x37335): Mirror 1: 0xe854803f Mirror 35: 0xe854803f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37335: skipped 000901 1783136509.996448 mapread 0x005331 thru 0x007fb6 (0x2c86 bytes) 000902 1783136510.026432 trunc from 0x037335 to 0x03e788 (0x7454 bytes) 000906 1783136510.123559 write 0x02ac89 thru 0x03ab33 (0xfeab bytes) 000907 1783136510.137913 mapwrite 0x017717 thru 0x021e3c (0xa726 bytes) 000909 1783136510.212921 write 0x02dd63 thru 0x03c798 (0xea36 bytes) 000912 1783136510.220675 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3e789): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e789): Mirror 19: 0x5cff6e50 (stale) Mirror 20: 0x5cff6e50 (stale) Mirror 23: 0x5cff6e50 (stale) Mirror 24: 0x5cff6e50 (stale) Mirror 26: 0x5cff6e50 (stale) Mirror 27: 0x5cff6e50 (stale) Mirror 29: 0x5cff6e50 (stale) Mirror 32: 0x5cff6e50 (stale) Mirror 33: 0x5cff6e50 (stale) Mirror 34: 0x64816917 (stale) Mirror 35: 0x8813ae5e (stale) CRC-32 checksum value for chunk [0, 0x3e789): Mirror 1: 0x4c8557d5 Mirror 36: 0x4c8557d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e789: skipped 000913 1783136510.983352 mapwrite 0x00d2a0 thru 0x00ddd8 (0xb39 bytes) 000916 1783136511.312019 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e789): Mirror 19: 0x5cff6e50 (stale) Mirror 20: 0x5cff6e50 (stale) Mirror 23: 0x5cff6e50 (stale) Mirror 24: 0x5cff6e50 (stale) Mirror 26: 0x5cff6e50 (stale) Mirror 29: 0x5cff6e50 (stale) Mirror 32: 0x5cff6e50 (stale) Mirror 33: 0x5cff6e50 (stale) Mirror 34: 0x64816917 (stale) Mirror 35: 0x8813ae5e (stale) Mirror 36: 0x4c8557d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e789): Mirror 19: 0x5cff6e50 (stale) Mirror 20: 0x5cff6e50 (stale) Mirror 23: 0x5cff6e50 (stale) Mirror 24: 0x5cff6e50 (stale) Mirror 26: 0x5cff6e50 (stale) Mirror 29: 0x5cff6e50 (stale) Mirror 32: 0x5cff6e50 (stale) Mirror 33: 0x5cff6e50 (stale) Mirror 34: 0x64816917 (stale) Mirror 35: 0x8813ae5e (stale) Mirror 36: 0x4c8557d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e789: skipped 000917 1783136512.062406 read 0x00d32a thru 0x0197a2 (0xc479 bytes) 000919 1783136512.071564 write 0x010818 thru 0x015330 (0x4b19 bytes) 000920 1783136512.072807 mapwrite 0x01c860 thru 0x027001 (0xa7a2 bytes) 000922 1783136512.176111 trunc from 0x03e789 to 0x03b8b2 (0x2ed6 bytes) 000928 1783136512.224543 read 0x034a3e thru 0x03b8b2 (0x6e75 bytes) 000929 1783136512.236370 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8b3): Mirror 20: 0xe333364 (stale) Mirror 23: 0xe333364 (stale) Mirror 24: 0xe333364 (stale) Mirror 26: 0xe333364 (stale) Mirror 29: 0xe333364 (stale) Mirror 32: 0xe333364 (stale) Mirror 33: 0xe333364 (stale) Mirror 34: 0x72a1d59e (stale) Mirror 35: 0x827b5353 (stale) Mirror 36: 0x7e1299fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8b3): Mirror 20: 0xe333364 (stale) Mirror 23: 0xe333364 (stale) Mirror 24: 0xe333364 (stale) Mirror 26: 0xe333364 (stale) Mirror 29: 0xe333364 (stale) Mirror 32: 0xe333364 (stale) Mirror 33: 0xe333364 (stale) Mirror 34: 0x72a1d59e (stale) Mirror 35: 0x827b5353 (stale) Mirror 36: 0x7e1299fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8b3: skipped 000930 1783136513.005891 read 0x028fbf thru 0x0333aa (0xa3ec bytes) 000934 1783136513.022553 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8b3): Mirror 20: 0xe333364 (stale) Mirror 23: 0xe333364 (stale) Mirror 24: 0xe333364 (stale) Mirror 26: 0xe333364 (stale) Mirror 32: 0xe333364 (stale) Mirror 33: 0xe333364 (stale) Mirror 34: 0x72a1d59e (stale) Mirror 35: 0x827b5353 (stale) Mirror 36: 0x7e1299fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8b3): Mirror 20: 0xe333364 (stale) Mirror 23: 0xe333364 (stale) Mirror 24: 0xe333364 (stale) Mirror 26: 0xe333364 (stale) Mirror 32: 0xe333364 (stale) Mirror 33: 0xe333364 (stale) Mirror 34: 0x72a1d59e (stale) Mirror 35: 0x827b5353 (stale) Mirror 36: 0x7e1299fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8b3: skipped 000936 1783136513.562757 mapwrite 0x03b905 thru 0x03ffff (0x46fb bytes) 000937 1783136513.734447 read 0x0304c2 thru 0x036f5a (0x6a99 bytes) 000938 1783136513.762863 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x25936a4a (stale) Mirror 23: 0x25936a4a (stale) Mirror 24: 0x25936a4a (stale) Mirror 26: 0x25936a4a (stale) Mirror 32: 0x25936a4a (stale) Mirror 33: 0x25936a4a (stale) Mirror 34: 0x7068fd71 (stale) Mirror 36: 0xea4682d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x25936a4a (stale) Mirror 23: 0x25936a4a (stale) Mirror 24: 0x25936a4a (stale) Mirror 26: 0x25936a4a (stale) Mirror 32: 0x25936a4a (stale) Mirror 33: 0x25936a4a (stale) Mirror 34: 0x7068fd71 (stale) Mirror 36: 0xea4682d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000942 1783136514.418371 mapwrite 0x01d524 thru 0x024361 (0x6e3e bytes) 000946 1783136514.495363 read 0x039c64 thru 0x03ffff (0x639c bytes) 000949 1783136514.507109 mapread 0x0138a5 thru 0x016bbd (0x3319 bytes) 000950 1783136514.515700 read 0x02a08b thru 0x030453 (0x63c9 bytes) 000953 1783136514.536975 mapread 0x02d8e3 thru 0x02f35f (0x1a7d bytes) 000954 1783136514.538179 read 0x012c7b thru 0x019927 (0x6cad bytes) 000955 1783136514.548029 MIRROR_RESYNC 9 mirrors 000956 1783136516.267921 mapread 0x02b2c2 thru 0x038cef (0xda2e bytes) 000957 1783136516.286307 read 0x031def thru 0x03b7bf (0x99d1 bytes) 000958 1783136516.286740 read 0x00ed5c thru 0x01aba8 (0xbe4d bytes) 000959 1783136516.298734 read 0x02f87c thru 0x03169b (0x1e20 bytes) 000960 1783136516.299468 trunc from 0x040000 to 0x03569b (0xa964 bytes) 000961 1783136516.425112 mapwrite 0x00b777 thru 0x019076 (0xd900 bytes) 000963 1783136516.547422 mapwrite 0x00f442 thru 0x012e12 (0x39d1 bytes) 000964 1783136516.624562 mapread 0x0197e2 thru 0x01f683 (0x5ea2 bytes) 000968 1783136516.635857 write 0x03e03a thru 0x03ffff (0x1fc6 bytes) 000969 1783136516.640133 mapread 0x027996 thru 0x02c015 (0x4680 bytes) 000970 1783136516.641903 read 0x00e4f6 thru 0x010c14 (0x271f bytes) 000972 1783136516.645508 mapwrite 0x0266b4 thru 0x027a39 (0x1386 bytes) 000973 1783136516.715559 read 0x01d776 thru 0x0246bb (0x6f46 bytes) 000975 1783136516.721941 write 0x024b0c thru 0x02e162 (0x9657 bytes) 000978 1783136516.724533 mapwrite 0x03dd50 thru 0x03ffff (0x22b0 bytes) 000979 1783136516.797445 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x64318de4 (stale) Mirror 24: 0x64318de4 (stale) Mirror 26: 0x64318de4 (stale) Mirror 32: 0x64318de4 (stale) Mirror 33: 0x64318de4 (stale) Mirror 34: 0x64318de4 (stale) Mirror 36: 0x64318de4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x64318de4 (stale) Mirror 24: 0x64318de4 (stale) Mirror 26: 0x64318de4 (stale) Mirror 32: 0x64318de4 (stale) Mirror 33: 0x64318de4 (stale) Mirror 34: 0x64318de4 (stale) Mirror 36: 0x64318de4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000980 1783136517.512789 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x64318de4 (stale) Mirror 26: 0x64318de4 (stale) Mirror 32: 0x64318de4 (stale) Mirror 33: 0x64318de4 (stale) Mirror 34: 0x64318de4 (stale) Mirror 36: 0x64318de4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x64318de4 (stale) Mirror 26: 0x64318de4 (stale) Mirror 32: 0x64318de4 (stale) Mirror 33: 0x64318de4 (stale) Mirror 34: 0x64318de4 (stale) Mirror 36: 0x64318de4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000981 1783136517.941699 mapwrite 0x0091e3 thru 0x017da0 (0xebbe bytes) 000982 1783136518.025456 mapread 0x006066 thru 0x0135d3 (0xd56e bytes) 000983 1783136518.032908 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x64318de4 (stale) Mirror 26: 0x64318de4 (stale) Mirror 32: 0x64318de4 (stale) Mirror 33: 0x64318de4 (stale) Mirror 34: 0x64318de4 (stale) Mirror 36: 0x64318de4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8e003210 Mirror 37: 0x8e003210 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000984 1783136519.028296 mapwrite 0x0058fe thru 0x00aa2f (0x5132 bytes) 000988 1783136519.313512 trunc from 0x040000 to 0x017f18 (0x280e7 bytes) 000991 1783136519.334030 read 0x016c47 thru 0x017f18 (0x12d2 bytes) 000994 1783136519.339846 mapread 0x00e4bc thru 0x01427b (0x5dc0 bytes) 000995 1783136519.355615 mapread 0x0022c2 thru 0x00eb76 (0xc8b5 bytes) 000996 1783136519.356901 mapread 0x012aac thru 0x013f3e (0x1493 bytes) 001000 1783136519.360919 write 0x03d89f thru 0x03ffff (0x2761 bytes) All operations completed A-OK! PASS 70a (111s) == sanity-flr test 71: check mirror extend parent fid ==== 23:42:04 (1783136524) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 23:42:07 (1783136527) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.356281 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146397 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 23:42:16 (1783136536) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4038304 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1904320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6490944 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4424064 .. Extending file size to 5350464 .. Extending file size to 8174592 .. Extending file size to 8408864 .. Extending file size to 8102656 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8250240 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8431936 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8968736 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9139904 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56950 56952 56954 56955 56956 done 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) 192.168.202.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:43:41 (1783136621) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8112928 .. Extending file size to 2620544 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8261344 .. Extending file size to 8709248 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8287104 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8697344 .. Waiting 58428 58429 58430 58433 58434 done 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) 192.168.202.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:45:01 (1783136701) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113601 s, 18.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.282013 s, 7.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xebc80ad8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:45:20 (1783136720) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109478 s, 19.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 23:45:27 (1783136727) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x200:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:45:34 (1783136734) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202528 s, 10.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:45:42 (1783136742) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.326251 s, 6.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:45:52 (1783136752) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15401 s, 13.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:46:27 (1783136787) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117661 s, 8.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:47:30 (1783136850) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.470649 s, 4.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:48:28 (1783136908) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg213-server mds-ost sync done. Creating new pool test_204f oleg213-server: Pool lustre.test_204f created Adding targets to pool oleg213-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg213-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.508986 s, 4.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg213-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg213-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg213-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:49:45 (1783136985) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:49:55 (1783136995) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:50:05 (1783137005) oleg213-server: Pool lustre.test_206 created Adding targets to pool oleg213-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg213-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg213-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg213-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg213-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:50:30 (1783137030) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915617 s, 11.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739303 s, 14.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x605ea085 Mirror 3: 0x605ea085 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:50:42 (1783137042) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:50:46 (1783137046) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:50:50 (1783137050) Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.164553 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2650000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2650000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2650000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a50000 Read this (4 bytes): BBBB 192.168.202.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg213-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:51:33 (1783137093) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00975048 s, 840 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:51:40 (1783137100) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601152 s, 17.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x225 lmm_fid: [0x200000404:0x225:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x225 lmm_fid: [0x200000404:0x225:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834469 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-03 23:51:41.000000000 -0400 Modify: 2026-07-03 23:51:41.000000000 -0400 Change: 2026-07-03 23:51:43.000000000 -0400 Birth: 2026-07-03 23:51:41.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:51:48 (1783137108) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:51:52 (1783137112) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.594231 s, 68.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.130843 s, 31.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 3 23:51 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 3 23:51 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:52:01 (1783137121) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:52:08 (1783137128) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1757 sec =========== 23:52:16 (1783137136) === sanity-flr: start cleanup 23:52:19 (1783137139) === === sanity-flr: finish cleanup 23:52:24 (1783137144) ===