-----============= acceptance-small: sanity-flr ============----- Thu Apr 9 15:28:24 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg210-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:28:45 (1775762925) === oleg210-client.virtnet: executing check_config_client /mnt/lustre oleg210-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg210-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f149003d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f149003d000.idle_timeout=debug disable quota as required oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:29:25 (1775762965) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8196] == sanity-flr test 0a: lfs mirror create with -N option == 15:29:27 (1775762967) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:29:43 (1775762983) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:29:47 (1775762987) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:29:51 (1775762991) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:30:17 (1775763017) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:30:21 (1775763021) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:30:25 (1775763025) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f149003d000.stats=clear osc.lustre-OST0001-osc-ffff9f149003d000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:30:33 (1775763033) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:30:44 (1775763044) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:30:52 (1775763052) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:30:55 (1775763055) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 15:31:05 (1775763065) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:31:12 (1775763072) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:31:20 (1775763080) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:31:29 (1775763089) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:31:41 (1775763101) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:31:48 (1775763108) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.38289 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989239 s, 10.6 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:31:57 (1775763117) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0715774 s, 43.9 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:32:06 (1775763126) Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:33:23 (1775763203) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.215559 s, 19.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:34:07 (1775763247) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:35:30 (1775763330) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.7987 s, 18.7 MB/s -rw-r--r-- 1 root root 52428800 Apr 9 15:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.282s user 0m0.000s sys 0m0.192s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f149003d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.306s user 0m0.000s sys 0m0.180s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.009s user 0m0.001s sys 0m0.183s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f149003d000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:37:10 (1775763430) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:37:14 (1775763434) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:37:18 (1775763438) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 15:37:22 (1775763442) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 15:37:30 (1775763450) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.805137 s, 15.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:37:45 (1775763465) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406224 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624323 s, 16.8 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:37:54 (1775763474) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:38:03 (1775763483) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0014884 s, 688 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147851 s, 693 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000670718 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000947085 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 15:38:14 (1775763494) create test files with size 15740013 .. ac637c8f09a3c8af5d9c85bb2112a97d - 940b3e221a81f4449790fcd85c775971 - df8ff187f2c6c08b9ac32e6c20ba9738 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 15:38:43 (1775763523) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:38:59 (1775763539) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:39:07 (1775763547) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709586 s, 14.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184515 s, 56.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 15:39:17 (1775763557) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0391172 s, 53.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.282943 s, 14.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150685 s, 13.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 15:39:34 (1775763574) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 15:39:39 (1775763579) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:39:43 (1775763583) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393974 s, 26.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:39:51 (1775763591) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.21551 s, 14.6 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:40:03 (1775763603) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:40:16 (1775763616) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.508638 s, 20.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:40:34 (1775763634) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f14875cb800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.369 s, 28.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9f14902b3800.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:40:54 (1775763654) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (58s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:41:51 (1775763711) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (48s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:42:39 (1775763759) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:42:43 (1775763763) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.286155 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.246354 s, 12.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5bc58440 (stale) Mirror 2: 0x5bc58440 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 22c010de0246c5e859354ed45c412830 - mirror 1 checksum: 600612c5fbcbf410434bdcdbb557bc2e - mirror 2 checksum: 22c010de0246c5e859354ed45c412830 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 9 15:42:47 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 9 15:42:47 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 22c010de0246c5e859354ed45c412830 - mirror 1 checksum: 600612c5fbcbf410434bdcdbb557bc2e - mirror 2 checksum: 600612c5fbcbf410434bdcdbb557bc2e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 9 15:42:47 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 9 15:42:47 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 48 (14s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:42:57 (1775763777) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555419 s, 18.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:43:07 (1775763787) Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 15:43:22 (1775763802) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 15:43:58 (1775763838) targets are mounted 15:43:58 (1775763838) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356158 s, 29.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 50A (70s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:44:17 (1775763857) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0805568 s, 254 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0409824 s, 500 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (16s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:44:33 (1775763873) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.116806 s, 175 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0153355 s, 1.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (14s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:44:47 (1775763887) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0209008 s, 784 kB/s PASS 50c (13s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:45:00 (1775763900) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0643125 s, 318 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (12s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:45:13 (1775763913) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0286443 s, 143 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0448071 s, 91.4 kB/s PASS 60a (13s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:45:25 (1775763925) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:45:52 (1775763952) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:46:24 (1775763984) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:46:51 (1775764011) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:47:18 (1775764038) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 15:47:28 (1775764048) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55950 mirror split pid 55951 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55950 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55951 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (72s) == sanity-flr test 70a: flr mode fsx test ================ 15:48:40 (1775764120) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4123 skipping zero size read skipping zero length punch hole 000003 1775764124.291977 write 0x02ea66 thru 0x036979 (0x7f14 bytes) 000005 1775764124.300312 punch from 0x02e90e to 0x036979 (0x806c bytes) truncating to largest ever: 0x243ef 000006 1775764124.334430 trunc from 0x03697a to 0x0243ee (0x1258b bytes) 000008 1775764124.375607 fallocate 0x0145fb thru 0x023352 (0xed58 bytes) 000012 1775764124.389967 mapread 0x000abf thru 0x00c17d (0xb6bf bytes) 000013 1775764124.444445 mapwrite 0x0067b7 thru 0x00839e (0x1be8 bytes) 000015 1775764124.473175 fallocate 0x012a83 thru 0x019a10 (0x6f8e bytes) 000016 1775764124.483686 punch from 0x015df3 to 0x0217f2 (0xba00 bytes) 000017 1775764124.491496 punch from 0x015a74 to 0x0215ee (0xbb7b bytes) 000019 1775764124.504724 mapread 0x012f39 thru 0x02247e (0xf546 bytes) 000020 1775764124.543438 trunc from 0x006110 to 0x0098db (0x37cc bytes) 000021 1775764124.553548 punch from 0x015d38 to 0x0225b4 (0xc87d bytes) 000022 1775764124.560186 write 0x016a09 thru 0x01c365 (0x595d bytes) 000023 1775764124.579175 write 0x00eeb9 thru 0x01e6d8 (0xf820 bytes) 000024 1775764124.591368 MIRROR_EXTEND to 2 mirrors 000025 1775764125.444122 fallocate 0x014f96 thru 0x01c138 (0x71a3 bytes) 000026 1775764125.497111 fallocate 0x01c1f8 thru 0x0213d2 (0x51db bytes) 000027 1775764125.502984 write 0x01b69b thru 0x026de5 (0xb74b bytes) 000028 1775764125.515361 mapread 0x004c0e thru 0x0051ad (0x5a0 bytes) 000029 1775764125.541252 trunc from 0x026de6 to 0x0096e7 (0x1d6fe bytes) 000030 1775764125.572064 mapwrite 0x01a7a7 thru 0x021a61 (0x72bb bytes) 000032 1775764125.630640 MIRROR_RESYNC 2 mirrors 000033 1775764126.247589 read 0x00046d thru 0x005017 (0x4bab bytes) 000034 1775764126.259191 punch from 0x021933 to 0x021a61 (0x12f bytes) 000035 1775764126.332844 trunc from 0x00358e to 0x008585 (0x4ff8 bytes) 000037 1775764126.340647 trunc from 0x018d2a to 0x01f532 (0x6809 bytes) 000038 1775764126.348363 mapread 0x004829 thru 0x013f5e (0xf736 bytes) truncating to largest ever: 0x33053 000039 1775764126.407062 trunc from 0x021a62 to 0x033052 (0x115f1 bytes) 000040 1775764126.466847 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) 000042 1775764126.649521 write 0x0288c0 thru 0x0380a5 (0xf7e6 bytes) 000043 1775764126.663094 read 0x03412c thru 0x0380a5 (0x3f7a bytes) 000045 1775764126.664770 write 0x033174 thru 0x03b2e6 (0x8173 bytes) 000047 1775764126.668208 mapwrite 0x00e237 thru 0x00fed5 (0x1c9f bytes) 000048 1775764126.708341 mapwrite 0x01580e thru 0x0197b8 (0x3fab bytes) 000049 1775764126.736317 write 0x031796 thru 0x035c40 (0x44ab bytes) 000050 1775764126.738422 trunc from 0x03b2e7 to 0x018e31 (0x224b5 bytes) 000051 1775764126.758208 trunc from 0x003b52 to 0x00c8bd (0x8d6c bytes) 000053 1775764126.766329 read 0x00283b thru 0x009d72 (0x7538 bytes) 000054 1775764126.774195 mapwrite 0x0022bb thru 0x00c350 (0xa096 bytes) 000055 1775764126.803923 write 0x00f7ed thru 0x012e5a (0x366e bytes) 000057 1775764126.805728 read 0x0006d7 thru 0x00e29c (0xdbc6 bytes) fallocating to largest ever: 0x34882 000058 1775764126.806428 fallocate 0x02aad0 thru 0x034881 (0x9db2 bytes) 000059 1775764126.814395 trunc from 0x0152af to 0x018e31 (0x3b83 bytes) 000060 1775764126.821200 fallocate 0x00be91 thru 0x01bce8 (0xfe58 bytes) 000061 1775764126.827363 trunc from 0x00d1bf to 0x018b46 (0xb988 bytes) 000062 1775764126.847636 MIRROR_EXTEND to 2 mirrors 000063 1775764127.987653 punch from 0x00aed1 to 0x015920 (0xaa50 bytes) 000065 1775764128.035350 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1bce9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bce9): Mirror 2: 0x735120b4 (stale) CRC-32 checksum value for chunk [0, 0x1bce9): Mirror 1: 0x91955e65 Mirror 3: 0x91955e65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bce9: skipped 000066 1775764129.710717 punch from 0x01868c to 0x01bce8 (0x365d bytes) 000067 1775764129.785984 trunc from 0x00df0c to 0x01bce8 (0xdddd bytes) 000068 1775764129.800055 trunc from 0x01bce9 to 0x033646 (0x1795e bytes) 000069 1775764129.849341 MIRROR_RESYNC 3 mirrors 000070 1775764130.873193 trunc from 0x004565 to 0x00b1c0 (0x6c5c bytes) 000072 1775764130.924506 MIRROR_RESYNC 3 mirrors 000073 1775764132.220176 punch from 0x02006f to 0x022d13 (0x2ca5 bytes) 000074 1775764132.263636 write 0x022554 thru 0x02d4be (0xaf6b bytes) 000075 1775764132.281953 write 0x02f6cc thru 0x034176 (0x4aab bytes) 000076 1775764132.309237 read 0x014998 thru 0x01b184 (0x67ed bytes) 000077 1775764132.334628 write 0x016800 thru 0x0253e6 (0xebe7 bytes) 000078 1775764132.342251 punch from 0x020278 to 0x024493 (0x421c bytes) 000079 1775764132.381398 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34177): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34177): Mirror 2: 0xd1e33d9 (stale) Mirror 3: 0xd1e33d9 (stale) CRC-32 checksum value for chunk [0, 0x34177): Mirror 1: 0xb95566ea Mirror 4: 0xb95566ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34177: skipped 000080 1775764133.893879 trunc from 0x00c8c0 to 0x01bb48 (0xf289 bytes) 000081 1775764134.028298 punch from 0x014881 to 0x01abd5 (0x6355 bytes) 000082 1775764134.059783 punch from 0x010d14 to 0x012af7 (0x1de4 bytes) fallocating to largest ever: 0x3e53d 000083 1775764134.078357 fallocate 0x03e04c thru 0x03e53c (0x4f1 bytes) 000085 1775764134.129627 trunc from 0x00fad6 to 0x012870 (0x2d9b bytes) 000086 1775764134.146051 trunc from 0x034177 to 0x00efed (0x25189 bytes) 000087 1775764134.215348 mapwrite 0x00d98e thru 0x01b1b0 (0xd823 bytes) 000088 1775764134.477332 mapread 0x005881 thru 0x0143d5 (0xeb55 bytes) 000089 1775764134.497594 read 0x001cfa thru 0x00c937 (0xac3e bytes) 000090 1775764134.510065 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b1b1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b1b1): Mirror 2: 0xb4329c97 (stale) Mirror 3: 0xb4329c97 (stale) Mirror 4: 0x9d170c15 (stale) CRC-32 checksum value for chunk [0, 0x1b1b1): Mirror 1: 0x3801a42a Mirror 5: 0x3801a42a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b1b1: skipped 000091 1775764136.233004 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b1b1): Mirror 2: 0xb4329c97 (stale) Mirror 3: 0xb4329c97 (stale) Mirror 4: 0x9d170c15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b1b1): Mirror 2: 0xb4329c97 (stale) Mirror 3: 0xb4329c97 (stale) Mirror 4: 0x9d170c15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b1b1: skipped 000092 1775764137.128795 write 0x00ddc2 thru 0x01bbcd (0xde0c bytes) 000094 1775764137.170540 mapwrite 0x012c87 thru 0x01cdff (0xa179 bytes) 000095 1775764137.262111 fallocate 0x032188 thru 0x034610 (0x2489 bytes) 000096 1775764137.272273 punch from 0x0259e4 to 0x031c16 (0xc233 bytes) 000097 1775764137.279599 punch from 0x02f77e to 0x03216d (0x29f0 bytes) 000099 1775764137.287137 trunc from 0x034611 to 0x026e0e (0xd802 bytes) 000100 1775764137.311967 mapwrite 0x016d0b thru 0x024a33 (0xdd29 bytes) 000101 1775764137.351201 fallocate 0x033c77 thru 0x0352fa (0x1684 bytes) 000102 1775764137.358087 mapwrite 0x025edb thru 0x032a9b (0xcbc1 bytes) 000103 1775764137.414399 MIRROR_RESYNC 4 mirrors fallocating to largest ever: 0x40000 000104 1775764139.226898 fallocate 0x03e1a7 thru 0x03ffff (0x1e59 bytes) 000105 1775764139.295103 fallocate 0x025032 thru 0x032d00 (0xdccf bytes) 000106 1775764139.319146 read 0x000700 thru 0x004eb1 (0x47b2 bytes) 000107 1775764139.341105 fallocate 0x00e753 thru 0x01501a (0x68c8 bytes) 000108 1775764139.357107 write 0x02c269 thru 0x02d994 (0x172c bytes) 000109 1775764139.387237 trunc from 0x0352fb to 0x037925 (0x262b bytes) 000110 1775764139.446746 write 0x02e9bc thru 0x03a70e (0xbd53 bytes) 000111 1775764139.475601 trunc from 0x034f37 to 0x03a70e (0x57d8 bytes) 000112 1775764139.513138 read 0x01dc49 thru 0x02d537 (0xf8ef bytes) 000113 1775764139.531019 write 0x02ae74 thru 0x036e2e (0xbfbb bytes) 000114 1775764139.535414 fallocate 0x036ecd thru 0x037b5d (0xc91 bytes) 000115 1775764139.550795 trunc from 0x03a70f to 0x033a47 (0x6cc7 bytes) 000116 1775764139.644255 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: 0x46c930ad (stale) Mirror 4: 0x46c930ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: 0x46c930ad (stale) Mirror 4: 0x46c930ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a48: skipped 000117 1775764140.220163 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: 0x46c930ad (stale) Mirror 4: 0x46c930ad (stale) CRC-32 checksum value for chunk [0, 0x33a48): Mirror 1: 0xc5cf369 Mirror 5: 0xc5cf369 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a48: skipped 000118 1775764141.603074 mapread 0x00e95b thru 0x017c51 (0x92f7 bytes) 000119 1775764141.703921 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: 0x46c930ad (stale) Mirror 4: 0x46c930ad (stale) CRC-32 checksum value for chunk [0, 0x33a48): Mirror 1: 0xc5cf369 Mirror 5: 0xc5cf369 Mirror 6: 0xc5cf369 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a48: skipped 000120 1775764143.420271 fallocate 0x00bb69 thru 0x01a7f8 (0xec90 bytes) 000121 1775764143.477375 fallocate 0x01399e thru 0x0229a6 (0xf009 bytes) 000122 1775764143.501698 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a48): Mirror 2: 0x46c930ad (stale) Mirror 4: 0x46c930ad (stale) Mirror 5: 0xc5cf369 (stale) Mirror 6: 0xc5cf369 (stale) CRC-32 checksum value for chunk [0, 0x33a48): Mirror 1: 0xc5cf369 Mirror 7: 0xc5cf369 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a48: skipped 000123 1775764144.875410 trunc from 0x033a48 to 0x02d48e (0x65b9 bytes) 000124 1775764144.968806 mapread 0x00a1e5 thru 0x0117d5 (0x75f1 bytes) 000125 1775764145.006092 trunc from 0x02a76c to 0x02aa0c (0x2a1 bytes) 000126 1775764145.054491 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d48f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d48f): Mirror 2: 0x4b86a26 (stale) Mirror 4: 0x4b86a26 (stale) Mirror 5: 0x7036ce93 (stale) Mirror 6: 0x7036ce93 (stale) Mirror 7: 0x7036ce93 (stale) CRC-32 checksum value for chunk [0, 0x2d48f): Mirror 1: 0x6623deaf Mirror 8: 0x6623deaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d48f: skipped 000127 1775764145.918877 punch from 0x019273 to 0x01bf11 (0x2c9f bytes) 000128 1775764145.963903 read 0x023bb0 thru 0x02c6da (0x8b2b bytes) 000129 1775764145.975337 punch from 0x014b0e to 0x019846 (0x4d39 bytes) 000130 1775764145.980864 read 0x028e8e thru 0x02d48e (0x4601 bytes) 000131 1775764145.991101 mapread 0x018eef thru 0x01a299 (0x13ab bytes) 000132 1775764145.993173 write 0x00db3d thru 0x0179e5 (0x9ea9 bytes) 000133 1775764145.999191 mapread 0x01c546 thru 0x02067f (0x413a bytes) 000134 1775764146.006010 MIRROR_RESYNC 7 mirrors 000135 1775764148.259461 trunc from 0x02d48f to 0x01bc49 (0x11845 bytes) 000136 1775764148.318403 punch from 0x0006bf to 0x00e8f4 (0xe236 bytes) 000137 1775764148.329114 fallocate 0x026b8f thru 0x032859 (0xbccb bytes) 000138 1775764148.336123 punch from 0x01a23f to 0x01bbf4 (0x19b6 bytes) 000139 1775764148.350279 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bc4a): Mirror 2: 0xb6496735 (stale) Mirror 5: 0xb6496735 (stale) Mirror 6: 0xb6496735 (stale) Mirror 7: 0xb6496735 (stale) Mirror 8: 0xb6496735 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc4a): Mirror 2: 0xb6496735 (stale) Mirror 5: 0xb6496735 (stale) Mirror 6: 0xb6496735 (stale) Mirror 7: 0xb6496735 (stale) Mirror 8: 0xb6496735 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc4a: skipped 000140 1775764148.816699 mapwrite 0x03d497 thru 0x03ffff (0x2b69 bytes) 000141 1775764148.883170 read 0x026750 thru 0x027bf0 (0x14a1 bytes) 000142 1775764148.892814 read 0x02df87 thru 0x0374e9 (0x9563 bytes) 000143 1775764148.908025 write 0x01976b thru 0x01f1db (0x5a71 bytes) 000144 1775764148.909552 read 0x032401 thru 0x03593c (0x353c bytes) 000145 1775764148.916985 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x59199770 (stale) Mirror 5: 0x59199770 (stale) Mirror 6: 0x59199770 (stale) Mirror 7: 0x59199770 (stale) Mirror 8: 0x59199770 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8721942f Mirror 9: 0x8721942f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000146 1775764150.046350 mapwrite 0x00043f thru 0x001c75 (0x1837 bytes) 000147 1775764150.192952 mapwrite 0x00382c thru 0x00c7ee (0x8fc3 bytes) 000148 1775764150.255143 trunc from 0x021a51 to 0x02486d (0x2e1d bytes) 000150 1775764150.262989 punch from 0x00245a to 0x00af0e (0x8ab5 bytes) 000151 1775764150.273123 read 0x016a62 thru 0x01fafe (0x909d bytes) 000152 1775764150.291991 trunc from 0x017f36 to 0x025d24 (0xddef bytes) 000153 1775764150.310855 MIRROR_RESYNC 7 mirrors 000154 1775764152.399425 punch from 0x03839f to 0x03d2f5 (0x4f57 bytes) 000155 1775764152.566882 MIRROR_RESYNC 7 mirrors 000156 1775764156.187650 trunc from 0x025be0 to 0x028ab5 (0x2ed6 bytes) 000157 1775764156.238910 fallocate 0x01f024 thru 0x022c12 (0x3bef bytes) 000158 1775764156.263975 trunc from 0x040000 to 0x02ff92 (0x1006d bytes) 000159 1775764156.305070 fallocate 0x003f89 thru 0x00a01b (0x6093 bytes) 000160 1775764156.319162 mapwrite 0x001b2c thru 0x00874b (0x6c20 bytes) 000161 1775764156.442995 trunc from 0x01361e to 0x021d77 (0xe75a bytes) 000162 1775764156.459424 trunc from 0x02ff93 to 0x01a71f (0x15873 bytes) 000163 1775764156.525318 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a720): Mirror 2: 0x2dd8e60e (stale) Mirror 5: 0x2dd8e60e (stale) Mirror 6: 0x2dd8e60e (stale) Mirror 7: 0x2dd8e60e (stale) Mirror 8: 0x2dd8e60e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a720): Mirror 2: 0x2dd8e60e (stale) Mirror 5: 0x2dd8e60e (stale) Mirror 6: 0x2dd8e60e (stale) Mirror 7: 0x2dd8e60e (stale) Mirror 8: 0x2dd8e60e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a720: skipped 000164 1775764157.486273 trunc from 0x004ed5 to 0x00b464 (0x6590 bytes) 000165 1775764157.497975 mapread 0x016e54 thru 0x01a71f (0x38cc bytes) 000166 1775764157.564141 read 0x005f75 thru 0x00e70f (0x879b bytes) 000167 1775764157.576439 trunc from 0x01a720 to 0x01ceb8 (0x2799 bytes) 000168 1775764157.636322 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1ceb9): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ceb9): Mirror 2: 0xfc3710ef (stale) Mirror 5: 0xfc3710ef (stale) Mirror 6: 0xfc3710ef (stale) Mirror 7: 0xfc3710ef (stale) Mirror 8: 0xfc3710ef (stale) CRC-32 checksum value for chunk [0, 0x1ceb9): Mirror 1: 0x83de3a Mirror 9: 0x83de3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ceb9: skipped 000170 1775764159.141465 trunc from 0x000160 to 0x00484d (0x46ee bytes) 000171 1775764159.174186 punch from 0x00ca88 to 0x01a994 (0xdf0d bytes) 000172 1775764159.183687 mapread 0x00f475 thru 0x014c48 (0x57d4 bytes) 000173 1775764159.197984 mapread 0x019b7c thru 0x01ceb8 (0x333d bytes) 000174 1775764159.203267 punch from 0x014e32 to 0x01ceb8 (0x8087 bytes) 000175 1775764159.207009 read 0x00eaa1 thru 0x0101cc (0x172c bytes) 000176 1775764159.207604 trunc from 0x003373 to 0x00878c (0x541a bytes) 000177 1775764159.210498 write 0x02a166 thru 0x030e85 (0x6d20 bytes) 000178 1775764159.213912 mapread 0x0212e4 thru 0x02db7a (0xc897 bytes) 000179 1775764159.220476 mapread 0x019fbc thru 0x029e21 (0xfe66 bytes) 000180 1775764159.229399 write 0x00e861 thru 0x01b4cb (0xcc6b bytes) 000181 1775764159.230539 write 0x01f8e2 thru 0x022373 (0x2a92 bytes) 000182 1775764159.231197 trunc from 0x030e86 to 0x0256f5 (0xb790 bytes) 000183 1775764159.249059 mapread 0x01671d thru 0x023242 (0xcb26 bytes) 000184 1775764159.250325 write 0x00bed5 thru 0x015055 (0x9181 bytes) 000185 1775764159.251871 fallocate 0x022841 thru 0x0319e6 (0xf1a6 bytes) 000186 1775764159.256860 punch from 0x010422 to 0x012d3f (0x291e bytes) 000188 1775764159.269176 fallocate 0x03330e thru 0x035af7 (0x27ea bytes) 000189 1775764159.278099 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35af8): Mirror 2: 0xd3f5164f (stale) Mirror 5: 0xd3f5164f (stale) Mirror 7: 0xd3f5164f (stale) Mirror 8: 0xd3f5164f (stale) Mirror 9: 0x9965918a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35af8): Mirror 2: 0xd3f5164f (stale) Mirror 5: 0xd3f5164f (stale) Mirror 7: 0xd3f5164f (stale) Mirror 8: 0xd3f5164f (stale) Mirror 9: 0x9965918a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35af8: skipped 000190 1775764160.063007 read 0x011808 thru 0x021193 (0xf98c bytes) 000191 1775764160.112475 mapwrite 0x02880f thru 0x02aa83 (0x2275 bytes) 000192 1775764160.215550 fallocate 0x01d1d0 thru 0x023bc3 (0x69f4 bytes) 000194 1775764160.223414 write 0x001c65 thru 0x00be15 (0xa1b1 bytes) 000195 1775764160.250102 mapread 0x011dd7 thru 0x01da12 (0xbc3c bytes) 000197 1775764160.253177 fallocate 0x00cb8e thru 0x012ef5 (0x6368 bytes) 000198 1775764160.259291 write 0x03857e thru 0x03b41b (0x2e9e bytes) 000199 1775764160.262130 mapwrite 0x03a2a4 thru 0x03ffff (0x5d5c bytes) 000200 1775764160.338607 trunc from 0x036e3f to 0x03ffff (0x91c1 bytes) 000201 1775764160.354247 trunc from 0x0343fd to 0x03e0a4 (0x9ca8 bytes) 000202 1775764160.373946 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf4650b24 (stale) Mirror 5: 0xf4650b24 (stale) Mirror 7: 0xf4650b24 (stale) Mirror 8: 0xf4650b24 (stale) Mirror 9: 0xda655402 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x469934d4 Mirror 10: 0x469934d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000203 1775764161.474198 fallocate 0x003504 thru 0x005b0c (0x2609 bytes) 000204 1775764161.556381 fallocate 0x00e468 thru 0x019c31 (0xb7ca bytes) 000205 1775764161.571808 punch from 0x0281bf to 0x0294ae (0x12f0 bytes) 000206 1775764161.583833 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf4650b24 (stale) Mirror 7: 0xf4650b24 (stale) Mirror 8: 0xf4650b24 (stale) Mirror 9: 0xda655402 (stale) Mirror 10: 0x469934d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf4650b24 (stale) Mirror 7: 0xf4650b24 (stale) Mirror 8: 0xf4650b24 (stale) Mirror 9: 0xda655402 (stale) Mirror 10: 0x469934d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000207 1775764162.208693 read 0x03d1c9 thru 0x03ffff (0x2e37 bytes) 000208 1775764162.217278 mapwrite 0x004f56 thru 0x00958e (0x4639 bytes) 000209 1775764162.262722 trunc from 0x01c94c to 0x01d5fa (0xcaf bytes) 000210 1775764162.276158 mapwrite 0x012902 thru 0x01ed31 (0xc430 bytes) 000211 1775764162.320680 trunc from 0x036d49 to 0x03e048 (0x7300 bytes) 000212 1775764162.326676 read 0x033da7 thru 0x03e4d8 (0xa732 bytes) 000213 1775764162.335991 punch from 0x026929 to 0x02d813 (0x6eeb bytes) 000214 1775764162.346788 MIRROR_RESYNC 6 mirrors 000215 1775764164.328298 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) 000216 1775764164.581140 fallocate 0x01b469 thru 0x0251a1 (0x9d39 bytes) 000217 1775764164.610083 write 0x0034b2 thru 0x00e54c (0xb09b bytes) 000218 1775764164.624307 fallocate 0x021441 thru 0x023caa (0x286a bytes) 000219 1775764164.629447 trunc from 0x031451 to 0x033213 (0x1dc3 bytes) 000220 1775764164.634096 mapwrite 0x011339 thru 0x0174ca (0x6192 bytes) 000221 1775764164.659685 read 0x020863 thru 0x02cca3 (0xc441 bytes) 000222 1775764164.668306 punch from 0x033700 to 0x0379f2 (0x42f3 bytes) 000223 1775764164.672894 read 0x008c1c thru 0x00ae22 (0x2207 bytes) 000224 1775764164.673674 write 0x01ab4c thru 0x026eb8 (0xc36d bytes) 000225 1775764164.682062 fallocate 0x03fbb3 thru 0x03ffff (0x44d bytes) 000226 1775764164.687708 write 0x008ca3 thru 0x00a4f1 (0x184f bytes) 000227 1775764164.689027 write 0x002c50 thru 0x005b22 (0x2ed3 bytes) 000228 1775764164.690029 mapread 0x03e4f2 thru 0x03ffff (0x1b0e bytes) 000229 1775764164.691374 write 0x03826c thru 0x03ffff (0x7d94 bytes) 000230 1775764164.692974 mapwrite 0x01f41a thru 0x025079 (0x5c60 bytes) 000232 1775764164.722282 trunc from 0x040000 to 0x02e6aa (0x11955 bytes) 000233 1775764164.755902 trunc from 0x02e6ab to 0x01072a (0x1df80 bytes) 000234 1775764164.773861 mapwrite 0x01e29e thru 0x02247e (0x41e1 bytes) 000235 1775764164.834685 mapwrite 0x0355ba thru 0x03ffff (0xaa46 bytes) 000236 1775764164.988751 punch from 0x0398f3 to 0x03ffff (0x670d bytes) 000237 1775764164.996275 mapread 0x008125 thru 0x00bc08 (0x3ae4 bytes) 000238 1775764164.998176 write 0x0080b6 thru 0x0106b6 (0x8601 bytes) 000239 1775764165.000313 fallocate 0x027c80 thru 0x036d48 (0xf0c9 bytes) 000240 1775764165.008128 read 0x01d9d4 thru 0x026fb3 (0x95e0 bytes) 000241 1775764165.024188 read 0x01ec4b thru 0x02780a (0x8bc0 bytes) 000243 1775764165.026086 trunc from 0x01ac14 to 0x0273db (0xc7c8 bytes) 000245 1775764165.071711 MIRROR_RESYNC 5 mirrors 000246 1775764167.220146 fallocate 0x0269fc thru 0x02ee1d (0x8422 bytes) 000247 1775764167.253074 punch from 0x03dd18 to 0x03ffff (0x22e8 bytes) 000248 1775764167.271911 MIRROR_RESYNC 5 mirrors 000249 1775764169.425767 punch from 0x003ab9 to 0x00e365 (0xa8ad bytes) 000250 1775764169.511264 mapread 0x0028b9 thru 0x00ffc5 (0xd70d bytes) 000251 1775764169.634189 read 0x000fc8 thru 0x00df99 (0xcfd2 bytes) 000253 1775764169.659945 mapwrite 0x009018 thru 0x00b5d3 (0x25bc bytes) 000254 1775764169.707420 read 0x03a393 thru 0x03ffff (0x5c6d bytes) 000255 1775764169.766632 punch from 0x008c74 to 0x016b54 (0xdee1 bytes) 000256 1775764169.773648 fallocate 0x01a3ef thru 0x0261ce (0xbde0 bytes) 000257 1775764169.780258 punch from 0x03f9f7 to 0x03ffff (0x609 bytes) 000258 1775764169.823048 fallocate 0x03806b thru 0x03ffff (0x7f95 bytes) 000259 1775764169.845160 trunc from 0x0013b7 to 0x00fa06 (0xe650 bytes) 000260 1775764169.868259 fallocate 0x03efad thru 0x03ffff (0x1053 bytes) 000261 1775764169.888085 punch from 0x0076ad to 0x0169c0 (0xf314 bytes) 000262 1775764169.897342 mapwrite 0x01e7b9 thru 0x027d89 (0x95d1 bytes) 000263 1775764169.975760 punch from 0x0343cb to 0x038cbe (0x48f4 bytes) 000264 1775764169.999273 punch from 0x02cbc8 to 0x032076 (0x54af bytes) 000265 1775764170.018256 fallocate 0x0221f0 thru 0x0286d3 (0x64e4 bytes) 000266 1775764170.057812 mapwrite 0x035ea3 thru 0x036b2a (0xc88 bytes) 000267 1775764170.160022 write 0x036bdf thru 0x03bca0 (0x50c2 bytes) 000268 1775764170.161622 fallocate 0x0155d1 thru 0x01cde9 (0x7819 bytes) 000269 1775764170.179991 trunc from 0x020420 to 0x030136 (0xfd17 bytes) 000270 1775764170.195761 write 0x016abd thru 0x021001 (0xa545 bytes) 000271 1775764170.240809 punch from 0x02c268 to 0x035493 (0x922c bytes) 000272 1775764170.260436 fallocate 0x02a32a thru 0x02f9dd (0x56b4 bytes) 000273 1775764170.302971 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x466e9fad (stale) Mirror 7: 0x466e9fad (stale) Mirror 8: 0x466e9fad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x466e9fad (stale) Mirror 7: 0x466e9fad (stale) Mirror 8: 0x466e9fad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000274 1775764171.074766 trunc from 0x03a812 to 0x03ffff (0x57ee bytes) 000275 1775764171.089136 write 0x00ca8d thru 0x016250 (0x97c4 bytes) 000276 1775764171.116071 trunc from 0x040000 to 0x0351a7 (0xae58 bytes) 000277 1775764171.166214 mapread 0x033754 thru 0x0351a7 (0x1a54 bytes) 000278 1775764171.186702 write 0x0260d4 thru 0x02ce0f (0x6d3c bytes) 000279 1775764171.203206 write 0x026479 thru 0x02d14c (0x6cd4 bytes) 000280 1775764171.211130 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x351a8): Mirror 5: 0x92cc8be6 (stale) Mirror 8: 0x92cc8be6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351a8): Mirror 5: 0x92cc8be6 (stale) Mirror 8: 0x92cc8be6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351a8: skipped 000281 1775764171.923015 write 0x026037 thru 0x02fb6e (0x9b38 bytes) 000283 1775764171.953695 read 0x0331b2 thru 0x0340c3 (0xf12 bytes) 000284 1775764171.964328 mapread 0x00a845 thru 0x00c7cd (0x1f89 bytes) 000285 1775764171.980845 trunc from 0x01be86 to 0x02777a (0xb8f5 bytes) 000286 1775764172.011222 mapread 0x033012 thru 0x0351a7 (0x2196 bytes) 000288 1775764172.030275 mapread 0x019bdd thru 0x029437 (0xf85b bytes) 000289 1775764172.066703 trunc from 0x0351a8 to 0x01440c (0x20d9b bytes) 000290 1775764172.103014 trunc from 0x01199e to 0x01440c (0x2a6f bytes) 000291 1775764172.111050 trunc from 0x0076fd to 0x00a2ed (0x2bf1 bytes) 000292 1775764172.121940 read 0x0094c1 thru 0x00cb91 (0x36d1 bytes) 000293 1775764172.172029 trunc from 0x01440d to 0x011dea (0x2622 bytes) 000294 1775764172.204113 mapread 0x0085c1 thru 0x00ce30 (0x4870 bytes) 000295 1775764172.207378 trunc from 0x0061e1 to 0x011dea (0xbc0a bytes) 000296 1775764172.231262 mapwrite 0x01833b thru 0x01ea02 (0x66c8 bytes) 000297 1775764172.350399 fallocate 0x00d99a thru 0x01a8be (0xcf25 bytes) 000298 1775764172.361760 read 0x0024ac thru 0x003caa (0x17ff bytes) 000300 1775764172.362912 mapwrite 0x022191 thru 0x027e40 (0x5cb0 bytes) 000301 1775764172.506683 trunc from 0x027e41 to 0x032589 (0xa749 bytes) 000302 1775764172.587316 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3258a): Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3258a): Mirror 5: 0x3084d502 (stale) Mirror 8: 0x3084d502 (stale) CRC-32 checksum value for chunk [0, 0x3258a): Mirror 1: 0xbd2b3c40 Mirror 9: 0xbd2b3c40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3258a: skipped 000303 1775764174.698353 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3258a): Mirror 8: 0x3084d502 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3258a): Mirror 8: 0x3084d502 (stale) CRC-32 checksum value for chunk [0, 0x3258a): Mirror 1: 0xbd2b3c40 Mirror 9: 0xbd2b3c40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3258a: skipped 000304 1775764175.967122 fallocate 0x03ce18 thru 0x03ffff (0x31e8 bytes) 000305 1775764176.173188 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x5bbf161b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x5bbf161b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000306 1775764176.811850 punch from 0x02728f to 0x030923 (0x9695 bytes) 000307 1775764176.827666 write 0x0142c8 thru 0x0206d6 (0xc40f bytes) 000308 1775764176.865741 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) 000309 1775764177.183102 trunc from 0x040000 to 0x01db85 (0x2247a bytes) 000310 1775764177.246200 fallocate 0x02ac7c thru 0x0312eb (0x6670 bytes) 000311 1775764177.258004 mapread 0x01a2d0 thru 0x01db85 (0x38b6 bytes) 000312 1775764177.347319 punch from 0x0005c8 to 0x00f43f (0xee78 bytes) 000313 1775764177.361125 write 0x00d116 thru 0x013c42 (0x6b2d bytes) 000314 1775764177.409620 trunc from 0x01db86 to 0x024aa9 (0x6f24 bytes) 000315 1775764177.452634 mapwrite 0x02a66a thru 0x036964 (0xc2fb bytes) 000316 1775764177.624005 mapread 0x00b9c0 thru 0x015397 (0x99d8 bytes) 000317 1775764177.630588 read 0x017401 thru 0x024c6f (0xd86f bytes) 000318 1775764177.669697 fallocate 0x024fa1 thru 0x02cc06 (0x7c66 bytes) 000319 1775764177.688320 trunc from 0x005420 to 0x00a14b (0x4d2c bytes) 000320 1775764177.712934 mapread 0x024395 thru 0x033f45 (0xfbb1 bytes) 000321 1775764177.714347 fallocate 0x004d1a thru 0x013d31 (0xf018 bytes) 000322 1775764177.748781 MIRROR_EXTEND to 2 mirrors 000323 1775764179.459112 trunc from 0x001c72 to 0x00ef1f (0xd2ae bytes) 000325 1775764179.579365 fallocate 0x03687f thru 0x037862 (0xfe4 bytes) 000326 1775764179.597455 mapread 0x0240b3 thru 0x0307b2 (0xc700 bytes) 000327 1775764179.670234 write 0x0154b6 thru 0x01bafd (0x6648 bytes) 000328 1775764179.697302 mapwrite 0x02c318 thru 0x03b8f7 (0xf5e0 bytes) 000329 1775764179.919180 read 0x0169ef thru 0x022943 (0xbf55 bytes) 000330 1775764179.933179 MIRROR_RESYNC 2 mirrors 000331 1775764180.910974 mapread 0x00dd89 thru 0x012039 (0x42b1 bytes) 000332 1775764180.942337 mapwrite 0x033329 thru 0x03ed9d (0xba75 bytes) 000333 1775764181.039615 mapwrite 0x02a2a4 thru 0x03467f (0xa3dc bytes) 000334 1775764181.088155 write 0x02ab1c thru 0x030f17 (0x63fc bytes) 000335 1775764181.090289 fallocate 0x01c8b4 thru 0x029232 (0xc97f bytes) 000336 1775764181.096233 mapread 0x020de0 thru 0x02bc77 (0xae98 bytes) 000337 1775764181.116601 mapwrite 0x026779 thru 0x02fa31 (0x92b9 bytes) 000338 1775764181.154500 MIRROR_RESYNC 2 mirrors 000340 1775764181.910468 write 0x01695a thru 0x019eb1 (0x3558 bytes) 000341 1775764181.978773 punch from 0x03c929 to 0x03ed9d (0x2475 bytes) 000343 1775764181.988848 mapwrite 0x003f49 thru 0x00c216 (0x82ce bytes) 000344 1775764182.043746 trunc from 0x03ed9e to 0x037512 (0x788b bytes) 000345 1775764182.098788 fallocate 0x03a05c thru 0x03c067 (0x200c bytes) 000346 1775764182.108776 write 0x0326ed thru 0x037ad3 (0x53e7 bytes) 000347 1775764182.125902 mapread 0x029a37 thru 0x02b84b (0x1e15 bytes) 000348 1775764182.142836 fallocate 0x00e0b7 thru 0x015e19 (0x7d63 bytes) 000349 1775764182.150696 trunc from 0x03c068 to 0x03390b (0x875c bytes) 000350 1775764182.171388 mapwrite 0x031184 thru 0x0324e2 (0x135f bytes) 000351 1775764182.188778 trunc from 0x0038be to 0x01217c (0xe8bf bytes) 000352 1775764182.195486 read 0x02a31b thru 0x030505 (0x61eb bytes) 000353 1775764182.200672 mapread 0x02aaa9 thru 0x0334d8 (0x8a30 bytes) 000354 1775764182.202758 mapwrite 0x03b2ac thru 0x03ffff (0x4d54 bytes) 000355 1775764182.269157 mapwrite 0x0385f7 thru 0x03bd0a (0x3714 bytes) 000356 1775764182.286604 trunc from 0x040000 to 0x01e326 (0x21cd9 bytes) 000357 1775764182.307847 trunc from 0x01e327 to 0x00198b (0x1c99b bytes) 000359 1775764182.331752 mapread 0x000937 thru 0x00198b (0x1055 bytes) 000360 1775764182.339518 mapread 0x001125 thru 0x00198b (0x867 bytes) 000361 1775764182.341047 mapwrite 0x00e76f thru 0x011a25 (0x32b7 bytes) 000362 1775764182.391305 punch from 0x0016a7 to 0x00685c (0x51b6 bytes) 000363 1775764182.397898 trunc from 0x002bff to 0x00b1aa (0x85ac bytes) 000364 1775764182.404753 read 0x004d3a thru 0x0110d5 (0xc39c bytes) 000365 1775764182.419904 punch from 0x00cb49 to 0x01106b (0x4523 bytes) 000367 1775764182.430861 read 0x002161 thru 0x009c67 (0x7b07 bytes) 000368 1775764182.432060 fallocate 0x015713 thru 0x02483e (0xf12c bytes) 000369 1775764182.439403 mapwrite 0x00083a thru 0x00127d (0xa44 bytes) 000370 1775764182.465868 write 0x011eac thru 0x01abd5 (0x8d2a bytes) 000371 1775764182.487127 write 0x010aad thru 0x01af8a (0xa4de bytes) 000372 1775764182.490623 mapread 0x000fb0 thru 0x009d48 (0x8d99 bytes) 000373 1775764182.493278 punch from 0x00e66c to 0x01ac6a (0xc5ff bytes) 000374 1775764182.515363 write 0x030bee thru 0x035a05 (0x4e18 bytes) 000375 1775764182.522980 MIRROR_RESYNC 2 mirrors 000376 1775764183.617009 write 0x024486 thru 0x024a5a (0x5d5 bytes) 000377 1775764183.683177 MIRROR_RESYNC 2 mirrors 000378 1775764184.737307 trunc from 0x035a06 to 0x023ed5 (0x11b30 bytes) 000379 1775764184.797924 mapread 0x0227c6 thru 0x023ed5 (0x1710 bytes) 000380 1775764184.819672 mapwrite 0x0127b6 thru 0x01e346 (0xbb91 bytes) 000381 1775764184.889053 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) 000385 1775764185.080996 trunc from 0x023ed6 to 0x02bf6b (0x8096 bytes) 000386 1775764185.104462 trunc from 0x0123c8 to 0x015737 (0x3370 bytes) 000387 1775764185.111895 fallocate 0x0348a3 thru 0x0381ec (0x394a bytes) 000388 1775764185.124266 mapwrite 0x0018cb thru 0x00e604 (0xcd3a bytes) 000389 1775764185.210891 trunc from 0x0157a9 to 0x01f54b (0x9da3 bytes) 000390 1775764185.220017 fallocate 0x02a1f6 thru 0x03844e (0xe259 bytes) 000391 1775764185.226116 punch from 0x01cf9b to 0x02989f (0xc905 bytes) 000392 1775764185.231991 trunc from 0x0381ed to 0x02499a (0x13852 bytes) 000393 1775764185.259912 punch from 0x016f5f to 0x021a58 (0xaafa bytes) 000395 1775764185.272826 mapread 0x01da25 thru 0x021271 (0x384d bytes) 000396 1775764185.288924 write 0x006bba thru 0x00f23c (0x8683 bytes) 000397 1775764185.292817 trunc from 0x017b3b to 0x017fb1 (0x477 bytes) 000398 1775764185.298941 mapwrite 0x013ca7 thru 0x016033 (0x238d bytes) 000399 1775764185.344138 MIRROR_EXTEND to 2 mirrors 000400 1775764186.044961 punch from 0x010eb6 to 0x016b08 (0x5c53 bytes) 000401 1775764186.090262 write 0x00acb7 thru 0x019a5f (0xeda9 bytes) 000402 1775764186.108137 mapwrite 0x005a43 thru 0x00a5c4 (0x4b82 bytes) 000403 1775764186.145709 mapwrite 0x014823 thru 0x021b14 (0xd2f2 bytes) 000404 1775764186.187350 fallocate 0x0398f0 thru 0x03ffff (0x6710 bytes) 000406 1775764186.195889 write 0x0221a8 thru 0x029acc (0x7925 bytes) 000408 1775764186.200012 fallocate 0x012aed thru 0x01ec13 (0xc127 bytes) 000410 1775764186.210239 trunc from 0x029acd to 0x02726a (0x2862 bytes) 000411 1775764186.260781 trunc from 0x02726b to 0x01b66a (0xbc00 bytes) 000413 1775764186.279988 mapwrite 0x02d6fc thru 0x02da6c (0x371 bytes) 000414 1775764186.322944 trunc from 0x0159c3 to 0x0209b3 (0xaff1 bytes) 000415 1775764186.340879 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) 000416 1775764186.552326 mapwrite 0x013060 thru 0x017825 (0x47c6 bytes) 000417 1775764186.621705 mapread 0x00f6d9 thru 0x00f704 (0x2c bytes) 000419 1775764186.625063 trunc from 0x02da6d to 0x014f88 (0x18ae4 bytes) 000421 1775764186.660829 trunc from 0x004f7e to 0x007669 (0x26ec bytes) 000422 1775764186.668308 mapwrite 0x00e2a1 thru 0x01c4de (0xe23e bytes) 000423 1775764186.740463 MIRROR_EXTEND to 2 mirrors 000424 1775764187.690126 fallocate 0x03aa27 thru 0x03ffff (0x55d9 bytes) 000425 1775764187.735133 trunc from 0x004193 to 0x00422c (0x9a bytes) 000426 1775764187.749949 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c4df): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c4df): Mirror 2: 0x497f9504 (stale) CRC-32 checksum value for chunk [0, 0x1c4df): Mirror 1: 0xecd164e4 Mirror 3: 0xecd164e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c4df: skipped 000427 1775764189.065520 mapwrite 0x0020c5 thru 0x00923d (0x7179 bytes) 000428 1775764189.260065 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1c4df): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c4df): Mirror 2: 0x497f9504 (stale) Mirror 3: 0xecd164e4 (stale) CRC-32 checksum value for chunk [0, 0x1c4df): Mirror 1: 0x7cac852 Mirror 4: 0x7cac852 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c4df: skipped 000429 1775764190.313684 fallocate 0x032731 thru 0x03ffff (0xd8cf bytes) 000430 1775764190.378870 MIRROR_RESYNC 4 mirrors 000432 1775764191.284607 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) 000433 1775764191.814773 punch from 0x017a37 to 0x01c3c7 (0x4991 bytes) 000434 1775764191.911631 write 0x0325a0 thru 0x03df76 (0xb9d7 bytes) 000435 1775764191.950499 read 0x03c284 thru 0x03ffff (0x3d7c bytes) 000436 1775764191.969232 punch from 0x032c14 to 0x03ffff (0xd3ec bytes) 000437 1775764192.014119 mapwrite 0x014348 thru 0x01df63 (0x9c1c bytes) 000438 1775764192.218615 read 0x032ac3 thru 0x036f3b (0x4479 bytes) 000439 1775764192.266438 trunc from 0x040000 to 0x0299b2 (0x1664d bytes) 000440 1775764192.339148 mapread 0x00ad41 thru 0x00fad7 (0x4d97 bytes) 000441 1775764192.344163 read 0x000583 thru 0x00a147 (0x9bc5 bytes) 000442 1775764192.346086 mapwrite 0x0302f8 thru 0x03e36a (0xe073 bytes) 000443 1775764192.511981 trunc from 0x03e36b to 0x0098c9 (0x34aa1 bytes) 000444 1775764192.555683 trunc from 0x008b8a to 0x0098c9 (0xd40 bytes) 000445 1775764192.566496 trunc from 0x0098ca to 0x018ea8 (0xf5df bytes) 000447 1775764192.599430 fallocate 0x00eddf thru 0x018a5b (0x9c7d bytes) 000448 1775764192.613877 mapwrite 0x002633 thru 0x00bd7f (0x974d bytes) 000449 1775764192.677666 punch from 0x0080bc to 0x00e5f8 (0x653d bytes) 000450 1775764192.695524 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18ea9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ea9): Mirror 3: 0x8bf3ff9d (stale) Mirror 4: 0x8bf3ff9d (stale) CRC-32 checksum value for chunk [0, 0x18ea9): Mirror 1: 0x76195480 Mirror 5: 0x76195480 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ea9: skipped 000451 1775764194.249505 mapread 0x00cebb thru 0x018ea8 (0xbfee bytes) 000453 1775764194.308203 mapread 0x005db0 thru 0x013ae3 (0xdd34 bytes) 000455 1775764194.328413 mapwrite 0x0367d7 thru 0x03d050 (0x687a bytes) 000456 1775764194.505591 trunc from 0x02c00c to 0x033da3 (0x7d98 bytes) 000457 1775764194.511866 trunc from 0x012f4f to 0x018db9 (0x5e6b bytes) 000458 1775764194.527098 mapread 0x034ede thru 0x03933d (0x4460 bytes) 000459 1775764194.554545 mapwrite 0x02a1ff thru 0x02cea1 (0x2ca3 bytes) 000460 1775764194.629174 write 0x037401 thru 0x03f03c (0x7c3c bytes) 000461 1775764194.652764 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f03d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f03d): Mirror 3: 0x41210847 (stale) Mirror 4: 0x41210847 (stale) Mirror 5: 0xca68d41d (stale) CRC-32 checksum value for chunk [0, 0x3f03d): Mirror 1: 0x410c8513 Mirror 6: 0x410c8513 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f03d: skipped 000462 1775764196.395512 write 0x013199 thru 0x015a83 (0x28eb bytes) 000463 1775764196.455885 trunc from 0x03f03d to 0x00d533 (0x31b09 bytes) 000464 1775764196.477595 mapwrite 0x00c6dc thru 0x00ff02 (0x3827 bytes) 000465 1775764196.546995 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xff03): Mirror 4: 0x5af96e17 (stale) Mirror 5: 0x157a3cdd (stale) Mirror 6: 0x157a3cdd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xff03): Mirror 4: 0x5af96e17 (stale) Mirror 5: 0x157a3cdd (stale) Mirror 6: 0x157a3cdd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xff03: skipped 000466 1775764196.980880 mapread 0x001ca0 thru 0x0071d1 (0x5532 bytes) 000467 1775764197.003671 mapwrite 0x01e16b thru 0x02667b (0x8511 bytes) 000468 1775764197.069999 punch from 0x0241bb to 0x02667b (0x24c1 bytes) 000469 1775764197.080262 mapread 0x0205bb thru 0x02667b (0x60c1 bytes) 000470 1775764197.094414 trunc from 0x02667c to 0x03051f (0x9ea4 bytes) 000471 1775764197.131492 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30520): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30520): Mirror 4: 0x370de552 (stale) Mirror 5: 0xc31119a1 (stale) Mirror 6: 0x5ff5265d (stale) CRC-32 checksum value for chunk [0, 0x30520): Mirror 1: 0x5c0f1823 Mirror 7: 0x5c0f1823 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30520: skipped 000472 1775764198.297334 MIRROR_RESYNC 5 mirrors 000473 1775764199.849127 MIRROR_EXTEND to 6 mirrors 000474 1775764201.346522 trunc from 0x030520 to 0x007f40 (0x285df bytes) 000475 1775764201.430934 mapread 0x00005a thru 0x007f40 (0x7ee7 bytes) 000476 1775764201.456126 write 0x00c457 thru 0x0113d4 (0x4f7e bytes) 000478 1775764201.459753 mapread 0x00a312 thru 0x00bff0 (0x1cdf bytes) 000479 1775764201.476207 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x113d5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x113d5): Mirror 4: 0x4ff1e5ba (stale) Mirror 5: 0x4ff1e5ba (stale) Mirror 6: 0x4ff1e5ba (stale) Mirror 7: 0x4ff1e5ba (stale) Mirror 8: 0x4ff1e5ba (stale) CRC-32 checksum value for chunk [0, 0x113d5): Mirror 1: 0xbe1f1f3b Mirror 9: 0xbe1f1f3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x113d5: skipped 000480 1775764202.769360 trunc from 0x005c53 to 0x00f9d7 (0x9d85 bytes) 000481 1775764202.820172 mapread 0x0087fc thru 0x0113d4 (0x8bd9 bytes) 000482 1775764202.849874 mapwrite 0x01c937 thru 0x02a263 (0xd92d bytes) 000483 1775764202.945161 trunc from 0x0225a2 to 0x02a263 (0x7cc2 bytes) 000484 1775764202.955508 trunc from 0x0298b7 to 0x02a263 (0x9ad bytes) 000485 1775764202.967991 trunc from 0x02a264 to 0x025bc1 (0x46a2 bytes) 000486 1775764203.019096 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25bc2): Mirror 4: 0x4232306e (stale) Mirror 5: 0x4232306e (stale) Mirror 6: 0x4232306e (stale) Mirror 8: 0x4232306e (stale) Mirror 9: 0x9ebb58d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25bc2): Mirror 4: 0x4232306e (stale) Mirror 5: 0x4232306e (stale) Mirror 6: 0x4232306e (stale) Mirror 8: 0x4232306e (stale) Mirror 9: 0x9ebb58d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25bc2: skipped 000489 1775764203.597084 punch from 0x01f641 to 0x025bc1 (0x6581 bytes) 000490 1775764203.603190 read 0x00eece thru 0x01e3a9 (0xf4dc bytes) 000491 1775764203.625377 write 0x0310ed thru 0x03ffff (0xef13 bytes) 000492 1775764203.636433 fallocate 0x004ebb thru 0x00cf5a (0x80a0 bytes) 000493 1775764203.647295 write 0x01b67f thru 0x01c964 (0x12e6 bytes) 000494 1775764203.649389 read 0x022166 thru 0x02e18c (0xc027 bytes) 000495 1775764203.672495 read 0x00db3c thru 0x01826f (0xa734 bytes) 000496 1775764203.673652 punch from 0x0319eb to 0x03fdc2 (0xe3d8 bytes) 000497 1775764203.713474 fallocate 0x00f07f thru 0x0134df (0x4461 bytes) 000498 1775764203.719971 trunc from 0x040000 to 0x026590 (0x19a6f bytes) 000499 1775764203.743604 write 0x00ab44 thru 0x0153e9 (0xa8a6 bytes) 000501 1775764203.745706 trunc from 0x026591 to 0x037fcb (0x11a3b bytes) 000502 1775764203.768167 fallocate 0x01ab00 thru 0x01addc (0x2dd bytes) 000504 1775764203.775099 fallocate 0x02026d thru 0x02c294 (0xc028 bytes) 000506 1775764203.784768 trunc from 0x037fcc to 0x02d01d (0xafae bytes) 000507 1775764203.805488 write 0x006202 thru 0x00e923 (0x8722 bytes) 000508 1775764203.808460 write 0x028aea thru 0x0299d0 (0xee7 bytes) 000509 1775764203.818430 write 0x03f7c1 thru 0x03ffff (0x83f bytes) 000510 1775764203.819878 trunc from 0x040000 to 0x0092cc (0x36d33 bytes) 000511 1775764203.848770 mapwrite 0x034113 thru 0x03cc77 (0x8b65 bytes) 000512 1775764203.914975 read 0x00f2bb thru 0x011bd3 (0x2919 bytes) 000513 1775764203.931367 read 0x013d97 thru 0x016b81 (0x2deb bytes) 000514 1775764203.932287 trunc from 0x021b12 to 0x02eb3a (0xd029 bytes) 000515 1775764203.938336 mapwrite 0x01ce9a thru 0x024740 (0x78a7 bytes) 000516 1775764203.975667 trunc from 0x03cc78 to 0x029b9d (0x130da bytes) 000517 1775764203.999123 mapwrite 0x038f94 thru 0x03febc (0x6f29 bytes) 000518 1775764204.081181 write 0x035278 thru 0x03eb8f (0x9918 bytes) 000519 1775764204.094919 punch from 0x005456 to 0x00b053 (0x5bfe bytes) 000520 1775764204.105360 fallocate 0x018c5c thru 0x025c03 (0xcfa8 bytes) 000521 1775764204.114383 punch from 0x032d56 to 0x03febc (0xd167 bytes) 000522 1775764204.137465 fallocate 0x01140f thru 0x0115dc (0x1ce bytes) 000523 1775764204.151713 punch from 0x02737e to 0x032570 (0xb1f3 bytes) 000524 1775764204.173069 fallocate 0x01f3ee thru 0x027bf8 (0x880b bytes) 000525 1775764204.181452 fallocate 0x035960 thru 0x03ffff (0xa6a0 bytes) 000527 1775764204.190032 trunc from 0x040000 to 0x03841f (0x7be0 bytes) 000528 1775764204.216662 trunc from 0x038420 to 0x039c01 (0x17e2 bytes) 000529 1775764204.254012 MIRROR_RESYNC 6 mirrors 000530 1775764206.184167 trunc from 0x032532 to 0x039c01 (0x76d0 bytes) 000531 1775764206.244730 mapwrite 0x00331e thru 0x009615 (0x62f8 bytes) 000532 1775764206.306134 read 0x011bc5 thru 0x01fa29 (0xde65 bytes) 000533 1775764206.332665 mapread 0x022e18 thru 0x02743e (0x4627 bytes) 000534 1775764206.350448 trunc from 0x039c02 to 0x011660 (0x285a1 bytes) 000535 1775764206.394707 trunc from 0x011661 to 0x03941a (0x27dba bytes) 000536 1775764206.427875 trunc from 0x00b128 to 0x0152e7 (0xa1c0 bytes) 000537 1775764206.437370 fallocate 0x003419 thru 0x00e310 (0xaef8 bytes) 000538 1775764206.449050 write 0x003046 thru 0x00b94a (0x8905 bytes) 000539 1775764206.465853 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3941b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3941b): Mirror 4: 0x54b59ea (stale) Mirror 5: 0x54b59ea (stale) Mirror 6: 0x54b59ea (stale) Mirror 8: 0x54b59ea (stale) Mirror 9: 0x54b59ea (stale) CRC-32 checksum value for chunk [0, 0x3941b): Mirror 1: 0xfc53b4d9 Mirror 10: 0xfc53b4d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3941b: skipped 000540 1775764207.988080 mapread 0x001c5e thru 0x010731 (0xead4 bytes) 000541 1775764208.058428 fallocate 0x00eb8c thru 0x01819f (0x9614 bytes) 000543 1775764208.099804 read 0x02aa15 thru 0x03941a (0xea06 bytes) 000544 1775764208.112091 trunc from 0x03941b to 0x03489d (0x4b7d bytes) 000545 1775764208.165522 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3489e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3489e): Mirror 4: 0x47f3d1c5 (stale) Mirror 5: 0x47f3d1c5 (stale) Mirror 6: 0x47f3d1c5 (stale) Mirror 8: 0x47f3d1c5 (stale) Mirror 9: 0x47f3d1c5 (stale) Mirror 10: 0x6caa5da2 (stale) CRC-32 checksum value for chunk [0, 0x3489e): Mirror 1: 0x6caa5da2 Mirror 11: 0x6caa5da2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3489e: skipped 000546 1775764209.848698 write 0x00b59d thru 0x014e2d (0x9891 bytes) 000547 1775764209.962346 mapread 0x00d22c thru 0x018d43 (0xbb18 bytes) 000548 1775764209.996981 trunc from 0x03489e to 0x0110ed (0x237b0 bytes) 000549 1775764210.056762 trunc from 0x010e3c to 0x0110ed (0x2b2 bytes) 000551 1775764210.073916 trunc from 0x0110ee to 0x00c6fb (0x49f2 bytes) 000552 1775764210.108616 punch from 0x005313 to 0x00c6fb (0x73e9 bytes) 000553 1775764210.121123 trunc from 0x009183 to 0x00c6fb (0x3579 bytes) 000555 1775764210.131457 read 0x0029ed thru 0x00a590 (0x7ba4 bytes) 000556 1775764210.139920 trunc from 0x00c6fc to 0x005aa4 (0x6c57 bytes) 000557 1775764210.161539 trunc from 0x003574 to 0x005aa4 (0x2531 bytes) 000559 1775764210.170058 mapwrite 0x009160 thru 0x009e36 (0xcd7 bytes) 000560 1775764210.227490 fallocate 0x006113 thru 0x007f1c (0x1e0a bytes) 000561 1775764210.235862 punch from 0x00633f to 0x009506 (0x31c8 bytes) 000562 1775764210.243326 trunc from 0x009e37 to 0x024dcd (0x1af97 bytes) 000563 1775764210.276903 write 0x037be8 thru 0x03ffff (0x8418 bytes) 000564 1775764210.281753 write 0x00a434 thru 0x0192fd (0xeeca bytes) 000565 1775764210.312031 MIRROR_RESYNC 8 mirrors 000566 1775764213.752058 trunc from 0x0198c6 to 0x01c296 (0x29d1 bytes) 000567 1775764213.792985 mapwrite 0x03f1c9 thru 0x03ffff (0xe37 bytes) 000568 1775764213.813178 trunc from 0x040000 to 0x01f711 (0x208ee bytes) 000569 1775764213.839416 mapread 0x00d82e thru 0x01d251 (0xfa24 bytes) 000570 1775764213.869445 read 0x000950 thru 0x0059d4 (0x5085 bytes) 000571 1775764213.879272 fallocate 0x029cd7 thru 0x02c6cd (0x29f7 bytes) 000572 1775764213.885462 mapwrite 0x00d261 thru 0x01d0bf (0xfe5f bytes) 000573 1775764213.928444 fallocate 0x00a9a0 thru 0x00fd07 (0x5368 bytes) 000574 1775764213.935346 read 0x00e655 thru 0x01b8a3 (0xd24f bytes) 000575 1775764213.940821 mapread 0x00abdf thru 0x010334 (0x5756 bytes) 000576 1775764213.943204 punch from 0x005749 to 0x00c59b (0x6e53 bytes) 000577 1775764213.957766 trunc from 0x02c6ce to 0x010533 (0x1c19a bytes) 000578 1775764213.986925 trunc from 0x00020c to 0x00d527 (0xd31c bytes) 000579 1775764213.996052 trunc from 0x004c1a to 0x00eb7a (0x9f61 bytes) 000580 1775764214.007658 fallocate 0x01c3ef thru 0x021281 (0x4e93 bytes) 000581 1775764214.020900 trunc from 0x010534 to 0x010928 (0x3f5 bytes) 000582 1775764214.053285 mapwrite 0x03072f thru 0x03ee74 (0xe746 bytes) 000583 1775764214.136799 trunc from 0x033f8f to 0x03ee74 (0xaee6 bytes) 000584 1775764214.143098 read 0x02d232 thru 0x03c7a7 (0xf576 bytes) 000585 1775764214.155023 write 0x005539 thru 0x010242 (0xad0a bytes) 000586 1775764214.157685 punch from 0x03adc6 to 0x03ee74 (0x40af bytes) 000587 1775764214.167490 punch from 0x020eeb to 0x02a498 (0x95ae bytes) 000588 1775764214.181527 read 0x03804c thru 0x03ee74 (0x6e29 bytes) 000589 1775764214.206475 trunc from 0x03ee75 to 0x0102b0 (0x2ebc4 bytes) 000590 1775764214.242939 mapwrite 0x03b7b2 thru 0x03cd29 (0x1578 bytes) 000591 1775764214.379833 mapread 0x02dc42 thru 0x03bae3 (0xdea2 bytes) 000592 1775764214.465529 fallocate 0x01aab5 thru 0x027516 (0xca62 bytes) 000593 1775764214.473279 read 0x028d2e thru 0x02e619 (0x58ec bytes) 000594 1775764214.509472 mapwrite 0x029acf thru 0x02cc97 (0x31c9 bytes) 000595 1775764214.549288 read 0x024bad thru 0x0319e6 (0xce3a bytes) 000598 1775764214.573902 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd2a): Mirror 5: 0x9d479fb7 (stale) Mirror 6: 0x9d479fb7 (stale) Mirror 8: 0x9d479fb7 (stale) Mirror 9: 0x9d479fb7 (stale) Mirror 10: 0x9d479fb7 (stale) Mirror 11: 0x9d479fb7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd2a): Mirror 5: 0x9d479fb7 (stale) Mirror 6: 0x9d479fb7 (stale) Mirror 8: 0x9d479fb7 (stale) Mirror 9: 0x9d479fb7 (stale) Mirror 10: 0x9d479fb7 (stale) Mirror 11: 0x9d479fb7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd2a: skipped 000599 1775764215.602043 mapwrite 0x00fe2e thru 0x01bf49 (0xc11c bytes) 000600 1775764215.697694 mapwrite 0x000473 thru 0x0030c8 (0x2c56 bytes) 000601 1775764215.760330 trunc from 0x03cd2a to 0x008626 (0x34703 bytes) 000602 1775764215.800273 MIRROR_RESYNC 7 mirrors 000603 1775764216.939210 fallocate 0x02b8ef thru 0x0366cc (0xadde bytes) 000604 1775764216.984842 trunc from 0x008627 to 0x0207bb (0x18195 bytes) 000605 1775764217.019486 punch from 0x017a2e to 0x01f7bf (0x7d92 bytes) 000606 1775764217.030476 trunc from 0x0207bc to 0x026e91 (0x66d6 bytes) 000607 1775764217.066692 mapwrite 0x024277 thru 0x031ed5 (0xdc5f bytes) 000608 1775764217.196920 write 0x01a388 thru 0x028c19 (0xe892 bytes) 000609 1775764217.211242 trunc from 0x031ed6 to 0x02be2a (0x60ab bytes) 000610 1775764217.243189 read 0x027096 thru 0x02b677 (0x45e2 bytes) 000611 1775764217.243947 punch from 0x02a7f7 to 0x02be2a (0x1634 bytes) 000612 1775764217.273836 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2be2b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be2b): Mirror 5: 0x3a43f9c (stale) Mirror 6: 0x3a43f9c (stale) Mirror 8: 0x3a43f9c (stale) Mirror 9: 0x3a43f9c (stale) Mirror 10: 0x3a43f9c (stale) Mirror 11: 0x3a43f9c (stale) CRC-32 checksum value for chunk [0, 0x2be2b): Mirror 1: 0xdd687687 Mirror 12: 0xdd687687 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be2b: skipped 000613 1775764218.765941 write 0x03c6aa thru 0x03ffff (0x3956 bytes) 000614 1775764218.849635 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xec933e9 (stale) Mirror 6: 0xec933e9 (stale) Mirror 8: 0xec933e9 (stale) Mirror 9: 0xec933e9 (stale) Mirror 10: 0xec933e9 (stale) Mirror 11: 0xec933e9 (stale) Mirror 12: 0xaca193a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7347481b Mirror 13: 0x7347481b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000615 1775764220.281765 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xec933e9 (stale) Mirror 8: 0xec933e9 (stale) Mirror 9: 0xec933e9 (stale) Mirror 10: 0xec933e9 (stale) Mirror 11: 0xec933e9 (stale) Mirror 12: 0xaca193a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xec933e9 (stale) Mirror 8: 0xec933e9 (stale) Mirror 9: 0xec933e9 (stale) Mirror 10: 0xec933e9 (stale) Mirror 11: 0xec933e9 (stale) Mirror 12: 0xaca193a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7347481b Mirror 13: 0x7347481b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000616 1775764221.441769 mapread 0x01f0ae thru 0x02b36e (0xc2c1 bytes) 000617 1775764221.601224 read 0x033908 thru 0x0392f3 (0x59ec bytes) 000618 1775764221.661429 write 0x010f52 thru 0x018140 (0x71ef bytes) 000619 1775764221.812883 trunc from 0x0315c0 to 0x039904 (0x8345 bytes) 000620 1775764221.817991 trunc from 0x03d1df to 0x03ffff (0x2e21 bytes) 000621 1775764221.854741 fallocate 0x0121df thru 0x012c05 (0xa27 bytes) 000624 1775764221.872444 mapread 0x010990 thru 0x01d144 (0xc7b5 bytes) 000625 1775764221.902506 trunc from 0x03f120 to 0x03ffff (0xee0 bytes) 000626 1775764221.913932 read 0x00a0ac thru 0x014328 (0xa27d bytes) 000627 1775764221.923328 trunc from 0x00a760 to 0x014387 (0x9c28 bytes) 000628 1775764221.964010 mapread 0x001d9c thru 0x003df6 (0x205b bytes) 000629 1775764221.979531 punch from 0x00a6c5 to 0x01703a (0xc976 bytes) 000630 1775764221.985604 trunc from 0x00acfb to 0x00fb13 (0x4e19 bytes) 000631 1775764221.996702 read 0x001ef0 thru 0x008548 (0x6659 bytes) 000634 1775764222.024273 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xec933e9 (stale) Mirror 8: 0xec933e9 (stale) Mirror 9: 0xec933e9 (stale) Mirror 10: 0xec933e9 (stale) Mirror 11: 0xec933e9 (stale) Mirror 12: 0xaca193a6 (stale) Mirror 13: 0x7347481b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa8b6b822 Mirror 14: 0xa8b6b822 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1775764224.304654 mapwrite 0x024369 thru 0x0283b4 (0x404c bytes) 000636 1775764224.443408 write 0x002839 thru 0x009029 (0x67f1 bytes) 000637 1775764224.462783 trunc from 0x040000 to 0x034a1f (0xb5e0 bytes) 000638 1775764224.492390 mapread 0x01f037 thru 0x02b32b (0xc2f5 bytes) 000639 1775764224.495696 write 0x023f23 thru 0x02bf8b (0x8069 bytes) 000640 1775764224.512793 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a20): Mirror 5: 0x4624600c (stale) Mirror 8: 0x4624600c (stale) Mirror 9: 0x4624600c (stale) Mirror 10: 0x4624600c (stale) Mirror 11: 0x4624600c (stale) Mirror 13: 0x1bb001dc (stale) Mirror 14: 0x7f2e9e07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a20): Mirror 5: 0x4624600c (stale) Mirror 8: 0x4624600c (stale) Mirror 9: 0x4624600c (stale) Mirror 10: 0x4624600c (stale) Mirror 11: 0x4624600c (stale) Mirror 13: 0x1bb001dc (stale) Mirror 14: 0x7f2e9e07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a20: skipped 000641 1775764225.189231 mapread 0x004059 thru 0x009b17 (0x5abf bytes) 000643 1775764225.219701 trunc from 0x02b154 to 0x034a1f (0x98cc bytes) 000644 1775764225.227720 read 0x01d1f9 thru 0x0227a4 (0x55ac bytes) 000645 1775764225.239108 write 0x000f6f thru 0x00e207 (0xd299 bytes) 000647 1775764225.260104 read 0x031628 thru 0x034a1f (0x33f8 bytes) 000648 1775764225.284722 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a20): Mirror 5: 0x4624600c (stale) Mirror 8: 0x4624600c (stale) Mirror 9: 0x4624600c (stale) Mirror 11: 0x4624600c (stale) Mirror 13: 0x1bb001dc (stale) Mirror 14: 0x7f2e9e07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a20): Mirror 5: 0x4624600c (stale) Mirror 8: 0x4624600c (stale) Mirror 9: 0x4624600c (stale) Mirror 11: 0x4624600c (stale) Mirror 13: 0x1bb001dc (stale) Mirror 14: 0x7f2e9e07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a20: skipped 000650 1775764226.026371 fallocate 0x031224 thru 0x039f96 (0x8d73 bytes) 000651 1775764226.046714 read 0x007851 thru 0x01010c (0x88bc bytes) 000653 1775764226.070830 fallocate 0x0362e6 thru 0x03add7 (0x4af2 bytes) 000654 1775764226.089695 trunc from 0x01b17d to 0x02285f (0x76e3 bytes) 000655 1775764226.108225 mapread 0x009438 thru 0x013862 (0xa42b bytes) 000656 1775764226.162425 mapwrite 0x033027 thru 0x03ffff (0xcfd9 bytes) 000658 1775764226.360467 read 0x02bc37 thru 0x032996 (0x6d60 bytes) 000659 1775764226.403162 mapwrite 0x00cf3f thru 0x01c871 (0xf933 bytes) 000660 1775764226.463111 read 0x02c1a1 thru 0x02dbf9 (0x1a59 bytes) 000661 1775764226.463793 read 0x01b0d5 thru 0x02384b (0x8777 bytes) 000662 1775764226.464421 fallocate 0x00b473 thru 0x017544 (0xc0d2 bytes) 000663 1775764226.482590 fallocate 0x03edfa thru 0x03ffff (0x1206 bytes) 000664 1775764226.491533 trunc from 0x040000 to 0x00b937 (0x346c8 bytes) 000665 1775764226.537250 punch from 0x00b162 to 0x00b937 (0x7d6 bytes) 000666 1775764226.549520 write 0x037296 thru 0x03ffff (0x8d6a bytes) 000667 1775764226.553369 mapread 0x01f405 thru 0x026ac1 (0x76bd bytes) 000668 1775764226.567739 mapwrite 0x01dde0 thru 0x02cdfa (0xf01b bytes) 000669 1775764226.610423 mapwrite 0x016d5d thru 0x01a562 (0x3806 bytes) 000670 1775764226.653470 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xec933e9 (stale) Mirror 8: 0xec933e9 (stale) Mirror 9: 0xec933e9 (stale) Mirror 11: 0xec933e9 (stale) Mirror 13: 0x7347481b (stale) Mirror 14: 0xa8b6b822 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3b05866 Mirror 15: 0x3b05866 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000673 1775764228.218465 write 0x0011e9 thru 0x00c8ef (0xb707 bytes) 000674 1775764228.312931 read 0x0379ea thru 0x03ffff (0x8616 bytes) 000676 1775764228.331878 punch from 0x016c37 to 0x0193d6 (0x27a0 bytes) 000677 1775764228.344473 fallocate 0x03e3dc thru 0x03ffff (0x1c24 bytes) 000678 1775764228.354103 mapwrite 0x015e0b thru 0x023143 (0xd339 bytes) 000679 1775764228.467779 read 0x0337ce thru 0x03ed5b (0xb58e bytes) 000680 1775764228.495709 MIRROR_RESYNC 8 mirrors 000681 1775764230.868037 mapread 0x01cbde thru 0x022bf7 (0x601a bytes) 000682 1775764230.893530 mapread 0x034c1d thru 0x03ffff (0xb3e3 bytes) 000683 1775764230.929208 fallocate 0x000f50 thru 0x0092c0 (0x8371 bytes) 000684 1775764230.973206 mapwrite 0x03cd46 thru 0x03ffff (0x32ba bytes) 000685 1775764231.017607 mapread 0x02fd20 thru 0x03d750 (0xda31 bytes) 000686 1775764231.042824 mapwrite 0x01fba7 thru 0x02b3e0 (0xb83a bytes) 000688 1775764231.096844 write 0x027698 thru 0x02d6b4 (0x601d bytes) 000689 1775764231.100178 mapwrite 0x0379f6 thru 0x03e9a4 (0x6faf bytes) 000690 1775764231.142292 mapwrite 0x0138a1 thru 0x01918d (0x58ed bytes) 000691 1775764231.216011 trunc from 0x01756d to 0x01830c (0xda0 bytes) 000692 1775764231.223136 write 0x038282 thru 0x03dcfd (0x5a7c bytes) 000693 1775764231.225917 mapwrite 0x00589b thru 0x008219 (0x297f bytes) 000694 1775764231.261270 mapwrite 0x038a33 thru 0x03b15b (0x2729 bytes) 000696 1775764231.290657 MIRROR_RESYNC 8 mirrors 000697 1775764234.189275 read 0x020834 thru 0x022149 (0x1916 bytes) 000698 1775764234.228594 read 0x02cdeb thru 0x039b56 (0xcd6c bytes) 000699 1775764234.261005 trunc from 0x035fee to 0x03a8be (0x48d1 bytes) 000700 1775764234.344042 punch from 0x02b1a3 to 0x030f55 (0x5db3 bytes) 000702 1775764234.359880 punch from 0x02a94c to 0x02d2b3 (0x2968 bytes) 000703 1775764234.374997 trunc from 0x02e6ab to 0x036cfd (0x8653 bytes) 000704 1775764234.390620 mapwrite 0x00d5b7 thru 0x01b9b9 (0xe403 bytes) 000705 1775764234.564731 mapread 0x018da1 thru 0x01c137 (0x3397 bytes) 000706 1775764234.571809 trunc from 0x040000 to 0x004b27 (0x3b4d8 bytes) 000708 1775764234.609702 mapwrite 0x00c7e8 thru 0x019fee (0xd807 bytes) 000709 1775764234.766294 mapread 0x0134a9 thru 0x0187ec (0x5344 bytes) 000710 1775764234.767496 write 0x01204e thru 0x01678a (0x473d bytes) 000711 1775764234.772528 fallocate 0x03d23c thru 0x03ffff (0x2dc4 bytes) 000712 1775764234.780159 trunc from 0x014c80 to 0x019fee (0x536f bytes) 000713 1775764234.806211 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x19fef): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19fef): Mirror 5: 0xb17843de (stale) Mirror 8: 0xb17843de (stale) Mirror 9: 0xb17843de (stale) Mirror 11: 0xb17843de (stale) Mirror 13: 0xb17843de (stale) Mirror 14: 0xb17843de (stale) Mirror 15: 0xb17843de (stale) CRC-32 checksum value for chunk [0, 0x19fef): Mirror 1: 0x108325f Mirror 16: 0x108325f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19fef: skipped 000714 1775764237.149299 write 0x03732b thru 0x03ffff (0x8cd5 bytes) 000715 1775764237.325576 write 0x03e4d6 thru 0x03ffff (0x1b2a bytes) 000716 1775764237.351876 trunc from 0x040000 to 0x01fad7 (0x20528 bytes) 000717 1775764237.448494 trunc from 0x0021b4 to 0x005152 (0x2f9f bytes) 000718 1775764237.466702 mapread 0x01f734 thru 0x01fad7 (0x3a4 bytes) 000719 1775764237.499721 trunc from 0x01fad8 to 0x00243b (0x1d69c bytes) 000720 1775764237.546438 trunc from 0x000307 to 0x0015a3 (0x129d bytes) 000721 1775764237.568430 punch from 0x001255 to 0x00243b (0x11e7 bytes) 000722 1775764237.577290 write 0x039e66 thru 0x03a84f (0x9ea bytes) 000723 1775764237.578699 fallocate 0x01dfc9 thru 0x0225fc (0x4634 bytes) 000724 1775764237.600969 read 0x01ca9e thru 0x024e25 (0x8388 bytes) 000725 1775764237.629636 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a850): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a850): Mirror 5: 0xdf885f70 (stale) Mirror 8: 0xdf885f70 (stale) Mirror 9: 0xdf885f70 (stale) Mirror 11: 0xdf885f70 (stale) Mirror 13: 0xdf885f70 (stale) Mirror 14: 0xdf885f70 (stale) Mirror 15: 0xdf885f70 (stale) Mirror 16: 0x7e6555c9 (stale) CRC-32 checksum value for chunk [0, 0x3a850): Mirror 1: 0x4504248 Mirror 17: 0x4504248 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a850: skipped 000726 1775764239.805536 trunc from 0x016711 to 0x01e487 (0x7d77 bytes) 000727 1775764239.914245 punch from 0x0398ca to 0x03a84f (0xf86 bytes) 000728 1775764239.939228 mapwrite 0x035de9 thru 0x03ffff (0xa217 bytes) 000729 1775764240.185258 trunc from 0x040000 to 0x024909 (0x1b6f6 bytes) 000730 1775764240.225369 write 0x03d32a thru 0x03ffff (0x2cd6 bytes) 000734 1775764240.235440 mapwrite 0x005875 thru 0x008701 (0x2e8d bytes) 000735 1775764240.316741 mapwrite 0x01c527 thru 0x02206e (0x5b48 bytes) 000736 1775764240.390122 punch from 0x01e351 to 0x028301 (0x9fb1 bytes) 000738 1775764240.406399 fallocate 0x0351fd thru 0x03ffff (0xae03 bytes) 000739 1775764240.436679 mapread 0x014915 thru 0x018001 (0x36ed bytes) 000740 1775764240.459086 mapwrite 0x024491 thru 0x0300f9 (0xbc69 bytes) 000741 1775764240.520149 punch from 0x02be2f to 0x02d3b1 (0x1583 bytes) 000742 1775764240.527933 write 0x027c17 thru 0x037667 (0xfa51 bytes) 000743 1775764240.535265 mapwrite 0x039029 thru 0x03ffff (0x6fd7 bytes) 000744 1775764240.594930 trunc from 0x040000 to 0x012b9b (0x2d464 bytes) 000745 1775764240.641006 trunc from 0x0032e6 to 0x0129bd (0xf6d8 bytes) 000746 1775764240.650252 write 0x009d51 thru 0x00cb68 (0x2e18 bytes) 000748 1775764240.676803 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x12b9c): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12b9c): Mirror 5: 0x5ef08863 (stale) Mirror 8: 0x5ef08863 (stale) Mirror 9: 0x5ef08863 (stale) Mirror 11: 0x5ef08863 (stale) Mirror 13: 0x5ef08863 (stale) Mirror 14: 0x5ef08863 (stale) Mirror 15: 0x5ef08863 (stale) Mirror 16: 0xe219983c (stale) Mirror 17: 0x684fbb24 (stale) CRC-32 checksum value for chunk [0, 0x12b9c): Mirror 1: 0x26c56aa4 Mirror 18: 0x26c56aa4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12b9c: skipped 000749 1775764242.971810 fallocate 0x01631d thru 0x024557 (0xe23b bytes) 000751 1775764243.056034 mapread 0x00de3a thru 0x012b9b (0x4d62 bytes) 000752 1775764243.133827 trunc from 0x012b9c to 0x006939 (0xc262 bytes) 000753 1775764243.188377 punch from 0x003855 to 0x003cb8 (0x464 bytes) 000754 1775764243.208030 trunc from 0x00693a to 0x011d8c (0xb453 bytes) 000756 1775764243.305455 fallocate 0x03e9f8 thru 0x03ffff (0x1608 bytes) 000757 1775764243.335331 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5425fc4b (stale) Mirror 9: 0x5425fc4b (stale) Mirror 11: 0x5425fc4b (stale) Mirror 13: 0x5425fc4b (stale) Mirror 14: 0x5425fc4b (stale) Mirror 15: 0x5425fc4b (stale) Mirror 16: 0x7961eb35 (stale) Mirror 17: 0xa0cad54 (stale) Mirror 18: 0x77c04b9b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5425fc4b (stale) Mirror 9: 0x5425fc4b (stale) Mirror 11: 0x5425fc4b (stale) Mirror 13: 0x5425fc4b (stale) Mirror 14: 0x5425fc4b (stale) Mirror 15: 0x5425fc4b (stale) Mirror 16: 0x7961eb35 (stale) Mirror 17: 0xa0cad54 (stale) Mirror 18: 0x77c04b9b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000758 1775764244.784706 trunc from 0x040000 to 0x00e85c (0x317a3 bytes) 000759 1775764244.816307 mapread 0x0040d3 thru 0x00ae2f (0x6d5d bytes) 000760 1775764244.841994 read 0x002f1b thru 0x0064a3 (0x3589 bytes) 000761 1775764244.843673 trunc from 0x00e85d to 0x025f59 (0x176fd bytes) 000762 1775764244.866009 trunc from 0x01eeef to 0x025f59 (0x706b bytes) 000763 1775764244.872156 mapread 0x019c97 thru 0x02174b (0x7ab5 bytes) 000765 1775764244.884965 trunc from 0x00108d to 0x002b77 (0x1aeb bytes) 000766 1775764244.890666 trunc from 0x003ce1 to 0x00d6f6 (0x9a16 bytes) 000767 1775764244.897067 mapread 0x002f31 thru 0x00d7b4 (0xa884 bytes) 000768 1775764244.915527 read 0x017dc1 thru 0x01b4cb (0x370b bytes) 000771 1775764244.919261 read 0x005235 thru 0x013967 (0xe733 bytes) 000772 1775764244.933525 mapread 0x00ca53 thru 0x00eeaa (0x2458 bytes) 000773 1775764244.935633 mapwrite 0x0051d7 thru 0x010aed (0xb917 bytes) 000774 1775764244.977064 read 0x008a0f thru 0x010aef (0x80e1 bytes) 000775 1775764244.981263 trunc from 0x025f5a to 0x002542 (0x23a17 bytes) 000776 1775764245.029901 punch from 0x0005bf to 0x001e63 (0x18a5 bytes) 000779 1775764245.044635 punch from 0x00237d to 0x002542 (0x1c6 bytes) 000780 1775764245.051809 mapread 0x00180a thru 0x002542 (0xd39 bytes) 000781 1775764245.063058 trunc from 0x002543 to 0x00443a (0x1ef8 bytes) 000782 1775764245.088241 trunc from 0x00443b to 0x016b48 (0x1270e bytes) 000783 1775764245.175819 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x16b49): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16b49): Mirror 5: 0x20425493 (stale) Mirror 9: 0x20425493 (stale) Mirror 11: 0x20425493 (stale) Mirror 13: 0x20425493 (stale) Mirror 14: 0x20425493 (stale) Mirror 15: 0x20425493 (stale) Mirror 16: 0x3c06a341 (stale) Mirror 17: 0xee531730 (stale) Mirror 18: 0xa22727e6 (stale) CRC-32 checksum value for chunk [0, 0x16b49): Mirror 1: 0xee531730 Mirror 19: 0xee531730 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16b49: skipped 000785 1775764247.818328 trunc from 0x016b49 to 0x028462 (0x1191a bytes) 000786 1775764247.890153 mapwrite 0x03bc5a thru 0x03ffff (0x43a6 bytes) 000787 1775764248.000418 mapwrite 0x03021b thru 0x031d80 (0x1b66 bytes) 000788 1775764248.061119 trunc from 0x040000 to 0x032fca (0xd035 bytes) 000789 1775764248.118295 mapwrite 0x02d2b5 thru 0x039e32 (0xcb7e bytes) 000790 1775764248.267494 MIRROR_RESYNC 11 mirrors 000791 1775764251.259267 MIRROR_RESYNC 11 mirrors 000793 1775764252.639309 trunc from 0x039e33 to 0x01890c (0x21526 bytes) 000794 1775764252.760622 punch from 0x00cad7 to 0x01890c (0xbe36 bytes) 000795 1775764252.766908 read 0x0043ae thru 0x005be8 (0x183b bytes) 000796 1775764252.779349 fallocate 0x029683 thru 0x033218 (0x9b96 bytes) 000799 1775764252.799315 fallocate 0x0243d9 thru 0x02d295 (0x8ebd bytes) 000800 1775764252.813577 trunc from 0x033219 to 0x031aca (0x174e bytes) 000801 1775764252.920154 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31acb): Mirror 5: 0x9e21181a (stale) Mirror 9: 0x9e21181a (stale) Mirror 11: 0x9e21181a (stale) Mirror 13: 0x9e21181a (stale) Mirror 14: 0x9e21181a (stale) Mirror 15: 0x9e21181a (stale) Mirror 16: 0x9e21181a (stale) Mirror 17: 0x9e21181a (stale) Mirror 19: 0x9e21181a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31acb): Mirror 5: 0x9e21181a (stale) Mirror 9: 0x9e21181a (stale) Mirror 11: 0x9e21181a (stale) Mirror 13: 0x9e21181a (stale) Mirror 14: 0x9e21181a (stale) Mirror 15: 0x9e21181a (stale) Mirror 16: 0x9e21181a (stale) Mirror 17: 0x9e21181a (stale) Mirror 19: 0x9e21181a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31acb: skipped 000802 1775764253.826056 read 0x01aeeb thru 0x01af55 (0x6b bytes) 000803 1775764253.832876 read 0x01cf3d thru 0x026e06 (0x9eca bytes) 000804 1775764253.848276 read 0x015897 thru 0x024afc (0xf266 bytes) 000805 1775764253.849170 fallocate 0x006868 thru 0x00ff55 (0x96ee bytes) 000807 1775764253.857245 mapwrite 0x01c1d0 thru 0x021577 (0x53a8 bytes) 000809 1775764253.889067 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x31acb): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31acb): Mirror 5: 0x9e21181a (stale) Mirror 9: 0x9e21181a (stale) Mirror 11: 0x9e21181a (stale) Mirror 13: 0x9e21181a (stale) Mirror 14: 0x9e21181a (stale) Mirror 15: 0x9e21181a (stale) Mirror 16: 0x9e21181a (stale) Mirror 17: 0x9e21181a (stale) Mirror 19: 0x9e21181a (stale) CRC-32 checksum value for chunk [0, 0x31acb): Mirror 1: 0xf701ff5b Mirror 20: 0xf701ff5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31acb: skipped 000810 1775764255.594270 fallocate 0x02201b thru 0x0261c6 (0x41ac bytes) 000811 1775764255.702480 read 0x01dddf thru 0x0298cb (0xbaed bytes) 000812 1775764255.745501 fallocate 0x035862 thru 0x035aa5 (0x244 bytes) 000813 1775764255.765772 trunc from 0x00734e to 0x0122b5 (0xaf68 bytes) 000814 1775764255.771739 write 0x006cbf thru 0x00c096 (0x53d8 bytes) 000815 1775764255.810369 trunc from 0x031acb to 0x03c9f1 (0xaf27 bytes) 000816 1775764255.836143 trunc from 0x03c9f2 to 0x03528e (0x7763 bytes) 000817 1775764255.866172 read 0x00db48 thru 0x01ba36 (0xdeef bytes) 000818 1775764255.886157 write 0x025f92 thru 0x02bebd (0x5f2c bytes) 000819 1775764255.889391 read 0x02f4d6 thru 0x02fd0c (0x837 bytes) 000820 1775764255.890280 read 0x0256c6 thru 0x02bf3b (0x6876 bytes) 000821 1775764255.896847 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3528f): Mirror 5: 0xde6473ab (stale) Mirror 9: 0xde6473ab (stale) Mirror 13: 0xde6473ab (stale) Mirror 14: 0xde6473ab (stale) Mirror 15: 0xde6473ab (stale) Mirror 16: 0xde6473ab (stale) Mirror 17: 0xde6473ab (stale) Mirror 19: 0xde6473ab (stale) Mirror 20: 0xa3f85015 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3528f): Mirror 5: 0xde6473ab (stale) Mirror 9: 0xde6473ab (stale) Mirror 13: 0xde6473ab (stale) Mirror 14: 0xde6473ab (stale) Mirror 15: 0xde6473ab (stale) Mirror 16: 0xde6473ab (stale) Mirror 17: 0xde6473ab (stale) Mirror 19: 0xde6473ab (stale) Mirror 20: 0xa3f85015 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3528f: skipped 000822 1775764257.108277 punch from 0x01cefa to 0x0214a8 (0x45af bytes) 000823 1775764257.120815 fallocate 0x034f35 thru 0x03ffff (0xb0cb bytes) 000824 1775764257.137123 mapwrite 0x03606c thru 0x037877 (0x180c bytes) 000825 1775764257.203964 fallocate 0x02d896 thru 0x0317d8 (0x3f43 bytes) 000826 1775764257.222231 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6485ab83 (stale) Mirror 9: 0x6485ab83 (stale) Mirror 13: 0x6485ab83 (stale) Mirror 14: 0x6485ab83 (stale) Mirror 15: 0x6485ab83 (stale) Mirror 16: 0x6485ab83 (stale) Mirror 17: 0x6485ab83 (stale) Mirror 19: 0x6485ab83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6485ab83 (stale) Mirror 9: 0x6485ab83 (stale) Mirror 13: 0x6485ab83 (stale) Mirror 14: 0x6485ab83 (stale) Mirror 15: 0x6485ab83 (stale) Mirror 16: 0x6485ab83 (stale) Mirror 17: 0x6485ab83 (stale) Mirror 19: 0x6485ab83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000827 1775764258.012523 punch from 0x023c29 to 0x031a0a (0xdde2 bytes) 000828 1775764258.035146 mapread 0x00e2c1 thru 0x01603f (0x7d7f bytes) 000829 1775764258.073188 trunc from 0x040000 to 0x03ccf3 (0x330c bytes) 000830 1775764258.128437 write 0x010406 thru 0x01f31b (0xef16 bytes) 000831 1775764258.140863 mapread 0x015602 thru 0x023ed2 (0xe8d1 bytes) 000832 1775764258.171179 read 0x00f1b3 thru 0x01172b (0x2579 bytes) 000833 1775764258.171919 write 0x0315f8 thru 0x0362f2 (0x4cfb bytes) 000834 1775764258.196759 mapwrite 0x0108d0 thru 0x01fd07 (0xf438 bytes) 000835 1775764258.264533 mapwrite 0x0267f6 thru 0x02ed32 (0x853d bytes) 000836 1775764258.349335 trunc from 0x01d382 to 0x0243fe (0x707d bytes) 000837 1775764258.360157 write 0x003209 thru 0x013126 (0xff1e bytes) 000838 1775764258.432402 MIRROR_RESYNC 9 mirrors 000840 1775764262.620357 punch from 0x02bcb2 to 0x0374e7 (0xb836 bytes) 000842 1775764262.695285 mapread 0x01d67e thru 0x021517 (0x3e9a bytes) 000843 1775764262.735468 MIRROR_RESYNC 9 mirrors 000844 1775764266.385903 fallocate 0x004cda thru 0x005f83 (0x12aa bytes) 000845 1775764266.428828 punch from 0x01cfe1 to 0x0204da (0x34fa bytes) 000846 1775764266.435946 fallocate 0x00ae67 thru 0x015444 (0xa5de bytes) 000847 1775764266.441342 punch from 0x008e2d to 0x00d8c2 (0x4a96 bytes) 000848 1775764266.449805 mapread 0x0286b9 thru 0x02b0af (0x29f7 bytes) 000849 1775764266.468997 write 0x038142 thru 0x03f005 (0x6ec4 bytes) 000850 1775764266.483897 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f006): Mirror 5: 0x26aa2d72 (stale) Mirror 9: 0x26aa2d72 (stale) Mirror 13: 0x26aa2d72 (stale) Mirror 14: 0x26aa2d72 (stale) Mirror 16: 0x26aa2d72 (stale) Mirror 17: 0x26aa2d72 (stale) Mirror 19: 0x26aa2d72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f006): Mirror 5: 0x26aa2d72 (stale) Mirror 9: 0x26aa2d72 (stale) Mirror 13: 0x26aa2d72 (stale) Mirror 14: 0x26aa2d72 (stale) Mirror 16: 0x26aa2d72 (stale) Mirror 17: 0x26aa2d72 (stale) Mirror 19: 0x26aa2d72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f006: skipped 000851 1775764267.393278 trunc from 0x03f006 to 0x02eff1 (0x10014 bytes) 000852 1775764267.467177 mapread 0x005857 thru 0x00bd69 (0x6513 bytes) 000853 1775764267.546347 punch from 0x0178fd to 0x0241cf (0xc8d3 bytes) 000854 1775764267.581477 fallocate 0x02f594 thru 0x03993f (0xa3ac bytes) 000855 1775764267.608424 trunc from 0x02308c to 0x02c2f2 (0x9267 bytes) 000856 1775764267.671290 mapread 0x035010 thru 0x036939 (0x192a bytes) 000857 1775764267.709334 mapwrite 0x011eea thru 0x01b00f (0x9126 bytes) 000858 1775764267.835499 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39940): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39940): Mirror 5: 0x86e32456 (stale) Mirror 9: 0x86e32456 (stale) Mirror 13: 0x86e32456 (stale) Mirror 14: 0x86e32456 (stale) Mirror 16: 0x86e32456 (stale) Mirror 17: 0x86e32456 (stale) Mirror 19: 0x86e32456 (stale) CRC-32 checksum value for chunk [0, 0x39940): Mirror 1: 0x36d034fc Mirror 20: 0x36d034fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39940: skipped 000859 1775764270.051357 read 0x0286b5 thru 0x038115 (0xfa61 bytes) 000860 1775764270.118359 mapwrite 0x03f199 thru 0x03f2f4 (0x15c bytes) 000861 1775764270.209850 write 0x0085bf thru 0x00adb4 (0x27f6 bytes) 000862 1775764270.248225 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f2f5): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2f5): Mirror 5: 0xb8717481 (stale) Mirror 9: 0xb8717481 (stale) Mirror 13: 0xb8717481 (stale) Mirror 14: 0xb8717481 (stale) Mirror 16: 0xb8717481 (stale) Mirror 17: 0xb8717481 (stale) Mirror 19: 0xb8717481 (stale) Mirror 20: 0x646e4e3c (stale) CRC-32 checksum value for chunk [0, 0x3f2f5): Mirror 1: 0x9ac27a32 Mirror 21: 0x9ac27a32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2f5: skipped 000863 1775764272.119401 trunc from 0x036c27 to 0x03f2f4 (0x86ce bytes) 000864 1775764272.189324 trunc from 0x027a86 to 0x0320a7 (0xa622 bytes) 000865 1775764272.200318 punch from 0x01b290 to 0x029768 (0xe4d9 bytes) 000866 1775764272.211726 punch from 0x030feb to 0x03d835 (0xc84b bytes) 000867 1775764272.217131 mapwrite 0x038ca7 thru 0x03ffff (0x7359 bytes) 000868 1775764272.345077 trunc from 0x040000 to 0x01e0a0 (0x21f5f bytes) 000870 1775764272.376246 read 0x00a4b9 thru 0x013f44 (0x9a8c bytes) 000871 1775764272.386530 punch from 0x0174c8 to 0x01e0a0 (0x6bd9 bytes) 000872 1775764272.398052 mapread 0x001a86 thru 0x00e93f (0xceba bytes) 000873 1775764272.427758 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e0a1): Mirror 9: 0xf4a4a140 (stale) Mirror 13: 0xf4a4a140 (stale) Mirror 14: 0xf4a4a140 (stale) Mirror 16: 0xf4a4a140 (stale) Mirror 17: 0xf4a4a140 (stale) Mirror 19: 0xf4a4a140 (stale) Mirror 20: 0xc3f70b8e (stale) Mirror 21: 0x5321a85b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e0a1): Mirror 9: 0xf4a4a140 (stale) Mirror 13: 0xf4a4a140 (stale) Mirror 14: 0xf4a4a140 (stale) Mirror 16: 0xf4a4a140 (stale) Mirror 17: 0xf4a4a140 (stale) Mirror 19: 0xf4a4a140 (stale) Mirror 20: 0xc3f70b8e (stale) Mirror 21: 0x5321a85b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e0a1: skipped 000874 1775764273.523521 punch from 0x01d99c to 0x01e0a0 (0x705 bytes) 000875 1775764273.545969 punch from 0x003b1a to 0x0136a0 (0xfb87 bytes) 000876 1775764273.575233 mapread 0x005798 thru 0x00f377 (0x9be0 bytes) 000877 1775764273.679176 trunc from 0x01e0a1 to 0x00664a (0x17a56 bytes) 000878 1775764273.738915 punch from 0x00661f to 0x00664a (0x2c bytes) 000879 1775764273.753912 read 0x00478a thru 0x00664a (0x1ec1 bytes) 000880 1775764273.768006 write 0x001535 thru 0x006cac (0x5778 bytes) 000881 1775764273.778250 trunc from 0x006ac3 to 0x006cac (0x1ea bytes) 000882 1775764273.823936 read 0x00304f thru 0x0062a5 (0x3257 bytes) 000883 1775764273.841464 trunc from 0x0039dc to 0x006cac (0x32d1 bytes) 000884 1775764273.877044 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6cad): Mirror 9: 0x9f6b4b69 (stale) Mirror 13: 0x9f6b4b69 (stale) Mirror 16: 0x9f6b4b69 (stale) Mirror 17: 0x9f6b4b69 (stale) Mirror 19: 0x9f6b4b69 (stale) Mirror 20: 0x9f6b4b69 (stale) Mirror 21: 0x9f6b4b69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6cad): Mirror 9: 0x9f6b4b69 (stale) Mirror 13: 0x9f6b4b69 (stale) Mirror 16: 0x9f6b4b69 (stale) Mirror 17: 0x9f6b4b69 (stale) Mirror 19: 0x9f6b4b69 (stale) Mirror 20: 0x9f6b4b69 (stale) Mirror 21: 0x9f6b4b69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6cad: skipped 000885 1775764274.855587 mapwrite 0x013849 thru 0x019a64 (0x621c bytes) 000886 1775764275.119323 read 0x00fa73 thru 0x015bc2 (0x6150 bytes) 000887 1775764275.125884 punch from 0x0081b0 to 0x00eca2 (0x6af3 bytes) 000888 1775764275.152322 trunc from 0x019a65 to 0x009f88 (0xfadc bytes) 000889 1775764275.209833 trunc from 0x009f89 to 0x037052 (0x2d0ca bytes) 000890 1775764275.264750 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37053): Mirror 13: 0x74bbcf6 (stale) Mirror 16: 0x74bbcf6 (stale) Mirror 17: 0x74bbcf6 (stale) Mirror 19: 0x74bbcf6 (stale) Mirror 20: 0x1bf7c99d (stale) Mirror 21: 0xe1b39b2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37053): Mirror 13: 0x74bbcf6 (stale) Mirror 16: 0x74bbcf6 (stale) Mirror 17: 0x74bbcf6 (stale) Mirror 19: 0x74bbcf6 (stale) Mirror 20: 0x1bf7c99d (stale) Mirror 21: 0xe1b39b2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37053: skipped 000891 1775764276.296618 punch from 0x001aca to 0x005733 (0x3c6a bytes) 000892 1775764276.329974 write 0x03daee thru 0x03ffff (0x2512 bytes) 000893 1775764276.389985 trunc from 0x001f12 to 0x004031 (0x2120 bytes) 000895 1775764276.417339 trunc from 0x040000 to 0x0007af (0x3f850 bytes) 000896 1775764276.484372 read 0x0006d7 thru 0x0007af (0xd9 bytes) 000897 1775764276.500880 mapread 0x00005e thru 0x0007af (0x752 bytes) 000898 1775764276.502421 read 0x000175 thru 0x0007af (0x63b bytes) 000899 1775764276.507849 trunc from 0x000671 to 0x0007af (0x13f bytes) 000900 1775764276.517438 mapread 0x000067 thru 0x0007af (0x749 bytes) 000901 1775764276.527620 read 0x00003e thru 0x0007af (0x772 bytes) 000902 1775764276.529042 write 0x013b92 thru 0x01ae3b (0x72aa bytes) 000903 1775764276.532233 read 0x0081e3 thru 0x009a2d (0x184b bytes) 000904 1775764276.547174 trunc from 0x01ae3c to 0x02ac2a (0xfdef bytes) 000905 1775764276.573791 fallocate 0x009381 thru 0x01041e (0x709e bytes) 000906 1775764276.588786 mapread 0x0022cb thru 0x00881b (0x6551 bytes) 000907 1775764276.590045 mapread 0x00c087 thru 0x0154bf (0x9439 bytes) 000908 1775764276.602190 MIRROR_RESYNC 7 mirrors 000909 1775764280.143073 punch from 0x01cf64 to 0x02684f (0x98ec bytes) 000910 1775764280.205527 mapread 0x020a3c thru 0x02ac2a (0xa1ef bytes) 000911 1775764280.246423 punch from 0x01be26 to 0x02ac2a (0xee05 bytes) 000913 1775764280.260215 fallocate 0x0359fc thru 0x03ab0c (0x5111 bytes) 000915 1775764280.279115 mapwrite 0x01edcb thru 0x02d978 (0xebae bytes) 000916 1775764280.356775 trunc from 0x00a867 to 0x016ca0 (0xc43a bytes) 000917 1775764280.363909 mapwrite 0x00be62 thru 0x0159be (0x9b5d bytes) 000918 1775764280.489480 fallocate 0x014792 thru 0x021ae5 (0xd354 bytes) 000919 1775764280.495820 trunc from 0x03ab0d to 0x0122b3 (0x28859 bytes) 000921 1775764280.538792 trunc from 0x001f4c to 0x00b79e (0x9853 bytes) 000922 1775764280.557443 fallocate 0x004c61 thru 0x014344 (0xf6e4 bytes) 000923 1775764280.573263 mapwrite 0x01386e thru 0x01d84b (0x9fde bytes) 000924 1775764280.713195 read 0x01277f thru 0x01a312 (0x7b94 bytes) 000925 1775764280.713952 punch from 0x00ef1b to 0x01d84b (0xe931 bytes) 000928 1775764280.733819 mapwrite 0x03af9d thru 0x03bf01 (0xf65 bytes) 000929 1775764280.864015 fallocate 0x02962b thru 0x03450e (0xaee4 bytes) 000930 1775764280.889982 trunc from 0x01cd9a to 0x02974d (0xc9b4 bytes) 000931 1775764280.905846 mapread 0x021a8c thru 0x027e05 (0x637a bytes) 000932 1775764280.942824 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf02): Mirror 13: 0x5e14a4b6 (stale) Mirror 16: 0x5e14a4b6 (stale) Mirror 17: 0x5e14a4b6 (stale) Mirror 19: 0x5e14a4b6 (stale) Mirror 20: 0x5e14a4b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf02): Mirror 13: 0x5e14a4b6 (stale) Mirror 16: 0x5e14a4b6 (stale) Mirror 17: 0x5e14a4b6 (stale) Mirror 19: 0x5e14a4b6 (stale) Mirror 20: 0x5e14a4b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf02: skipped 000933 1775764281.703110 fallocate 0x03fde0 thru 0x03ffff (0x220 bytes) 000934 1775764281.710113 write 0x012e92 thru 0x017bc5 (0x4d34 bytes) 000935 1775764281.725173 read 0x0224de thru 0x027c6f (0x5792 bytes) 000936 1775764281.737789 fallocate 0x0132e3 thru 0x0164c7 (0x31e5 bytes) 000938 1775764281.746850 trunc from 0x03bf02 to 0x02f2be (0xcc43 bytes) 000939 1775764281.763260 mapread 0x004b47 thru 0x00fd18 (0xb1d2 bytes) 000940 1775764281.784121 write 0x01ba08 thru 0x024dc3 (0x93bc bytes) 000941 1775764281.797668 trunc from 0x02f2bf to 0x02f8ca (0x60c bytes) 000942 1775764281.818036 mapwrite 0x006075 thru 0x00dc68 (0x7bf4 bytes) 000943 1775764281.865889 trunc from 0x02f8cb to 0x01ca3d (0x12e8d bytes) 000944 1775764281.909001 fallocate 0x01253b thru 0x01cba1 (0xa667 bytes) 000945 1775764281.930701 mapwrite 0x01a00c thru 0x029a9b (0xfa90 bytes) 000946 1775764282.092099 fallocate 0x03824d thru 0x03ffff (0x7db3 bytes) 000947 1775764282.127525 mapwrite 0x01022c thru 0x013eae (0x3c83 bytes) 000948 1775764282.236977 MIRROR_RESYNC 6 mirrors 000949 1775764286.618713 write 0x037ad3 thru 0x037e5c (0x38a bytes) 000950 1775764286.755322 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37e5d): Mirror 13: 0xa8bcd0dd (stale) Mirror 16: 0xa8bcd0dd (stale) Mirror 17: 0xa8bcd0dd (stale) Mirror 19: 0xa8bcd0dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e5d): Mirror 13: 0xa8bcd0dd (stale) Mirror 16: 0xa8bcd0dd (stale) Mirror 17: 0xa8bcd0dd (stale) Mirror 19: 0xa8bcd0dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e5d: skipped 000951 1775764287.722601 trunc from 0x0012a4 to 0x00df88 (0xcce5 bytes) 000953 1775764287.772852 punch from 0x00bc32 to 0x00c001 (0x3d0 bytes) 000956 1775764287.792794 trunc from 0x037e5d to 0x0287b7 (0xf6a5 bytes) 000957 1775764287.828846 trunc from 0x01dfa1 to 0x0287b7 (0xa817 bytes) 000958 1775764287.845727 write 0x027fb6 thru 0x0300ac (0x80f7 bytes) 000959 1775764287.864113 mapwrite 0x026b3b thru 0x02a567 (0x3a2d bytes) 000960 1775764287.935098 trunc from 0x0233b9 to 0x02f931 (0xc579 bytes) 000961 1775764287.946179 write 0x00154e thru 0x00d0a1 (0xbb54 bytes) 000962 1775764287.970657 trunc from 0x0300ad to 0x015d5e (0x1a34e bytes) 000963 1775764288.044709 trunc from 0x015d5f to 0x027c97 (0x11f39 bytes) 000964 1775764288.071838 trunc from 0x006358 to 0x013af6 (0xd79f bytes) 000965 1775764288.113977 trunc from 0x02531e to 0x027c97 (0x297a bytes) 000966 1775764288.131141 fallocate 0x00b954 thru 0x010e86 (0x5533 bytes) 000967 1775764288.150016 trunc from 0x027c98 to 0x010a42 (0x17255 bytes) 000968 1775764288.178059 mapwrite 0x002c05 thru 0x00a174 (0x7570 bytes) 000969 1775764288.244761 mapread 0x00ebf2 thru 0x010a42 (0x1e51 bytes) 000970 1775764288.246790 trunc from 0x01036a to 0x010a42 (0x6d9 bytes) 000971 1775764288.263834 read 0x002aca thru 0x010a42 (0xdf79 bytes) 000972 1775764288.277086 fallocate 0x017f9a thru 0x023d14 (0xbd7b bytes) 000973 1775764288.293981 mapread 0x016651 thru 0x02267d (0xc02d bytes) 000975 1775764288.376928 fallocate 0x033129 thru 0x03ffff (0xced7 bytes) 000976 1775764288.392977 mapwrite 0x03d488 thru 0x03ffff (0x2b78 bytes) 000977 1775764288.486009 mapread 0x004ea7 thru 0x00930c (0x4466 bytes) 000978 1775764288.499033 MIRROR_RESYNC 5 mirrors 000979 1775764290.333516 read 0x001c03 thru 0x005198 (0x3596 bytes) 000980 1775764290.361735 fallocate 0x0399dd thru 0x03ffff (0x6623 bytes) 000982 1775764290.430529 trunc from 0x00aa9b to 0x017fe8 (0xd54e bytes) 000983 1775764290.448475 write 0x02ad11 thru 0x02dd36 (0x3026 bytes) 000984 1775764290.478304 punch from 0x0271eb to 0x0327db (0xb5f1 bytes) 000985 1775764290.519988 trunc from 0x022073 to 0x027784 (0x5712 bytes) 000986 1775764290.542022 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x139:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x25889714 (stale) Mirror 16: 0x25889714 (stale) Mirror 17: 0x25889714 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x25889714 (stale) Mirror 16: 0x25889714 (stale) Mirror 17: 0x25889714 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1775764291.318428 write 0x029f60 thru 0x0375ed (0xd68e bytes) 000988 1775764291.343738 mapread 0x022de1 thru 0x0306a2 (0xd8c2 bytes) 000989 1775764291.392788 punch from 0x009ef9 to 0x00e27d (0x4385 bytes) 000990 1775764291.405017 mapread 0x0200ef thru 0x021025 (0xf37 bytes) 000992 1775764291.406903 trunc from 0x040000 to 0x02d0e4 (0x12f1b bytes) 000993 1775764291.465182 trunc from 0x02030a to 0x02d0e4 (0xcddb bytes) 000994 1775764291.512706 write 0x0191fa thru 0x01b15a (0x1f61 bytes) 000995 1775764291.527913 mapwrite 0x00d088 thru 0x01c152 (0xf0cb bytes) 000996 1775764291.581890 write 0x009737 thru 0x012db4 (0x967e bytes) 000997 1775764291.585968 read 0x021055 thru 0x02a86b (0x9817 bytes) 000998 1775764291.599801 mapread 0x028dfe thru 0x02d0e4 (0x42e7 bytes) 001000 1775764291.607773 write 0x00f009 thru 0x01d538 (0xe530 bytes) All operations completed A-OK! PASS 70a (178s) == sanity-flr test 71: check mirror extend parent fid ==== 15:51:38 (1775764298) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x52:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 15:51:51 (1775764311) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.325294 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921497 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 15:52:02 (1775764322) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1390176 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6272288 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4983264 .. Extending file size to 5487488 .. Extending file size to 8036032 .. Extending file size to 7501120 .. Extending file size to 9054208 .. Extending file size to 8906688 .. Extending file size to 8941792 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9182272 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59619 59620 59621 59624 59627 done 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (93s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:53:35 (1775764415) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8037504 .. Extending file size to 4714016 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9229888 .. failed Extending file size to 5987328 .. Extending file size to 7887360 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8922880 .. Extending file size to 8967072 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61180 61181 61182 61184 61187 done 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:55:01 (1775764501) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2200 1269104 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18712 3588308 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24344 3582676 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43056 7170984 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2200 1269104 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18712 3588308 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24344 3582676 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43056 7170984 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0911375 s, 23.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.356443 s, 5.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xdde188db (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) PASS 200c (27s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:55:31 (1775764531) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248584 s, 8.4 MB/s PASS 202 (11s) == sanity-flr test 203: mirror file preserve mirror ID === 15:55:43 (1775764543) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:55:51 (1775764551) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0860445 s, 24.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:56:03 (1775764563) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104322 s, 20.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:56:15 (1775764575) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186251 s, 11.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:56:51 (1775764611) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101528 s, 10.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:57:49 (1775764669) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194356 s, 10.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:58:49 (1775764729) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg210-server: Pool lustre.test_204f created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154234 s, 13.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg210-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:00:10 (1775764810) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:00:19 (1775764819) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:00:28 (1775764828) oleg210-server: Pool lustre.test_206 created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg210-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:00:59 (1775764859) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107027 s, 9.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660914 s, 15.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa46e7966 Mirror 3: 0xa46e7966 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:01:12 (1775764872) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:01:16 (1775764876) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:01:20 (1775764880) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0422963 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xfa0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process