-----============= acceptance-small: sanity-flr ============----- Tue Jun 23 05:20:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg326-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 05:20:53 (1782206453) === oleg326-client.virtnet: executing check_config_client /mnt/lustre oleg326-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg326-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d0dc6145800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d0dc6145800.idle_timeout=debug disable quota as required oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:21:24 (1782206484) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8211] == sanity-flr test 0a: lfs mirror create with -N option == 05:21:25 (1782206485) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:21:37 (1782206497) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:21:39 (1782206499) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:21:41 (1782206501) PASS 0d (18s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:21:59 (1782206519) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:22:02 (1782206522) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 05:22:05 (1782206525) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9d0dc6145800.stats=clear osc.lustre-OST0001-osc-ffff9d0dc6145800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:22:10 (1782206530) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:22:18 (1782206538) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:22:24 (1782206544) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 05:22:27 (1782206547) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:22:32 (1782206552) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 05:22:39 (1782206559) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:22:44 (1782206564) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:22:50 (1782206570) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:22:56 (1782206576) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:23:04 (1782206584) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:23:10 (1782206590) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179339 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489772 s, 21.4 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:23:17 (1782206597) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.08 s, 39.3 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 05:23:24 (1782206604) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:23:31 (1782206611) Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (57s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:24:28 (1782206668) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130808 s, 16.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:25:04 (1782206704) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:26:16 (1782206776) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.93339 s, 13.3 MB/s -rw-r--r-- 1 root root 52428800 Jun 23 05:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.020s user 0m0.000s sys 0m0.183s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d0dc6145800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.785s user 0m0.001s sys 0m0.162s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.108s user 0m0.000s sys 0m0.184s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d0dc6145800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:27:52 (1782206872) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:27:56 (1782206876) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:28:01 (1782206881) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 05:28:04 (1782206884) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 05:28:12 (1782206892) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.516512 s, 24.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:28:25 (1782206905) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389077 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070825 s, 14.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:28:33 (1782206913) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:28:40 (1782206920) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00183154 s, 559 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00084738 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012964 s, 790 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00184098 s, 556 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 05:28:51 (1782206931) create test files with size 15757792 .. 50b0f1ac2e9104b230e7d815c63a8a5a - 8a8e20ca14d54b705bf875c0fa04b74e - be531c379c1fe189cee0d406f60a99fe - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 05:29:15 (1782206955) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:29:29 (1782206969) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:29:35 (1782206975) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069397 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470044 s, 22.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 05:29:44 (1782206984) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0850011 s, 24.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.366186 s, 11.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198033 s, 10.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 05:29:57 (1782206997) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 05:30:00 (1782207000) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:30:03 (1782207003) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268058 s, 39.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 05:30:11 (1782207011) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.183979 s, 17.1 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:30:20 (1782207020) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:30:29 (1782207029) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.356515 s, 29.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:30:41 (1782207041) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9d0dc33c3000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.337912 s, 31.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9d0dd2c95800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:30:56 (1782207056) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:31:33 (1782207093) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:32:13 (1782207133) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 05:32:16 (1782207136) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.166483 s, 18.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.235199 s, 13.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5fda8f36 (nosync,stale) Mirror 2: 0x5fda8f36 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f54b26befd7a8b7a37f4ebd3ea011c5d - mirror 1 checksum: b27b03af11508951937cd320e5447a4a - mirror 2 checksum: f54b26befd7a8b7a37f4ebd3ea011c5d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T05:32:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T05:32:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f54b26befd7a8b7a37f4ebd3ea011c5d - mirror 1 checksum: b27b03af11508951937cd320e5447a4a - mirror 2 checksum: b27b03af11508951937cd320e5447a4a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T05:32:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T05:32:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:32:27 (1782207147) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307373 s, 34.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:32:35 (1782207155) Failing mds1 on oleg326-server Stopping /mnt/lustre-mds1 (opts:) on oleg326-server 05:32:48 (1782207168) shut down facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server Failover mds1 to oleg326-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:20 (1782207200) targets are mounted 05:33:20 (1782207200) facet_failover done oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223484 s, 46.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:33:36 (1782207216) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0602613 s, 340 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.014131 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:33:44 (1782207224) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0555822 s, 368 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0077482 s, 2.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:33:54 (1782207234) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00712503 s, 2.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:34:01 (1782207241) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0704093 s, 291 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:34:10 (1782207250) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00780329 s, 525 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0573824 s, 71.4 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:34:18 (1782207258) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:34:41 (1782207281) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:35:10 (1782207310) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:35:35 (1782207335) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:35:59 (1782207359) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 05:36:07 (1782207367) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57352 mirror split pid 57353 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57352 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57353 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 05:37:14 (1782207434) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 7436 skipping zero length punch hole skipping zero length zero range skipping zero size read skipping zero length zero range skipping zero length zero range 000006 1782207437.145560 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x10015 000007 1782207437.855434 fallocate 0x00cfe0 thru 0x010014 (0x3035 bytes) truncating to largest ever: 0x1073a 000008 1782207437.912347 trunc from 00000000 to 0x010739 (0x1073a bytes) 000009 1782207437.959312 punch from 0x008f5a to 0x00c851 (0x38f8 bytes) 000011 1782207437.969936 mapread 0x00bb68 thru 0x010739 (0x4bd2 bytes) 000013 1782207438.024397 trunc from 0x006d4d to 0x010739 (0x99ed bytes) fallocating to largest ever: 0x349f7 000014 1782207438.035228 fallocate 0x02e773 thru 0x0349f6 (0x6284 bytes) 000015 1782207438.050278 mapread 0x027b6f thru 0x0349f6 (0xce88 bytes) 000016 1782207438.073491 write 0x02807b thru 0x02e132 (0x60b8 bytes) 000017 1782207438.084243 mapread 0x01fa74 thru 0x029410 (0x999d bytes) 000018 1782207438.125729 mapread 0x031942 thru 0x0349f6 (0x30b5 bytes) 000019 1782207438.131357 mapwrite 0x00e929 thru 0x015e1e (0x74f6 bytes) 000020 1782207438.221159 mapwrite 0x000584 thru 0x004c75 (0x46f2 bytes) 000021 1782207438.329281 trunc from 0x012deb to 0x018c35 (0x5e4b bytes) 000022 1782207438.337071 fallocate 0x01570d thru 0x022cdb (0xd5cf bytes) truncating to largest ever: 0x35d34 000023 1782207438.342202 trunc from 0x0349f7 to 0x035d33 (0x133d bytes) 000024 1782207438.389900 mapread 0x027e49 thru 0x02872f (0x8e7 bytes) 000025 1782207438.391007 fallocate 0x0231d1 thru 0x02dc6a (0xaa9a bytes) 000027 1782207438.408671 punch from 0x022dd9 to 0x029569 (0x6791 bytes) 000028 1782207438.426947 trunc from 0x00eea4 to 0x01eb4c (0xfca9 bytes) 000029 1782207438.435090 trunc from 0x035d34 to 0x0139e3 (0x22350 bytes) 000030 1782207438.470086 write 0x0162c8 thru 0x01eab7 (0x87f0 bytes) 000031 1782207438.471542 mapread 0x00f18d thru 0x015cef (0x6b63 bytes) 000032 1782207438.495137 punch from 0x00d33a to 0x01ae53 (0xdb1a bytes) 000033 1782207438.523181 fallocate 0x029e9c thru 0x0316ee (0x7853 bytes) 000034 1782207438.545453 MIRROR_RESYNC 2 mirrors fallocating to largest ever: 0x36296 000036 1782207439.530378 fallocate 0x0357b1 thru 0x036295 (0xae5 bytes) 000037 1782207439.604328 read 0x0039e0 thru 0x00816f (0x4790 bytes) 000038 1782207439.646999 read 0x01c273 thru 0x020d36 (0x4ac4 bytes) 000039 1782207439.675601 fallocate 0x00a2b7 thru 0x0100d7 (0x5e21 bytes) 000040 1782207439.688983 read 0x019208 thru 0x02052c (0x7325 bytes) 000041 1782207439.690701 read 0x02ab29 thru 0x02f928 (0x4e00 bytes) 000042 1782207439.696341 read 0x00abb2 thru 0x019255 (0xe6a4 bytes) 000043 1782207439.698599 fallocate 0x011568 thru 0x0131b4 (0x1c4d bytes) 000044 1782207439.708735 trunc from 0x0316ef to 0x019b88 (0x17b66 bytes) 000045 1782207439.751549 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) 000047 1782207440.023641 punch from 0x006623 to 0x0066e1 (0xbf bytes) 000048 1782207440.030069 trunc from 0x019b89 to 0x00cc13 (0xcf75 bytes) 000049 1782207440.056558 read 0x005891 thru 0x00858c (0x2cfc bytes) 000051 1782207440.075040 trunc from 0x002837 to 0x00cc13 (0xa3dd bytes) 000052 1782207440.091391 trunc from 0x001fcb to 0x009531 (0x7567 bytes) 000053 1782207440.102625 mapread 0x00c02c thru 0x00cc13 (0xbe8 bytes) 000054 1782207440.117393 read 0x00bacc thru 0x00cc13 (0x1148 bytes) 000055 1782207440.137396 write 0x03bcd1 thru 0x03e210 (0x2540 bytes) 000057 1782207440.141289 mapwrite 0x03a29d thru 0x03ffff (0x5d63 bytes) 000058 1782207440.256565 mapread 0x03e849 thru 0x03ffff (0x17b7 bytes) 000059 1782207440.257658 mapwrite 0x00e1a9 thru 0x0198e8 (0xb740 bytes) 000060 1782207440.348658 read 0x00d9d2 thru 0x014803 (0x6e32 bytes) 000061 1782207440.358365 punch from 0x016c3a to 0x020706 (0x9acd bytes) 000062 1782207440.371774 trunc from 0x0096ad to 0x00d02d (0x3981 bytes) 000063 1782207440.383917 write 0x000b1b thru 0x00760e (0x6af4 bytes) 000064 1782207440.388731 punch from 0x009fc8 to 0x0162b2 (0xc2eb bytes) 000065 1782207440.406619 mapread 0x00e778 thru 0x01a59c (0xbe25 bytes) 000066 1782207440.429390 write 0x021ec7 thru 0x028e4b (0x6f85 bytes) 000067 1782207440.432216 trunc from 0x021595 to 0x030e92 (0xf8fe bytes) 000068 1782207440.459579 trunc from 0x040000 to 0x02bcf8 (0x14307 bytes) 000071 1782207440.498155 mapwrite 0x0147f8 thru 0x021391 (0xcb9a bytes) 000072 1782207440.553124 trunc from 0x029b81 to 0x02bcf8 (0x2178 bytes) 000073 1782207440.579294 punch from 0x027f6d to 0x02bcf8 (0x3d8c bytes) 000074 1782207440.596683 punch from 0x00c51e to 0x01a813 (0xe2f6 bytes) 000075 1782207440.615769 MIRROR_EXTEND to 2 mirrors 000076 1782207441.693883 write 0x01058f thru 0x015916 (0x5388 bytes) 000078 1782207441.804879 trunc from 0x02bcf9 to 0x019778 (0x12580 bytes) 000079 1782207441.842210 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) 000080 1782207442.086544 trunc from 0x00e4e6 to 0x010757 (0x2272 bytes) 000082 1782207442.095231 read 0x011a6d thru 0x01755e (0x5af2 bytes) 000083 1782207442.120017 mapwrite 0x008641 thru 0x011847 (0x9207 bytes) 000086 1782207442.154734 mapread 0x00defa thru 0x014ae0 (0x6be7 bytes) 000088 1782207442.158650 read 0x0063f2 thru 0x0117e4 (0xb3f3 bytes) 000089 1782207442.160063 read 0x00f3eb thru 0x019778 (0xa38e bytes) 000090 1782207442.170494 fallocate 0x007640 thru 0x01481a (0xd1db bytes) 000091 1782207442.175444 trunc from 0x008d04 to 0x018cf8 (0xfff5 bytes) 000092 1782207442.181516 fallocate 0x013a65 thru 0x01c40d (0x89a9 bytes) 000093 1782207442.189466 punch from 0x001488 to 0x005aba (0x4633 bytes) 000094 1782207442.193861 punch from 0x005e4f to 0x006365 (0x517 bytes) 000095 1782207442.200225 mapwrite 0x0281a7 thru 0x02a838 (0x2692 bytes) 000098 1782207442.291398 trunc from 0x0132c9 to 0x015dad (0x2ae5 bytes) 000099 1782207442.311295 read 0x01f976 thru 0x01fada (0x165 bytes) 000100 1782207442.313000 fallocate 0x021639 thru 0x0227c8 (0x1190 bytes) 000101 1782207442.324149 mapread 0x006fed thru 0x0125dc (0xb5f0 bytes) 000102 1782207442.346613 mapread 0x00407d thru 0x00e564 (0xa4e8 bytes) 000103 1782207442.348267 mapwrite 0x03ad78 thru 0x03ffff (0x5288 bytes) 000104 1782207442.457509 trunc from 0x040000 to 0x02093f (0x1f6c0 bytes) 000105 1782207442.498757 punch from 0x01f15f to 0x02093f (0x17e1 bytes) 000106 1782207442.504267 mapwrite 0x014967 thru 0x019ba1 (0x523b bytes) 000108 1782207442.538385 write 0x014e5b thru 0x024d8b (0xff31 bytes) 000109 1782207442.541383 trunc from 0x024d8c to 0x027a1d (0x2c92 bytes) 000110 1782207442.566077 trunc from 0x006732 to 0x016724 (0xfff3 bytes) 000112 1782207442.615457 mapwrite 0x039ad4 thru 0x03ffff (0x652c bytes) 000114 1782207442.734967 mapread 0x005d5a thru 0x007e57 (0x20fe bytes) 000115 1782207442.748018 trunc from 0x0141db to 0x01f304 (0xb12a bytes) 000116 1782207442.761475 write 0x026957 thru 0x027e22 (0x14cc bytes) 000117 1782207442.770971 mapwrite 0x032fcd thru 0x03fec7 (0xcefb bytes) 000118 1782207442.854439 read 0x0094b5 thru 0x01710e (0xdc5a bytes) 000121 1782207442.892511 mapread 0x039e55 thru 0x03ac1c (0xdc8 bytes) 000123 1782207442.921297 MIRROR_EXTEND to 2 mirrors 000125 1782207443.998191 punch from 0x00a8c5 to 0x00b246 (0x982 bytes) 000126 1782207444.030042 mapread 0x02493d thru 0x028de9 (0x44ad bytes) fallocating to largest ever: 0x3e88d 000127 1782207444.056043 fallocate 0x032c86 thru 0x03e88c (0xbc07 bytes) 000128 1782207444.060308 punch from 0x01aa18 to 0x0200e5 (0x56ce bytes) 000129 1782207444.068424 read 0x0194c3 thru 0x025675 (0xc1b3 bytes) 000130 1782207444.090561 trunc from 0x040000 to 0x039c0b (0x63f4 bytes) 000131 1782207444.126375 trunc from 0x036fa8 to 0x039c0b (0x2c64 bytes) 000132 1782207444.133335 fallocate 0x01bf52 thru 0x0210a6 (0x5155 bytes) 000133 1782207444.139448 punch from 0x01981d to 0x01e0e6 (0x48ca bytes) 000134 1782207444.146498 trunc from 0x033001 to 0x0347a0 (0x17a0 bytes) 000136 1782207444.165077 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) 000138 1782207444.309333 trunc from 0x039c0c to 0x020f45 (0x18cc6 bytes) 000139 1782207444.367681 mapwrite 0x01eeaf thru 0x026570 (0x76c2 bytes) 000140 1782207444.438661 fallocate 0x03553e thru 0x03a8c6 (0x5389 bytes) 000141 1782207444.442455 mapwrite 0x00e02c thru 0x01628c (0x8261 bytes) 000142 1782207444.467106 trunc from 0x03a8c7 to 0x00a6f9 (0x301cd bytes) 000144 1782207444.479620 mapwrite 0x00435c thru 0x00b2ce (0x6f73 bytes) 000145 1782207444.531703 write 0x02e3c5 thru 0x03aae6 (0xc722 bytes) 000146 1782207444.535127 read 0x038b05 thru 0x03aae6 (0x1fe2 bytes) 000147 1782207444.535702 read 0x00f9b5 thru 0x0130fd (0x3749 bytes) 000148 1782207444.547861 write 0x022d01 thru 0x0296c0 (0x69c0 bytes) 000149 1782207444.556634 read 0x01a7ee thru 0x0210b5 (0x68c8 bytes) 000150 1782207444.559481 write 0x008283 thru 0x015432 (0xd1b0 bytes) 000151 1782207444.564667 mapwrite 0x0232cb thru 0x02d680 (0xa3b6 bytes) 000152 1782207444.589688 trunc from 0x03aae7 to 0x0159d0 (0x25116 bytes) 000153 1782207444.606246 trunc from 0x009d9a to 0x01015b (0x63c2 bytes) 000154 1782207444.615516 fallocate 0x021077 thru 0x02ebac (0xdb36 bytes) 000155 1782207444.620359 trunc from 0x02ebad to 0x009a2f (0x2517d bytes) 000156 1782207444.643553 punch from 0x0072cf to 0x009a2f (0x2761 bytes) 000157 1782207444.647313 mapread 0x006691 thru 0x009a2f (0x339f bytes) 000158 1782207444.652390 fallocate 0x019922 thru 0x01d860 (0x3f3f bytes) 000159 1782207444.656422 trunc from 0x006e97 to 0x009a2f (0x2b99 bytes) 000160 1782207444.660238 mapwrite 0x0365b4 thru 0x03e968 (0x83b5 bytes) 000161 1782207444.707155 punch from 0x005433 to 0x00b6a4 (0x6272 bytes) 000162 1782207444.717267 MIRROR_EXTEND to 2 mirrors 000164 1782207445.511351 mapwrite 0x036c90 thru 0x039550 (0x28c1 bytes) 000165 1782207445.626217 mapread 0x01afc0 thru 0x01baec (0xb2d bytes) 000167 1782207445.660424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) 000168 1782207445.794754 mapwrite 0x0247ea thru 0x02828c (0x3aa3 bytes) 000169 1782207445.879877 read 0x036142 thru 0x03b57a (0x5439 bytes) 000171 1782207445.888024 mapwrite 0x03a00e thru 0x03ffff (0x5ff2 bytes) 000172 1782207445.950195 mapread 0x01a2e3 thru 0x020e79 (0x6b97 bytes) 000173 1782207445.960992 mapwrite 0x03c640 thru 0x03ffff (0x39c0 bytes) 000174 1782207445.997601 fallocate 0x0178a9 thru 0x01c089 (0x47e1 bytes) 000175 1782207446.005299 mapread 0x008fe6 thru 0x00be45 (0x2e60 bytes) 000176 1782207446.015171 trunc from 0x040000 to 0x03d767 (0x2898 bytes) 000177 1782207446.037213 trunc from 0x03d768 to 0x00f0bc (0x2e6ab bytes) 000179 1782207446.063143 trunc from 0x00f0bd to 0x0370c1 (0x28005 bytes) 000180 1782207446.091671 fallocate 0x025232 thru 0x032fe8 (0xddb7 bytes) 000182 1782207446.099956 punch from 0x020559 to 0x023935 (0x33dd bytes) 000183 1782207446.104357 mapwrite 0x01ed61 thru 0x01f24f (0x4ef bytes) 000184 1782207446.144318 mapwrite 0x035918 thru 0x03c13f (0x6828 bytes) 000185 1782207446.239910 fallocate 0x02a499 thru 0x034915 (0xa47d bytes) 000186 1782207446.252846 trunc from 0x03c140 to 0x02db0e (0xe631 bytes) 000187 1782207446.266347 mapwrite 0x01acb9 thru 0x028946 (0xdc8e bytes) fallocating to largest ever: 0x40000 000188 1782207446.302070 fallocate 0x038f85 thru 0x03ffff (0x707b bytes) 000189 1782207446.307958 trunc from 0x019e60 to 0x01c205 (0x23a6 bytes) 000191 1782207446.313740 punch from 0x01e07f to 0x027a36 (0x99b8 bytes) 000192 1782207446.319364 mapwrite 0x03e15b thru 0x03e55e (0x404 bytes) 000193 1782207446.339949 read 0x03d2ec thru 0x03ffff (0x2d14 bytes) 000194 1782207446.340619 trunc from 0x00b91e to 0x018648 (0xcd2b bytes) 000195 1782207446.345294 trunc from 0x040000 to 0x0146df (0x2b920 bytes) 000197 1782207446.363529 read 0x0047f1 thru 0x00f2ab (0xaabb bytes) 000198 1782207446.374882 write 0x00d80c thru 0x00ec3a (0x142f bytes) 000199 1782207446.377201 punch from 0x008c82 to 0x013363 (0xa6e2 bytes) 000200 1782207446.389343 trunc from 0x013fab to 0x0146df (0x735 bytes) 000201 1782207446.395658 trunc from 0x0138d0 to 0x0146df (0xe10 bytes) 000202 1782207446.400885 mapread 0x0073be thru 0x01060d (0x9250 bytes) 000203 1782207446.444044 mapread 0x013f11 thru 0x0146df (0x7cf bytes) 000204 1782207446.459968 MIRROR_EXTEND to 2 mirrors 000205 1782207447.478067 mapwrite 0x02f74f thru 0x0336d6 (0x3f88 bytes) 000206 1782207447.592921 mapwrite 0x03f06d thru 0x03fadb (0xa6f bytes) 000207 1782207447.637436 mapwrite 0x01da4d thru 0x024ea3 (0x7457 bytes) 000208 1782207447.674894 mapwrite 0x025eb6 thru 0x029f56 (0x40a1 bytes) 000209 1782207447.696682 write 0x02a3d1 thru 0x034b56 (0xa786 bytes) 000210 1782207447.700834 trunc from 0x03fadc to 0x030f88 (0xeb53 bytes) 000211 1782207447.723729 trunc from 0x030f89 to 0x0290d1 (0x7eb7 bytes) 000212 1782207447.757335 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) 000213 1782207447.979478 trunc from 0x0290d2 to 0x0128f8 (0x167d9 bytes) 000214 1782207448.033651 trunc from 0x00d098 to 0x010e43 (0x3dac bytes) 000215 1782207448.038710 write 0x03860c thru 0x0398be (0x12b3 bytes) 000216 1782207448.042242 trunc from 0x0398bf to 0x03b602 (0x1d44 bytes) 000217 1782207448.058306 trunc from 0x020893 to 0x02d12f (0xc89d bytes) 000218 1782207448.063680 fallocate 0x02a3ef thru 0x0363ca (0xbfdc bytes) 000219 1782207448.070145 mapread 0x00c25e thru 0x0197c5 (0xd568 bytes) 000220 1782207448.107949 punch from 0x011a1a to 0x01ced6 (0xb4bd bytes) 000222 1782207448.127036 punch from 0x026033 to 0x027d67 (0x1d35 bytes) 000223 1782207448.133004 fallocate 0x03413c thru 0x03daf4 (0x99b9 bytes) 000224 1782207448.140674 mapread 0x01012a thru 0x01d5a4 (0xd47b bytes) 000225 1782207448.161967 mapread 0x027e45 thru 0x02ed1b (0x6ed7 bytes) 000226 1782207448.178286 punch from 0x023c92 to 0x02e378 (0xa6e7 bytes) 000227 1782207448.183226 trunc from 0x015dc2 to 0x021147 (0xb386 bytes) 000228 1782207448.188123 trunc from 0x03882e to 0x03b602 (0x2dd5 bytes) 000229 1782207448.198232 write 0x0012f0 thru 0x0053cc (0x40dd bytes) 000230 1782207448.217871 mapread 0x012274 thru 0x014489 (0x2216 bytes) 000231 1782207448.221264 read 0x0272ce thru 0x035490 (0xe1c3 bytes) 000232 1782207448.232372 mapwrite 0x025c6a thru 0x02df84 (0x831b bytes) 000233 1782207448.262991 punch from 0x01ed56 to 0x026638 (0x78e3 bytes) 000234 1782207448.270560 write 0x025c14 thru 0x02d30c (0x76f9 bytes) 000235 1782207448.275944 mapread 0x0280ba thru 0x02d41b (0x5362 bytes) 000236 1782207448.277272 mapwrite 0x00b126 thru 0x015a1d (0xa8f8 bytes) 000237 1782207448.309521 trunc from 0x009da2 to 0x00a02f (0x28e bytes) 000238 1782207448.315185 punch from 0x0066a1 to 0x014484 (0xdde4 bytes) 000239 1782207448.322767 trunc from 0x03b603 to 0x028d31 (0x128d1 bytes) 000240 1782207448.339250 write 0x0094a6 thru 0x00b799 (0x22f4 bytes) 000241 1782207448.348302 mapread 0x009f7a thru 0x01132d (0x73b4 bytes) 000243 1782207448.357188 trunc from 0x005e5b to 0x00917a (0x3320 bytes) 000244 1782207448.375607 mapwrite 0x03774c thru 0x03ffff (0x88b4 bytes) 000245 1782207448.454042 MIRROR_EXTEND to 2 mirrors 000246 1782207449.424232 mapwrite 0x025e70 thru 0x0285bb (0x274c bytes) 000248 1782207449.532525 trunc from 0x040000 to 0x003208 (0x3cdf7 bytes) 000249 1782207449.573356 punch from 0x000e5d to 0x003208 (0x23ac bytes) 000250 1782207449.584621 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3209): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3209): Mirror 2: 0x4a3ea07e (stale) CRC-32 checksum value for chunk [0, 0x3209): Mirror 1: 0xc0ea8e7d Mirror 3: 0xc0ea8e7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3209: skipped 000251 1782207450.546626 mapread 0x0006ba thru 0x003208 (0x2b4f bytes) 000252 1782207450.603723 write 0x0252de thru 0x03442c (0xf14f bytes) 000253 1782207450.671602 write 0x015c85 thru 0x02255e (0xc8da bytes) 000254 1782207450.691747 mapwrite 0x026313 thru 0x02f0ce (0x8dbc bytes) 000255 1782207450.754494 read 0x02a74b thru 0x032bac (0x8462 bytes) 000256 1782207450.762209 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3442d): Mirror 2: 0xcaf73cc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3442d): Mirror 2: 0xcaf73cc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3442d: skipped 000257 1782207451.123288 punch from 0x017b98 to 0x02288b (0xacf4 bytes) 000258 1782207451.128640 trunc from 0x00098b to 0x004831 (0x3ea7 bytes) 000259 1782207451.142592 trunc from 0x020f53 to 0x02cca4 (0xbd52 bytes) 000260 1782207451.150940 write 0x018f50 thru 0x02733f (0xe3f0 bytes) 000261 1782207451.185135 trunc from 0x01adc9 to 0x0208f1 (0x5b29 bytes) 000262 1782207451.203751 write 0x01bd72 thru 0x01ddc6 (0x2055 bytes) 000263 1782207451.216595 mapwrite 0x007a86 thru 0x011765 (0x9ce0 bytes) 000264 1782207451.291294 mapwrite 0x0344c6 thru 0x037700 (0x323b bytes) 000265 1782207451.334921 punch from 0x02fc65 to 0x037700 (0x7a9c bytes) 000266 1782207451.342017 fallocate 0x00affe thru 0x00bb28 (0xb2b bytes) 000267 1782207451.349395 trunc from 0x037701 to 0x027917 (0xfde9 bytes) 000268 1782207451.364461 write 0x02f4e5 thru 0x03df36 (0xea52 bytes) 000269 1782207451.369153 trunc from 0x0041c3 to 0x006456 (0x2294 bytes) 000270 1782207451.375019 mapwrite 0x01b365 thru 0x023a1c (0x86b8 bytes) 000271 1782207451.405857 fallocate 0x03311e thru 0x03689d (0x3780 bytes) 000273 1782207451.411453 punch from 0x01d7be to 0x02c03d (0xe880 bytes) 000274 1782207451.417218 mapwrite 0x005e67 thru 0x012957 (0xcaf1 bytes) 000275 1782207451.452944 trunc from 0x03df37 to 0x0150f4 (0x28e42 bytes) 000276 1782207451.466251 trunc from 0x0150f5 to 0x00f89b (0x5859 bytes) 000277 1782207451.484926 mapwrite 0x01e3cc thru 0x01ec9a (0x8cf bytes) 000278 1782207451.538858 mapwrite 0x0278ba thru 0x031c43 (0xa38a bytes) 000279 1782207451.616626 MIRROR_RESYNC 2 mirrors 000280 1782207452.466003 punch from 0x0310af to 0x031c43 (0xb95 bytes) 000281 1782207452.537574 read 0x009407 thru 0x00c1b2 (0x2dac bytes) 000282 1782207452.556491 fallocate 0x00bcb6 thru 0x00fb07 (0x3e52 bytes) 000283 1782207452.563755 mapwrite 0x03cb51 thru 0x03ffff (0x34af bytes) 000284 1782207452.665267 fallocate 0x02171b thru 0x029125 (0x7a0b bytes) 000285 1782207452.678710 punch from 0x036a21 to 0x038d83 (0x2363 bytes) 000287 1782207452.687098 read 0x008afb thru 0x011ecd (0x93d3 bytes) 000288 1782207452.723864 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x40503f06 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6ce756ec Mirror 3: 0x6ce756ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000289 1782207453.773244 mapwrite 0x004432 thru 0x013a99 (0xf668 bytes) 000290 1782207453.917145 mapwrite 0x0033a1 thru 0x00c77e (0x93de bytes) 000291 1782207453.934619 mapread 0x01810c thru 0x0216b8 (0x95ad bytes) 000292 1782207453.943483 trunc from 0x0158f6 to 0x01c3f1 (0x6afc bytes) 000294 1782207453.948504 punch from 0x02e422 to 0x02eda9 (0x988 bytes) 000296 1782207453.953542 trunc from 0x020761 to 0x02ddfe (0xd69e bytes) 000298 1782207453.960574 trunc from 0x00022a to 0x000db4 (0xb8b bytes) 000300 1782207453.965689 mapread 0x01bb7f thru 0x01f0f7 (0x3579 bytes) 000301 1782207453.974781 mapread 0x03e71d thru 0x03ffff (0x18e3 bytes) 000303 1782207453.984326 fallocate 0x004d9c thru 0x013ec9 (0xf12e bytes) 000304 1782207453.989062 fallocate 0x00a0f6 thru 0x018899 (0xe7a4 bytes) 000305 1782207453.993549 fallocate 0x0235dd thru 0x025788 (0x21ac bytes) 000306 1782207453.998640 fallocate 0x011ab4 thru 0x0218c8 (0xfe15 bytes) 000307 1782207454.003161 trunc from 0x040000 to 0x01c1a4 (0x23e5b bytes) 000308 1782207454.027414 trunc from 0x01c1a5 to 0x00e95a (0xd84a bytes) 000309 1782207454.042070 mapread 0x00af5f thru 0x00cf3f (0x1fe1 bytes) 000310 1782207454.043353 mapwrite 0x006dad thru 0x011966 (0xabba bytes) 000311 1782207454.081712 fallocate 0x03e747 thru 0x03ffff (0x18b9 bytes) 000312 1782207454.093565 MIRROR_RESYNC 3 mirrors 000313 1782207454.723327 read 0x002f09 thru 0x00b271 (0x8369 bytes) 000314 1782207454.733093 mapread 0x00ce08 thru 0x011966 (0x4b5f bytes) 000315 1782207454.749775 read 0x00fd5f thru 0x011966 (0x1c08 bytes) 000316 1782207454.750510 trunc from 0x011967 to 0x039dfc (0x28496 bytes) 000317 1782207454.857338 fallocate 0x01b359 thru 0x022ad7 (0x777f bytes) 000318 1782207454.867268 write 0x021cbb thru 0x0263be (0x4704 bytes) 000319 1782207454.907059 trunc from 0x039dfd to 0x012a76 (0x27386 bytes) 000320 1782207454.938703 mapwrite 0x0367a6 thru 0x03ffff (0x985a bytes) 000321 1782207455.107172 trunc from 0x040000 to 0x00f251 (0x30dae bytes) 000322 1782207455.120569 trunc from 0x003a2c to 0x00f251 (0xb826 bytes) 000323 1782207455.125757 mapread 0x0018b4 thru 0x0043df (0x2b2c bytes) 000324 1782207455.138284 trunc from 0x006338 to 0x00f251 (0x8f1a bytes) 000326 1782207455.145623 punch from 0x00b9c7 to 0x00f251 (0x388b bytes) 000327 1782207455.150363 mapread 0x006986 thru 0x00eefd (0x8578 bytes) 000328 1782207455.202367 read 0x001206 thru 0x00365b (0x2456 bytes) 000329 1782207455.203229 read 0x0066ce thru 0x007e08 (0x173b bytes) 000330 1782207455.205432 write 0x023710 thru 0x03323a (0xfb2b bytes) 000331 1782207455.208987 mapread 0x0048e7 thru 0x009480 (0x4b9a bytes) 000332 1782207455.230770 fallocate 0x0259ba thru 0x025b95 (0x1dc bytes) 000333 1782207455.242126 trunc from 0x02a3f6 to 0x03012d (0x5d38 bytes) 000334 1782207455.283867 punch from 0x022610 to 0x02573e (0x312f bytes) 000335 1782207455.296172 read 0x02a75c thru 0x03323a (0x8adf bytes) 000336 1782207455.319991 mapwrite 0x0271ac thru 0x02ec77 (0x7acc bytes) 000337 1782207455.349292 trunc from 0x03323b to 0x01165e (0x21bdc bytes) 000338 1782207455.372393 punch from 0x0006c8 to 0x00b912 (0xb24b bytes) 000340 1782207455.383423 mapread 0x001d4b thru 0x00af2a (0x91e0 bytes) 000341 1782207455.478056 MIRROR_RESYNC 3 mirrors 000343 1782207456.341450 mapwrite 0x031772 thru 0x032c7c (0x150b bytes) 000344 1782207456.445305 write 0x036557 thru 0x03aa41 (0x44eb bytes) 000345 1782207456.446608 mapread 0x014718 thru 0x019208 (0x4af1 bytes) 000346 1782207456.457601 fallocate 0x006100 thru 0x010224 (0xa125 bytes) 000347 1782207456.462630 mapread 0x007844 thru 0x00a313 (0x2ad0 bytes) 000348 1782207456.476324 read 0x003278 thru 0x0116e4 (0xe46d bytes) 000349 1782207456.477897 trunc from 0x03aa42 to 0x02265a (0x183e7 bytes) 000350 1782207456.493316 mapwrite 0x0329f5 thru 0x036f1d (0x4529 bytes) 000351 1782207456.535121 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36f1e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f1e): Mirror 2: 0xb645b90c (stale) Mirror 3: 0xb645b90c (stale) CRC-32 checksum value for chunk [0, 0x36f1e): Mirror 1: 0x57bcb3ba Mirror 4: 0x57bcb3ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f1e: skipped 000352 1782207457.369391 mapwrite 0x01cbdc thru 0x0283da (0xb7ff bytes) 000353 1782207457.576451 trunc from 0x036f1e to 0x03496e (0x25af bytes) 000354 1782207457.638799 mapwrite 0x017210 thru 0x018a92 (0x1883 bytes) 000356 1782207457.664120 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3496f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3496f): Mirror 2: 0xbf040637 (stale) Mirror 3: 0xbf040637 (stale) Mirror 4: 0x5d232c9c (stale) CRC-32 checksum value for chunk [0, 0x3496f): Mirror 1: 0xb66747df Mirror 5: 0xb66747df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3496f: skipped 000357 1782207458.709080 mapwrite 0x0378bf thru 0x037b5c (0x29e bytes) 000359 1782207458.821112 write 0x02b15e thru 0x0395d5 (0xe478 bytes) 000360 1782207458.828429 punch from 0x00cf00 to 0x01c162 (0xf263 bytes) 000361 1782207458.834457 fallocate 0x02f177 thru 0x0391f9 (0xa083 bytes) 000362 1782207458.841159 trunc from 0x01e94e to 0x0228e0 (0x3f93 bytes) 000363 1782207458.865693 mapwrite 0x02230a thru 0x02bef1 (0x9be8 bytes) 000365 1782207458.937370 fallocate 0x01eb45 thru 0x02b3d7 (0xc893 bytes) 000366 1782207458.949369 punch from 0x0095b4 to 0x00c151 (0x2b9e bytes) 000367 1782207458.964525 write 0x0011f7 thru 0x003cdd (0x2ae7 bytes) 000368 1782207458.981401 punch from 0x014e98 to 0x015990 (0xaf9 bytes) 000369 1782207458.988555 mapwrite 0x02b520 thru 0x03920e (0xdcef bytes) 000370 1782207459.047133 fallocate 0x00b69d thru 0x00e122 (0x2a86 bytes) 000371 1782207459.055008 mapread 0x00c59c thru 0x015cb9 (0x971e bytes) 000372 1782207459.087435 punch from 0x00d299 to 0x016b5a (0x98c2 bytes) 000373 1782207459.102449 fallocate 0x03b6ac thru 0x03d7f3 (0x2148 bytes) 000375 1782207459.124459 write 0x019319 thru 0x01d677 (0x435f bytes) 000376 1782207459.129090 trunc from 0x0320bc to 0x03bc53 (0x9b98 bytes) 000377 1782207459.141041 fallocate 0x0114f9 thru 0x017c10 (0x6718 bytes) 000378 1782207459.159110 mapwrite 0x00db2a thru 0x0149ee (0x6ec5 bytes) 000379 1782207459.235774 mapread 0x0241b4 thru 0x024447 (0x294 bytes) 000380 1782207459.237500 trunc from 0x03d7f4 to 0x02804c (0x157a7 bytes) 000381 1782207459.257123 fallocate 0x015502 thru 0x01bd8c (0x688b bytes) 000382 1782207459.262937 trunc from 0x00bae9 to 0x013829 (0x7d41 bytes) 000383 1782207459.270597 trunc from 0x0263b2 to 0x02804c (0x1c9b bytes) 000384 1782207459.276891 write 0x0191c3 thru 0x026200 (0xd03e bytes) 000385 1782207459.284979 trunc from 0x02804d to 0x00ca99 (0x1b5b3 bytes) 000386 1782207459.317348 trunc from 0x00ca9a to 0x01429e (0x7805 bytes) 000387 1782207459.354791 mapwrite 0x024598 thru 0x02817c (0x3be5 bytes) 000388 1782207459.463661 read 0x018dc5 thru 0x0267df (0xda1b bytes) 000389 1782207459.480908 write 0x03402f thru 0x03ffff (0xbfd1 bytes) 000390 1782207459.483926 punch from 0x028015 to 0x03792a (0xf916 bytes) 000391 1782207459.517174 trunc from 0x040000 to 0x03e874 (0x178b bytes) 000392 1782207459.555671 punch from 0x00339d to 0x012e04 (0xfa68 bytes) 000393 1782207459.560685 trunc from 0x03e875 to 0x038d48 (0x5b2c bytes) 000394 1782207459.598147 punch from 0x02bd24 to 0x02ea72 (0x2d4f bytes) 000395 1782207459.608962 trunc from 0x038d49 to 0x015b74 (0x231d4 bytes) 000397 1782207459.636373 read 0x003fa1 thru 0x0138bf (0xf91f bytes) 000399 1782207459.667412 read 0x00bd22 thru 0x00d870 (0x1b4f bytes) 000400 1782207459.672185 punch from 0x00974b to 0x015b74 (0xc42a bytes) 000401 1782207459.690521 fallocate 0x014cba thru 0x01dcaa (0x8ff1 bytes) 000402 1782207459.702020 trunc from 0x00723e to 0x00a3c7 (0x318a bytes) 000403 1782207459.718944 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15b75): Mirror 2: 0xa1f0c233 (stale) Mirror 3: 0xa1f0c233 (stale) Mirror 5: 0xa1f0c233 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15b75): Mirror 2: 0xa1f0c233 (stale) Mirror 3: 0xa1f0c233 (stale) Mirror 5: 0xa1f0c233 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15b75: skipped 000404 1782207460.029135 punch from 0x000565 to 0x006aeb (0x6587 bytes) 000405 1782207460.036072 mapwrite 0x039526 thru 0x03e42f (0x4f0a bytes) 000406 1782207460.091391 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e430): Mirror 2: 0x9b80335a (stale) Mirror 3: 0x9b80335a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e430): Mirror 2: 0x9b80335a (stale) Mirror 3: 0x9b80335a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e430: skipped 000407 1782207460.542951 read 0x016e37 thru 0x01cacc (0x5c96 bytes) 000408 1782207460.573725 fallocate 0x01a100 thru 0x0270e3 (0xcfe4 bytes) 000410 1782207460.587167 mapwrite 0x01af92 thru 0x027a7d (0xcaec bytes) 000411 1782207460.666055 mapwrite 0x02769c thru 0x029f72 (0x28d7 bytes) 000412 1782207460.691190 trunc from 0x03e430 to 0x0113bb (0x2d074 bytes) 000413 1782207460.733580 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x113bc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x113bc): Mirror 2: 0xb6e0882f (stale) Mirror 3: 0xb6e0882f (stale) CRC-32 checksum value for chunk [0, 0x113bc): Mirror 1: 0xb6e0882f Mirror 4: 0xb6e0882f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x113bc: skipped 000415 1782207461.672231 trunc from 0x0113bc to 0x027a4c (0x16691 bytes) 000416 1782207461.744204 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 2: 0x5a965caf (stale) Mirror 3: 0x5a965caf (stale) Mirror 4: 0x5a965caf (stale) CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 1: 0x5a965caf Mirror 5: 0x5a965caf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a4d: skipped 000417 1782207462.726186 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 2: 0x5a965caf (stale) Mirror 3: 0x5a965caf (stale) Mirror 4: 0x5a965caf (stale) CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 1: 0x5a965caf Mirror 5: 0x5a965caf Mirror 6: 0x5a965caf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a4d: skipped 000418 1782207463.750809 mapread 0x023f85 thru 0x027a4c (0x3ac8 bytes) 000419 1782207463.788099 mapread 0x00189c thru 0x00eb25 (0xd28a bytes) 000420 1782207463.809846 punch from 0x024eef to 0x027a4c (0x2b5e bytes) 000421 1782207463.839497 fallocate 0x0331d4 thru 0x03765c (0x4489 bytes) 000422 1782207463.847883 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 2: 0x5a965caf (stale) Mirror 3: 0x5a965caf (stale) Mirror 4: 0x5a965caf (stale) Mirror 5: 0x5a965caf (stale) Mirror 6: 0x5a965caf (stale) CRC-32 checksum value for chunk [0, 0x27a4d): Mirror 1: 0x5a965caf Mirror 7: 0x5a965caf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a4d: skipped 000423 1782207464.929900 read 0x010554 thru 0x01d090 (0xcb3d bytes) 000424 1782207464.955226 read 0x010755 thru 0x014f6f (0x481b bytes) 000425 1782207464.964991 read 0x01b394 thru 0x0239d5 (0x8642 bytes) 000426 1782207465.006989 trunc from 0x01ab92 to 0x025bb3 (0xb022 bytes) 000427 1782207465.059948 trunc from 0x01adad to 0x01f475 (0x46c9 bytes) 000428 1782207465.066056 mapwrite 0x02c3f5 thru 0x02d906 (0x1512 bytes) 000429 1782207465.115537 write 0x0163a3 thru 0x01e4ab (0x8109 bytes) 000430 1782207465.130558 MIRROR_RESYNC 7 mirrors 000431 1782207467.274492 read 0x009132 thru 0x00c1e1 (0x30b0 bytes) 000432 1782207467.279849 mapread 0x015788 thru 0x024be6 (0xf45f bytes) 000433 1782207467.308795 mapread 0x003dd9 thru 0x00ae2d (0x7055 bytes) 000434 1782207467.320259 MIRROR_EXTEND to 8 mirrors 000435 1782207468.218311 punch from 0x0164c4 to 0x01dd65 (0x78a2 bytes) 000436 1782207468.359481 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d907): Mirror 2: 0x29493554 (stale) Mirror 3: 0x29493554 (stale) Mirror 4: 0x29493554 (stale) Mirror 5: 0x29493554 (stale) Mirror 6: 0x29493554 (stale) Mirror 8: 0x29493554 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d907): Mirror 2: 0x29493554 (stale) Mirror 3: 0x29493554 (stale) Mirror 4: 0x29493554 (stale) Mirror 5: 0x29493554 (stale) Mirror 6: 0x29493554 (stale) Mirror 8: 0x29493554 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d907: skipped 000437 1782207469.140341 mapwrite 0x016ba1 thru 0x01c47b (0x58db bytes) 000438 1782207469.194124 mapwrite 0x03a331 thru 0x03ca5f (0x272f bytes) 000439 1782207469.238501 read 0x022fd2 thru 0x02974a (0x6779 bytes) 000440 1782207469.262040 MIRROR_RESYNC 7 mirrors 000441 1782207471.571185 trunc from 0x033003 to 0x03ca5f (0x9a5d bytes) 000443 1782207471.629595 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca60): Mirror 3: 0xcfbbe05b (stale) Mirror 4: 0xcfbbe05b (stale) Mirror 5: 0xcfbbe05b (stale) Mirror 6: 0xcfbbe05b (stale) Mirror 8: 0xcfbbe05b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca60): Mirror 3: 0xcfbbe05b (stale) Mirror 4: 0xcfbbe05b (stale) Mirror 5: 0xcfbbe05b (stale) Mirror 6: 0xcfbbe05b (stale) Mirror 8: 0xcfbbe05b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca60: skipped 000444 1782207472.137362 trunc from 0x03980b to 0x03ca5f (0x3255 bytes) 000446 1782207472.148254 punch from 0x001e49 to 0x009ce2 (0x7e9a bytes) 000447 1782207472.154321 punch from 0x0136b3 to 0x0183db (0x4d29 bytes) 000448 1782207472.161776 read 0x018452 thru 0x0224b4 (0xa063 bytes) 000449 1782207472.169735 mapread 0x031a01 thru 0x032a8b (0x108b bytes) 000450 1782207472.186282 write 0x03737b thru 0x03ffff (0x8c85 bytes) 000451 1782207472.198202 mapread 0x0008c3 thru 0x007dab (0x74e9 bytes) 000452 1782207472.217625 trunc from 0x040000 to 0x00ed7f (0x31280 bytes) 000453 1782207472.239694 punch from 0x00a6ec to 0x00ae88 (0x79d bytes) 000454 1782207472.252365 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xed80): Mirror 4: 0x36140c22 (stale) Mirror 5: 0x36140c22 (stale) Mirror 6: 0x36140c22 (stale) Mirror 8: 0x36140c22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xed80): Mirror 4: 0x36140c22 (stale) Mirror 5: 0x36140c22 (stale) Mirror 6: 0x36140c22 (stale) Mirror 8: 0x36140c22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xed80: skipped 000455 1782207472.670182 mapread 0x00141b thru 0x00e328 (0xcf0e bytes) 000456 1782207472.697613 fallocate 0x00b988 thru 0x00bc49 (0x2c2 bytes) 000457 1782207472.702579 mapread 0x007df0 thru 0x00ed7f (0x6f90 bytes) 000458 1782207472.704636 fallocate 0x02b05a thru 0x03183a (0x67e1 bytes) 000459 1782207472.713449 MIRROR_RESYNC 5 mirrors 000460 1782207473.543301 write 0x027367 thru 0x027dd7 (0xa71 bytes) 000461 1782207473.590437 trunc from 0x000f75 to 0x004ef4 (0x3f80 bytes) 000462 1782207473.598912 read 0x012ebb thru 0x0144e0 (0x1626 bytes) 000463 1782207473.606053 mapread 0x01e95d thru 0x0223e7 (0x3a8b bytes) 000464 1782207473.627169 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27dd8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27dd8): Mirror 4: 0x367b5630 (stale) Mirror 5: 0x367b5630 (stale) Mirror 6: 0x367b5630 (stale) Mirror 8: 0x367b5630 (stale) CRC-32 checksum value for chunk [0, 0x27dd8): Mirror 1: 0x3e6a50d6 Mirror 9: 0x3e6a50d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27dd8: skipped 000465 1782207474.597978 read 0x021a68 thru 0x027dd7 (0x6370 bytes) 000466 1782207474.653660 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27dd8): Mirror 5: 0x367b5630 (stale) Mirror 6: 0x367b5630 (stale) Mirror 8: 0x367b5630 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27dd8): Mirror 5: 0x367b5630 (stale) Mirror 6: 0x367b5630 (stale) Mirror 8: 0x367b5630 (stale) CRC-32 checksum value for chunk [0, 0x27dd8): Mirror 1: 0x3e6a50d6 Mirror 9: 0x3e6a50d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27dd8: skipped 000467 1782207475.367279 mapread 0x026aac thru 0x027dd7 (0x132c bytes) 000468 1782207475.391268 fallocate 0x0077ab thru 0x008bd7 (0x142d bytes) 000470 1782207475.445880 mapwrite 0x03d937 thru 0x03ffff (0x26c9 bytes) 000471 1782207475.516847 mapread 0x010b7c thru 0x01c341 (0xb7c6 bytes) 000472 1782207475.535210 trunc from 0x03a678 to 0x03ffff (0x5988 bytes) 000473 1782207475.540267 trunc from 0x022700 to 0x031d5c (0xf65d bytes) 000474 1782207475.545171 punch from 0x02fd68 to 0x039baa (0x9e43 bytes) 000475 1782207475.550641 trunc from 0x040000 to 0x003122 (0x3cedd bytes) 000476 1782207475.569207 trunc from 0x003123 to 0x036d52 (0x33c30 bytes) 000477 1782207475.584103 punch from 0x0252dc to 0x02a5e7 (0x530c bytes) 000479 1782207475.591295 trunc from 0x010b63 to 0x0194c4 (0x8962 bytes) 000480 1782207475.597593 mapread 0x008e04 thru 0x015eea (0xd0e7 bytes) 000481 1782207475.625801 write 0x01a276 thru 0x027cea (0xda75 bytes) 000482 1782207475.641476 mapwrite 0x02ea61 thru 0x02f632 (0xbd2 bytes) 000483 1782207475.700948 fallocate 0x01787d thru 0x01c7eb (0x4f6f bytes) 000484 1782207475.715707 mapread 0x002859 thru 0x0091b5 (0x695d bytes) 000485 1782207475.721871 punch from 0x00645a to 0x00e53d (0x80e4 bytes) 000486 1782207475.746763 MIRROR_RESYNC 5 mirrors 000487 1782207477.630381 mapread 0x01b48b thru 0x02870f (0xd285 bytes) 000488 1782207477.705176 MIRROR_RESYNC 5 mirrors 000489 1782207478.276893 trunc from 0x036d53 to 0x018d9e (0x1dfb4 bytes) 000490 1782207478.328969 trunc from 0x018d9f to 0x03bd72 (0x22fd4 bytes) 000491 1782207478.365012 mapread 0x020117 thru 0x024699 (0x4583 bytes) 000493 1782207478.400150 fallocate 0x03d101 thru 0x03ffff (0x2eff bytes) 000496 1782207478.419035 fallocate 0x03060a thru 0x0377b2 (0x71a9 bytes) 000497 1782207478.440201 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bd73): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd73): Mirror 5: 0xd67a9a5 (stale) Mirror 6: 0xd67a9a5 (stale) Mirror 8: 0xd67a9a5 (stale) Mirror 9: 0xd67a9a5 (stale) CRC-32 checksum value for chunk [0, 0x3bd73): Mirror 1: 0xb532438f Mirror 10: 0xb532438f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd73: skipped 000498 1782207479.664657 trunc from 0x02ba38 to 0x036a75 (0xb03e bytes) 000499 1782207479.755804 fallocate 0x00278f thru 0x00c942 (0xa1b4 bytes) 000502 1782207479.774689 mapread 0x00b1eb thru 0x00b407 (0x21d bytes) 000503 1782207479.786140 mapwrite 0x033da0 thru 0x03f5f8 (0xb859 bytes) 000504 1782207479.933202 MIRROR_RESYNC 6 mirrors 000505 1782207481.107419 mapwrite 0x02b916 thru 0x037e9d (0xc588 bytes) 000506 1782207481.234179 trunc from 0x01fd8a to 0x02e7cd (0xea44 bytes) 000507 1782207481.241211 trunc from 0x007dff to 0x010190 (0x8392 bytes) 000508 1782207481.246668 fallocate 0x0280b6 thru 0x030086 (0x7fd1 bytes) 000509 1782207481.258087 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f5f9): Mirror 6: 0x21afae17 (stale) Mirror 8: 0x21afae17 (stale) Mirror 9: 0x21afae17 (stale) Mirror 10: 0x21afae17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5f9): Mirror 6: 0x21afae17 (stale) Mirror 8: 0x21afae17 (stale) Mirror 9: 0x21afae17 (stale) Mirror 10: 0x21afae17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5f9: skipped 000510 1782207481.753986 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f5f9): Mirror 8: 0x21afae17 (stale) Mirror 9: 0x21afae17 (stale) Mirror 10: 0x21afae17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5f9): Mirror 8: 0x21afae17 (stale) Mirror 9: 0x21afae17 (stale) Mirror 10: 0x21afae17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5f9: skipped 000511 1782207482.127031 read 0x02cde8 thru 0x02d0ac (0x2c5 bytes) 000512 1782207482.132420 read 0x033eff thru 0x03457a (0x67c bytes) 000513 1782207482.155660 read 0x0231d4 thru 0x025b2b (0x2958 bytes) 000514 1782207482.156370 mapwrite 0x019ef2 thru 0x025060 (0xb16f bytes) 000516 1782207482.193052 fallocate 0x006839 thru 0x00b4f4 (0x4cbc bytes) 000517 1782207482.200673 mapwrite 0x01245a thru 0x01bfb0 (0x9b57 bytes) 000518 1782207482.234699 fallocate 0x00bc1b thru 0x01b441 (0xf827 bytes) 000519 1782207482.239731 mapwrite 0x02857d thru 0x02c2dc (0x3d60 bytes) 000520 1782207482.269020 fallocate 0x0116f2 thru 0x01ccf1 (0xb600 bytes) 000522 1782207482.281158 write 0x021af4 thru 0x02f8d2 (0xdddf bytes) 000524 1782207482.290559 mapwrite 0x035caf thru 0x03ffff (0xa351 bytes) 000525 1782207482.368559 trunc from 0x001183 to 0x0057ff (0x467d bytes) 000526 1782207482.373900 trunc from 0x040000 to 0x00d6f0 (0x3290f bytes) 000527 1782207482.391156 punch from 0x00c4cd to 0x00d6f0 (0x1224 bytes) 000528 1782207482.396060 punch from 0x001197 to 0x00d6f0 (0xc55a bytes) 000529 1782207482.405595 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd6f1): Mirror 9: 0x5463eaae (stale) Mirror 10: 0x5463eaae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6f1): Mirror 9: 0x5463eaae (stale) Mirror 10: 0x5463eaae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6f1: skipped 000530 1782207482.756617 mapwrite 0x002293 thru 0x00fea4 (0xdc12 bytes) 000531 1782207482.812175 read 0x00c5b4 thru 0x00fea4 (0x38f1 bytes) 000532 1782207482.815099 fallocate 0x015054 thru 0x01c5e9 (0x7596 bytes) 000533 1782207482.819734 fallocate 0x014b6f thru 0x01b34f (0x67e1 bytes) 000534 1782207482.831390 write 0x006260 thru 0x0123d0 (0xc171 bytes) 000535 1782207482.843102 trunc from 0x01b350 to 0x02f84a (0x144fb bytes) 000536 1782207482.883891 trunc from 0x02f84b to 0x02f649 (0x201 bytes) 000537 1782207482.909147 trunc from 0x0207a6 to 0x025359 (0x4bb4 bytes) 000538 1782207482.919035 trunc from 0x009727 to 0x00b4ab (0x1d85 bytes) 000539 1782207482.935320 mapread 0x014463 thru 0x022b92 (0xe730 bytes) 000541 1782207482.956765 mapwrite 0x02bc6a thru 0x039d76 (0xe10d bytes) 000542 1782207483.032222 fallocate 0x00bf65 thru 0x00fa0e (0x3aaa bytes) 000543 1782207483.040170 trunc from 0x00a70d to 0x01570b (0xafff bytes) 000544 1782207483.047217 trunc from 0x001266 to 0x00c8a7 (0xb642 bytes) 000545 1782207483.055237 punch from 0x00fdf3 to 0x017dc1 (0x7fcf bytes) 000546 1782207483.063451 mapread 0x026b8e thru 0x030292 (0x9705 bytes) 000547 1782207483.064757 write 0x02eb49 thru 0x034371 (0x5829 bytes) 000548 1782207483.065890 fallocate 0x029108 thru 0x02fdc2 (0x6cbb bytes) 000549 1782207483.071771 fallocate 0x028fcd thru 0x03896b (0xf99f bytes) 000551 1782207483.077650 trunc from 0x039d77 to 0x016700 (0x23676 bytes) 000552 1782207483.102612 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16701): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16701): Mirror 9: 0x10ccd4ad (stale) Mirror 10: 0x10ccd4ad (stale) CRC-32 checksum value for chunk [0, 0x16701): Mirror 1: 0x10ccd4ad Mirror 11: 0x10ccd4ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16701: skipped 000553 1782207484.401076 read 0x00fd38 thru 0x016700 (0x69c9 bytes) 000554 1782207484.438436 fallocate 0x0281df thru 0x02a78a (0x25ac bytes) 000555 1782207484.525151 mapwrite 0x014ab3 thru 0x01be78 (0x73c6 bytes) 000556 1782207484.605089 trunc from 0x01be79 to 0x03dc6d (0x21df5 bytes) 000557 1782207484.632693 mapwrite 0x024752 thru 0x029abd (0x536c bytes) 000558 1782207484.729507 mapwrite 0x02b71f thru 0x02d871 (0x2153 bytes) 000559 1782207484.774887 fallocate 0x02ac61 thru 0x0331c3 (0x8563 bytes) 000560 1782207484.782578 punch from 0x004722 to 0x005cfa (0x15d9 bytes) 000561 1782207484.797795 write 0x001627 thru 0x00b7fd (0xa1d7 bytes) 000562 1782207484.831701 punch from 0x00a334 to 0x019daa (0xfa77 bytes) 000563 1782207484.854103 read 0x01073c thru 0x020219 (0xfade bytes) 000564 1782207484.870569 read 0x0324dc thru 0x034da0 (0x28c5 bytes) 000565 1782207484.872188 punch from 0x01d6dc to 0x01e3e6 (0xd0b bytes) 000566 1782207484.881305 mapread 0x01b2ae thru 0x0227a9 (0x74fc bytes) 000567 1782207484.886907 fallocate 0x02d0da thru 0x03459c (0x74c3 bytes) 000568 1782207484.893027 trunc from 0x03dc6e to 0x004eff (0x38d6e bytes) 000570 1782207484.914035 mapread 0x004430 thru 0x004eff (0xad0 bytes) 000571 1782207484.918992 trunc from 0x004f00 to 0x03bb51 (0x36c52 bytes) 000572 1782207484.946011 MIRROR_RESYNC 4 mirrors 000573 1782207485.933560 trunc from 0x03bb52 to 0x01830f (0x23842 bytes) 000574 1782207485.984102 trunc from 0x0050a5 to 0x010776 (0xb6d2 bytes) 000575 1782207485.991786 punch from 0x0099de to 0x018197 (0xe7ba bytes) 000577 1782207485.997948 mapread 0x006007 thru 0x0095a3 (0x359d bytes) 000578 1782207486.035500 trunc from 0x00aa4e to 0x00bdf6 (0x13a9 bytes) 000580 1782207486.042527 read 0x00f316 thru 0x017639 (0x8324 bytes) 000581 1782207486.055211 fallocate 0x01c100 thru 0x01ddf0 (0x1cf1 bytes) 000582 1782207486.067300 mapread 0x00e7b8 thru 0x01830f (0x9b58 bytes) 000583 1782207486.077363 punch from 0x0152d0 to 0x01830f (0x3040 bytes) 000584 1782207486.082723 fallocate 0x020d64 thru 0x0223d6 (0x1673 bytes) 000585 1782207486.089944 fallocate 0x038664 thru 0x03ffff (0x799c bytes) 000586 1782207486.101296 trunc from 0x00e7af to 0x011d4b (0x359d bytes) 000587 1782207486.115033 mapread 0x01fe46 thru 0x0223d6 (0x2591 bytes) 000588 1782207486.137024 MIRROR_RESYNC 4 mirrors 000589 1782207487.107035 trunc from 0x002018 to 0x008c63 (0x6c4c bytes) 000590 1782207487.145468 trunc from 0x007c3a to 0x013280 (0xb647 bytes) 000591 1782207487.159759 trunc from 0x011c1e to 0x01b737 (0x9b1a bytes) 000592 1782207487.175499 write 0x00af0a thru 0x00f3dc (0x44d3 bytes) 000593 1782207487.196350 write 0x03b3b0 thru 0x03ffff (0x4c50 bytes) 000594 1782207487.201529 MIRROR_RESYNC 4 mirrors 000595 1782207488.299157 read 0x03a7ff thru 0x03ffff (0x5801 bytes) 000596 1782207488.308471 read 0x0061b6 thru 0x00d3c0 (0x720b bytes) 000597 1782207488.327324 punch from 0x017358 to 0x01879b (0x1444 bytes) 000598 1782207488.358756 trunc from 0x040000 to 0x03ed35 (0x12ca bytes) 000599 1782207488.391065 trunc from 0x03ed36 to 0x00ae25 (0x33f10 bytes) 000600 1782207488.417601 mapread 0x00a9f0 thru 0x00ae25 (0x436 bytes) 000601 1782207488.442114 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xae26): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae26): Mirror 9: 0xa7b84abd (stale) Mirror 10: 0xa7b84abd (stale) Mirror 11: 0xa7b84abd (stale) CRC-32 checksum value for chunk [0, 0xae26): Mirror 1: 0xa7b84abd Mirror 12: 0xa7b84abd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae26: skipped 000602 1782207489.809533 mapwrite 0x018e8d thru 0x025e8e (0xd002 bytes) 000603 1782207489.962269 MIRROR_RESYNC 5 mirrors 000605 1782207491.518531 mapwrite 0x02cd42 thru 0x02e5eb (0x18aa bytes) 000606 1782207491.601597 mapread 0x0031fd thru 0x003b0b (0x90f bytes) 000607 1782207491.621621 write 0x031723 thru 0x0364db (0x4db9 bytes) 000608 1782207491.626382 mapread 0x020235 thru 0x027e0d (0x7bd9 bytes) 000609 1782207491.652236 punch from 0x00df69 to 0x0107dd (0x2875 bytes) 000610 1782207491.671640 mapwrite 0x02891b thru 0x02ff50 (0x7636 bytes) 000611 1782207491.735230 fallocate 0x021206 thru 0x02a1ec (0x8fe7 bytes) 000612 1782207491.743693 read 0x02be5c thru 0x03177e (0x5923 bytes) 000613 1782207491.747498 fallocate 0x018faf thru 0x01d16e (0x41c0 bytes) 000614 1782207491.753325 punch from 0x018bb7 to 0x027ccc (0xf116 bytes) 000615 1782207491.759568 read 0x013d10 thru 0x018e11 (0x5102 bytes) 000616 1782207491.774457 punch from 0x0195a7 to 0x021177 (0x7bd1 bytes) 000617 1782207491.779506 trunc from 0x0364dc to 0x005468 (0x31073 bytes) 000618 1782207491.817334 MIRROR_RESYNC 5 mirrors 000619 1782207492.883757 trunc from 0x005469 to 0x03a083 (0x34c1b bytes) 000620 1782207492.931762 trunc from 0x02eb02 to 0x02f634 (0xb33 bytes) 000621 1782207492.936452 mapread 0x0306c7 thru 0x03a083 (0x99bd bytes) 000622 1782207492.961287 trunc from 0x0350cf to 0x03a083 (0x4fb5 bytes) 000623 1782207492.968950 punch from 0x02afff to 0x0338db (0x88dd bytes) 000624 1782207492.974689 trunc from 0x028ad4 to 0x02f233 (0x6760 bytes) 000625 1782207492.979074 mapread 0x0117f0 thru 0x017b80 (0x6391 bytes) 000626 1782207492.989363 mapwrite 0x02cde0 thru 0x030c9c (0x3ebd bytes) 000627 1782207493.016652 read 0x00525f thru 0x007c2b (0x29cd bytes) 000628 1782207493.023305 trunc from 0x03a084 to 0x01f79a (0x1a8e9 bytes) 000629 1782207493.052586 trunc from 0x000a65 to 0x010775 (0xfd11 bytes) 000630 1782207493.063848 fallocate 0x013031 thru 0x01925b (0x622b bytes) 000631 1782207493.068022 trunc from 0x007dcb to 0x010b42 (0x8d78 bytes) 000632 1782207493.071467 mapread 0x00b4fa thru 0x012481 (0x6f88 bytes) 000633 1782207493.078974 mapwrite 0x00dacd thru 0x0146e3 (0x6c17 bytes) 000634 1782207493.102677 MIRROR_RESYNC 5 mirrors 000635 1782207493.998909 mapwrite 0x0040ec thru 0x0048b5 (0x7ca bytes) 000636 1782207494.118707 mapread 0x00c860 thru 0x00edd4 (0x2575 bytes) 000637 1782207494.128197 read 0x00d827 thru 0x017b79 (0xa353 bytes) 000638 1782207494.134741 trunc from 0x012006 to 0x015ee5 (0x3ee0 bytes) 000639 1782207494.140787 trunc from 0x01f79b to 0x00f9a5 (0xfdf5 bytes) 000640 1782207494.183069 trunc from 0x00f9a6 to 0x011619 (0x1c74 bytes) 000641 1782207494.198347 trunc from 0x01161a to 0x008413 (0x9206 bytes) 000642 1782207494.217525 MIRROR_RESYNC 5 mirrors 000643 1782207495.119274 MIRROR_RESYNC 5 mirrors 000644 1782207495.632766 write 0x02ee72 thru 0x02ff56 (0x10e5 bytes) 000645 1782207495.686005 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 9: 0x3ce086a6 (stale) Mirror 10: 0x3ce086a6 (stale) Mirror 11: 0x3ce086a6 (stale) Mirror 12: 0x3ce086a6 (stale) CRC-32 checksum value for chunk [0, 0x2ff57): Mirror 1: 0xffeecb75 Mirror 13: 0xffeecb75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff57: skipped 000646 1782207497.030438 mapwrite 0x026f6e thru 0x0323ec (0xb47f bytes) 000647 1782207497.148263 punch from 0x020108 to 0x027fd9 (0x7ed2 bytes) 000648 1782207497.157719 mapwrite 0x03b4b0 thru 0x03f98c (0x44dd bytes) 000649 1782207497.206718 read 0x029053 thru 0x02d94c (0x48fa bytes) 000650 1782207497.207933 punch from 0x00837e to 0x00bb01 (0x3784 bytes) 000651 1782207497.214118 write 0x014d7c thru 0x016287 (0x150c bytes) 000652 1782207497.224983 mapread 0x01860b thru 0x022a44 (0xa43a bytes) 000653 1782207497.245062 read 0x033ac9 thru 0x03e565 (0xaa9d bytes) 000654 1782207497.259070 write 0x0139db thru 0x02064b (0xcc71 bytes) 000655 1782207497.262014 mapwrite 0x0360c0 thru 0x03a55e (0x449f bytes) 000656 1782207497.297188 fallocate 0x005304 thru 0x00c1cf (0x6ecc bytes) 000657 1782207497.318715 trunc from 0x03f98d to 0x018c0f (0x26d7d bytes) 000658 1782207497.364304 punch from 0x015dc6 to 0x018c0f (0x2e4a bytes) 000660 1782207497.376180 fallocate 0x02b63d thru 0x038f7f (0xd943 bytes) 000661 1782207497.394666 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38f80): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f80): Mirror 9: 0xf9aa92df (stale) Mirror 10: 0xf9aa92df (stale) Mirror 11: 0xf9aa92df (stale) Mirror 12: 0xf9aa92df (stale) Mirror 13: 0x82a1c0d7 (stale) CRC-32 checksum value for chunk [0, 0x38f80): Mirror 1: 0xb1d8c880 Mirror 14: 0xb1d8c880 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f80: skipped 000662 1782207498.523910 fallocate 0x032a41 thru 0x037f7a (0x553a bytes) 000663 1782207498.573244 write 0x030537 thru 0x0324f3 (0x1fbd bytes) 000664 1782207498.590734 trunc from 0x00288c to 0x00b265 (0x89da bytes) 000665 1782207498.595778 write 0x03387c thru 0x034b3a (0x12bf bytes) 000666 1782207498.606091 read 0x01048e thru 0x014b03 (0x4676 bytes) 000667 1782207498.618923 punch from 0x017d50 to 0x01fa80 (0x7d31 bytes) 000668 1782207498.624717 read 0x02f042 thru 0x03513e (0x60fd bytes) 000669 1782207498.690669 fallocate 0x0270a5 thru 0x02b53f (0x449b bytes) 000670 1782207498.696010 mapread 0x02f14c thru 0x0368ac (0x7761 bytes) 000671 1782207498.698782 trunc from 0x01a5d3 to 0x02851a (0xdf48 bytes) 000672 1782207498.705120 punch from 0x00ae74 to 0x015a3a (0xabc7 bytes) 000673 1782207498.710976 write 0x01fa08 thru 0x023fcb (0x45c4 bytes) 000674 1782207498.739513 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f80): Mirror 9: 0xf9aa92df (stale) Mirror 10: 0xf9aa92df (stale) Mirror 11: 0xf9aa92df (stale) Mirror 13: 0x82a1c0d7 (stale) Mirror 14: 0xb1d8c880 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f80): Mirror 9: 0xf9aa92df (stale) Mirror 10: 0xf9aa92df (stale) Mirror 11: 0xf9aa92df (stale) Mirror 13: 0x82a1c0d7 (stale) Mirror 14: 0xb1d8c880 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f80: skipped 000676 1782207499.416218 write 0x001099 thru 0x00e932 (0xd89a bytes) 000677 1782207499.463522 read 0x03778c thru 0x038f7f (0x17f4 bytes) 000678 1782207499.479581 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38f80): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f80): Mirror 9: 0xf9aa92df (stale) Mirror 10: 0xf9aa92df (stale) Mirror 11: 0xf9aa92df (stale) Mirror 13: 0x82a1c0d7 (stale) Mirror 14: 0xb1d8c880 (stale) CRC-32 checksum value for chunk [0, 0x38f80): Mirror 1: 0xd16cbecc Mirror 15: 0xd16cbecc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f80: skipped 000679 1782207500.970166 write 0x01c6ab thru 0x0273de (0xad34 bytes) 000680 1782207501.044577 read 0x024d1f thru 0x02db20 (0x8e02 bytes) 000681 1782207501.065394 trunc from 0x038f80 to 0x03d035 (0x40b6 bytes) 000682 1782207501.103734 mapwrite 0x023e4d thru 0x02d54d (0x9701 bytes) 000684 1782207501.172222 write 0x0294b3 thru 0x0341ce (0xad1c bytes) 000685 1782207501.187609 mapread 0x013fc0 thru 0x01bf42 (0x7f83 bytes) 000686 1782207501.222581 trunc from 0x03d036 to 0x00687f (0x367b6 bytes) 000687 1782207501.252275 trunc from 0x006880 to 0x02bd34 (0x254b5 bytes) 000688 1782207501.316288 trunc from 0x026c52 to 0x02bd34 (0x50e3 bytes) 000689 1782207501.330101 fallocate 0x022194 thru 0x022a00 (0x86d bytes) 000690 1782207501.339391 trunc from 0x026ec6 to 0x02b98d (0x4ac8 bytes) 000691 1782207501.357656 mapread 0x002016 thru 0x00e0e1 (0xc0cc bytes) 000692 1782207501.370071 trunc from 0x0034da to 0x005944 (0x246b bytes) 000693 1782207501.375881 trunc from 0x02bd35 to 0x03a6c1 (0xe98d bytes) 000694 1782207501.394004 mapwrite 0x01f125 thru 0x026021 (0x6efd bytes) 000695 1782207501.483374 mapwrite 0x003f2c thru 0x01075e (0xc833 bytes) 000696 1782207501.559586 fallocate 0x008461 thru 0x00c60a (0x41aa bytes) 000697 1782207501.576244 write 0x00bb20 thru 0x015e92 (0xa373 bytes) 000700 1782207501.596303 punch from 0x018753 to 0x021295 (0x8b43 bytes) 000702 1782207501.612634 read 0x00c5e0 thru 0x00ea64 (0x2485 bytes) 000703 1782207501.613796 mapwrite 0x03b4dd thru 0x03ffff (0x4b23 bytes) 000704 1782207501.719624 write 0x023ebe thru 0x028e4f (0x4f92 bytes) 000705 1782207501.721234 trunc from 0x040000 to 0x0187b6 (0x27849 bytes) 000706 1782207501.735905 write 0x00917c thru 0x0132f5 (0xa17a bytes) 000707 1782207501.737374 write 0x013bf0 thru 0x018d52 (0x5163 bytes) 000708 1782207501.751935 trunc from 0x018d53 to 0x02c788 (0x13a36 bytes) 000709 1782207501.779120 punch from 0x01df31 to 0x022ade (0x4bae bytes) 000710 1782207501.785226 read 0x000a66 thru 0x00366d (0x2c08 bytes) 000711 1782207501.785944 write 0x00825d thru 0x00f04f (0x6df3 bytes) 000712 1782207501.790028 trunc from 0x02c789 to 0x026b13 (0x5c75 bytes) 000713 1782207501.822216 trunc from 0x00325b to 0x00ec0e (0xb9b4 bytes) 000715 1782207501.837404 punch from 0x0094ec to 0x00f5f8 (0x610d bytes) 000716 1782207501.842461 punch from 0x002f2b to 0x00b8a4 (0x897a bytes) 000717 1782207501.848101 trunc from 0x026b14 to 0x03120f (0xa6fc bytes) 000718 1782207501.861601 mapread 0x0293bc thru 0x02c846 (0x348b bytes) 000722 1782207501.872551 punch from 0x00938d to 0x00f4e5 (0x6159 bytes) 000723 1782207501.887653 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31210): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31210): Mirror 9: 0x6b56f64f (stale) Mirror 10: 0x6b56f64f (stale) Mirror 11: 0x6b56f64f (stale) Mirror 13: 0xa999da7c (stale) Mirror 14: 0xcf39b8ff (stale) Mirror 15: 0xe16d0ab3 (stale) CRC-32 checksum value for chunk [0, 0x31210): Mirror 1: 0x166e7651 Mirror 16: 0x166e7651 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31210: skipped 000724 1782207502.845294 write 0x025c56 thru 0x02fea0 (0xa24b bytes) 000725 1782207502.941375 trunc from 0x031210 to 0x00bdbb (0x25454 bytes) 000726 1782207502.975912 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbdbc): Mirror 9: 0x60407897 (stale) Mirror 10: 0x60407897 (stale) Mirror 11: 0x60407897 (stale) Mirror 14: 0x60407897 (stale) Mirror 15: 0xcba5f5e1 (stale) Mirror 16: 0x258ad0d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbdbc): Mirror 9: 0x60407897 (stale) Mirror 10: 0x60407897 (stale) Mirror 11: 0x60407897 (stale) Mirror 14: 0x60407897 (stale) Mirror 15: 0xcba5f5e1 (stale) Mirror 16: 0x258ad0d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbdbc: skipped 000727 1782207503.514655 trunc from 0x0042b4 to 0x00ad7a (0x6ac7 bytes) 000728 1782207503.524339 MIRROR_RESYNC 7 mirrors 000729 1782207504.881872 mapread 0x002cbd thru 0x009620 (0x6964 bytes) 000730 1782207504.905249 trunc from 0x00bdbc to 0x00b480 (0x93b bytes) 000731 1782207504.948464 trunc from 0x00b481 to 0x02eb76 (0x236f6 bytes) 000732 1782207504.966223 mapwrite 0x01889f thru 0x01c3b7 (0x3b19 bytes) 000734 1782207505.001099 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2eb77): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb77): Mirror 9: 0x263c853f (stale) Mirror 10: 0x263c853f (stale) Mirror 11: 0x263c853f (stale) Mirror 14: 0x263c853f (stale) Mirror 15: 0x263c853f (stale) Mirror 16: 0x263c853f (stale) CRC-32 checksum value for chunk [0, 0x2eb77): Mirror 1: 0xe6f627ad Mirror 17: 0xe6f627ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb77: skipped 000735 1782207506.140981 read 0x021e39 thru 0x0236fb (0x18c3 bytes) 000736 1782207506.154170 mapwrite 0x0276af thru 0x02cfb6 (0x5908 bytes) 000737 1782207506.352518 punch from 0x002f9b to 0x00c43a (0x94a0 bytes) 000738 1782207506.383698 MIRROR_RESYNC 8 mirrors 000739 1782207508.371157 punch from 0x01aabf to 0x0233f0 (0x8932 bytes) 000740 1782207508.405064 read 0x024078 thru 0x02a6bf (0x6648 bytes) 000741 1782207508.423114 MIRROR_RESYNC 8 mirrors 000743 1782207510.409631 read 0x0279aa thru 0x02ce6c (0x54c3 bytes) 000744 1782207510.417120 write 0x02b568 thru 0x038043 (0xcadc bytes) 000745 1782207510.473284 mapread 0x019b98 thru 0x027c83 (0xe0ec bytes) 000747 1782207510.513141 mapread 0x00c791 thru 0x019795 (0xd005 bytes) 000748 1782207510.521973 write 0x00b0c8 thru 0x01964c (0xe585 bytes) 000749 1782207510.524440 read 0x014710 thru 0x01a185 (0x5a76 bytes) 000750 1782207510.533162 MIRROR_RESYNC 8 mirrors 000751 1782207512.390934 fallocate 0x02cbb6 thru 0x03269f (0x5aea bytes) 000752 1782207512.441702 punch from 0x027aba to 0x036e6f (0xf3b6 bytes) 000753 1782207512.447844 trunc from 0x038044 to 0x03d89d (0x585a bytes) 000754 1782207512.471796 mapread 0x030654 thru 0x034946 (0x42f3 bytes) 000756 1782207512.504561 trunc from 0x000495 to 0x0065bb (0x6127 bytes) 000757 1782207512.515777 write 0x026973 thru 0x02fdb5 (0x9443 bytes) 000758 1782207512.524699 mapwrite 0x02c5d2 thru 0x03a148 (0xdb77 bytes) 000759 1782207512.580455 punch from 0x029e00 to 0x02b2ff (0x1500 bytes) 000760 1782207512.590997 write 0x01d630 thru 0x027289 (0x9c5a bytes) 000761 1782207512.596705 mapwrite 0x038e7a thru 0x03ffff (0x7186 bytes) 000762 1782207512.674198 trunc from 0x038a12 to 0x03efd2 (0x65c1 bytes) 000763 1782207512.683908 trunc from 0x040000 to 0x03b9a1 (0x465e bytes) 000764 1782207512.716711 mapwrite 0x010f4f thru 0x01b9f4 (0xaaa6 bytes) 000765 1782207512.865835 read 0x0135a1 thru 0x017ba4 (0x4604 bytes) 000766 1782207512.869742 mapread 0x02e425 thru 0x034b74 (0x6750 bytes) 000767 1782207512.871788 write 0x015767 thru 0x0234f1 (0xdd8b bytes) 000768 1782207512.875795 read 0x011659 thru 0x0167ea (0x5192 bytes) 000769 1782207512.881548 fallocate 0x03474c thru 0x03ffff (0xb8b4 bytes) 000770 1782207512.888475 write 0x027273 thru 0x0300c9 (0x8e57 bytes) 000771 1782207512.896367 mapwrite 0x026bb6 thru 0x030924 (0x9d6f bytes) 000772 1782207512.986484 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9ca6cfb5 (stale) Mirror 10: 0x9ca6cfb5 (stale) Mirror 11: 0x9ca6cfb5 (stale) Mirror 14: 0x9ca6cfb5 (stale) Mirror 15: 0x9ca6cfb5 (stale) Mirror 16: 0x9ca6cfb5 (stale) Mirror 17: 0x9ca6cfb5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc2e7ba2b Mirror 18: 0xc2e7ba2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1782207514.385987 trunc from 0x018c87 to 0x0227b9 (0x9b33 bytes) 000774 1782207514.449132 write 0x03cc85 thru 0x03ffff (0x337b bytes) 000775 1782207514.464204 mapread 0x008d62 thru 0x01536e (0xc60d bytes) 000776 1782207514.499708 read 0x01718d thru 0x0209d9 (0x984d bytes) 000777 1782207514.520251 mapread 0x03d891 thru 0x03ffff (0x276f bytes) 000778 1782207514.523470 punch from 0x00d8ea to 0x017b4a (0xa261 bytes) 000779 1782207514.543997 trunc from 0x040000 to 0x000a3b (0x3f5c4 bytes) 000781 1782207514.598497 mapread 0x0001d2 thru 0x000a3b (0x86a bytes) 000782 1782207514.607981 mapwrite 0x03a5c1 thru 0x03af33 (0x973 bytes) 000784 1782207514.677611 write 0x018d61 thru 0x01aba5 (0x1e45 bytes) 000785 1782207514.685516 read 0x02114c thru 0x0238f5 (0x27aa bytes) 000786 1782207514.701264 mapwrite 0x001044 thru 0x00c47a (0xb437 bytes) 000787 1782207514.751258 read 0x00c383 thru 0x00f9bb (0x3639 bytes) 000788 1782207514.752572 read 0x012d8c thru 0x021ac3 (0xed38 bytes) 000790 1782207514.754856 mapwrite 0x002238 thru 0x00cd1f (0xaae8 bytes) 000791 1782207514.799536 mapread 0x022d34 thru 0x030cb4 (0xdf81 bytes) 000792 1782207514.803086 punch from 0x0058a3 to 0x006ad0 (0x122e bytes) 000793 1782207514.813786 punch from 0x032a3f to 0x03af33 (0x84f5 bytes) 000794 1782207514.819383 read 0x02a26d thru 0x02b460 (0x11f4 bytes) 000795 1782207514.826881 MIRROR_RESYNC 9 mirrors 000796 1782207516.946241 punch from 0x021dc0 to 0x027c16 (0x5e57 bytes) 000797 1782207517.006747 trunc from 0x034449 to 0x03af33 (0x6aeb bytes) 000798 1782207517.017571 trunc from 0x036825 to 0x03af33 (0x470f bytes) 000799 1782207517.024325 mapread 0x02cc36 thru 0x03af33 (0xe2fe bytes) 000800 1782207517.066791 trunc from 0x03af34 to 0x036308 (0x4c2b bytes) 000801 1782207517.102118 fallocate 0x003c63 thru 0x00c7fc (0x8b9a bytes) 000802 1782207517.111582 write 0x01a48e thru 0x0270dc (0xcc4f bytes) 000803 1782207517.121903 fallocate 0x012f18 thru 0x01dd1a (0xae03 bytes) 000804 1782207517.129311 punch from 0x0279c5 to 0x02934b (0x1987 bytes) 000805 1782207517.144472 trunc from 0x002585 to 0x00587b (0x32f7 bytes) 000806 1782207517.159488 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36309): Mirror 10: 0x294a00ef (stale) Mirror 11: 0x294a00ef (stale) Mirror 14: 0x294a00ef (stale) Mirror 15: 0x294a00ef (stale) Mirror 16: 0x294a00ef (stale) Mirror 17: 0x294a00ef (stale) Mirror 18: 0x294a00ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36309): Mirror 10: 0x294a00ef (stale) Mirror 11: 0x294a00ef (stale) Mirror 14: 0x294a00ef (stale) Mirror 15: 0x294a00ef (stale) Mirror 16: 0x294a00ef (stale) Mirror 17: 0x294a00ef (stale) Mirror 18: 0x294a00ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36309: skipped 000807 1782207517.861599 fallocate 0x02b7e4 thru 0x032ee3 (0x7700 bytes) 000808 1782207517.874263 punch from 0x0036af to 0x005753 (0x20a5 bytes) 000809 1782207517.897657 trunc from 0x036309 to 0x0256ab (0x10c5d bytes) 000810 1782207517.955493 mapread 0x0092d6 thru 0x00d0a8 (0x3dd3 bytes) 000811 1782207518.007150 read 0x024da8 thru 0x0256ab (0x904 bytes) 000812 1782207518.019734 mapread 0x014797 thru 0x021837 (0xd0a1 bytes) 000814 1782207518.046949 write 0x0033de thru 0x0084ab (0x50ce bytes) 000815 1782207518.058175 write 0x00e010 thru 0x01dee5 (0xfed6 bytes) 000817 1782207518.063076 punch from 0x022306 to 0x0256ab (0x33a6 bytes) 000818 1782207518.071151 punch from 0x00495f to 0x00ee6b (0xa50d bytes) 000819 1782207518.091683 trunc from 0x0256ac to 0x00bb0f (0x19b9c bytes) 000821 1782207518.130975 trunc from 0x00bb10 to 0x0137ac (0x7c9d bytes) 000822 1782207518.158472 read 0x0088ac thru 0x00ec3a (0x638f bytes) 000823 1782207518.187164 punch from 0x01066b to 0x0137ac (0x3142 bytes) 000824 1782207518.194509 mapread 0x00dce4 thru 0x0137ac (0x5ac9 bytes) 000825 1782207518.222528 trunc from 0x0137ad to 0x003804 (0xffa8 bytes) 000826 1782207518.255388 mapwrite 0x00d554 thru 0x0127f0 (0x529d bytes) 000827 1782207518.421163 fallocate 0x02bed7 thru 0x02e54a (0x2674 bytes) 000828 1782207518.435699 mapwrite 0x03aac5 thru 0x03be64 (0x13a0 bytes) 000829 1782207518.509788 trunc from 0x03be65 to 0x014714 (0x27750 bytes) 000830 1782207518.530752 fallocate 0x022a04 thru 0x02cc9f (0xa29c bytes) 000831 1782207518.539155 read 0x00f165 thru 0x0196b9 (0xa555 bytes) 000832 1782207518.556577 mapread 0x005f07 thru 0x00b9dc (0x5ad6 bytes) 000833 1782207518.557949 mapread 0x02b239 thru 0x02b3d9 (0x1a1 bytes) 000834 1782207518.561667 mapread 0x00f2e3 thru 0x0135b3 (0x42d1 bytes) 000836 1782207518.571310 trunc from 0x024a47 to 0x02cc9f (0x8259 bytes) 000837 1782207518.577189 read 0x02a310 thru 0x02a365 (0x56 bytes) 000838 1782207518.606999 mapread 0x01280f thru 0x014de9 (0x25db bytes) 000839 1782207518.620971 MIRROR_RESYNC 8 mirrors 000840 1782207520.171306 punch from 0x00848b to 0x00b287 (0x2dfd bytes) 000842 1782207520.236874 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 10: 0x2bb9fa81 (stale) Mirror 11: 0x2bb9fa81 (stale) Mirror 14: 0x2bb9fa81 (stale) Mirror 15: 0x2bb9fa81 (stale) Mirror 16: 0x2bb9fa81 (stale) Mirror 17: 0x2bb9fa81 (stale) Mirror 18: 0x2bb9fa81 (stale) CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 1: 0x2bb9fa81 Mirror 19: 0x2bb9fa81 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cca0: skipped 000843 1782207521.341257 trunc from 0x01f70c to 0x02701e (0x7913 bytes) 000844 1782207521.390546 read 0x01b5bf thru 0x026606 (0xb048 bytes) 000845 1782207521.405449 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 10: 0x2bb9fa81 (stale) Mirror 11: 0x2bb9fa81 (stale) Mirror 15: 0x2bb9fa81 (stale) Mirror 16: 0x2bb9fa81 (stale) Mirror 17: 0x2bb9fa81 (stale) Mirror 18: 0x2bb9fa81 (stale) Mirror 19: 0x2bb9fa81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 10: 0x2bb9fa81 (stale) Mirror 11: 0x2bb9fa81 (stale) Mirror 15: 0x2bb9fa81 (stale) Mirror 16: 0x2bb9fa81 (stale) Mirror 17: 0x2bb9fa81 (stale) Mirror 18: 0x2bb9fa81 (stale) Mirror 19: 0x2bb9fa81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cca0: skipped 000846 1782207522.047371 punch from 0x0227d1 to 0x026322 (0x3b52 bytes) 000847 1782207522.064708 trunc from 0x027442 to 0x0278d9 (0x498 bytes) 000848 1782207522.072255 trunc from 0x011e25 to 0x017a12 (0x5bee bytes) 000849 1782207522.082242 write 0x0231ed thru 0x02a830 (0x7644 bytes) 000850 1782207522.129742 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 10: 0x2bb9fa81 (stale) Mirror 11: 0x2bb9fa81 (stale) Mirror 15: 0x2bb9fa81 (stale) Mirror 16: 0x2bb9fa81 (stale) Mirror 17: 0x2bb9fa81 (stale) Mirror 18: 0x2bb9fa81 (stale) Mirror 19: 0x2bb9fa81 (stale) CRC-32 checksum value for chunk [0, 0x2cca0): Mirror 1: 0x64cbfea Mirror 20: 0x64cbfea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cca0: skipped 000852 1782207522.955243 read 0x001691 thru 0x00e88e (0xd1fe bytes) 000853 1782207522.972067 fallocate 0x03085b thru 0x036b80 (0x6326 bytes) 000854 1782207523.022443 mapread 0x00eead thru 0x01bbf8 (0xcd4c bytes) 000855 1782207523.049018 write 0x00caa3 thru 0x01826f (0xb7cd bytes) 000856 1782207523.054632 write 0x018268 thru 0x020501 (0x829a bytes) 000857 1782207523.056596 mapread 0x002f2e thru 0x007ce1 (0x4db4 bytes) 000858 1782207523.065797 trunc from 0x02cca0 to 0x032fc4 (0x6325 bytes) 000859 1782207523.088984 read 0x017440 thru 0x026fa3 (0xfb64 bytes) 000860 1782207523.103226 write 0x02d843 thru 0x033602 (0x5dc0 bytes) 000862 1782207523.104996 punch from 0x00a392 to 0x015806 (0xb475 bytes) 000863 1782207523.123912 trunc from 0x033603 to 0x004044 (0x2f5be bytes) 000864 1782207523.143094 fallocate 0x022b35 thru 0x02f3e9 (0xc8b5 bytes) 000865 1782207523.150588 mapwrite 0x006b2f thru 0x012344 (0xb816 bytes) 000866 1782207523.218229 fallocate 0x0112e2 thru 0x01939f (0x80be bytes) 000867 1782207523.222796 mapread 0x01d9e3 thru 0x01e542 (0xb60 bytes) 000868 1782207523.232726 read 0x002346 thru 0x00d63b (0xb2f6 bytes) 000870 1782207523.258563 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f3ea): Mirror 10: 0xa7e50ef6 (stale) Mirror 15: 0xa7e50ef6 (stale) Mirror 16: 0xa7e50ef6 (stale) Mirror 17: 0xa7e50ef6 (stale) Mirror 18: 0xa7e50ef6 (stale) Mirror 19: 0xa7e50ef6 (stale) Mirror 20: 0xa0b53509 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f3ea): Mirror 10: 0xa7e50ef6 (stale) Mirror 15: 0xa7e50ef6 (stale) Mirror 16: 0xa7e50ef6 (stale) Mirror 17: 0xa7e50ef6 (stale) Mirror 18: 0xa7e50ef6 (stale) Mirror 19: 0xa7e50ef6 (stale) Mirror 20: 0xa0b53509 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f3ea: skipped 000871 1782207523.692268 punch from 0x00dc4b to 0x01b641 (0xd9f7 bytes) 000872 1782207523.697464 trunc from 0x02f3ea to 0x014914 (0x1aad5 bytes) 000873 1782207523.716748 read 0x008a61 thru 0x010341 (0x78e1 bytes) 000874 1782207523.725485 punch from 0x01287f to 0x014914 (0x2096 bytes) 000875 1782207523.729736 mapread 0x00be49 thru 0x00db9d (0x1d55 bytes) 000876 1782207523.730884 mapread 0x0122be thru 0x014914 (0x2657 bytes) 000877 1782207523.745572 trunc from 0x014915 to 0x0063ba (0xe55a bytes) 000878 1782207523.762992 punch from 0x0002dc to 0x000d3f (0xa64 bytes) 000879 1782207523.769996 trunc from 0x004876 to 0x0063ba (0x1b45 bytes) 000880 1782207523.776605 mapread 0x0014d1 thru 0x0063ba (0x4eea bytes) 000881 1782207523.787529 trunc from 0x0063bb to 0x025cbf (0x1f905 bytes) 000882 1782207523.813158 write 0x027356 thru 0x02b690 (0x433b bytes) 000883 1782207523.814635 write 0x01c82d thru 0x0245d5 (0x7da9 bytes) 000884 1782207523.836504 trunc from 0x02b691 to 0x0142a7 (0x173e9 bytes) 000885 1782207523.871259 read 0x008d35 thru 0x00de68 (0x5134 bytes) 000886 1782207523.886454 write 0x0105a2 thru 0x018a2e (0x848d bytes) 000887 1782207523.889422 trunc from 0x007210 to 0x0134d0 (0xc2c1 bytes) 000888 1782207523.909837 trunc from 0x018a2f to 0x0188fd (0x131 bytes) 000889 1782207523.923750 read 0x007a17 thru 0x00b261 (0x384b bytes) 000890 1782207523.946805 read 0x016ddb thru 0x0188fd (0x1b23 bytes) 000891 1782207523.949601 fallocate 0x00f7e0 thru 0x011782 (0x1fa3 bytes) 000892 1782207523.955793 mapread 0x0011d9 thru 0x009f74 (0x8d9c bytes) 000893 1782207523.957583 trunc from 0x0188fe to 0x02f07a (0x1677d bytes) 000894 1782207523.980720 write 0x004399 thru 0x00e062 (0x9cca bytes) 000895 1782207523.983542 mapread 0x01d5a7 thru 0x0292d0 (0xbd2a bytes) 000896 1782207523.996656 write 0x031718 thru 0x03cd5e (0xb647 bytes) 000897 1782207524.003061 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 10: 0x13441ab6 (stale) Mirror 15: 0x13441ab6 (stale) Mirror 16: 0x13441ab6 (stale) Mirror 17: 0x13441ab6 (stale) Mirror 18: 0x13441ab6 (stale) Mirror 19: 0x13441ab6 (stale) Mirror 20: 0x4aa07350 (stale) CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 1: 0x49a0a46c Mirror 21: 0x49a0a46c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd5f: skipped 000898 1782207525.089752 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 10: 0x13441ab6 (stale) Mirror 15: 0x13441ab6 (stale) Mirror 16: 0x13441ab6 (stale) Mirror 17: 0x13441ab6 (stale) Mirror 18: 0x13441ab6 (stale) Mirror 19: 0x13441ab6 (stale) Mirror 20: 0x4aa07350 (stale) CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 1: 0x49a0a46c Mirror 21: 0x49a0a46c Mirror 22: 0x49a0a46c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd5f: skipped 000899 1782207526.054905 fallocate 0x027dbb thru 0x02c91f (0x4b65 bytes) 000901 1782207526.102130 write 0x0284de thru 0x02bae9 (0x360c bytes) 000902 1782207526.138577 mapread 0x0038c4 thru 0x007224 (0x3961 bytes) 000903 1782207526.184680 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 10: 0x13441ab6 (stale) Mirror 15: 0x13441ab6 (stale) Mirror 16: 0x13441ab6 (stale) Mirror 17: 0x13441ab6 (stale) Mirror 19: 0x13441ab6 (stale) Mirror 20: 0x4aa07350 (stale) Mirror 21: 0x49a0a46c (stale) Mirror 22: 0x49a0a46c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd5f): Mirror 10: 0x13441ab6 (stale) Mirror 15: 0x13441ab6 (stale) Mirror 16: 0x13441ab6 (stale) Mirror 17: 0x13441ab6 (stale) Mirror 19: 0x13441ab6 (stale) Mirror 20: 0x4aa07350 (stale) Mirror 21: 0x49a0a46c (stale) Mirror 22: 0x49a0a46c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd5f: skipped 000904 1782207526.827793 punch from 0x00060d to 0x00e6ff (0xe0f3 bytes) 000905 1782207526.833426 mapwrite 0x0210dd thru 0x02ea00 (0xd924 bytes) 000906 1782207526.892430 read 0x03c827 thru 0x03cd5e (0x538 bytes) 000907 1782207526.897580 punch from 0x01e399 to 0x024c31 (0x6899 bytes) 000908 1782207526.903874 trunc from 0x03cd5f to 0x00343a (0x39924 bytes) 000909 1782207526.927727 trunc from 0x00143d to 0x00343a (0x1ffe bytes) 000910 1782207526.932774 trunc from 0x00343b to 0x008eb8 (0x5a7e bytes) 000911 1782207526.952641 trunc from 0x0014de to 0x008eb8 (0x79db bytes) 000913 1782207526.964999 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x8eb9): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8eb9): Mirror 10: 0xe11472dd (stale) Mirror 15: 0xe11472dd (stale) Mirror 16: 0xe11472dd (stale) Mirror 17: 0xe11472dd (stale) Mirror 19: 0xe11472dd (stale) Mirror 20: 0xe11472dd (stale) Mirror 21: 0x4e9a3201 (stale) Mirror 22: 0x4e9a3201 (stale) CRC-32 checksum value for chunk [0, 0x8eb9): Mirror 1: 0x1fbb4627 Mirror 23: 0x1fbb4627 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8eb9: skipped 000914 1782207528.025602 trunc from 0x0024de to 0x008eb8 (0x69db bytes) 000915 1782207528.065303 punch from 0x00103c to 0x008eb8 (0x7e7d bytes) 000916 1782207528.070298 mapread 0x00455c thru 0x008eb8 (0x495d bytes) 000917 1782207528.090609 trunc from 0x008eb9 to 0x03e40e (0x35556 bytes) 000918 1782207528.117717 write 0x034c38 thru 0x03dc07 (0x8fd0 bytes) 000919 1782207528.134292 MIRROR_RESYNC 10 mirrors 000920 1782207530.049801 read 0x0140c9 thru 0x017474 (0x33ac bytes) 000921 1782207530.060090 fallocate 0x036382 thru 0x03e2a0 (0x7f1f bytes) 000922 1782207530.104103 MIRROR_RESYNC 10 mirrors 000923 1782207532.059927 trunc from 0x03e40f to 0x0303d8 (0xe036 bytes) 000924 1782207532.093562 write 0x025b9c thru 0x02c578 (0x69dd bytes) 000925 1782207532.106427 punch from 0x02cb44 to 0x0303d8 (0x3895 bytes) 000927 1782207532.122492 write 0x02098c thru 0x02d23e (0xc8b3 bytes) 000928 1782207532.134264 trunc from 0x0064b4 to 0x00a6f5 (0x4242 bytes) 000929 1782207532.139567 mapwrite 0x03cf88 thru 0x03ffff (0x3078 bytes) 000930 1782207532.220569 punch from 0x02846f to 0x03376c (0xb2fe bytes) 000932 1782207532.239179 write 0x024e0a thru 0x031547 (0xc73e bytes) 000933 1782207532.246849 fallocate 0x02a1e6 thru 0x02de0a (0x3c25 bytes) 000934 1782207532.254432 fallocate 0x00bdb5 thru 0x016f98 (0xb1e4 bytes) 000935 1782207532.260467 fallocate 0x01eded thru 0x020fb8 (0x21cc bytes) 000936 1782207532.267508 read 0x012daa thru 0x0229a4 (0xfbfb bytes) 000937 1782207532.285646 trunc from 0x040000 to 0x0116ad (0x2e952 bytes) 000938 1782207532.315170 read 0x011671 thru 0x0116ad (0x3d bytes) 000939 1782207532.327864 read 0x00b0c5 thru 0x0116ad (0x65e9 bytes) 000940 1782207532.329752 trunc from 0x00a186 to 0x00abab (0xa26 bytes) 000941 1782207532.335348 fallocate 0x02de44 thru 0x02f3d1 (0x158e bytes) 000942 1782207532.341492 trunc from 0x0116ae to 0x028cbc (0x1760f bytes) 000943 1782207532.371494 trunc from 0x028cbd to 0x025601 (0x36bb bytes) 000944 1782207532.394586 read 0x00e4b8 thru 0x010ceb (0x2834 bytes) 000945 1782207532.395643 mapwrite 0x02b7c3 thru 0x031845 (0x6083 bytes) 000946 1782207532.461943 write 0x003a66 thru 0x00a676 (0x6c11 bytes) 000947 1782207532.471401 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31846): Mirror 10: 0x9e056278 (stale) Mirror 16: 0x9e056278 (stale) Mirror 17: 0x9e056278 (stale) Mirror 19: 0x9e056278 (stale) Mirror 20: 0x9e056278 (stale) Mirror 21: 0x9e056278 (stale) Mirror 22: 0x9e056278 (stale) Mirror 23: 0x9e056278 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31846): Mirror 10: 0x9e056278 (stale) Mirror 16: 0x9e056278 (stale) Mirror 17: 0x9e056278 (stale) Mirror 19: 0x9e056278 (stale) Mirror 20: 0x9e056278 (stale) Mirror 21: 0x9e056278 (stale) Mirror 22: 0x9e056278 (stale) Mirror 23: 0x9e056278 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31846: skipped 000948 1782207533.022750 trunc from 0x00ebee to 0x011807 (0x2c1a bytes) 000949 1782207533.027413 fallocate 0x006c81 thru 0x0113b5 (0xa735 bytes) 000950 1782207533.033598 punch from 0x01fafe to 0x02675d (0x6c60 bytes) 000951 1782207533.037571 punch from 0x02e972 to 0x031124 (0x27b3 bytes) 000952 1782207533.042492 punch from 0x02da78 to 0x02ddf4 (0x37d bytes) 000953 1782207533.047335 mapread 0x02f320 thru 0x031845 (0x2526 bytes) 000954 1782207533.065653 write 0x0084a9 thru 0x0131ec (0xad44 bytes) 000956 1782207533.086751 fallocate 0x03b29a thru 0x03ffff (0x4d66 bytes) 000957 1782207533.092349 trunc from 0x040000 to 0x03442c (0xbbd3 bytes) 000958 1782207533.109433 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3442d): Mirror 10: 0xddbfe2aa (stale) Mirror 16: 0xddbfe2aa (stale) Mirror 17: 0xddbfe2aa (stale) Mirror 20: 0xddbfe2aa (stale) Mirror 21: 0xddbfe2aa (stale) Mirror 22: 0xddbfe2aa (stale) Mirror 23: 0xddbfe2aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3442d): Mirror 10: 0xddbfe2aa (stale) Mirror 16: 0xddbfe2aa (stale) Mirror 17: 0xddbfe2aa (stale) Mirror 20: 0xddbfe2aa (stale) Mirror 21: 0xddbfe2aa (stale) Mirror 22: 0xddbfe2aa (stale) Mirror 23: 0xddbfe2aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3442d: skipped 000960 1782207533.577609 write 0x00e8e6 thru 0x0133dd (0x4af8 bytes) 000961 1782207533.590200 trunc from 0x00e814 to 0x01c599 (0xdd86 bytes) 000962 1782207533.602739 write 0x0155e4 thru 0x01f6c6 (0xa0e3 bytes) 000963 1782207533.613083 mapread 0x012c8f thru 0x018007 (0x5379 bytes) 000964 1782207533.629446 read 0x024a35 thru 0x02a341 (0x590d bytes) 000965 1782207533.638595 mapread 0x00bca9 thru 0x00e258 (0x25b0 bytes) 000969 1782207533.651353 trunc from 0x03442d to 0x024bfe (0xf82e bytes) 000970 1782207533.666609 mapwrite 0x0369c3 thru 0x038395 (0x19d3 bytes) 000971 1782207533.707257 trunc from 0x038396 to 0x016f23 (0x21472 bytes) 000972 1782207533.721478 trunc from 0x016f24 to 0x0330c1 (0x1c19e bytes) 000973 1782207533.749837 punch from 0x00bed6 to 0x0133a9 (0x74d4 bytes) 000974 1782207533.760191 fallocate 0x03448f thru 0x03ffff (0xbb71 bytes) 000975 1782207533.765538 trunc from 0x040000 to 0x0280ad (0x17f52 bytes) 000976 1782207533.780540 punch from 0x009503 to 0x014f33 (0xba31 bytes) 000977 1782207533.785304 mapwrite 0x02c58c thru 0x03212e (0x5ba3 bytes) 000978 1782207533.840731 write 0x01ac1c thru 0x022f3e (0x8323 bytes) 000979 1782207533.855236 trunc from 0x0141ae to 0x01fadd (0xb930 bytes) 000980 1782207533.871741 mapread 0x026e64 thru 0x028c90 (0x1e2d bytes) 000981 1782207533.874250 trunc from 0x03212f to 0x0046f3 (0x2da3b bytes) 000982 1782207533.891501 punch from 0x001aec to 0x0046f3 (0x2c08 bytes) 000983 1782207533.900692 mapwrite 0x037e26 thru 0x03ffff (0x81da bytes) 000984 1782207533.954054 punch from 0x0224e2 to 0x0272fe (0x4e1d bytes) 000986 1782207533.964002 trunc from 0x040000 to 0x01b338 (0x24cc7 bytes) 000987 1782207533.991347 punch from 0x00ccb2 to 0x01b338 (0xe687 bytes) 000989 1782207533.999136 read 0x01289b thru 0x01b338 (0x8a9e bytes) 000990 1782207534.011178 write 0x02d069 thru 0x03be26 (0xedbe bytes) 000991 1782207534.016722 trunc from 0x00bee1 to 0x01a724 (0xe844 bytes) 000993 1782207534.029236 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3be27): Mirror 10: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be27): Mirror 10: 0xb6700e12 (stale) Mirror 16: 0xb6700e12 (stale) Mirror 17: 0xb6700e12 (stale) Mirror 20: 0xb6700e12 (stale) Mirror 21: 0xb6700e12 (stale) Mirror 22: 0xb6700e12 (stale) Mirror 23: 0xb6700e12 (stale) CRC-32 checksum value for chunk [0, 0x3be27): Mirror 1: 0xe02613ee Mirror 24: 0xe02613ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be27: skipped 000994 1782207535.118749 mapwrite 0x001d73 thru 0x00626e (0x44fc bytes) 000996 1782207535.194980 punch from 0x02ac78 to 0x038ed4 (0xe25d bytes) 000997 1782207535.200429 trunc from 0x03be27 to 0x01a982 (0x214a4 bytes) 000998 1782207535.240612 mapread 0x007b6b thru 0x00df11 (0x63a7 bytes) 000999 1782207535.249791 punch from 0x007a41 to 0x00f569 (0x7b29 bytes) 001000 1782207535.261102 fallocate 0x01b431 thru 0x022858 (0x7428 bytes) All operations completed A-OK! PASS 70a (106s) == sanity-flr test 71: check mirror extend parent fid ==== 05:39:00 (1782207540) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1cb:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 05:39:09 (1782207549) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144956 s, 14.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112619 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 05:39:17 (1782207557) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5908576 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8120768 .. Extending file size to 6193216 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8501280 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9383040 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8915040 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9227776 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61129 61131 61133 61134 61136 done 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:40:37 (1782207637) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3538432 .. Extending file size to 5356928 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7177632 .. Extending file size to 9002624 .. Extending file size to 8063712 .. Extending file size to 9151008 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8085600 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8113600 .. Waiting 62500 62501 62502 62505 62506 done 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:42:01 (1782207721) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1269284 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25676 3581344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17852 3589168 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43528 7170512 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2020 1269284 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25676 3581344 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17852 3589168 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43528 7170512 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0942415 s, 22.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240319 s, 8.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5cae58bd (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:42:20 (1782207740) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105856 s, 19.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 05:42:28 (1782207748) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1da:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x315:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x315:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x316:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x313:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:42:37 (1782207757) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106544 s, 19.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:42:46 (1782207766) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117609 s, 17.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:42:54 (1782207774) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0807994 s, 26.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:43:26 (1782207806) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562077 s, 18.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:44:13 (1782207853) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129036 s, 16.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:45:02 (1782207902) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg326-server: Pool lustre.test_204f created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102639 s, 20.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg326-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:46:18 (1782207978) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:46:26 (1782207986) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:46:33 (1782207993) oleg326-server: Pool lustre.test_206 created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg326-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:47:03 (1782208023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727322 s, 14.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415035 s, 25.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9a6ead47 Mirror 3: 0x9a6ead47 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:47:15 (1782208035) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:47:20 (1782208040) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:47:24 (1782208044) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.058876 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1f00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2470000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2470000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2470000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9b0000 Read this (4 bytes): BBBB 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:48:01 (1782208081) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.026733 s, 306 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:48:08 (1782208088) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655419 s, 16.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f7 lmm_fid: [0x200000405:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f7 lmm_fid: [0x200000405:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x320:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611639 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-23 05:48:10.000000000 -0400 Modify: 2026-06-23 05:48:10.000000000 -0400 Change: 2026-06-23 05:48:11.000000000 -0400 Birth: 2026-06-23 05:48:11.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:48:17 (1782208097) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017901 s, 58.6 MB/s oleg326-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg326-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg326-server: O/280000401/d1/321: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 210c: write to a file with foreign component ========================================================== 05:48:40 (1782208120) fail_loc=0x1440 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 65 lmm_objects: - 0: { l_ost_idx: 65, l_fid: [0x280000401:0x322:0x0] } - lcme_id: 2 dd: error writing '/mnt/lustre/f210c.sanity-flr': No data available 1+0 records in 0+0 records out 0 bytes copied, 0.0160434 s, 0.0 kB/s PASS 210c (187s) == sanity-flr test 210d: truncate a file with foreign component ========================================================== 05:51:47 (1782208307) fail_loc=0x1440 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 65 lmm_objects: - 0: { l_ost_idx: 65, l_fid: [0x2c0000401:0x324:0x0] } - lcme_id: 2 truncate: cannot truncate '/mnt/lustre/f210d.sanity-flr' to length 1048576: No data available PASS 210d (188s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:54:55 (1782208495) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.318212 s, 129 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.017152 s, 239 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 23 05:54 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ff:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 23 05:54 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:55:05 (1782208505) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:55:13 (1782208513) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2084 sec =========== 05:55:22 (1782208522) === sanity-flr: start cleanup 05:55:24 (1782208524) === === sanity-flr: finish cleanup 05:55:31 (1782208531) ===