-----============= acceptance-small: sanity-flr ============----- Thu Sep 3 20:51:48 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg345-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:52:07 (1788483127) === oleg345-client.virtnet: executing check_config_client /mnt/lustre oleg345-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg345-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff893c484c0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff893c484c0000.idle_timeout=debug disable quota as required oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:52:41 (1788483161) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8264] == sanity-flr test 0a: lfs mirror create with -N option == 20:52:43 (1788483163) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:52:59 (1788483179) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:53:02 (1788483182) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:53:05 (1788483185) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:53:28 (1788483208) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:53:31 (1788483211) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:53:34 (1788483214) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff893c484c0000.stats=clear osc.lustre-OST0001-osc-ffff893c484c0000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:53:41 (1788483221) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:53:51 (1788483231) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:53:59 (1788483239) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:54:02 (1788483242) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:54:09 (1788483249) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 20:54:18 (1788483258) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:54:24 (1788483264) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:54:31 (1788483271) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:54:39 (1788483279) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:54:49 (1788483289) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:54:55 (1788483295) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.960826 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070992 s, 14.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:55:04 (1788483304) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050946 s, 20.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:55:12 (1788483312) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:55:22 (1788483322) Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg345-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:56:36 (1788483396) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.402233 s, 13.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:57:17 (1788483437) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg345-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:58:43 (1788483523) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.58929 s, 14.6 MB/s -rw-r--r-- 1 root root 52428800 Sep 3 20:58 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.620s user 0m0.001s sys 0m0.199s Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff893c484c0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.187s user 0m0.000s sys 0m0.196s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.064s user 0m0.000s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg345-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff893c484c0000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:00:24 (1788483624) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:00:27 (1788483627) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:00:30 (1788483630) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 21:00:34 (1788483634) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:00:41 (1788483641) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.616524 s, 20.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:00:53 (1788483653) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477455 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02622 s, 40.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:01:00 (1788483660) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:01:10 (1788483670) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000806197 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000753202 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000820828 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000779718 s, 1.3 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 21:01:18 (1788483678) create test files with size 15758434 .. 611b5fc08e4dcdb9344116872988e226 - 5d652a0de543127062de7325f6512788 - 7c7b6b68bd4c8a32ebee66d777cfa035 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 21:01:40 (1788483700) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:01:52 (1788483712) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:02:00 (1788483720) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970205 s, 10.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261017 s, 40.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 21:02:08 (1788483728) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0677232 s, 31.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.436254 s, 9.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148262 s, 14.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 21:02:24 (1788483744) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 21:02:28 (1788483748) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:02:32 (1788483752) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521851 s, 20.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 21:02:42 (1788483762) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.332204 s, 9.5 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:02:53 (1788483773) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:03:04 (1788483784) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31665 s, 33.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:03:15 (1788483795) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff893c50518800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.397785 s, 26.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff893c5e375800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:03:31 (1788483811) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:04:10 (1788483850) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:04:48 (1788483888) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 21:04:51 (1788483891) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.245269 s, 12.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.190358 s, 16.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xdd53defb (nosync,stale) Mirror 2: 0xdd53defb (nosync,stale) ** verify mirror 2 doesn't change original checksum: ebf1f6f5301df1a394c63b1f47b62e83 - mirror 1 checksum: da99b313fb0b4ad6a1948cfb63cfcabd - mirror 2 checksum: ebf1f6f5301df1a394c63b1f47b62e83 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-03T21:04:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-03T21:04:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ebf1f6f5301df1a394c63b1f47b62e83 - mirror 1 checksum: da99b313fb0b4ad6a1948cfb63cfcabd - mirror 2 checksum: da99b313fb0b4ad6a1948cfb63cfcabd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-03T21:04:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-03T21:04:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:05:00 (1788483900) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04502 s, 23.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:05:07 (1788483907) Failing mds1 on oleg345-server Stopping /mnt/lustre-mds1 (opts:) on oleg345-server 21:05:18 (1788483918) shut down facet: mds1 facet_host: oleg345-server facet_failover_host: oleg345-server Failover mds1 to oleg345-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 21:05:41 (1788483941) targets are mounted 21:05:41 (1788483941) facet_failover done oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047976 s, 21.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:06:04 (1788483964) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0577995 s, 354 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0247972 s, 826 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:06:17 (1788483977) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0825115 s, 248 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0099551 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:06:27 (1788483987) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00696268 s, 2.4 MB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:06:39 (1788483999) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.041457 s, 494 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:06:48 (1788484008) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0096832 s, 423 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0685512 s, 59.8 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:06:56 (1788484016) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:07:19 (1788484039) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:07:48 (1788484068) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:08:13 (1788484093) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:08:39 (1788484119) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 21:08:47 (1788484127) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57616 mirror split pid 57617 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57616 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57617 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 21:09:55 (1788484195) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4197 skipping zero size read fallocating to largest ever: 0x1288f 000002 1788484198.340597 fallocate 0x00c0c2 thru 0x01288e (0x67cd bytes) 000003 1788484198.354316 fallocate 0x006d28 thru 0x008bd6 (0x1eaf bytes) truncating to largest ever: 0x36e9c 000006 1788484198.364828 trunc from 00000000 to 0x036e9b (0x36e9c bytes) 000007 1788484198.420015 MIRROR_EXTEND to 2 mirrors 000008 1788484199.155830 fallocate 0x00973b thru 0x013267 (0x9b2d bytes) 000009 1788484199.193257 mapread 0x005a8b thru 0x00f412 (0x9988 bytes) 000010 1788484199.214535 write 0x03aefd thru 0x03ffff (0x5103 bytes) 000011 1788484199.221270 read 0x014ac7 thru 0x021509 (0xca43 bytes) 000012 1788484199.239219 read 0x03165d thru 0x03acf7 (0x969b bytes) 000014 1788484199.254841 trunc from 0x00adcd to 0x011ce8 (0x6f1c bytes) 000016 1788484199.261126 write 0x018ee7 thru 0x0229e9 (0x9b03 bytes) 000017 1788484199.262388 read 0x01716f thru 0x023e9e (0xcd30 bytes) 000018 1788484199.262973 fallocate 0x029461 thru 0x031502 (0x80a2 bytes) 000019 1788484199.268631 trunc from 0x040000 to 0x013a9d (0x2c562 bytes) 000020 1788484199.296165 read 0x00b129 thru 0x00df6b (0x2e43 bytes) 000021 1788484199.306773 punch from 0x00fb0b to 0x013a9d (0x3f93 bytes) 000022 1788484199.311937 trunc from 0x013a9e to 0x01d62e (0x9b91 bytes) 000023 1788484199.341648 MIRROR_RESYNC 2 mirrors 000024 1788484199.849099 trunc from 0x01d62f to 0x023e80 (0x6852 bytes) 000025 1788484199.922244 mapwrite 0x030af9 thru 0x03ee1f (0xe327 bytes) 000026 1788484199.995110 write 0x01f01f thru 0x02ce19 (0xddfb bytes) 000027 1788484200.006394 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) 000028 1788484200.160449 trunc from 0x0190d1 to 0x01969e (0x5ce bytes) 000029 1788484200.166354 write 0x00f3aa thru 0x01a78d (0xb3e4 bytes) 000030 1788484200.183159 trunc from 0x0277a8 to 0x034c13 (0xd46c bytes) 000031 1788484200.189363 mapread 0x03a7c1 thru 0x03ee1f (0x465f bytes) 000032 1788484200.214813 mapwrite 0x02e1b7 thru 0x03bd4c (0xdb96 bytes) 000034 1788484200.277001 trunc from 0x03ee20 to 0x02d0a7 (0x11d78 bytes) 000035 1788484200.300116 mapwrite 0x036806 thru 0x036ede (0x6d9 bytes) 000036 1788484200.387434 punch from 0x00737a to 0x014e86 (0xdb0d bytes) truncating to largest ever: 0x3a7c8 000037 1788484200.399928 trunc from 0x036edf to 0x03a7c7 (0x38e9 bytes) 000039 1788484200.436124 mapread 0x028fc4 thru 0x0335e6 (0xa623 bytes) 000041 1788484200.453263 read 0x026443 thru 0x033e3d (0xd9fb bytes) 000042 1788484200.460782 trunc from 0x03a7c8 to 0x01e751 (0x1c076 bytes) 000043 1788484200.505732 write 0x029f68 thru 0x02fccf (0x5d68 bytes) 000044 1788484200.514977 MIRROR_EXTEND to 2 mirrors 000045 1788484201.363696 trunc from 0x02fcd0 to 0x022085 (0xdc4a bytes) 000046 1788484201.442210 punch from 0x0166f3 to 0x01cf4e (0x685c bytes) 000047 1788484201.448239 read 0x01803b thru 0x02127a (0x9240 bytes) 000048 1788484201.461088 mapwrite 0x030514 thru 0x03fbee (0xf6db bytes) 000049 1788484201.601854 trunc from 0x03fbef to 0x02d69a (0x12554 bytes) 000050 1788484201.628274 mapwrite 0x0304c1 thru 0x038433 (0x7f73 bytes) 000051 1788484201.690248 write 0x006131 thru 0x01015f (0xa02f bytes) 000052 1788484201.711101 trunc from 0x038434 to 0x00af62 (0x2d4d1 bytes) 000053 1788484201.741565 mapwrite 0x03c222 thru 0x03ded6 (0x1cb5 bytes) 000055 1788484201.798681 mapwrite 0x00c36d thru 0x00f91e (0x35b2 bytes) 000057 1788484201.825803 trunc from 0x03ded7 to 0x02f0f8 (0xedde bytes) 000058 1788484201.848835 fallocate 0x02248c thru 0x023ec7 (0x1a3c bytes) 000059 1788484201.853918 punch from 0x006185 to 0x007da7 (0x1c23 bytes) 000060 1788484201.859392 mapread 0x002e7f thru 0x009906 (0x6a88 bytes) 000061 1788484201.868495 mapread 0x024180 thru 0x02a2f2 (0x6173 bytes) 000062 1788484201.878183 write 0x00e93d thru 0x0146a5 (0x5d69 bytes) 000063 1788484201.879424 write 0x025200 thru 0x031574 (0xc375 bytes) 000064 1788484201.881948 mapread 0x0055d9 thru 0x00a697 (0x50bf bytes) 000065 1788484201.883442 mapread 0x00cd15 thru 0x010494 (0x3780 bytes) 000068 1788484201.886408 trunc from 0x0009cc to 0x00d914 (0xcf49 bytes) 000069 1788484201.892258 fallocate 0x002256 thru 0x008a90 (0x683b bytes) 000070 1788484201.896796 fallocate 0x02acbe thru 0x02ad49 (0x8c bytes) 000071 1788484201.907672 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) 000072 1788484202.068915 mapread 0x0133ee thru 0x019cc8 (0x68db bytes) 000073 1788484202.105176 trunc from 0x028609 to 0x031574 (0x8f6c bytes) 000074 1788484202.115402 trunc from 0x031575 to 0x012401 (0x1f173 bytes) 000075 1788484202.146404 punch from 0x00619b to 0x011e7d (0xbce3 bytes) 000076 1788484202.152535 fallocate 0x0253f2 thru 0x025cac (0x8bb bytes) 000077 1788484202.157938 trunc from 0x025cad to 0x022c81 (0x302b bytes) 000079 1788484202.178129 trunc from 0x022c82 to 0x018b5f (0xa122 bytes) 000080 1788484202.212632 punch from 0x005754 to 0x00fa38 (0xa2e5 bytes) fallocating to largest ever: 0x3e39a 000081 1788484202.219392 fallocate 0x03296e thru 0x03e399 (0xba2c bytes) 000082 1788484202.224900 read 0x033d6d thru 0x036c61 (0x2ef5 bytes) 000083 1788484202.231592 fallocate 0x017d77 thru 0x020061 (0x82eb bytes) 000085 1788484202.239503 read 0x0140e4 thru 0x01553f (0x145c bytes) 000087 1788484202.275593 mapwrite 0x0287ab thru 0x034d07 (0xc55d bytes) 000088 1788484202.294964 mapread 0x02286e thru 0x02b8a9 (0x903c bytes) 000089 1788484202.297917 punch from 0x00de14 to 0x018add (0xacca bytes) 000090 1788484202.302844 mapread 0x007cf0 thru 0x008e98 (0x11a9 bytes) 000091 1788484202.303740 read 0x02cfed thru 0x03a23a (0xd24e bytes) 000092 1788484202.305033 write 0x0395f9 thru 0x03a9bf (0x13c7 bytes) 000093 1788484202.306389 punch from 0x01a8af to 0x02360c (0x8d5e bytes) 000094 1788484202.313866 trunc from 0x0068bd to 0x016715 (0xfe59 bytes) 000096 1788484202.323548 fallocate 0x03b1a2 thru 0x03ccf5 (0x1b54 bytes) 000097 1788484202.330118 fallocate 0x02b8cc thru 0x030352 (0x4a87 bytes) 000098 1788484202.336172 write 0x00dd60 thru 0x01c7ab (0xea4c bytes) 000099 1788484202.351868 MIRROR_EXTEND to 2 mirrors 000100 1788484203.167888 mapread 0x00e874 thru 0x011938 (0x30c5 bytes) 000101 1788484203.217488 trunc from 0x008158 to 0x0095e9 (0x1492 bytes) 000103 1788484203.258765 write 0x0248b7 thru 0x0302aa (0xb9f4 bytes) 000104 1788484203.273353 trunc from 0x03e39a to 0x033315 (0xb084 bytes) 000105 1788484203.309693 trunc from 0x033316 to 0x028397 (0xaf7e bytes) 000106 1788484203.330054 read 0x022583 thru 0x028397 (0x5e15 bytes) 000107 1788484203.342887 punch from 0x001952 to 0x01112b (0xf7da bytes) 000108 1788484203.353110 trunc from 0x00d2bf to 0x011a08 (0x474a bytes) truncating to largest ever: 0x3eb89 000110 1788484203.359269 trunc from 0x028398 to 0x03eb88 (0x167f1 bytes) 000111 1788484203.384494 punch from 0x01dd09 to 0x01f223 (0x151b bytes) 000112 1788484203.389914 fallocate 0x004e9e thru 0x008165 (0x32c8 bytes) 000113 1788484203.400431 mapread 0x017cdc thru 0x01a608 (0x292d bytes) 000114 1788484203.416384 trunc from 0x03eb89 to 0x0242d1 (0x1a8b7 bytes) 000115 1788484203.444532 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x242d2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242d2): Mirror 2: 0xef1b32ba (stale) CRC-32 checksum value for chunk [0, 0x242d2): Mirror 1: 0x15c83fa8 Mirror 3: 0x15c83fa8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242d2: skipped 000116 1788484204.419380 punch from 0x020cc3 to 0x0242d1 (0x360f bytes) 000117 1788484204.496731 trunc from 0x016f46 to 0x02006c (0x9127 bytes) 000119 1788484204.508223 mapread 0x019128 thru 0x01fcba (0x6b93 bytes) 000120 1788484204.542095 fallocate 0x01ab59 thru 0x028bea (0xe092 bytes) 000121 1788484204.554107 mapwrite 0x024298 thru 0x025dec (0x1b55 bytes) 000122 1788484204.588354 punch from 0x00a642 to 0x015120 (0xaadf bytes) 000123 1788484204.612490 read 0x020480 thru 0x028bea (0x876b bytes) 000124 1788484204.619181 mapread 0x01bcfd thru 0x021a49 (0x5d4d bytes) 000125 1788484204.624389 mapwrite 0x00d1e8 thru 0x01a4d4 (0xd2ed bytes) 000126 1788484204.676392 mapwrite 0x00fcfc thru 0x01ce35 (0xd13a bytes) truncating to largest ever: 0x3f089 000127 1788484204.745771 trunc from 0x028beb to 0x03f088 (0x1649e bytes) 000129 1788484204.836774 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f089): Mirror 2: 0x2c7b2c6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f089): Mirror 2: 0x2c7b2c6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f089: skipped 000131 1788484205.265430 read 0x021fe0 thru 0x02838c (0x63ad bytes) 000132 1788484205.274700 write 0x02e8b1 thru 0x0362d5 (0x7a25 bytes) 000133 1788484205.298806 trunc from 0x0355ae to 0x03f088 (0x9adb bytes) 000134 1788484205.325168 punch from 0x02914d to 0x029495 (0x349 bytes) 000136 1788484205.329268 trunc from 0x03f089 to 0x0218b0 (0x1d7d8 bytes) 000137 1788484205.341079 trunc from 0x0218b1 to 0x01fcce (0x1be2 bytes) 000138 1788484205.364254 MIRROR_RESYNC 2 mirrors 000139 1788484206.106903 mapwrite 0x004d95 thru 0x0093ee (0x465a bytes) 000140 1788484206.312368 mapread 0x005d6f thru 0x0136d6 (0xd968 bytes) 000141 1788484206.344487 read 0x005dd0 thru 0x013e98 (0xe0c9 bytes) 000142 1788484206.351441 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1fccf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fccf): Mirror 2: 0xbc09dcf4 (stale) CRC-32 checksum value for chunk [0, 0x1fccf): Mirror 1: 0x937d4d31 Mirror 3: 0x937d4d31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fccf: skipped 000143 1788484207.231239 fallocate 0x025f4f thru 0x032f8b (0xd03d bytes) 000144 1788484207.289977 fallocate 0x00e6c7 thru 0x013c7e (0x55b8 bytes) 000145 1788484207.296698 mapread 0x023554 thru 0x03007a (0xcb27 bytes) 000146 1788484207.357249 trunc from 0x032f8c to 0x03aeeb (0x7f60 bytes) 000148 1788484207.428693 mapread 0x02ece6 thru 0x037705 (0x8a20 bytes) 000149 1788484207.448966 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 2: 0x8fea19e (stale) Mirror 3: 0xe95b8bc4 (stale) CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 1: 0xe95b8bc4 Mirror 4: 0xe95b8bc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aeec: skipped 000150 1788484208.778261 read 0x011537 thru 0x01f0d3 (0xdb9d bytes) 000151 1788484208.796545 mapread 0x035cf3 thru 0x036506 (0x814 bytes) 000152 1788484208.837141 write 0x02705d thru 0x02ad62 (0x3d06 bytes) 000153 1788484208.918797 punch from 0x01fb92 to 0x022479 (0x28e8 bytes) 000154 1788484208.929551 trunc from 0x036e4a to 0x03aeeb (0x40a2 bytes) 000155 1788484208.936826 fallocate 0x01a3a1 thru 0x0241b3 (0x9e13 bytes) 000156 1788484208.943898 punch from 0x02ac80 to 0x039b29 (0xeeaa bytes) 000157 1788484208.972280 read 0x039b81 thru 0x03aeeb (0x136b bytes) 000158 1788484209.001484 mapwrite 0x00045b thru 0x004abd (0x4663 bytes) 000159 1788484209.051397 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 2: 0x8fea19e (stale) Mirror 3: 0xe95b8bc4 (stale) Mirror 4: 0xe95b8bc4 (stale) CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 1: 0x32131dfc Mirror 5: 0x32131dfc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aeec: skipped 000160 1788484210.213398 write 0x01d38c thru 0x01f477 (0x20ec bytes) 000161 1788484210.257896 mapread 0x010d5d thru 0x0208cd (0xfb71 bytes) 000164 1788484210.293344 mapread 0x00b036 thru 0x019cf0 (0xecbb bytes) 000165 1788484210.294270 trunc from 0x03aeec to 0x03a57d (0x96e bytes) 000166 1788484210.339051 mapread 0x02c8ae thru 0x03a57d (0xdcd0 bytes) 000167 1788484210.363034 trunc from 0x017e93 to 0x01ab68 (0x2cd6 bytes) 000168 1788484210.376241 mapread 0x0124f5 thru 0x017a1e (0x552a bytes) 000169 1788484210.392358 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a57e): Mirror 3: 0x1c73b52c (stale) Mirror 4: 0x1c73b52c (stale) Mirror 5: 0x92b15eec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a57e): Mirror 3: 0x1c73b52c (stale) Mirror 4: 0x1c73b52c (stale) Mirror 5: 0x92b15eec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a57e: skipped 000170 1788484210.808488 trunc from 0x0386ea to 0x038e14 (0x72b bytes) 000171 1788484210.816774 write 0x02a3f7 thru 0x0380b0 (0xdcba bytes) 000172 1788484210.832269 punch from 0x02f3db to 0x03a57d (0xb1a3 bytes) 000174 1788484210.847799 trunc from 0x0296e1 to 0x037967 (0xe287 bytes) 000175 1788484210.853269 mapread 0x020098 thru 0x02d693 (0xd5fc bytes) 000176 1788484210.879792 write 0x012194 thru 0x01f478 (0xd2e5 bytes) 000177 1788484210.889366 MIRROR_RESYNC 4 mirrors 000178 1788484212.183401 trunc from 0x03a57e to 0x0193c2 (0x211bb bytes) 000180 1788484212.236724 write 0x00de2f thru 0x014c4e (0x6e20 bytes) 000181 1788484212.254698 trunc from 0x0193c3 to 0x01d39b (0x3fd9 bytes) 000183 1788484212.280189 mapwrite 0x015323 thru 0x021b2a (0xc808 bytes) 000184 1788484212.363010 trunc from 0x002625 to 0x011c0d (0xf5e9 bytes) 000185 1788484212.370795 trunc from 0x017b12 to 0x01c283 (0x4772 bytes) 000186 1788484212.380514 MIRROR_RESYNC 4 mirrors 000187 1788484213.444504 trunc from 0x021b2b to 0x00dd73 (0x13db7 bytes) 000188 1788484213.487176 write 0x03b2ad thru 0x03ffff (0x4d53 bytes) 000189 1788484213.495426 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0e443c7 (stale) Mirror 5: 0xb0e443c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0e443c7 (stale) Mirror 5: 0xb0e443c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000190 1788484213.810260 write 0x018e8f thru 0x024945 (0xbab7 bytes) 000191 1788484213.833219 read 0x005855 thru 0x00c638 (0x6de4 bytes) 000192 1788484213.849220 write 0x0024f3 thru 0x008481 (0x5f8f bytes) 000193 1788484213.859453 punch from 0x023f68 to 0x026e60 (0x2ef9 bytes) 000195 1788484213.877802 read 0x017ab7 thru 0x026f57 (0xf4a1 bytes) 000196 1788484213.887262 trunc from 0x033ce7 to 0x03b0fb (0x7415 bytes) 000197 1788484213.892271 read 0x01d8e4 thru 0x02609f (0x87bc bytes) 000199 1788484213.894387 mapread 0x011e9c thru 0x018bcc (0x6d31 bytes) 000200 1788484213.895982 trunc from 0x040000 to 0x032bc3 (0xd43c bytes) 000201 1788484213.916675 fallocate 0x0018fe thru 0x00253d (0xc40 bytes) 000202 1788484213.926642 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32bc4): Mirror 3: 0x66481c1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32bc4): Mirror 3: 0x66481c1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32bc4: skipped 000203 1788484214.278839 fallocate 0x022283 thru 0x02b167 (0x8ee5 bytes) 000204 1788484214.293165 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) 000205 1788484214.493893 write 0x00f605 thru 0x0190e9 (0x9ae5 bytes) 000206 1788484214.509574 write 0x006073 thru 0x014e80 (0xee0e bytes) 000207 1788484214.517816 trunc from 0x032bc4 to 0x017643 (0x1b580 bytes) 000208 1788484214.535749 punch from 0x008d05 to 0x0167e2 (0xdade bytes) 000209 1788484214.551575 trunc from 0x017644 to 0x018c28 (0x15e5 bytes) 000212 1788484214.567386 mapread 0x002ac0 thru 0x004f95 (0x24d6 bytes) 000213 1788484214.583717 punch from 0x0098bf to 0x011ad8 (0x821a bytes) 000214 1788484214.588371 mapwrite 0x00273a thru 0x0101dd (0xdaa4 bytes) 000215 1788484214.617048 punch from 0x012476 to 0x0171fd (0x4d88 bytes) 000216 1788484214.621754 trunc from 0x0115b7 to 0x018c28 (0x7672 bytes) 000217 1788484214.626592 read 0x001e52 thru 0x00b190 (0x933f bytes) 000218 1788484214.627237 mapread 0x0004d4 thru 0x0045d0 (0x40fd bytes) 000223 1788484214.635129 mapread 0x014e53 thru 0x01520e (0x3bc bytes) 000224 1788484214.643511 mapread 0x003245 thru 0x00f8ce (0xc68a bytes) fallocating to largest ever: 0x40000 000225 1788484214.644715 fallocate 0x03bfea thru 0x03ffff (0x4016 bytes) 000226 1788484214.652690 MIRROR_EXTEND to 2 mirrors 000227 1788484215.381969 trunc from 0x00d8fb to 0x011686 (0x3d8c bytes) 000228 1788484215.409218 fallocate 0x039a40 thru 0x03b5db (0x1b9c bytes) 000230 1788484215.418131 write 0x010451 thru 0x01fa1b (0xf5cb bytes) 000232 1788484215.446738 write 0x01db74 thru 0x022bbc (0x5049 bytes) 000233 1788484215.452752 trunc from 0x03e4e5 to 0x03ffff (0x1b1b bytes) 000234 1788484215.462829 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x980b4480 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5d53c280 Mirror 3: 0x5d53c280 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000235 1788484216.249006 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x980b4480 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5d53c280 Mirror 3: 0x5d53c280 Mirror 4: 0x5d53c280 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1788484217.095859 mapwrite 0x0031c6 thru 0x009f20 (0x6d5b bytes) 000237 1788484217.214469 fallocate 0x02eda8 thru 0x02f1b7 (0x410 bytes) 000238 1788484217.225131 read 0x017dc8 thru 0x0186a6 (0x8df bytes) 000239 1788484217.236476 read 0x01479b thru 0x01bd9e (0x7604 bytes) 000240 1788484217.291898 fallocate 0x00e367 thru 0x00ec93 (0x92d bytes) 000241 1788484217.322084 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x980b4480 (stale) Mirror 3: 0x5d53c280 (stale) Mirror 4: 0x5d53c280 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe06f9d83 Mirror 5: 0xe06f9d83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000242 1788484218.671703 trunc from 0x00731a to 0x00f4c7 (0x81ae bytes) 000243 1788484218.716382 fallocate 0x0344ac thru 0x03ffff (0xbb54 bytes) 000245 1788484218.723465 mapread 0x0252c2 thru 0x02b460 (0x619f bytes) 000247 1788484218.744691 punch from 0x023522 to 0x02447d (0xf5c bytes) 000248 1788484218.749978 fallocate 0x0005a8 thru 0x0104cc (0xff25 bytes) 000249 1788484218.757762 fallocate 0x03d166 thru 0x03df9e (0xe39 bytes) 000250 1788484218.768739 trunc from 0x040000 to 0x03343e (0xcbc1 bytes) 000251 1788484218.837582 mapread 0x0074f8 thru 0x008254 (0xd5d bytes) 000252 1788484218.853778 punch from 0x01d3a8 to 0x02516a (0x7dc3 bytes) 000253 1788484218.859785 fallocate 0x025684 thru 0x02ad87 (0x5704 bytes) 000256 1788484218.865648 punch from 0x0002b4 to 0x00049c (0x1e9 bytes) 000257 1788484218.870118 punch from 0x00e618 to 0x011638 (0x3021 bytes) 000258 1788484218.875987 read 0x009ef7 thru 0x00b699 (0x17a3 bytes) 000259 1788484218.876672 trunc from 0x03343f to 0x025af8 (0xd946 bytes) 000260 1788484218.896189 trunc from 0x025af9 to 0x0287e8 (0x2cf0 bytes) 000261 1788484218.915085 fallocate 0x01ecf6 thru 0x0208e5 (0x1bf0 bytes) 000262 1788484218.924032 MIRROR_RESYNC 5 mirrors 000263 1788484220.388651 punch from 0x00e824 to 0x01d4c7 (0xeca4 bytes) 000264 1788484220.439559 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x287e9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x287e9): Mirror 2: 0x1158958f (stale) Mirror 3: 0x1158958f (stale) Mirror 4: 0x1158958f (stale) Mirror 5: 0x1158958f (stale) CRC-32 checksum value for chunk [0, 0x287e9): Mirror 1: 0x55a48d50 Mirror 6: 0x55a48d50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x287e9: skipped 000265 1788484221.534453 read 0x02364f thru 0x0287e8 (0x519a bytes) 000266 1788484221.552068 punch from 0x002b4c to 0x00c95c (0x9e11 bytes) 000267 1788484221.648029 fallocate 0x035980 thru 0x03ffff (0xa680 bytes) 000268 1788484221.655517 trunc from 0x01eefe to 0x0287e8 (0x98eb bytes) 000269 1788484221.666185 mapwrite 0x0292f9 thru 0x031fa1 (0x8ca9 bytes) 000270 1788484221.751058 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31fa2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31fa2): Mirror 2: 0x2c5499be (stale) Mirror 3: 0x2c5499be (stale) Mirror 4: 0x2c5499be (stale) Mirror 5: 0x2c5499be (stale) Mirror 6: 0xd7b29d28 (stale) CRC-32 checksum value for chunk [0, 0x31fa2): Mirror 1: 0xcce27bf5 Mirror 7: 0xcce27bf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31fa2: skipped 000271 1788484222.752702 trunc from 0x031fa2 to 0x015725 (0x1c87c bytes) 000272 1788484222.812910 trunc from 0x015726 to 0x0259ba (0x10295 bytes) 000273 1788484222.849649 write 0x00b131 thru 0x0150d5 (0x9fa5 bytes) 000274 1788484222.871631 trunc from 0x0259bb to 0x0359af (0xfff5 bytes) 000275 1788484222.935014 trunc from 0x0359b0 to 0x00fe68 (0x25b47 bytes) 000276 1788484222.971817 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfe69): Mirror 3: 0xe1932b6c (stale) Mirror 4: 0xe1932b6c (stale) Mirror 5: 0xe1932b6c (stale) Mirror 6: 0xe1932b6c (stale) Mirror 7: 0x57ad07f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfe69): Mirror 3: 0xe1932b6c (stale) Mirror 4: 0xe1932b6c (stale) Mirror 5: 0xe1932b6c (stale) Mirror 6: 0xe1932b6c (stale) Mirror 7: 0x57ad07f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfe69: skipped 000277 1788484223.514795 read 0x002b24 thru 0x0061ab (0x3688 bytes) 000280 1788484223.527509 mapwrite 0x01c2e7 thru 0x01fa68 (0x3782 bytes) 000281 1788484223.578714 read 0x01976b thru 0x01fa68 (0x62fe bytes) 000282 1788484223.584372 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fa69): Mirror 3: 0x21c27e42 (stale) Mirror 4: 0x21c27e42 (stale) Mirror 5: 0x21c27e42 (stale) Mirror 6: 0xb962bc27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fa69): Mirror 3: 0x21c27e42 (stale) Mirror 4: 0x21c27e42 (stale) Mirror 5: 0x21c27e42 (stale) Mirror 6: 0xb962bc27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fa69: skipped 000284 1788484223.994059 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1fa69): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fa69): Mirror 3: 0x21c27e42 (stale) Mirror 4: 0x21c27e42 (stale) Mirror 5: 0x21c27e42 (stale) Mirror 6: 0xb962bc27 (stale) CRC-32 checksum value for chunk [0, 0x1fa69): Mirror 1: 0x522d499d Mirror 7: 0x522d499d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fa69: skipped 000285 1788484225.056631 punch from 0x00fa51 to 0x01315e (0x370e bytes) 000286 1788484225.122770 read 0x010f3a thru 0x01de83 (0xcf4a bytes) 000287 1788484225.141367 mapread 0x01f6ba thru 0x01fa68 (0x3af bytes) 000288 1788484225.146617 trunc from 0x01fa69 to 0x005f93 (0x19ad5 bytes) 000289 1788484225.185937 fallocate 0x035a34 thru 0x03ffff (0xa5cc bytes) 000290 1788484225.202607 read 0x0055cd thru 0x005f93 (0x9c7 bytes) 000291 1788484225.218406 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5f94): Mirror 3: 0x54c9ef61 (stale) Mirror 4: 0x54c9ef61 (stale) Mirror 5: 0x54c9ef61 (stale) Mirror 7: 0x6192eb8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5f94): Mirror 3: 0x54c9ef61 (stale) Mirror 4: 0x54c9ef61 (stale) Mirror 5: 0x54c9ef61 (stale) Mirror 7: 0x6192eb8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5f94: skipped 000294 1788484225.591876 punch from 0x005748 to 0x005f93 (0x84c bytes) 000295 1788484225.597191 fallocate 0x031b2f thru 0x03371e (0x1bf0 bytes) 000296 1788484225.602589 read 0x002620 thru 0x005f93 (0x3974 bytes) 000297 1788484225.613624 MIRROR_RESYNC 5 mirrors 000298 1788484226.709146 MIRROR_RESYNC 5 mirrors 000299 1788484227.175602 MIRROR_EXTEND to 6 mirrors 000300 1788484227.962313 mapwrite 0x031b95 thru 0x03f062 (0xd4ce bytes) 000301 1788484228.078622 write 0x02c89a thru 0x03bfb3 (0xf71a bytes) 000302 1788484228.081119 write 0x013075 thru 0x021677 (0xe603 bytes) 000303 1788484228.115811 write 0x02e7df thru 0x034e72 (0x6694 bytes) 000304 1788484228.119726 read 0x02ccbf thru 0x036dd4 (0xa116 bytes) 000305 1788484228.131661 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f063): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f063): Mirror 3: 0x3c687a24 (stale) Mirror 4: 0x3c687a24 (stale) Mirror 5: 0x3c687a24 (stale) Mirror 7: 0x3c687a24 (stale) Mirror 8: 0x3c687a24 (stale) CRC-32 checksum value for chunk [0, 0x3f063): Mirror 1: 0xfba7cd18 Mirror 9: 0xfba7cd18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f063: skipped 000306 1788484229.109786 mapwrite 0x027ef7 thru 0x02f0c8 (0x71d2 bytes) 000307 1788484229.252698 read 0x004a06 thru 0x013115 (0xe710 bytes) 000308 1788484229.267144 fallocate 0x03436e thru 0x039faf (0x5c42 bytes) 000309 1788484229.275344 MIRROR_RESYNC 7 mirrors 000311 1788484230.968044 trunc from 0x033275 to 0x034a5d (0x17e9 bytes) 000313 1788484230.997167 trunc from 0x03f063 to 0x02966b (0x159f7 bytes) 000314 1788484231.026944 mapread 0x00b4e8 thru 0x00ed61 (0x387a bytes) 000315 1788484231.072620 trunc from 0x02966c to 0x020fba (0x86b1 bytes) 000316 1788484231.116119 MIRROR_RESYNC 7 mirrors 000317 1788484232.466981 fallocate 0x000622 thru 0x00af57 (0xa936 bytes) 000318 1788484232.510615 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x20fbb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20fbb): Mirror 3: 0xc4f88a2f (stale) Mirror 4: 0xc4f88a2f (stale) Mirror 5: 0xc4f88a2f (stale) Mirror 7: 0xc4f88a2f (stale) Mirror 8: 0xc4f88a2f (stale) Mirror 9: 0xc4f88a2f (stale) CRC-32 checksum value for chunk [0, 0x20fbb): Mirror 1: 0xc4f88a2f Mirror 10: 0xc4f88a2f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20fbb: skipped 000320 1788484233.492929 mapread 0x006c5a thru 0x008e09 (0x21b0 bytes) 000321 1788484233.555768 write 0x01b778 thru 0x0259ee (0xa277 bytes) 000322 1788484233.625908 punch from 0x01b3b2 to 0x0206ca (0x5319 bytes) 000323 1788484233.647520 mapwrite 0x0266f6 thru 0x027f48 (0x1853 bytes) 000324 1788484233.770510 mapread 0x01293e thru 0x018ff9 (0x66bc bytes) 000325 1788484233.790630 fallocate 0x0316b5 thru 0x03ffff (0xe94b bytes) 000326 1788484233.804684 trunc from 0x027f49 to 0x033e8b (0xbf43 bytes) 000327 1788484233.826724 read 0x02d00c thru 0x033e8b (0x6e80 bytes) 000328 1788484233.847209 write 0x01cd51 thru 0x021747 (0x49f7 bytes) 000329 1788484233.868030 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33e8c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e8c): Mirror 3: 0x31f8b7b6 (stale) Mirror 4: 0x31f8b7b6 (stale) Mirror 5: 0x31f8b7b6 (stale) Mirror 7: 0x31f8b7b6 (stale) Mirror 8: 0x31f8b7b6 (stale) Mirror 9: 0x31f8b7b6 (stale) Mirror 10: 0x31f8b7b6 (stale) CRC-32 checksum value for chunk [0, 0x33e8c): Mirror 1: 0xf6879fb Mirror 11: 0xf6879fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e8c: skipped 000331 1788484234.915636 read 0x031142 thru 0x033e8b (0x2d4a bytes) 000332 1788484234.940448 mapread 0x016720 thru 0x01adc9 (0x46aa bytes) 000333 1788484234.990496 mapwrite 0x00e8f2 thru 0x01c345 (0xda54 bytes) 000334 1788484235.119956 trunc from 0x00224f to 0x003974 (0x1726 bytes) 000337 1788484235.129551 trunc from 0x033e8c to 0x032d24 (0x1167 bytes) 000338 1788484235.159089 read 0x00017f thru 0x003fe3 (0x3e65 bytes) 000340 1788484235.169116 mapread 0x00da66 thru 0x01887f (0xae1a bytes) 000341 1788484235.170677 mapwrite 0x03155d thru 0x035806 (0x42aa bytes) 000342 1788484235.209964 mapwrite 0x006058 thru 0x00ba8c (0x5a35 bytes) 000343 1788484235.232406 MIRROR_RESYNC 9 mirrors 000344 1788484238.330178 punch from 0x01f80f to 0x02b50d (0xbcff bytes) 000346 1788484238.372329 mapwrite 0x02ac5c thru 0x03714e (0xc4f3 bytes) 000348 1788484238.455346 mapread 0x029b24 thru 0x02fd6c (0x6249 bytes) 000349 1788484238.457987 write 0x02c394 thru 0x02c4a6 (0x113 bytes) 000350 1788484238.459664 fallocate 0x031eb6 thru 0x03c93a (0xaa85 bytes) 000351 1788484238.465332 write 0x00d418 thru 0x01a251 (0xce3a bytes) 000352 1788484238.478077 MIRROR_RESYNC 9 mirrors 000353 1788484241.857913 read 0x015cb9 thru 0x018364 (0x26ac bytes) 000354 1788484241.865006 write 0x02dda4 thru 0x0370fa (0x9357 bytes) 000355 1788484241.957121 mapwrite 0x02fd32 thru 0x033135 (0x3404 bytes) 000356 1788484242.011150 read 0x02407f thru 0x02ca5d (0x89df bytes) 000357 1788484242.037105 trunc from 0x03c93b to 0x00be0c (0x30b2e bytes) 000358 1788484242.095547 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xbe0d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe0d): Mirror 3: 0x1093736 (stale) Mirror 4: 0x1093736 (stale) Mirror 5: 0x1093736 (stale) Mirror 7: 0x1093736 (stale) Mirror 8: 0x1093736 (stale) Mirror 9: 0x1093736 (stale) Mirror 10: 0x1093736 (stale) Mirror 11: 0x1093736 (stale) CRC-32 checksum value for chunk [0, 0xbe0d): Mirror 1: 0x1093736 Mirror 12: 0x1093736 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe0d: skipped 000359 1788484243.526364 write 0x03e0a3 thru 0x03ffff (0x1f5d bytes) 000361 1788484243.591522 fallocate 0x020788 thru 0x02f37b (0xebf4 bytes) 000362 1788484243.598497 fallocate 0x00fcb6 thru 0x01d516 (0xd861 bytes) 000363 1788484243.611322 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1825a7db (stale) Mirror 4: 0x1825a7db (stale) Mirror 5: 0x1825a7db (stale) Mirror 7: 0x1825a7db (stale) Mirror 8: 0x1825a7db (stale) Mirror 10: 0x1825a7db (stale) Mirror 11: 0x1825a7db (stale) Mirror 12: 0xb94d0734 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1825a7db (stale) Mirror 4: 0x1825a7db (stale) Mirror 5: 0x1825a7db (stale) Mirror 7: 0x1825a7db (stale) Mirror 8: 0x1825a7db (stale) Mirror 10: 0x1825a7db (stale) Mirror 11: 0x1825a7db (stale) Mirror 12: 0xb94d0734 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000364 1788484244.782692 mapwrite 0x01d2a8 thru 0x02100f (0x3d68 bytes) 000366 1788484244.875506 punch from 0x012a88 to 0x01b4bb (0x8a34 bytes) 000368 1788484244.888935 punch from 0x00c16e to 0x015416 (0x92a9 bytes) 000369 1788484244.896339 mapread 0x023921 thru 0x023d8f (0x46f bytes) 000370 1788484244.898150 read 0x032ad3 thru 0x03ffff (0xd52d bytes) 000371 1788484244.913080 trunc from 0x01bdf0 to 0x025585 (0x9796 bytes) 000372 1788484244.922618 mapwrite 0x0293a2 thru 0x029c92 (0x8f1 bytes) 000373 1788484244.948145 trunc from 0x003560 to 0x008b2d (0x55ce bytes) 000374 1788484244.953190 mapwrite 0x003a4f thru 0x00c34c (0x88fe bytes) 000375 1788484244.988894 fallocate 0x035433 thru 0x037421 (0x1fef bytes) 000376 1788484244.999621 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1825a7db (stale) Mirror 4: 0x1825a7db (stale) Mirror 5: 0x1825a7db (stale) Mirror 7: 0x1825a7db (stale) Mirror 8: 0x1825a7db (stale) Mirror 10: 0x1825a7db (stale) Mirror 11: 0x1825a7db (stale) Mirror 12: 0xb94d0734 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x12ac9f53 Mirror 13: 0x12ac9f53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000377 1788484246.110250 mapwrite 0x00db36 thru 0x018ddf (0xb2aa bytes) 000378 1788484246.280903 mapwrite 0x006905 thru 0x01496e (0xe06a bytes) 000379 1788484246.315451 mapwrite 0x034434 thru 0x036e9e (0x2a6b bytes) 000380 1788484246.341901 mapwrite 0x00e2cd thru 0x0177ff (0x9533 bytes) 000381 1788484246.361262 trunc from 0x022f86 to 0x023494 (0x50f bytes) 000382 1788484246.368023 punch from 0x02d87b to 0x0304d5 (0x2c5b bytes) 000383 1788484246.374223 mapread 0x03c2bd thru 0x03da10 (0x1754 bytes) 000385 1788484246.381972 read 0x0127be thru 0x0151f2 (0x2a35 bytes) 000386 1788484246.382656 punch from 0x036104 to 0x03a786 (0x4683 bytes) 000387 1788484246.393236 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1825a7db (stale) Mirror 4: 0x1825a7db (stale) Mirror 5: 0x1825a7db (stale) Mirror 7: 0x1825a7db (stale) Mirror 8: 0x1825a7db (stale) Mirror 10: 0x1825a7db (stale) Mirror 11: 0x1825a7db (stale) Mirror 12: 0xb94d0734 (stale) Mirror 13: 0x12ac9f53 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ea786a1 Mirror 14: 0x1ea786a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1788484247.698762 punch from 0x0127a7 to 0x017c83 (0x54dd bytes) 000389 1788484247.759493 read 0x03a28f thru 0x03e833 (0x45a5 bytes) 000390 1788484247.772350 read 0x023eb0 thru 0x02e4f0 (0xa641 bytes) 000391 1788484247.785523 read 0x03145b thru 0x039046 (0x7bec bytes) 000393 1788484247.786971 trunc from 0x040000 to 0x001ee8 (0x3e117 bytes) 000394 1788484247.862182 read 0x000ab8 thru 0x001ee8 (0x1431 bytes) 000395 1788484247.863929 trunc from 0x001ee9 to 0x00dd80 (0xbe98 bytes) 000396 1788484247.890267 read 0x002d7d thru 0x006aa4 (0x3d28 bytes) 000397 1788484247.908929 fallocate 0x03d797 thru 0x03ffff (0x2869 bytes) 000398 1788484247.918635 fallocate 0x03c3e2 thru 0x03ffff (0x3c1e bytes) 000399 1788484247.926155 punch from 0x00ea3d to 0x01adac (0xc370 bytes) 000400 1788484247.938533 MIRROR_RESYNC 11 mirrors 000401 1788484249.989032 trunc from 0x040000 to 0x0168f8 (0x29707 bytes) 000402 1788484250.026793 mapread 0x00f00c thru 0x0168f8 (0x78ed bytes) 000403 1788484250.043081 punch from 0x00ebe8 to 0x01672e (0x7b47 bytes) 000405 1788484250.048363 trunc from 0x0168f9 to 0x006b93 (0xfd65 bytes) 000406 1788484250.068989 mapread 0x00504c thru 0x006b93 (0x1b48 bytes) 000407 1788484250.077854 read 0x0002cf thru 0x006b93 (0x68c5 bytes) 000408 1788484250.084231 trunc from 0x006b94 to 0x012cad (0xc11a bytes) 000409 1788484250.104004 trunc from 0x00ad7f to 0x0127e4 (0x7a66 bytes) 000410 1788484250.111240 mapwrite 0x039c17 thru 0x03ffff (0x63e9 bytes) 000411 1788484250.157946 write 0x00c0f7 thru 0x012683 (0x658d bytes) 000412 1788484250.169603 punch from 0x008a94 to 0x00b531 (0x2a9e bytes) 000413 1788484250.179654 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x59da4ada (stale) Mirror 5: 0x59da4ada (stale) Mirror 7: 0x59da4ada (stale) Mirror 8: 0x59da4ada (stale) Mirror 10: 0x59da4ada (stale) Mirror 11: 0x59da4ada (stale) Mirror 12: 0x59da4ada (stale) Mirror 13: 0x59da4ada (stale) Mirror 14: 0x59da4ada (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x59da4ada (stale) Mirror 5: 0x59da4ada (stale) Mirror 7: 0x59da4ada (stale) Mirror 8: 0x59da4ada (stale) Mirror 10: 0x59da4ada (stale) Mirror 11: 0x59da4ada (stale) Mirror 12: 0x59da4ada (stale) Mirror 13: 0x59da4ada (stale) Mirror 14: 0x59da4ada (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000414 1788484251.029960 mapread 0x01d863 thru 0x028189 (0xa927 bytes) 000415 1788484251.062823 trunc from 0x030c3e to 0x039d7c (0x913f bytes) 000416 1788484251.068446 mapread 0x01407c thru 0x02388a (0xf80f bytes) 000417 1788484251.079225 mapwrite 0x00a3d5 thru 0x00d731 (0x335d bytes) 000419 1788484251.099558 mapread 0x02fd1c thru 0x03d1ae (0xd493 bytes) 000420 1788484251.120025 read 0x03bde5 thru 0x03ffff (0x421b bytes) 000421 1788484251.121234 trunc from 0x040000 to 0x0347dc (0xb823 bytes) 000422 1788484251.149722 punch from 0x024fe9 to 0x02975f (0x4777 bytes) 000423 1788484251.157913 trunc from 0x0347dd to 0x02cb94 (0x7c48 bytes) 000424 1788484251.173807 punch from 0x003d24 to 0x00e843 (0xab20 bytes) 000425 1788484251.179732 punch from 0x01be2a to 0x01ca65 (0xc3c bytes) 000426 1788484251.185125 trunc from 0x02cb95 to 0x02cad3 (0xc1 bytes) 000428 1788484251.204847 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cad4): Mirror 4: 0x6ff20736 (stale) Mirror 5: 0x6ff20736 (stale) Mirror 7: 0x6ff20736 (stale) Mirror 8: 0x6ff20736 (stale) Mirror 10: 0x6ff20736 (stale) Mirror 12: 0x6ff20736 (stale) Mirror 13: 0x6ff20736 (stale) Mirror 14: 0x6ff20736 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cad4): Mirror 4: 0x6ff20736 (stale) Mirror 5: 0x6ff20736 (stale) Mirror 7: 0x6ff20736 (stale) Mirror 8: 0x6ff20736 (stale) Mirror 10: 0x6ff20736 (stale) Mirror 12: 0x6ff20736 (stale) Mirror 13: 0x6ff20736 (stale) Mirror 14: 0x6ff20736 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cad4: skipped 000429 1788484251.780239 mapwrite 0x000a24 thru 0x00c712 (0xbcef bytes) 000430 1788484251.855875 punch from 0x00be7a to 0x015545 (0x96cc bytes) 000431 1788484251.862225 mapwrite 0x027d4e thru 0x02b5b0 (0x3863 bytes) 000432 1788484251.909144 fallocate 0x00ae94 thru 0x017f5a (0xd0c7 bytes) 000433 1788484251.919485 mapwrite 0x029cb5 thru 0x03453b (0xa887 bytes) 000434 1788484252.017476 read 0x02c1ea thru 0x03453b (0x8352 bytes) 000435 1788484252.018985 read 0x013cd0 thru 0x021ce1 (0xe012 bytes) 000438 1788484252.034831 mapread 0x032102 thru 0x032c97 (0xb96 bytes) 000439 1788484252.040247 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3453c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3453c): Mirror 4: 0xfcc9b457 (stale) Mirror 5: 0xfcc9b457 (stale) Mirror 7: 0xfcc9b457 (stale) Mirror 8: 0xfcc9b457 (stale) Mirror 10: 0xfcc9b457 (stale) Mirror 12: 0xfcc9b457 (stale) Mirror 13: 0xfcc9b457 (stale) Mirror 14: 0xfcc9b457 (stale) CRC-32 checksum value for chunk [0, 0x3453c): Mirror 1: 0x3e8ecac7 Mirror 15: 0x3e8ecac7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3453c: skipped 000440 1788484253.280120 fallocate 0x036702 thru 0x03ffff (0x98fe bytes) 000441 1788484253.389122 mapread 0x017e5b thru 0x019d17 (0x1ebd bytes) 000442 1788484253.456736 trunc from 0x040000 to 0x038c9b (0x7364 bytes) 000443 1788484253.503554 write 0x035b6d thru 0x03ffff (0xa493 bytes) 000444 1788484253.517210 fallocate 0x02418a thru 0x0302ad (0xc124 bytes) 000445 1788484253.531945 MIRROR_RESYNC 10 mirrors 000446 1788484255.434769 MIRROR_RESYNC 10 mirrors 000447 1788484255.900785 fallocate 0x00ee3b thru 0x01ba33 (0xcbf9 bytes) 000448 1788484255.944907 read 0x00351a thru 0x0055b8 (0x209f bytes) 000449 1788484255.951563 write 0x01905d thru 0x02207c (0x9020 bytes) 000450 1788484255.965713 MIRROR_RESYNC 10 mirrors 000451 1788484258.508416 fallocate 0x02ad8c thru 0x03abf0 (0xfe65 bytes) 000452 1788484258.556919 write 0x02689d thru 0x029899 (0x2ffd bytes) 000453 1788484258.579176 trunc from 0x040000 to 0x03f8e1 (0x71e bytes) 000454 1788484258.607367 fallocate 0x03f246 thru 0x03ffff (0xdba bytes) 000455 1788484258.615703 mapread 0x02fdb1 thru 0x03791d (0x7b6d bytes) 000456 1788484258.640145 read 0x0144ad thru 0x0229f4 (0xe548 bytes) 000457 1788484258.661107 write 0x00b701 thru 0x00c41d (0xd1d bytes) 000458 1788484258.677413 MIRROR_RESYNC 10 mirrors 000459 1788484261.344596 trunc from 0x00d13d to 0x00f5ec (0x24b0 bytes) 000460 1788484261.385085 write 0x020e90 thru 0x02a594 (0x9705 bytes) 000461 1788484261.397039 mapread 0x0336c8 thru 0x03838c (0x4cc5 bytes) 000462 1788484261.420159 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1663aa43 (stale) Mirror 5: 0x1663aa43 (stale) Mirror 7: 0x1663aa43 (stale) Mirror 10: 0x1663aa43 (stale) Mirror 12: 0x1663aa43 (stale) Mirror 13: 0x1663aa43 (stale) Mirror 14: 0x1663aa43 (stale) Mirror 15: 0x1663aa43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1663aa43 (stale) Mirror 5: 0x1663aa43 (stale) Mirror 7: 0x1663aa43 (stale) Mirror 10: 0x1663aa43 (stale) Mirror 12: 0x1663aa43 (stale) Mirror 13: 0x1663aa43 (stale) Mirror 14: 0x1663aa43 (stale) Mirror 15: 0x1663aa43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000463 1788484262.198979 write 0x03c2a9 thru 0x03ffff (0x3d57 bytes) 000464 1788484262.237133 write 0x01ac98 thru 0x02a919 (0xfc82 bytes) 000465 1788484262.257521 punch from 0x00046e to 0x007d80 (0x7913 bytes) 000466 1788484262.263187 fallocate 0x02b15d thru 0x035215 (0xa0b9 bytes) 000467 1788484262.269131 read 0x000b11 thru 0x000cc4 (0x1b4 bytes) 000468 1788484262.277852 trunc from 0x02a685 to 0x02dcb3 (0x362f bytes) 000469 1788484262.303037 trunc from 0x040000 to 0x00094b (0x3f6b4 bytes) 000470 1788484262.327254 mapwrite 0x026b20 thru 0x02dd7b (0x725c bytes) 000471 1788484262.395864 fallocate 0x00f9e4 thru 0x012ef7 (0x3514 bytes) 000472 1788484262.404256 trunc from 0x016e2a to 0x01f973 (0x8b4a bytes) 000473 1788484262.410839 fallocate 0x00a514 thru 0x017c2c (0xd719 bytes) 000474 1788484262.416789 fallocate 0x02c2f1 thru 0x03a63e (0xe34e bytes) 000475 1788484262.421896 mapwrite 0x039776 thru 0x03f9c9 (0x6254 bytes) 000476 1788484262.474965 mapread 0x021f88 thru 0x031bcb (0xfc44 bytes) 000477 1788484262.484128 trunc from 0x03f9ca to 0x008319 (0x376b0 bytes) 000478 1788484262.519477 mapread 0x000a3d thru 0x0020fe (0x16c2 bytes) 000479 1788484262.544529 read 0x006e53 thru 0x0077a0 (0x94e bytes) 000480 1788484262.545679 fallocate 0x00136d thru 0x001396 (0x2a bytes) 000481 1788484262.551856 mapwrite 0x013c58 thru 0x014102 (0x4ab bytes) 000482 1788484262.664005 mapread 0x00cf38 thru 0x0102da (0x33a3 bytes) 000483 1788484262.665774 punch from 0x000dfc to 0x006666 (0x586b bytes) 000484 1788484262.677044 mapwrite 0x000098 thru 0x00b41d (0xb386 bytes) 000485 1788484262.804886 mapread 0x0021b8 thru 0x0080ea (0x5f33 bytes) 000487 1788484262.806975 write 0x004490 thru 0x0113b6 (0xcf27 bytes) 000488 1788484262.809243 mapwrite 0x01cb91 thru 0x0272b0 (0xa720 bytes) 000489 1788484262.877860 mapwrite 0x004281 thru 0x0043bc (0x13c bytes) 000490 1788484262.890643 read 0x023473 thru 0x0272b0 (0x3e3e bytes) 000491 1788484262.891265 trunc from 0x014576 to 0x015ab6 (0x1541 bytes) 000492 1788484262.898966 punch from 0x007555 to 0x00a8bd (0x3369 bytes) 000493 1788484262.908125 read 0x013166 thru 0x014688 (0x1523 bytes) 000494 1788484262.923180 trunc from 0x009a23 to 0x014db3 (0xb391 bytes) 000495 1788484262.930385 fallocate 0x009056 thru 0x00bf75 (0x2f20 bytes) 000496 1788484262.936502 read 0x0009f1 thru 0x010674 (0xfc84 bytes) 000497 1788484262.944815 mapread 0x026e1f thru 0x0272b0 (0x492 bytes) 000498 1788484262.947192 trunc from 0x00d2b3 to 0x01c5c5 (0xf313 bytes) 000499 1788484262.961449 mapwrite 0x012308 thru 0x0142b3 (0x1fac bytes) 000500 1788484263.007531 trunc from 0x0272b1 to 0x0239bc (0x38f4 bytes) 000501 1788484263.039924 fallocate 0x000b39 thru 0x00dfe1 (0xd4a9 bytes) 000502 1788484263.049520 fallocate 0x015f57 thru 0x01d1a4 (0x724e bytes) 000503 1788484263.059088 read 0x000b4c thru 0x000e08 (0x2bd bytes) 000504 1788484263.060079 mapwrite 0x00a05a thru 0x00d0ce (0x3075 bytes) 000505 1788484263.101881 trunc from 0x0239bd to 0x00e26b (0x15751 bytes) 000506 1788484263.125027 mapwrite 0x02fac1 thru 0x0367e2 (0x6d22 bytes) 000507 1788484263.191252 write 0x007290 thru 0x00e235 (0x6fa6 bytes) 000508 1788484263.193561 read 0x01944e thru 0x020a40 (0x75f3 bytes) 000509 1788484263.216911 punch from 0x00123f to 0x00b2c3 (0xa085 bytes) 000510 1788484263.245624 write 0x00f59d thru 0x010c68 (0x16cc bytes) 000512 1788484263.249743 mapread 0x0024d1 thru 0x0075f2 (0x5122 bytes) 000514 1788484263.298196 mapwrite 0x028932 thru 0x036001 (0xd6d0 bytes) 000516 1788484263.331556 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367e3): Mirror 5: 0x90632e5a (stale) Mirror 7: 0x90632e5a (stale) Mirror 10: 0x90632e5a (stale) Mirror 12: 0x90632e5a (stale) Mirror 13: 0x90632e5a (stale) Mirror 14: 0x90632e5a (stale) Mirror 15: 0x90632e5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367e3): Mirror 5: 0x90632e5a (stale) Mirror 7: 0x90632e5a (stale) Mirror 10: 0x90632e5a (stale) Mirror 12: 0x90632e5a (stale) Mirror 13: 0x90632e5a (stale) Mirror 14: 0x90632e5a (stale) Mirror 15: 0x90632e5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367e3: skipped 000517 1788484264.135585 trunc from 0x029010 to 0x031917 (0x8908 bytes) 000518 1788484264.145322 punch from 0x00e7cb to 0x0188c8 (0xa0fe bytes) 000520 1788484264.154996 read 0x017316 thru 0x01a3af (0x309a bytes) 000521 1788484264.188717 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367e3): Mirror 5: 0x90632e5a (stale) Mirror 7: 0x90632e5a (stale) Mirror 10: 0x90632e5a (stale) Mirror 12: 0x90632e5a (stale) Mirror 13: 0x90632e5a (stale) Mirror 14: 0x90632e5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367e3): Mirror 5: 0x90632e5a (stale) Mirror 7: 0x90632e5a (stale) Mirror 10: 0x90632e5a (stale) Mirror 12: 0x90632e5a (stale) Mirror 13: 0x90632e5a (stale) Mirror 14: 0x90632e5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367e3: skipped 000522 1788484264.799160 mapwrite 0x0125ff thru 0x015826 (0x3228 bytes) 000523 1788484264.850579 trunc from 0x00e9e3 to 0x010c68 (0x2286 bytes) 000524 1788484264.873271 write 0x030e2b thru 0x03b1d6 (0xa3ac bytes) 000525 1788484264.893554 write 0x010838 thru 0x018740 (0x7f09 bytes) 000526 1788484264.901889 mapwrite 0x00dc4a thru 0x013ffc (0x63b3 bytes) 000527 1788484264.975682 punch from 0x010adc to 0x016abc (0x5fe1 bytes) 000529 1788484264.988959 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b1d7): Mirror 5: 0xbf2d3777 (stale) Mirror 7: 0xbf2d3777 (stale) Mirror 12: 0xbf2d3777 (stale) Mirror 13: 0xbf2d3777 (stale) Mirror 14: 0xbf2d3777 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b1d7): Mirror 5: 0xbf2d3777 (stale) Mirror 7: 0xbf2d3777 (stale) Mirror 12: 0xbf2d3777 (stale) Mirror 13: 0xbf2d3777 (stale) Mirror 14: 0xbf2d3777 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b1d7: skipped 000530 1788484265.667044 read 0x02e632 thru 0x03740f (0x8dde bytes) 000531 1788484265.687770 fallocate 0x03bda1 thru 0x03ffff (0x425f bytes) 000532 1788484265.701312 mapwrite 0x01be4f thru 0x025e2f (0x9fe1 bytes) 000533 1788484265.781167 punch from 0x03f558 to 0x03ffff (0xaa8 bytes) 000534 1788484265.798172 read 0x013a37 thru 0x017957 (0x3f21 bytes) 000535 1788484265.826029 punch from 0x03fd63 to 0x03ffff (0x29d bytes) 000536 1788484265.836409 write 0x01f37b thru 0x0296e4 (0xa36a bytes) 000537 1788484265.855879 trunc from 0x040000 to 0x033898 (0xc767 bytes) 000538 1788484265.892168 punch from 0x002ffd to 0x005f7e (0x2f82 bytes) 000539 1788484265.903971 trunc from 0x033899 to 0x021582 (0x12316 bytes) 000540 1788484265.937362 write 0x036a1f thru 0x03a69c (0x3c7e bytes) 000541 1788484265.939851 mapread 0x010d47 thru 0x01cce0 (0xbf9a bytes) 000542 1788484265.942375 fallocate 0x02427a thru 0x02b92b (0x76b2 bytes) 000543 1788484265.948896 write 0x035bf2 thru 0x037324 (0x1733 bytes) 000544 1788484265.956809 trunc from 0x000c44 to 0x00f43b (0xe7f8 bytes) 000545 1788484265.965529 trunc from 0x03a69d to 0x01ea67 (0x1bc35 bytes) 000546 1788484266.002909 mapread 0x0168ac thru 0x0193d8 (0x2b2d bytes) 000547 1788484266.005804 mapwrite 0x0273ed thru 0x02ad64 (0x3978 bytes) 000548 1788484266.114127 mapwrite 0x0061e4 thru 0x00cf74 (0x6d91 bytes) 000549 1788484266.200808 mapwrite 0x03792b thru 0x03a6e3 (0x2db9 bytes) 000550 1788484266.346491 trunc from 0x03a6e4 to 0x03e982 (0x429f bytes) 000551 1788484266.372390 punch from 0x000501 to 0x0061f4 (0x5cf4 bytes) 000552 1788484266.384895 mapwrite 0x013507 thru 0x016ca8 (0x37a2 bytes) 000553 1788484266.418160 mapread 0x0004b0 thru 0x00a5f1 (0xa142 bytes) 000554 1788484266.473045 mapwrite 0x03ebfc thru 0x03ffff (0x1404 bytes) 000555 1788484266.573498 write 0x00d6fc thru 0x01a2e1 (0xcbe6 bytes) 000556 1788484266.590231 trunc from 0x040000 to 0x023656 (0x1c9a9 bytes) 000557 1788484266.623059 punch from 0x022c79 to 0x023656 (0x9de bytes) 000559 1788484266.634445 mapwrite 0x01311e thru 0x01bfbb (0x8e9e bytes) 000560 1788484266.664825 mapwrite 0x01d4f3 thru 0x020b85 (0x3693 bytes) 000561 1788484266.693637 MIRROR_RESYNC 6 mirrors 000562 1788484268.422952 read 0x00c87c thru 0x00f0d2 (0x2857 bytes) 000563 1788484268.430822 trunc from 0x023657 to 0x02dd3c (0xa6e6 bytes) 000564 1788484268.480392 mapwrite 0x00ee80 thru 0x019382 (0xa503 bytes) 000565 1788484268.543075 trunc from 0x007a8c to 0x00ca06 (0x4f7b bytes) 000566 1788484268.552225 mapwrite 0x000b1e thru 0x003b36 (0x3019 bytes) 000567 1788484268.616146 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd3d): Mirror 5: 0x5bd7e55d (stale) Mirror 12: 0x5bd7e55d (stale) Mirror 13: 0x5bd7e55d (stale) Mirror 14: 0x5bd7e55d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd3d): Mirror 5: 0x5bd7e55d (stale) Mirror 12: 0x5bd7e55d (stale) Mirror 13: 0x5bd7e55d (stale) Mirror 14: 0x5bd7e55d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd3d: skipped 000568 1788484269.166516 write 0x025190 thru 0x0255b8 (0x429 bytes) 000570 1788484269.178954 mapread 0x006954 thru 0x00a59b (0x3c48 bytes) 000572 1788484269.197472 mapwrite 0x02c8cf thru 0x03a8d0 (0xe002 bytes) 000573 1788484269.264876 read 0x029c6e thru 0x02be89 (0x221c bytes) 000574 1788484269.269931 read 0x02f256 thru 0x031f40 (0x2ceb bytes) 000575 1788484269.273992 trunc from 0x03a8d1 to 0x030321 (0xa5af bytes) 000576 1788484269.325652 trunc from 0x030322 to 0x00ff99 (0x20388 bytes) 000577 1788484269.360937 punch from 0x00209c to 0x00ff99 (0xdefe bytes) 000578 1788484269.369031 trunc from 0x00d9db to 0x00ff99 (0x25bf bytes) 000579 1788484269.375345 trunc from 0x0083d7 to 0x00ff99 (0x7bc3 bytes) 000580 1788484269.386066 mapread 0x006810 thru 0x008f8d (0x277e bytes) 000581 1788484269.400998 fallocate 0x0232cf thru 0x0258bc (0x25ee bytes) 000582 1788484269.420246 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x258bd): Mirror 12: 0xf3a1a76b (stale) Mirror 13: 0xf3a1a76b (stale) Mirror 14: 0xf3a1a76b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x258bd): Mirror 12: 0xf3a1a76b (stale) Mirror 13: 0xf3a1a76b (stale) Mirror 14: 0xf3a1a76b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x258bd: skipped 000583 1788484269.868233 mapread 0x01a5d5 thru 0x01b2f9 (0xd25 bytes) 000585 1788484269.893951 write 0x03ac35 thru 0x03d0ed (0x24b9 bytes) 000586 1788484269.903900 write 0x023ed8 thru 0x0244d6 (0x5ff bytes) 000587 1788484269.920110 read 0x010a28 thru 0x01d700 (0xccd9 bytes) 000588 1788484269.939407 trunc from 0x00bce3 to 0x01b5dc (0xf8fa bytes) 000589 1788484269.947846 write 0x01902b thru 0x028929 (0xf8ff bytes) 000590 1788484269.968203 mapread 0x033370 thru 0x03a2ba (0x6f4b bytes) 000591 1788484270.000493 punch from 0x03473f to 0x03d0ed (0x89af bytes) 000592 1788484270.037498 punch from 0x035c6e to 0x038267 (0x25fa bytes) 000593 1788484270.059946 MIRROR_RESYNC 4 mirrors 000594 1788484271.413340 write 0x00cfea thru 0x017e5d (0xae74 bytes) 000595 1788484271.445403 trunc from 0x022b55 to 0x02e46d (0xb919 bytes) 000596 1788484271.451926 trunc from 0x03d0ee to 0x00b733 (0x319ba bytes) 000598 1788484271.493009 mapread 0x0027bc thru 0x009682 (0x6ec7 bytes) 000600 1788484271.514047 fallocate 0x02d629 thru 0x0389a9 (0xb381 bytes) 000603 1788484271.521337 punch from 0x0099e2 to 0x00b733 (0x1d52 bytes) 000604 1788484271.526463 trunc from 0x00b734 to 0x0099d8 (0x1d5b bytes) 000605 1788484271.544435 read 0x007c75 thru 0x0099d8 (0x1d64 bytes) 000606 1788484271.553042 punch from 0x006d49 to 0x0099d8 (0x2c90 bytes) 000608 1788484271.559754 read 0x005d81 thru 0x0099d8 (0x3c58 bytes) 000611 1788484271.569725 write 0x024455 thru 0x0271ff (0x2dab bytes) 000612 1788484271.571420 mapwrite 0x027c38 thru 0x02f35c (0x7725 bytes) 000614 1788484271.623991 punch from 0x00410c to 0x00e4ff (0xa3f4 bytes) 000616 1788484271.634474 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f35d): Mirror 13: 0xb705847e (stale) Mirror 14: 0xb705847e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f35d): Mirror 13: 0xb705847e (stale) Mirror 14: 0xb705847e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f35d: skipped 000618 1788484271.999063 mapwrite 0x015a20 thru 0x01a707 (0x4ce8 bytes) 000620 1788484272.047820 read 0x006e5d thru 0x01071a (0x98be bytes) 000622 1788484272.074836 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f35d): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f35d): Mirror 13: 0xb705847e (stale) Mirror 14: 0xb705847e (stale) CRC-32 checksum value for chunk [0, 0x2f35d): Mirror 1: 0x7082a739 Mirror 15: 0x7082a739 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f35d: skipped 000623 1788484273.474589 punch from 0x0198b8 to 0x025c5b (0xc3a4 bytes) 000624 1788484273.537242 mapwrite 0x001594 thru 0x007a50 (0x64bd bytes) 000625 1788484273.594250 mapread 0x00b84e thru 0x00e1f1 (0x29a4 bytes) 000626 1788484273.599379 read 0x00b22f thru 0x01b161 (0xff33 bytes) 000627 1788484273.619799 trunc from 0x02f35d to 0x0022f0 (0x2d06c bytes) 000628 1788484273.717859 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22f1): Mirror 14: 0xb8cde2cd (stale) Mirror 15: 0xb8cde2cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22f1): Mirror 14: 0xb8cde2cd (stale) Mirror 15: 0xb8cde2cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22f1: skipped 000629 1788484274.119040 trunc from 0x000152 to 0x0022f0 (0x219f bytes) 000630 1788484274.156224 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22f1): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22f1): Mirror 14: 0xb8cde2cd (stale) Mirror 15: 0xb8cde2cd (stale) CRC-32 checksum value for chunk [0, 0x22f1): Mirror 1: 0x489ac16a Mirror 16: 0x489ac16a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22f1: skipped 000631 1788484275.217835 punch from 0x001190 to 0x0022f0 (0x1161 bytes) 000632 1788484275.282409 trunc from 0x001fda to 0x0022f0 (0x317 bytes) 000633 1788484275.294430 fallocate 0x025df8 thru 0x030f64 (0xb16d bytes) 000634 1788484275.306113 punch from 0x001a81 to 0x0022f0 (0x870 bytes) 000635 1788484275.314018 write 0x003c4e thru 0x00bfdd (0x8390 bytes) 000637 1788484275.321679 punch from 0x003226 to 0x0057dc (0x25b7 bytes) 000638 1788484275.361896 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbfde): Mirror 15: 0x5ed141f1 (stale) Mirror 16: 0xf0c4960a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbfde): Mirror 15: 0x5ed141f1 (stale) Mirror 16: 0xf0c4960a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbfde: skipped 000639 1788484275.789248 trunc from 0x00023c to 0x00883f (0x8604 bytes) 000640 1788484275.803982 mapread 0x00148e thru 0x00bfdd (0xab50 bytes) 000641 1788484275.856909 mapwrite 0x02b427 thru 0x02dcff (0x28d9 bytes) 000642 1788484275.918449 fallocate 0x004eca thru 0x013841 (0xe978 bytes) 000643 1788484275.924871 mapwrite 0x03336c thru 0x03f0d5 (0xbd6a bytes) 000644 1788484275.993429 fallocate 0x03f8ff thru 0x03ffff (0x701 bytes) 000645 1788484276.004089 mapwrite 0x000496 thru 0x0097fe (0x9369 bytes) 000646 1788484276.040049 mapread 0x00205a thru 0x004b10 (0x2ab7 bytes) 000647 1788484276.040935 punch from 0x026c6d to 0x032966 (0xbcfa bytes) 000648 1788484276.047009 read 0x021cc5 thru 0x030065 (0xe3a1 bytes) 000649 1788484276.060225 fallocate 0x0315f9 thru 0x032372 (0xd7a bytes) 000650 1788484276.072455 read 0x03bb9f thru 0x03ffff (0x4461 bytes) 000651 1788484276.075727 write 0x019eba thru 0x01f89c (0x59e3 bytes) 000653 1788484276.095039 fallocate 0x024642 thru 0x02a8d1 (0x6290 bytes) 000654 1788484276.101116 trunc from 0x040000 to 0x00ca40 (0x335bf bytes) 000655 1788484276.137909 read 0x004c76 thru 0x00a80f (0x5b9a bytes) 000656 1788484276.138730 fallocate 0x023459 thru 0x02537f (0x1f27 bytes) 000657 1788484276.147189 trunc from 0x00ca41 to 0x00c0e1 (0x95f bytes) 000658 1788484276.168950 write 0x020b73 thru 0x024f17 (0x43a5 bytes) 000660 1788484276.172042 mapwrite 0x00868c thru 0x0134d1 (0xae46 bytes) 000661 1788484276.249336 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24f18): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f18): Mirror 15: 0x968db307 (stale) Mirror 16: 0xc8bb2e40 (stale) CRC-32 checksum value for chunk [0, 0x24f18): Mirror 1: 0x864ec4d1 Mirror 17: 0x864ec4d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f18: skipped 000662 1788484277.521969 trunc from 0x024f18 to 0x03c481 (0x1756a bytes) 000663 1788484277.644761 read 0x002e8e thru 0x00d15b (0xa2ce bytes) 000665 1788484277.672141 mapwrite 0x00f450 thru 0x00f7eb (0x39c bytes) 000666 1788484277.710190 read 0x0392ca thru 0x03a99f (0x16d6 bytes) 000667 1788484277.760659 mapread 0x00d498 thru 0x00e9f3 (0x155c bytes) 000668 1788484277.761490 read 0x02a7f1 thru 0x038203 (0xda13 bytes) 000669 1788484277.762072 read 0x0378a1 thru 0x03c481 (0x4be1 bytes) 000670 1788484277.762608 punch from 0x0143cf to 0x01dd82 (0x99b4 bytes) 000671 1788484277.767436 trunc from 0x03c482 to 0x01fce6 (0x1c79b bytes) 000672 1788484277.789089 mapread 0x00d414 thru 0x019a31 (0xc61e bytes) 000673 1788484277.799134 punch from 0x00a62a to 0x01a29c (0xfc73 bytes) 000675 1788484277.804746 punch from 0x013793 to 0x016583 (0x2df1 bytes) 000676 1788484277.809827 mapwrite 0x00e341 thru 0x01b478 (0xd138 bytes) 000678 1788484278.018783 mapwrite 0x01e9ec thru 0x0239c2 (0x4fd7 bytes) 000679 1788484278.149922 trunc from 0x019595 to 0x0225da (0x9046 bytes) 000680 1788484278.161693 mapread 0x00deab thru 0x01db5e (0xfcb4 bytes) 000681 1788484278.217998 trunc from 0x023775 to 0x0239c2 (0x24e bytes) 000683 1788484278.228399 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x239c3): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x239c3): Mirror 15: 0x71a95370 (stale) Mirror 16: 0xbe18238d (stale) Mirror 17: 0x794501ab (stale) CRC-32 checksum value for chunk [0, 0x239c3): Mirror 1: 0x3dbf89c6 Mirror 18: 0x3dbf89c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x239c3: skipped 000684 1788484279.481971 fallocate 0x03dc09 thru 0x03ffff (0x23f7 bytes) 000685 1788484279.550655 trunc from 0x001a40 to 0x00fb51 (0xe112 bytes) 000686 1788484279.556768 fallocate 0x02a2cf thru 0x02c17a (0x1eac bytes) 000687 1788484279.562053 mapwrite 0x025fed thru 0x035fc6 (0xffda bytes) 000688 1788484279.736735 mapwrite 0x019bf6 thru 0x026bfa (0xd005 bytes) 000689 1788484279.772014 write 0x0164ae thru 0x01a8c2 (0x4415 bytes) 000690 1788484279.773757 trunc from 0x035fc7 to 0x00928b (0x2cd3b bytes) 000691 1788484279.805209 write 0x00bcd3 thru 0x018255 (0xc583 bytes) 000693 1788484279.820385 read 0x00c0bb thru 0x018255 (0xc19b bytes) 000694 1788484279.821424 mapread 0x006243 thru 0x00a1e2 (0x3fa0 bytes) 000695 1788484279.837009 fallocate 0x011844 thru 0x01e6b2 (0xce6f bytes) 000696 1788484279.842855 trunc from 0x01e6b3 to 0x00e975 (0xfd3d bytes) 000697 1788484279.866153 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe976): Mirror 16: 0x119896c4 (stale) Mirror 17: 0x3b4f3744 (stale) Mirror 18: 0x6b3a229e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe976): Mirror 16: 0x119896c4 (stale) Mirror 17: 0x3b4f3744 (stale) Mirror 18: 0x6b3a229e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe976: skipped 000698 1788484280.308630 mapwrite 0x01125e thru 0x01a584 (0x9327 bytes) 000699 1788484280.370681 trunc from 0x010401 to 0x011f87 (0x1b87 bytes) 000700 1788484280.377192 punch from 0x0127f5 to 0x017e5c (0x5668 bytes) 000702 1788484280.388204 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a585): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a585): Mirror 16: 0xc83678a5 (stale) Mirror 17: 0x6af8daf (stale) Mirror 18: 0xbd4ba177 (stale) CRC-32 checksum value for chunk [0, 0x1a585): Mirror 1: 0x2f2a8924 Mirror 19: 0x2f2a8924 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a585: skipped 000703 1788484281.571101 trunc from 0x00827f to 0x014918 (0xc69a bytes) 000704 1788484281.621608 fallocate 0x0208f2 thru 0x02be3b (0xb54a bytes) 000705 1788484281.628350 write 0x01d30c thru 0x029ef0 (0xcbe5 bytes) 000706 1788484281.640442 mapwrite 0x019749 thru 0x02416c (0xaa24 bytes) 000707 1788484281.689785 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29ef1): Mirror 16: 0xfd6f6b1b (stale) Mirror 17: 0x111397be (stale) Mirror 19: 0x2ab2adbf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ef1): Mirror 16: 0xfd6f6b1b (stale) Mirror 17: 0x111397be (stale) Mirror 19: 0x2ab2adbf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ef1: skipped 000708 1788484282.218966 fallocate 0x0384f8 thru 0x03ffff (0x7b08 bytes) 000709 1788484282.226713 write 0x01b986 thru 0x02200a (0x6685 bytes) 000710 1788484282.241887 fallocate 0x00d35b thru 0x0115a4 (0x424a bytes) 000711 1788484282.247702 write 0x0251ec thru 0x02df6a (0x8d7f bytes) 000712 1788484282.258588 read 0x0110eb thru 0x02092f (0xf845 bytes) 000713 1788484282.268036 trunc from 0x040000 to 0x02aae5 (0x1551a bytes) 000714 1788484282.310697 read 0x0271ee thru 0x0288cd (0x16e0 bytes) 000715 1788484282.311312 write 0x03dc3e thru 0x03ffff (0x23c2 bytes) 000717 1788484282.312764 mapwrite 0x0216bd thru 0x023252 (0x1b96 bytes) 000719 1788484282.357524 read 0x03df62 thru 0x03ffff (0x209e bytes) 000720 1788484282.366417 MIRROR_RESYNC 4 mirrors 000721 1788484284.049233 trunc from 0x040000 to 0x033b17 (0xc4e8 bytes) 000723 1788484284.140598 mapread 0x032c9e thru 0x033b17 (0xe7a bytes) 000724 1788484284.186896 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33b18): Mirror 16: 0x20aef380 (stale) Mirror 17: 0x20aef380 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b18): Mirror 16: 0x20aef380 (stale) Mirror 17: 0x20aef380 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b18: skipped 000725 1788484284.589850 mapread 0x0163a7 thru 0x01706b (0xcc5 bytes) 000726 1788484284.605151 mapread 0x0035c9 thru 0x010de7 (0xd81f bytes) 000727 1788484284.623874 trunc from 0x0108a0 to 0x017243 (0x69a4 bytes) 000729 1788484284.634787 punch from 0x00830e to 0x00b3b5 (0x30a8 bytes) 000730 1788484284.640873 trunc from 0x033b18 to 0x03faf3 (0xbfdc bytes) 000731 1788484284.672956 trunc from 0x00d055 to 0x01471e (0x76ca bytes) 000732 1788484284.689707 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3faf4): Mirror 16: 0xf990751a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3faf4): Mirror 16: 0xf990751a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3faf4: skipped 000733 1788484285.082130 fallocate 0x01f7e7 thru 0x023361 (0x3b7b bytes) 000734 1788484285.106788 trunc from 0x03faf4 to 0x038878 (0x727b bytes) 000735 1788484285.154185 punch from 0x01dc30 to 0x025dad (0x817e bytes) 000736 1788484285.165417 trunc from 0x038879 to 0x022e0e (0x15a6a bytes) 000737 1788484285.198081 fallocate 0x038fe9 thru 0x03ffff (0x7017 bytes) 000738 1788484285.252872 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22e0f): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e0f): Mirror 16: 0xe5a44f60 (stale) CRC-32 checksum value for chunk [0, 0x22e0f): Mirror 1: 0xea0ca16b Mirror 17: 0xea0ca16b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e0f: skipped 000739 1788484286.306984 fallocate 0x02a6d0 thru 0x032c43 (0x8574 bytes) 000740 1788484286.337656 trunc from 0x032c44 to 0x0213c1 (0x11882 bytes) 000741 1788484286.388964 fallocate 0x00d630 thru 0x015024 (0x79f5 bytes) 000742 1788484286.401722 mapread 0x00a3eb thru 0x00d6c4 (0x32da bytes) 000743 1788484286.452447 mapwrite 0x029a64 thru 0x035168 (0xb705 bytes) 000744 1788484286.587248 trunc from 0x02e9e5 to 0x035168 (0x6784 bytes) 000745 1788484286.596689 mapwrite 0x036f48 thru 0x03f648 (0x8701 bytes) 000746 1788484286.674698 trunc from 0x03f649 to 0x0048ce (0x3ad7a bytes) 000747 1788484286.701263 write 0x004611 thru 0x01437d (0xfd6d bytes) 000750 1788484286.718229 write 0x001c5b thru 0x00e6b9 (0xca5f bytes) 000751 1788484286.729205 mapread 0x000eea thru 0x002076 (0x118d bytes) 000752 1788484286.747158 trunc from 0x01437e to 0x010fab (0x33d2 bytes) 000753 1788484286.775767 read 0x00b242 thru 0x010fab (0x5d6a bytes) 000754 1788484286.776709 fallocate 0x02dfe8 thru 0x037ecf (0x9ee8 bytes) 000756 1788484286.790830 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10fac): Mirror 16: 0xad63f729 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10fac): Mirror 16: 0xad63f729 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10fac: skipped 000757 1788484287.245933 fallocate 0x02eabe thru 0x034348 (0x588b bytes) 000758 1788484287.254809 trunc from 0x0262d3 to 0x02ab38 (0x4866 bytes) 000759 1788484287.267101 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34349): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34349): Mirror 16: 0xda382a8 (stale) CRC-32 checksum value for chunk [0, 0x34349): Mirror 1: 0x4b703eca Mirror 17: 0x4b703eca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34349: skipped 000760 1788484288.422771 mapwrite 0x026f76 thru 0x0346df (0xd76a bytes) 000762 1788484288.565270 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x346e0): Mirror 17: 0x92079122 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346e0): Mirror 17: 0x92079122 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346e0: skipped 000763 1788484288.884847 mapread 0x02c380 thru 0x0346df (0x8360 bytes) 000764 1788484288.930342 trunc from 0x0346e0 to 0x0057b1 (0x2ef2e bytes) 000765 1788484288.963991 punch from 0x001b8f to 0x001dd0 (0x242 bytes) 000766 1788484288.971090 read 0x00317a thru 0x0057b1 (0x2638 bytes) 000767 1788484288.978002 punch from 0x000a19 to 0x0057b1 (0x4d99 bytes) 000768 1788484288.991876 punch from 0x0038b5 to 0x003cfa (0x446 bytes) 000769 1788484289.017599 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) 000770 1788484289.323274 mapread 0x003024 thru 0x0057b1 (0x278e bytes) 000771 1788484289.370785 mapread 0x00358b thru 0x0057b1 (0x2227 bytes) 000773 1788484289.372143 punch from 0x001b8a to 0x0057b1 (0x3c28 bytes) 000774 1788484289.390686 write 0x02bbb3 thru 0x02e59f (0x29ed bytes) 000776 1788484289.407218 fallocate 0x00b286 thru 0x01218c (0x6f07 bytes) 000777 1788484289.414188 trunc from 0x02e5a0 to 0x008991 (0x25c0e bytes) 000778 1788484289.448213 read 0x007f57 thru 0x008991 (0xa3b bytes) 000779 1788484289.457202 write 0x01ebdd thru 0x02e81f (0xfc43 bytes) 000780 1788484289.459350 write 0x02b78c thru 0x038ffb (0xd870 bytes) 000781 1788484289.463048 mapread 0x01e5aa thru 0x02b0fc (0xcb53 bytes) 000782 1788484289.467132 trunc from 0x038ffc to 0x01fea9 (0x19152 bytes) 000783 1788484289.518150 mapread 0x01c150 thru 0x01fea9 (0x3d5a bytes) 000784 1788484289.544685 mapread 0x01a7f1 thru 0x01fea9 (0x56b9 bytes) 000785 1788484289.545533 punch from 0x00fc71 to 0x01dc39 (0xdfc9 bytes) 000786 1788484289.558483 trunc from 0x01a171 to 0x01cb46 (0x29d6 bytes) 000788 1788484289.572153 mapread 0x00e764 thru 0x01c914 (0xe1b1 bytes) 000790 1788484289.613010 trunc from 0x0006b5 to 0x00f289 (0xebd5 bytes) 000792 1788484289.627103 punch from 0x019c55 to 0x01a0bb (0x467 bytes) 000794 1788484289.657527 trunc from 0x01feaa to 0x0253ca (0x5521 bytes) 000795 1788484289.706719 punch from 0x001ab5 to 0x00cf76 (0xb4c2 bytes) 000796 1788484289.722894 punch from 0x0028ed to 0x008e0d (0x6521 bytes) 000797 1788484289.732680 trunc from 0x0253cb to 0x03a099 (0x14ccf bytes) 000798 1788484289.768361 mapwrite 0x01c38a thru 0x02682c (0xa4a3 bytes) 000799 1788484289.848129 read 0x012674 thru 0x0126fa (0x87 bytes) 000800 1788484289.853066 trunc from 0x026ed1 to 0x03665a (0xf78a bytes) 000801 1788484289.882445 read 0x015edb thru 0x024efc (0xf022 bytes) 000802 1788484289.883222 read 0x01567a thru 0x020208 (0xab8f bytes) 000803 1788484289.892013 trunc from 0x00125c to 0x00435f (0x3104 bytes) 000805 1788484289.899200 punch from 0x0250b0 to 0x02ca09 (0x795a bytes) 000806 1788484289.922191 punch from 0x027229 to 0x028822 (0x15fa bytes) 000807 1788484289.930984 write 0x006dc3 thru 0x00c66f (0x58ad bytes) 000808 1788484289.951346 trunc from 0x0188c3 to 0x027e49 (0xf587 bytes) 000809 1788484289.966885 mapwrite 0x01fc76 thru 0x02cd3e (0xd0c9 bytes) 000810 1788484290.177188 punch from 0x002452 to 0x005d6c (0x391b bytes) 000812 1788484290.187307 mapread 0x02d698 thru 0x037a0e (0xa377 bytes) 000813 1788484290.193141 MIRROR_EXTEND to 2 mirrors 000814 1788484291.128399 write 0x023614 thru 0x02fee4 (0xc8d1 bytes) 000815 1788484291.236729 mapread 0x028215 thru 0x033b07 (0xb8f3 bytes) 000816 1788484291.269959 trunc from 0x03a09a to 0x00fb9f (0x2a4fa bytes) 000817 1788484291.348030 write 0x0296d7 thru 0x02efca (0x58f4 bytes) 000819 1788484291.354050 trunc from 0x02efcb to 0x008cb3 (0x26317 bytes) 000820 1788484291.389168 punch from 0x002456 to 0x008cb3 (0x685e bytes) 000821 1788484291.398280 write 0x03bb1f thru 0x03ffff (0x44e1 bytes) 000822 1788484291.409678 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xefb9e456 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x283f058e Mirror 3: 0x283f058e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000824 1788484293.010576 read 0x026e39 thru 0x034a8d (0xdc55 bytes) 000826 1788484293.045341 fallocate 0x03f5f6 thru 0x03ffff (0xa0a bytes) 000827 1788484293.082047 punch from 0x01eec8 to 0x0249fb (0x5b34 bytes) 000828 1788484293.089078 fallocate 0x03c4f7 thru 0x03ffff (0x3b09 bytes) 000829 1788484293.094608 read 0x034c17 thru 0x03952b (0x4915 bytes) 000830 1788484293.116900 write 0x0301e2 thru 0x031233 (0x1052 bytes) 000831 1788484293.122595 fallocate 0x013cfd thru 0x01b983 (0x7c87 bytes) 000832 1788484293.129206 write 0x00bc0d thru 0x019550 (0xd944 bytes) 000833 1788484293.130822 trunc from 0x040000 to 0x00291a (0x3d6e5 bytes) 000834 1788484293.185908 punch from 0x001141 to 0x001207 (0xc7 bytes) 000835 1788484293.195818 punch from 0x001ad6 to 0x00291a (0xe45 bytes) 000836 1788484293.207131 mapwrite 0x0200d1 thru 0x02887f (0x87af bytes) 000837 1788484293.382619 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28880): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28880): Mirror 2: 0xff479427 (stale) Mirror 3: 0x2b196562 (stale) CRC-32 checksum value for chunk [0, 0x28880): Mirror 1: 0xe913826 Mirror 4: 0xe913826 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28880: skipped 000839 1788484294.658059 trunc from 0x028880 to 0x03d576 (0x14cf7 bytes) 000840 1788484294.717219 read 0x0201d1 thru 0x026af4 (0x6924 bytes) 000841 1788484294.731254 mapwrite 0x029761 thru 0x03192a (0x81ca bytes) 000842 1788484294.783554 punch from 0x030280 to 0x037c13 (0x7994 bytes) 000843 1788484294.795245 mapwrite 0x02dd1c thru 0x039c39 (0xbf1e bytes) 000845 1788484294.842064 mapwrite 0x0365df thru 0x03fe49 (0x986b bytes) 000846 1788484294.935701 trunc from 0x03d0c2 to 0x03fe49 (0x2d88 bytes) 000847 1788484294.973205 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fe4a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe4a): Mirror 2: 0xf3324948 (stale) Mirror 3: 0x2c78ce92 (stale) Mirror 4: 0xb070d4bd (stale) CRC-32 checksum value for chunk [0, 0x3fe4a): Mirror 1: 0x690eb23f Mirror 5: 0x690eb23f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe4a: skipped 000848 1788484296.056676 trunc from 0x02d7dc to 0x030e9e (0x36c3 bytes) 000849 1788484296.150751 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe4a): Mirror 3: 0x2c78ce92 (stale) Mirror 4: 0xb070d4bd (stale) Mirror 5: 0x690eb23f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe4a): Mirror 3: 0x2c78ce92 (stale) Mirror 4: 0xb070d4bd (stale) Mirror 5: 0x690eb23f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe4a: skipped 000850 1788484296.729889 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe4a): Mirror 3: 0x2c78ce92 (stale) Mirror 4: 0xb070d4bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe4a): Mirror 3: 0x2c78ce92 (stale) Mirror 4: 0xb070d4bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe4a: skipped 000852 1788484297.256880 fallocate 0x0072c3 thru 0x00de1c (0x6b5a bytes) 000853 1788484297.269140 write 0x014366 thru 0x01ab2c (0x67c7 bytes) 000854 1788484297.293078 trunc from 0x03fe4a to 0x01614b (0x29cfe bytes) 000855 1788484297.330561 punch from 0x00a6a7 to 0x01614b (0xbaa5 bytes) 000856 1788484297.356053 mapread 0x0055b3 thru 0x00e184 (0x8bd2 bytes) 000857 1788484297.398105 write 0x019357 thru 0x01c7ff (0x34a9 bytes) 000858 1788484297.400952 punch from 0x005311 to 0x010919 (0xb609 bytes) 000859 1788484297.410962 trunc from 0x00870b to 0x00f95b (0x7251 bytes) 000860 1788484297.433708 MIRROR_RESYNC 3 mirrors 000862 1788484299.083867 fallocate 0x0171d4 thru 0x01f586 (0x83b3 bytes) 000863 1788484299.171124 MIRROR_RESYNC 3 mirrors 000864 1788484300.671821 MIRROR_EXTEND to 4 mirrors 000865 1788484301.565984 trunc from 0x01c800 to 0x0144f1 (0x830e bytes) 000866 1788484301.639987 write 0x03f5d3 thru 0x03ffff (0xa2d bytes) 000867 1788484301.642334 punch from 0x030da7 to 0x037669 (0x68c3 bytes) 000868 1788484301.649861 fallocate 0x0203a3 thru 0x02cd12 (0xc970 bytes) 000869 1788484301.657596 punch from 0x010b73 to 0x01b0d6 (0xa564 bytes) 000870 1788484301.666911 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8cc91db5 (stale) Mirror 4: 0x8cc91db5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8cc91db5 (stale) Mirror 4: 0x8cc91db5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000871 1788484302.155892 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8cc91db5 (stale) Mirror 4: 0x8cc91db5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4f62e6e4 Mirror 5: 0x4f62e6e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000872 1788484303.498161 fallocate 0x0369ab thru 0x03ffff (0x9655 bytes) 000874 1788484303.586474 fallocate 0x005d10 thru 0x0109e1 (0xacd2 bytes) 000875 1788484303.606119 read 0x0085ab thru 0x015ffa (0xda50 bytes) 000876 1788484303.620209 fallocate 0x0324a6 thru 0x03e8b5 (0xc410 bytes) 000877 1788484303.641860 trunc from 0x040000 to 0x008107 (0x37ef8 bytes) 000878 1788484303.732025 write 0x006ce8 thru 0x00ccf2 (0x600b bytes) 000879 1788484303.758256 punch from 0x00cac2 to 0x00ccf2 (0x231 bytes) 000880 1788484303.794036 mapread 0x004e72 thru 0x00ccf2 (0x7e81 bytes) 000881 1788484303.836041 trunc from 0x00ccf3 to 0x0287d4 (0x1bae2 bytes) 000882 1788484303.896158 punch from 0x021eb5 to 0x0287d4 (0x6920 bytes) 000883 1788484303.913525 fallocate 0x015df7 thru 0x01ca62 (0x6c6c bytes) 000884 1788484303.930695 mapwrite 0x02d8a8 thru 0x02e457 (0xbb0 bytes) 000885 1788484304.053035 trunc from 0x00972c to 0x017379 (0xdc4e bytes) 000886 1788484304.065064 read 0x01f123 thru 0x023484 (0x4362 bytes) 000887 1788484304.092179 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e458): Mirror 3: 0x34ba841b (stale) Mirror 4: 0x34ba841b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e458): Mirror 3: 0x34ba841b (stale) Mirror 4: 0x34ba841b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e458: skipped 000888 1788484304.758123 fallocate 0x01a65d thru 0x025bc8 (0xb56c bytes) 000889 1788484304.779021 trunc from 0x023c52 to 0x02ddd7 (0xa186 bytes) 000890 1788484304.789325 fallocate 0x002b58 thru 0x006ea5 (0x434e bytes) 000891 1788484304.797383 fallocate 0x0102f6 thru 0x015a93 (0x579e bytes) 000892 1788484304.809370 punch from 0x00f0de to 0x01526f (0x6192 bytes) 000894 1788484304.817088 read 0x0090ef thru 0x012b63 (0x9a75 bytes) 000896 1788484304.833021 read 0x01134e thru 0x012b61 (0x1814 bytes) 000897 1788484304.844694 write 0x0217bf thru 0x02d2dc (0xbb1e bytes) 000898 1788484304.896269 punch from 0x0060c4 to 0x007868 (0x17a5 bytes) 000899 1788484304.907090 trunc from 0x02e458 to 0x025d9e (0x86b9 bytes) 000900 1788484304.938911 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25d9f): Mirror 4: 0x6310eb00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25d9f): Mirror 4: 0x6310eb00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25d9f: skipped 000902 1788484305.352095 write 0x00b46d thru 0x01a6a1 (0xf235 bytes) 000903 1788484305.383410 punch from 0x025cc8 to 0x025d9e (0xd7 bytes) 000904 1788484305.389632 trunc from 0x003aba to 0x0043b3 (0x8fa bytes) 000905 1788484305.407210 mapread 0x01a6f1 thru 0x0218c2 (0x71d2 bytes) 000906 1788484305.448474 punch from 0x0074fd to 0x009588 (0x208c bytes) 000907 1788484305.453821 fallocate 0x0145e7 thru 0x0230cc (0xeae6 bytes) 000908 1788484305.462746 fallocate 0x03a578 thru 0x03f774 (0x51fd bytes) 000909 1788484305.472106 write 0x02b5a3 thru 0x0361cf (0xac2d bytes) 000910 1788484305.474866 mapwrite 0x017048 thru 0x0200e6 (0x909f bytes) 000911 1788484305.529124 mapwrite 0x026699 thru 0x02fbfd (0x9565 bytes) 000912 1788484305.573136 mapread 0x017792 thru 0x0254c6 (0xdd35 bytes) 000913 1788484305.580245 mapwrite 0x01a062 thru 0x022a4a (0x89e9 bytes) 000915 1788484305.621033 mapwrite 0x019cf5 thru 0x029cb4 (0xffc0 bytes) 000916 1788484305.648899 fallocate 0x036572 thru 0x037399 (0xe28 bytes) 000917 1788484305.653825 fallocate 0x01bd77 thru 0x022eb5 (0x713f bytes) 000920 1788484305.659726 write 0x02b74d thru 0x02f01c (0x38d0 bytes) 000921 1788484305.660992 trunc from 0x0361d0 to 0x03fad0 (0x9901 bytes) 000922 1788484305.673312 fallocate 0x02b177 thru 0x02ee88 (0x3d12 bytes) 000923 1788484305.682598 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 4: 0xd56b5ef1 (stale) CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 1: 0xaf940aa Mirror 5: 0xaf940aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fad1: skipped 000924 1788484306.737217 mapread 0x016d65 thru 0x01a1b0 (0x344c bytes) 000925 1788484306.820084 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 4: 0xd56b5ef1 (stale) CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 1: 0xaf940aa Mirror 5: 0xaf940aa Mirror 6: 0xaf940aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fad1: skipped 000926 1788484308.053126 write 0x017519 thru 0x02464f (0xd137 bytes) 000927 1788484308.179860 trunc from 0x00181f to 0x00fdac (0xe58e bytes) 000928 1788484308.187982 fallocate 0x005d24 thru 0x00672c (0xa09 bytes) 000929 1788484308.202957 fallocate 0x0208c0 thru 0x0259f0 (0x5131 bytes) 000930 1788484308.213245 mapwrite 0x030609 thru 0x030973 (0x36b bytes) 000933 1788484308.253417 read 0x029192 thru 0x029e39 (0xca8 bytes) 000934 1788484308.277395 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 4: 0xd56b5ef1 (stale) Mirror 5: 0xaf940aa (stale) Mirror 6: 0xaf940aa (stale) CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 1: 0xe46dc4be Mirror 7: 0xe46dc4be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fad1: skipped 000936 1788484309.429400 punch from 0x02bd2a to 0x031c34 (0x5f0b bytes) 000937 1788484309.515065 mapread 0x007ead thru 0x016f48 (0xf09c bytes) 000938 1788484309.609029 read 0x02e59a thru 0x030f60 (0x29c7 bytes) 000940 1788484309.617481 mapwrite 0x032dc0 thru 0x037420 (0x4661 bytes) 000941 1788484309.675889 punch from 0x02e98e to 0x03dfa2 (0xf615 bytes) 000942 1788484309.686441 trunc from 0x03fad1 to 0x01d6d9 (0x223f7 bytes) 000944 1788484309.768182 mapread 0x0010de thru 0x006b4c (0x5a6f bytes) 000945 1788484309.778947 mapwrite 0x0006d2 thru 0x0009ef (0x31e bytes) 000946 1788484309.808156 mapread 0x008635 thru 0x00a284 (0x1c50 bytes) 000947 1788484309.816932 MIRROR_RESYNC 5 mirrors 000948 1788484311.439032 fallocate 0x01559d thru 0x01debf (0x8923 bytes) 000949 1788484311.486717 trunc from 0x00bbf6 to 0x015215 (0x9620 bytes) 000950 1788484311.492799 mapread 0x015c41 thru 0x01b281 (0x5641 bytes) 000951 1788484311.520079 mapread 0x014033 thru 0x01debf (0x9e8d bytes) 000952 1788484311.524834 mapread 0x018d66 thru 0x019247 (0x4e2 bytes) 000954 1788484311.533681 read 0x01aa19 thru 0x01debf (0x34a7 bytes) 000955 1788484311.534410 punch from 0x00534a to 0x006167 (0xe1e bytes) 000956 1788484311.545631 fallocate 0x03afb6 thru 0x03f690 (0x46db bytes) 000957 1788484311.560577 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1dec0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dec0): Mirror 4: 0xc690825f (stale) Mirror 5: 0xc690825f (stale) Mirror 6: 0xc690825f (stale) Mirror 7: 0xc690825f (stale) CRC-32 checksum value for chunk [0, 0x1dec0): Mirror 1: 0xcfdd8d91 Mirror 8: 0xcfdd8d91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dec0: skipped 000958 1788484312.845454 write 0x02485a thru 0x029420 (0x4bc7 bytes) 000959 1788484312.943067 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x29421): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29421): Mirror 4: 0xfc96ca0b (stale) Mirror 5: 0xfc96ca0b (stale) Mirror 6: 0xfc96ca0b (stale) Mirror 7: 0xfc96ca0b (stale) Mirror 8: 0x4c57ac20 (stale) CRC-32 checksum value for chunk [0, 0x29421): Mirror 1: 0x5c2888c1 Mirror 9: 0x5c2888c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29421: skipped 000961 1788484314.452873 trunc from 0x001bf5 to 0x005cb0 (0x40bc bytes) 000962 1788484314.536951 trunc from 0x029421 to 0x018472 (0x10fae bytes) 000964 1788484314.612147 mapwrite 0x035801 thru 0x039c8a (0x448a bytes) 000965 1788484314.705764 read 0x0333a4 thru 0x039c8a (0x68e7 bytes) 000966 1788484314.706511 trunc from 0x039c8b to 0x0267c6 (0x134c4 bytes) 000967 1788484314.721615 mapread 0x003b30 thru 0x00e581 (0xaa52 bytes) 000968 1788484314.737892 punch from 0x0203e3 to 0x0267c6 (0x63e4 bytes) 000969 1788484314.743694 mapread 0x021d85 thru 0x0261d6 (0x4452 bytes) 000970 1788484314.760983 trunc from 0x0267c7 to 0x031f4c (0xb786 bytes) 000972 1788484314.784618 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31f4d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f4d): Mirror 4: 0xa77743d9 (stale) Mirror 5: 0xa77743d9 (stale) Mirror 6: 0xa77743d9 (stale) Mirror 7: 0xa77743d9 (stale) Mirror 8: 0xd41af9a5 (stale) Mirror 9: 0x7d35a7f7 (stale) CRC-32 checksum value for chunk [0, 0x31f4d): Mirror 1: 0xa67dc408 Mirror 10: 0xa67dc408 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f4d: skipped 000973 1788484315.986020 read 0x00bacd thru 0x012757 (0x6c8b bytes) 000974 1788484316.027240 write 0x01032a thru 0x01a106 (0x9ddd bytes) 000975 1788484316.104896 read 0x02ab28 thru 0x02d873 (0x2d4c bytes) 000976 1788484316.122973 write 0x005cc2 thru 0x00af5e (0x529d bytes) 000977 1788484316.124738 mapread 0x002992 thru 0x003e29 (0x1498 bytes) 000978 1788484316.125979 mapread 0x0038ad thru 0x0109bf (0xd113 bytes) 000979 1788484316.127705 mapread 0x031ab3 thru 0x031f4c (0x49a bytes) 000980 1788484316.129045 trunc from 0x028ed4 to 0x029aa7 (0xbd4 bytes) 000981 1788484316.134086 write 0x03d06f thru 0x03dfb9 (0xf4b bytes) 000982 1788484316.135774 write 0x011b7d thru 0x014ec9 (0x334d bytes) 000983 1788484316.137712 trunc from 0x03dfba to 0x005278 (0x38d41 bytes) 000984 1788484316.161950 mapread 0x000a97 thru 0x005278 (0x47e2 bytes) 000985 1788484316.163107 punch from 0x004d29 to 0x005278 (0x550 bytes) 000986 1788484316.179703 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x5279): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5279): Mirror 4: 0xe6047161 (stale) Mirror 5: 0xe6047161 (stale) Mirror 6: 0xe6047161 (stale) Mirror 7: 0xe6047161 (stale) Mirror 8: 0xe6047161 (stale) Mirror 9: 0xe6047161 (stale) Mirror 10: 0xe6047161 (stale) CRC-32 checksum value for chunk [0, 0x5279): Mirror 1: 0xe6047161 Mirror 11: 0xe6047161 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5279: skipped 000987 1788484317.275786 trunc from 0x000bb9 to 0x005278 (0x46c0 bytes) 000989 1788484317.358497 mapwrite 0x0173f6 thru 0x019adf (0x26ea bytes) 000990 1788484317.567984 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x19ae0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19ae0): Mirror 4: 0x2f4dd5d8 (stale) Mirror 5: 0x2f4dd5d8 (stale) Mirror 6: 0x2f4dd5d8 (stale) Mirror 7: 0x2f4dd5d8 (stale) Mirror 8: 0xe7eb4aa7 (stale) Mirror 9: 0xe7eb4aa7 (stale) Mirror 10: 0x14ef5154 (stale) Mirror 11: 0xeb4565a4 (stale) CRC-32 checksum value for chunk [0, 0x19ae0): Mirror 1: 0xc0eda780 Mirror 12: 0xc0eda780 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19ae0: skipped 000991 1788484318.603112 punch from 0x01387f to 0x01895a (0x50dc bytes) 000992 1788484318.700144 MIRROR_RESYNC 10 mirrors 000993 1788484321.117811 mapwrite 0x030670 thru 0x034101 (0x3a92 bytes) 000994 1788484321.305488 fallocate 0x009988 thru 0x0176cf (0xdd48 bytes) 000995 1788484321.310811 trunc from 0x027bf6 to 0x031d00 (0xa10b bytes) 000996 1788484321.322396 read 0x0209da thru 0x03044a (0xfa71 bytes) 000997 1788484321.341619 punch from 0x021be0 to 0x02aed8 (0x92f9 bytes) 000998 1788484321.347828 trunc from 0x029974 to 0x034101 (0xa78e bytes) 001000 1788484321.363176 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34102): Mirror 4: 0x49e35a16 (stale) Mirror 5: 0x49e35a16 (stale) Mirror 6: 0x49e35a16 (stale) Mirror 7: 0x49e35a16 (stale) Mirror 8: 0x49e35a16 (stale) Mirror 9: 0x49e35a16 (stale) Mirror 10: 0x49e35a16 (stale) Mirror 11: 0x49e35a16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34102): Mirror 4: 0x49e35a16 (stale) Mirror 5: 0x49e35a16 (stale) Mirror 6: 0x49e35a16 (stale) Mirror 7: 0x49e35a16 (stale) Mirror 8: 0x49e35a16 (stale) Mirror 9: 0x49e35a16 (stale) Mirror 10: 0x49e35a16 (stale) Mirror 11: 0x49e35a16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34102: skipped All operations completed A-OK! PASS 70a (134s) == sanity-flr test 71: check mirror extend parent fid ==== 21:12:09 (1788484329) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x52:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 21:12:22 (1788484342) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19421 s, 10.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150793 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 21:12:32 (1788484352) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 721376 .. Extending file size to 5193408 .. Extending file size to 7540000 .. Extending file size to 5532480 .. Extending file size to 8516960 .. Extending file size to 8512672 .. Extending file size to 8564768 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8610144 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8884960 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8856064 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61360 61362 61364 61365 61366 done 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) 192.168.203.145@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:13:58 (1788484438) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1429856 .. Extending file size to 7912256 .. Extending file size to 2035552 .. Extending file size to 2255328 .. Extending file size to 7926624 .. Extending file size to 4344512 .. Extending file size to 5915328 .. Extending file size to 7307840 .. Extending file size to 8360704 .. Extending file size to 8551008 .. Extending file size to 9181440 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9232000 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8598848 .. Waiting 62824 62825 62829 62830 62831 done 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) 192.168.203.145@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:15:24 (1788484524) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17852 3589168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25884 3581136 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43736 7170304 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17852 3589168 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25884 3581136 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43736 7170304 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0775286 s, 27.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257121 s, 8.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1c32d335 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:15:49 (1788484549) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0907991 s, 23.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 21:15:57 (1788484557) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:16:04 (1788484564) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100811 s, 20.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:16:12 (1788484572) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.091783 s, 22.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:16:20 (1788484580) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141186 s, 14.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:16:54 (1788484614) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11955 s, 8.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:17:42 (1788484662) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121398 s, 17.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:18:40 (1788484720) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg345-server: Pool lustre.test_204f created Adding targets to pool oleg345-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg345-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172927 s, 12.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg345-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg345-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg345-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:19:59 (1788484799) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:20:08 (1788484808) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:20:19 (1788484819) oleg345-server: Pool lustre.test_206 created Adding targets to pool oleg345-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg345-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg345-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg345-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg345-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:20:49 (1788484849) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118952 s, 8.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869619 s, 12.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x783ec218 Mirror 3: 0x783ec218 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:21:03 (1788484863) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:21:10 (1788484870) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:21:15 (1788484875) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0834934 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1bf0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2230000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2170000 Read this (4 bytes): BBBB 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) PASS 209a (46s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:22:01 (1788484921) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.015393 s, 532 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:22:09 (1788484929) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493679 s, 21.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611268 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 21:22:11.000000000 -0400 Modify: 2026-09-03 21:22:11.000000000 -0400 Change: 2026-09-03 21:22:13.000000000 -0400 Birth: 2026-09-03 21:22:11.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:22:19 (1788484939) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106502 s, 9.8 MB/s oleg345-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg345-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg345-server: O/280000401/d31/13f: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:22:44 (1788484964) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.347938 s, 118 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0249793 s, 164 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 3 21:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x89:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 3 21:22 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:22:53 (1788484973) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:23:02 (1788484982) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1883 sec =========== 21:23:12 (1788484992) === sanity-flr: start cleanup 21:23:14 (1788484994) === === sanity-flr: finish cleanup 21:23:25 (1788485005) ===