-----============= acceptance-small: sanity-flr ============----- Mon Mar 2 11:40:56 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg315-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 11:41:20 (1772469680) === oleg315-client.virtnet: executing check_config_client /mnt/lustre oleg315-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg315-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c91c70b0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c91c70b0000.idle_timeout=debug disable quota as required oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 11:42:02 (1772469722) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8236] == sanity-flr test 0a: lfs mirror create with -N option == 11:42:04 (1772469724) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:42:26 (1772469746) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:42:30 (1772469750) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:42:34 (1772469754) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:43:00 (1772469780) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:43:04 (1772469784) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 11:43:09 (1772469789) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c91c70b0000.stats=clear osc.lustre-OST0001-osc-ffff9c91c70b0000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:43:18 (1772469798) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:43:30 (1772469810) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:43:39 (1772469819) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:43:43 (1772469823) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (13s) == sanity-flr test 2: create components from existing files ========================================================== 11:43:56 (1772469836) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:44:05 (1772469845) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:44:14 (1772469854) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:44:22 (1772469862) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:44:34 (1772469874) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:44:41 (1772469881) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.39858 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118329 s, 8.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:44:51 (1772469891) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.156727 s, 26.8 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:45:01 (1772469901) Stopping /mnt/lustre-ost1 (opts:) on oleg315-server oleg315-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg315-server: oleg315-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg315-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg315-server oleg315-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg315-server: oleg315-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg315-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:46:18 (1772469978) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.709052 s, 14.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg315-server oleg315-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg315-server: oleg315-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg315-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:47:02 (1772470022) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg315-server oleg315-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg315-server: oleg315-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg315-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg315-server oleg315-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg315-server: oleg315-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg315-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:48:30 (1772470110) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.90881 s, 13.4 MB/s -rw-r--r-- 1 root root 52428800 Mar 2 11:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.220s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg315-server oleg315-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c91c70b0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.464s user 0m0.001s sys 0m0.168s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg315-server: oleg315-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg315-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg315-server oleg315-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.377s user 0m0.000s sys 0m0.183s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg315-server: oleg315-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg315-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c91c70b0000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:50:10 (1772470210) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:50:15 (1772470215) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:50:20 (1772470220) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 11:50:24 (1772470224) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 11:50:31 (1772470231) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.595082 s, 21.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:50:45 (1772470245) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221274 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305082 s, 34.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:50:52 (1772470252) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:51:01 (1772470261) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163246 s, 627 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00069636 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00264592 s, 387 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00229142 s, 447 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 11:51:13 (1772470273) create test files with size 15748436 .. e9d7845cf7cbaa358711d85847182d6c - a61ad95ceba8c306faac960ceae74d4b - 8d6695de9939825c5b2dbb524d17c8a1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 11:51:45 (1772470305) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:52:00 (1772470320) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:52:08 (1772470328) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120829 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445178 s, 23.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 11:52:17 (1772470337) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158346 s, 13.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.54362 s, 7.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240264 s, 8.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 11:52:37 (1772470357) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 11:52:41 (1772470361) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:52:46 (1772470366) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462597 s, 22.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 11:52:55 (1772470375) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.410523 s, 7.7 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:53:07 (1772470387) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:53:19 (1772470399) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.403012 s, 26.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:53:33 (1772470413) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9c91c70de800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.482114 s, 21.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9c91c3b42800.xattr_cache=1 PASS 44e (22s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:53:54 (1772470434) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:54:42 (1772470482) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:55:23 (1772470523) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 11:55:26 (1772470526) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.389531 s, 8.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.198575 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbb2f70f5 (stale) Mirror 2: 0xbb2f70f5 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 2e293d35328feb0f7e45471d535364a1 - mirror 1 checksum: bf5a623b963749ff757f8157bf8845c5 - mirror 2 checksum: 2e293d35328feb0f7e45471d535364a1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 2 11:55:30 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 2 11:55:30 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2e293d35328feb0f7e45471d535364a1 - mirror 1 checksum: bf5a623b963749ff757f8157bf8845c5 - mirror 2 checksum: bf5a623b963749ff757f8157bf8845c5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 2 11:55:30 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 2 11:55:30 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 11:55:39 (1772470539) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536864 s, 19.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:55:47 (1772470547) Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 11:55:59 (1772470559) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 11:56:35 (1772470595) targets are mounted 11:56:35 (1772470595) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605515 s, 17.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:56:53 (1772470613) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0805443 s, 254 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0232122 s, 882 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:57:06 (1772470626) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0486811 s, 421 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0106549 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:57:16 (1772470636) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00882342 s, 1.9 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:57:26 (1772470646) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0875569 s, 234 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:57:34 (1772470654) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0208044 s, 197 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0343595 s, 119 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:57:42 (1772470662) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:58:04 (1772470684) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:58:36 (1772470716) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:59:01 (1772470741) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:59:25 (1772470765) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 11:59:33 (1772470773) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55982 mirror split pid 55984 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55982 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55984 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:00:41 (1772470841) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 844 000002 1772470844.233719 mapwrite 0x00b239 thru 0x00c46b (0x1233 bytes) 000003 1772470844.309710 trunc from 0x00120d to 0x007726 (0x651a bytes) 000007 1772470844.326530 read 0x001304 thru 0x00b917 (0xa614 bytes) 000008 1772470844.338293 trunc from 0x009824 to 0x00c46b (0x2c48 bytes) 000009 1772470844.343434 mapwrite 0x013130 thru 0x015e94 (0x2d65 bytes) 000010 1772470844.408377 mapread 0x0117dd thru 0x015e94 (0x46b8 bytes) truncating to largest ever: 0x26b64 000011 1772470844.410730 trunc from 0x015e95 to 0x026b63 (0x10ccf bytes) 000012 1772470844.433240 mapread 0x01e6f0 thru 0x023d6f (0x5680 bytes) 000013 1772470844.455618 trunc from 0x001019 to 0x00dd4b (0xcd33 bytes) 000014 1772470844.460949 mapwrite 0x022e67 thru 0x032b53 (0xfced bytes) 000015 1772470844.545039 mapwrite 0x005642 thru 0x0155f6 (0xffb5 bytes) 000016 1772470844.579681 trunc from 0x032b54 to 0x009022 (0x29b31 bytes) fallocating to largest ever: 0x40000 000017 1772470844.599250 fallocate 0x03d3ab thru 0x03ffff (0x2c55 bytes) 000018 1772470844.609732 write 0x03a007 thru 0x03ed74 (0x4d6e bytes) 000019 1772470844.611290 mapwrite 0x030849 thru 0x035193 (0x494b bytes) 000020 1772470844.650295 mapread 0x026827 thru 0x030e5d (0xa637 bytes) 000021 1772470844.661931 trunc from 0x03ed75 to 0x0039fc (0x3b378 bytes) 000024 1772470844.683644 read 0x0038b4 thru 0x0039fc (0x149 bytes) 000025 1772470844.684854 mapwrite 0x006339 thru 0x011488 (0xb150 bytes) 000026 1772470844.733606 write 0x03660d thru 0x03ffff (0x99f3 bytes) 000030 1772470844.739879 trunc from 0x040000 to 0x02560d (0x1a9f2 bytes) 000031 1772470844.761740 read 0x01d9f0 thru 0x02560d (0x7c1e bytes) 000032 1772470844.776738 MIRROR_EXTEND to 2 mirrors 000033 1772470845.534303 read 0x0061c6 thru 0x00bc8e (0x5ac9 bytes) 000034 1772470845.586252 trunc from 0x02560e to 0x0174c4 (0xe149 bytes) 000035 1772470845.650820 mapread 0x0023d2 thru 0x00ea0e (0xc63d bytes) 000036 1772470845.689231 trunc from 0x0174c5 to 0x03cc4a (0x25786 bytes) 000037 1772470845.730752 mapread 0x03c9b2 thru 0x03cc4a (0x299 bytes) 000038 1772470845.741230 trunc from 0x03cc4b to 0x018ff3 (0x23c57 bytes) 000039 1772470845.759835 write 0x006a3a thru 0x0134f5 (0xcabc bytes) 000040 1772470845.778850 write 0x03bf1c thru 0x03ffff (0x40e4 bytes) 000041 1772470845.780184 mapread 0x00f2d1 thru 0x016d22 (0x7a52 bytes) 000042 1772470845.802701 write 0x038b55 thru 0x03ac83 (0x212f bytes) 000043 1772470845.815103 trunc from 0x024da0 to 0x02fabc (0xad1d bytes) 000044 1772470845.821171 fallocate 0x03e01e thru 0x03ffff (0x1fe2 bytes) 000045 1772470845.827079 write 0x000cf8 thru 0x00701a (0x6323 bytes) 000046 1772470845.837884 write 0x002ca8 thru 0x01150f (0xe868 bytes) 000047 1772470845.860825 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000048 1772470846.057824 fallocate 0x0234bd thru 0x028380 (0x4ec4 bytes) 000049 1772470846.064434 read 0x001c46 thru 0x00dbf1 (0xbfac bytes) 000050 1772470846.077636 write 0x00f8ab thru 0x0153e6 (0x5b3c bytes) 000051 1772470846.092352 trunc from 0x033dfc to 0x03ffff (0xc204 bytes) 000052 1772470846.098431 read 0x036d2c thru 0x03ffff (0x92d4 bytes) 000053 1772470846.129737 read 0x027ad4 thru 0x035325 (0xd852 bytes) 000054 1772470846.130714 read 0x035320 thru 0x03ffff (0xace0 bytes) 000055 1772470846.132013 mapread 0x011cfd thru 0x0162d7 (0x45db bytes) 000056 1772470846.133405 trunc from 0x003d9c to 0x012037 (0xe29c bytes) 000057 1772470846.150934 write 0x0355d9 thru 0x03ffff (0xaa27 bytes) 000058 1772470846.152443 mapread 0x0146c0 thru 0x01af2f (0x6870 bytes) 000059 1772470846.154353 punch from 0x02eb53 to 0x032139 (0x35e7 bytes) 000060 1772470846.159743 fallocate 0x0255ca thru 0x026bc8 (0x15ff bytes) 000061 1772470846.165386 trunc from 0x040000 to 0x021c2d (0x1e3d2 bytes) 000062 1772470846.185134 trunc from 0x021c2e to 0x0088ff (0x1932e bytes) 000063 1772470846.204165 MIRROR_EXTEND to 2 mirrors 000064 1772470846.935173 mapwrite 0x032721 thru 0x037109 (0x49e9 bytes) 000065 1772470847.136111 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3710a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3710a): Mirror 2: 0x3549ff3d (stale) CRC-32 checksum value for chunk [0, 0x3710a): Mirror 1: 0xb6319185 Mirror 3: 0xb6319185 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3710a: skipped 000066 1772470848.073284 mapwrite 0x02369b thru 0x032777 (0xf0dd bytes) 000067 1772470848.334925 trunc from 0x03710a to 0x036c00 (0x509 bytes) 000068 1772470848.409176 MIRROR_RESYNC 3 mirrors 000069 1772470849.211094 mapwrite 0x009c35 thru 0x00cce2 (0x30ae bytes) 000070 1772470849.303294 MIRROR_RESYNC 3 mirrors 000071 1772470850.334608 trunc from 0x032e74 to 0x036371 (0x34fe bytes) 000072 1772470850.394350 read 0x025c83 thru 0x02c55e (0x68dc bytes) 000073 1772470850.408424 mapread 0x025141 thru 0x025918 (0x7d8 bytes) 000074 1772470850.409471 read 0x00d8c8 thru 0x0151bb (0x78f4 bytes) 000075 1772470850.441985 punch from 0x01785a to 0x021c69 (0xa410 bytes) 000076 1772470850.450817 punch from 0x01f87a to 0x02c584 (0xcd0b bytes) 000077 1772470850.463140 write 0x00bd37 thru 0x00f1a3 (0x346d bytes) 000078 1772470850.469621 mapread 0x001b7b thru 0x007acd (0x5f53 bytes) 000079 1772470850.477182 read 0x0209ed thru 0x02b458 (0xaa6c bytes) 000080 1772470850.495534 read 0x022015 thru 0x0268d1 (0x48bd bytes) 000081 1772470850.497945 punch from 0x01d41e to 0x023ff8 (0x6bdb bytes) 000082 1772470850.505072 fallocate 0x003c50 thru 0x00f374 (0xb725 bytes) 000083 1772470850.526133 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36c01): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c01): Mirror 2: 0xf612d575 (stale) Mirror 3: 0xf612d575 (stale) CRC-32 checksum value for chunk [0, 0x36c01): Mirror 1: 0x343b8bfc Mirror 4: 0x343b8bfc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c01: skipped 000084 1772470851.742575 fallocate 0x03bafd thru 0x03ffff (0x4503 bytes) 000085 1772470851.785882 trunc from 0x036c01 to 0x023cc5 (0x12f3b bytes) 000086 1772470851.812572 write 0x00f882 thru 0x01d978 (0xe0f7 bytes) 000087 1772470851.827683 write 0x03aadc thru 0x03bdfb (0x1320 bytes) 000089 1772470851.829663 punch from 0x00e0c8 to 0x00f7d4 (0x170d bytes) 000090 1772470851.846676 trunc from 0x02f674 to 0x036c59 (0x75e6 bytes) 000091 1772470851.852724 mapwrite 0x006df7 thru 0x010dd0 (0x9fda bytes) 000092 1772470851.887055 write 0x01ea29 thru 0x020fe5 (0x25bd bytes) 000093 1772470851.899765 punch from 0x03107a to 0x032338 (0x12bf bytes) 000095 1772470851.904804 read 0x039370 thru 0x03bdfb (0x2a8c bytes) 000097 1772470851.911748 punch from 0x02292a to 0x02c0fa (0x97d1 bytes) 000099 1772470851.918520 write 0x00ba4b thru 0x0123b9 (0x696f bytes) 000100 1772470851.925834 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bdfc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdfc): Mirror 2: 0x8fe85e98 (stale) Mirror 3: 0x8fe85e98 (stale) Mirror 4: 0x7fb24bf2 (stale) CRC-32 checksum value for chunk [0, 0x3bdfc): Mirror 1: 0x114e836d Mirror 5: 0x114e836d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdfc: skipped 000101 1772470853.119101 mapwrite 0x027471 thru 0x028c5b (0x17eb bytes) 000102 1772470853.237824 mapwrite 0x0385ba thru 0x03ffff (0x7a46 bytes) 000103 1772470853.295118 trunc from 0x00c174 to 0x010155 (0x3fe2 bytes) 000104 1772470853.303915 trunc from 0x0155de to 0x020383 (0xada6 bytes) 000105 1772470853.313712 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfb24741a (stale) Mirror 4: 0xe305005 (stale) Mirror 5: 0xe273ac5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfb24741a (stale) Mirror 4: 0xe305005 (stale) Mirror 5: 0xe273ac5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000106 1772470853.990296 trunc from 0x03fe91 to 0x03ffff (0x16f bytes) 000107 1772470853.997246 read 0x034c7b thru 0x039bd6 (0x4f5c bytes) 000108 1772470854.032903 punch from 0x027ac4 to 0x0346c0 (0xcbfd bytes) 000109 1772470854.038094 fallocate 0x01f0ac thru 0x01f6a9 (0x5fe bytes) 000110 1772470854.046975 punch from 0x00f4fe to 0x017fac (0x8aaf bytes) 000111 1772470854.059658 write 0x033cca thru 0x038f15 (0x524c bytes) 000112 1772470854.079162 fallocate 0x03e11f thru 0x03ffff (0x1ee1 bytes) 000113 1772470854.090085 read 0x016524 thru 0x02563c (0xf119 bytes) 000114 1772470854.115856 mapread 0x002e5e thru 0x00a1cb (0x736e bytes) 000115 1772470854.118836 write 0x014c5b thru 0x01f366 (0xa70c bytes) 000116 1772470854.122965 punch from 0x02fa9e to 0x03f487 (0xf9ea bytes) 000117 1772470854.169502 MIRROR_RESYNC 4 mirrors 000119 1772470855.754670 fallocate 0x0105e1 thru 0x011b7b (0x159b bytes) 000120 1772470855.782849 punch from 0x036c46 to 0x03ffff (0x93ba bytes) 000121 1772470855.789756 read 0x0292e8 thru 0x02b47a (0x2193 bytes) 000122 1772470855.797309 fallocate 0x018405 thru 0x0217c1 (0x93bd bytes) 000123 1772470855.803178 trunc from 0x025790 to 0x02c26c (0x6add bytes) 000124 1772470855.820333 trunc from 0x040000 to 0x02c191 (0x13e6e bytes) 000125 1772470855.838981 fallocate 0x017a04 thru 0x022974 (0xaf71 bytes) 000127 1772470855.846148 fallocate 0x01968c thru 0x01d8c8 (0x423d bytes) 000129 1772470855.852472 fallocate 0x02e13c thru 0x03c05c (0xdf21 bytes) 000130 1772470855.858306 trunc from 0x02c192 to 0x010324 (0x1be6d bytes) 000131 1772470855.881204 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x10325): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10325): Mirror 3: 0xa41f77f2 (stale) Mirror 4: 0xa41f77f2 (stale) Mirror 5: 0xa41f77f2 (stale) CRC-32 checksum value for chunk [0, 0x10325): Mirror 1: 0xa41f77f2 Mirror 6: 0xa41f77f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10325: skipped 000132 1772470856.651251 trunc from 0x010325 to 0x01d642 (0xd31e bytes) 000133 1772470856.717083 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d643): Mirror 4: 0xed263dd4 (stale) Mirror 5: 0xed263dd4 (stale) Mirror 6: 0xcabfc0c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d643): Mirror 4: 0xed263dd4 (stale) Mirror 5: 0xed263dd4 (stale) Mirror 6: 0xcabfc0c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d643: skipped 000134 1772470857.007176 read 0x008c3b thru 0x00c3fd (0x37c3 bytes) 000135 1772470857.017133 fallocate 0x024405 thru 0x027819 (0x3415 bytes) 000136 1772470857.028351 trunc from 0x0116b8 to 0x0152ba (0x3c03 bytes) 000137 1772470857.033683 write 0x029079 thru 0x02c6d9 (0x3661 bytes) 000138 1772470857.040266 trunc from 0x02c6da to 0x026231 (0x64a8 bytes) 000139 1772470857.055072 mapwrite 0x011d37 thru 0x012128 (0x3f2 bytes) 000140 1772470857.098896 trunc from 0x0077c1 to 0x00a400 (0x2c40 bytes) 000141 1772470857.109243 trunc from 0x026232 to 0x004d16 (0x2151b bytes) 000142 1772470857.133868 fallocate 0x03f56d thru 0x03ffff (0xa93 bytes) 000143 1772470857.141528 trunc from 0x0029ac to 0x004d16 (0x236b bytes) 000144 1772470857.146555 fallocate 0x00ae8d thru 0x015761 (0xa8d5 bytes) 000145 1772470857.153461 trunc from 0x001a84 to 0x0098bb (0x7e38 bytes) 000147 1772470857.160312 read 0x002a85 thru 0x0091cc (0x6748 bytes) 000148 1772470857.175342 MIRROR_RESYNC 4 mirrors 000149 1772470857.873101 trunc from 0x00ee0d to 0x013fa8 (0x519c bytes) 000150 1772470857.911970 trunc from 0x015762 to 0x0040be (0x116a3 bytes) 000151 1772470857.956615 mapwrite 0x00b344 thru 0x015c02 (0xa8bf bytes) 000152 1772470858.013342 fallocate 0x027d62 thru 0x031a56 (0x9cf5 bytes) 000153 1772470858.020280 fallocate 0x01ff29 thru 0x02238a (0x2462 bytes) 000154 1772470858.027316 trunc from 0x0295a1 to 0x031a56 (0x84b6 bytes) 000155 1772470858.038581 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31a57): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a57): Mirror 4: 0xbf8ce8c5 (stale) Mirror 5: 0xbf8ce8c5 (stale) Mirror 6: 0xbf8ce8c5 (stale) CRC-32 checksum value for chunk [0, 0x31a57): Mirror 1: 0x127d074e Mirror 7: 0x127d074e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a57: skipped 000156 1772470859.031321 write 0x03e4db thru 0x03f09a (0xbc0 bytes) 000157 1772470859.099216 punch from 0x03c4b6 to 0x03f09a (0x2be5 bytes) 000158 1772470859.115523 mapread 0x023ba9 thru 0x029491 (0x58e9 bytes) 000159 1772470859.144220 mapwrite 0x03b484 thru 0x03ffff (0x4b7c bytes) 000160 1772470859.216932 trunc from 0x02cbac to 0x02dcf3 (0x1148 bytes) 000161 1772470859.222238 mapread 0x0096e6 thru 0x00e108 (0x4a23 bytes) 000162 1772470859.239131 punch from 0x022702 to 0x024dc2 (0x26c1 bytes) 000163 1772470859.244150 fallocate 0x01a25b thru 0x029380 (0xf126 bytes) 000164 1772470859.262145 mapread 0x017e3c thru 0x0224ac (0xa671 bytes) 000167 1772470859.302808 read 0x036b73 thru 0x03ffff (0x948d bytes) 000168 1772470859.303512 trunc from 0x040000 to 0x006887 (0x39778 bytes) 000169 1772470859.349956 fallocate 0x021ff8 thru 0x031b9c (0xfba5 bytes) 000170 1772470859.360846 mapwrite 0x031efa thru 0x03a50d (0x8614 bytes) 000172 1772470859.445100 fallocate 0x017998 thru 0x019f9a (0x2603 bytes) 000173 1772470859.451663 trunc from 0x03a50e to 0x005bb1 (0x3495c bytes) 000174 1772470859.478099 mapread 0x005316 thru 0x005bb1 (0x89c bytes) 000175 1772470859.479931 write 0x0318b6 thru 0x0355e5 (0x3d30 bytes) 000176 1772470859.482060 trunc from 0x03137a to 0x0355e5 (0x426c bytes) 000177 1772470859.508303 fallocate 0x014f01 thru 0x02199c (0xca9c bytes) 000178 1772470859.552175 mapwrite 0x03b50c thru 0x03c336 (0xe2b bytes) 000179 1772470859.635335 fallocate 0x021d0c thru 0x031ae8 (0xfddd bytes) 000180 1772470859.649518 mapread 0x0089e1 thru 0x00fd85 (0x73a5 bytes) 000181 1772470859.685297 mapwrite 0x002454 thru 0x00e0c5 (0xbc72 bytes) 000182 1772470859.767856 read 0x02a1df thru 0x037b35 (0xd957 bytes) 000184 1772470859.798428 punch from 0x00101e to 0x003c6b (0x2c4e bytes) 000185 1772470859.809857 mapread 0x013b43 thru 0x014249 (0x707 bytes) 000186 1772470859.811515 mapread 0x02fd3c thru 0x03be8f (0xc154 bytes) 000187 1772470859.817436 write 0x038cfd thru 0x03ffff (0x7303 bytes) 000188 1772470859.820193 trunc from 0x03f4b8 to 0x03ffff (0xb48 bytes) 000189 1772470859.841188 write 0x011a33 thru 0x01bba4 (0xa172 bytes) 000190 1772470859.843590 write 0x01c7a5 thru 0x026e15 (0xa671 bytes) 000191 1772470859.845260 mapwrite 0x02ff0b thru 0x037c79 (0x7d6f bytes) 000192 1772470859.889893 write 0x007a41 thru 0x00e9f0 (0x6fb0 bytes) 000193 1772470859.891387 fallocate 0x0047fd thru 0x00d3f3 (0x8bf7 bytes) 000195 1772470859.899168 write 0x03e2ec thru 0x03ffff (0x1d14 bytes) 000196 1772470859.900373 read 0x03aef9 thru 0x03d9b1 (0x2ab9 bytes) 000197 1772470859.914216 fallocate 0x03e913 thru 0x03ffff (0x16ed bytes) 000198 1772470859.934105 trunc from 0x03aaa6 to 0x03ffff (0x555a bytes) 000199 1772470860.011740 trunc from 0x040000 to 0x03aa92 (0x556d bytes) 000200 1772470860.047884 punch from 0x02abe7 to 0x033daf (0x91c9 bytes) 000202 1772470860.060333 fallocate 0x02bad7 thru 0x032264 (0x678e bytes) 000203 1772470860.069675 punch from 0x00eb74 to 0x00fefb (0x1388 bytes) 000205 1772470860.082721 mapwrite 0x02aad2 thru 0x031114 (0x6643 bytes) 000206 1772470860.134922 trunc from 0x03aa93 to 0x0210cc (0x199c6 bytes) 000207 1772470860.158108 trunc from 0x01b93c to 0x0210cc (0x5791 bytes) 000208 1772470860.164036 mapread 0x01d40c thru 0x0210cc (0x3cc1 bytes) 000209 1772470860.195761 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x210cd): Mirror 4: 0x68b91eb3 (stale) Mirror 5: 0x68b91eb3 (stale) Mirror 7: 0x106787b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210cd): Mirror 4: 0x68b91eb3 (stale) Mirror 5: 0x68b91eb3 (stale) Mirror 7: 0x106787b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210cd: skipped 000210 1772470860.592276 trunc from 0x0210cd to 0x012ac1 (0xe60b bytes) 000211 1772470860.638312 punch from 0x00eed2 to 0x012ac1 (0x3bf0 bytes) 000212 1772470860.646311 fallocate 0x009820 thru 0x015dbf (0xc5a0 bytes) 000213 1772470860.658049 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12ac2): Mirror 4: 0xe8345004 (stale) Mirror 7: 0xf5bc1409 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12ac2): Mirror 4: 0xe8345004 (stale) Mirror 7: 0xf5bc1409 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12ac2: skipped 000214 1772470861.187519 fallocate 0x016d37 thru 0x024b23 (0xdded bytes) 000215 1772470861.205659 write 0x0290ad thru 0x02f55d (0x64b1 bytes) 000216 1772470861.226284 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f55e): Mirror 4: 0x1e1eb373 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f55e): Mirror 4: 0x1e1eb373 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f55e: skipped 000217 1772470861.726109 mapread 0x0171db thru 0x020f13 (0x9d39 bytes) 000218 1772470861.754621 read 0x022b1a thru 0x02f55d (0xca44 bytes) 000219 1772470861.770618 write 0x02ca6c thru 0x02eb86 (0x211b bytes) 000220 1772470861.781701 read 0x0151d4 thru 0x02418a (0xefb7 bytes) 000221 1772470861.783364 trunc from 0x02761f to 0x02f55d (0x7f3f bytes) 000222 1772470861.794912 fallocate 0x014a95 thru 0x01e06b (0x95d7 bytes) 000223 1772470861.800281 read 0x02807f thru 0x02f55d (0x74df bytes) 000224 1772470861.811465 fallocate 0x0055be thru 0x00eef3 (0x9936 bytes) 000225 1772470861.816731 trunc from 0x0147f2 to 0x01dc1a (0x9429 bytes) 000226 1772470861.822981 mapwrite 0x01959b thru 0x0216af (0x8115 bytes) 000227 1772470861.871635 fallocate 0x02abf2 thru 0x038ae4 (0xdef3 bytes) 000229 1772470861.878523 read 0x0048c3 thru 0x00f61a (0xad58 bytes) 000230 1772470861.879223 mapread 0x0027f2 thru 0x0084e2 (0x5cf1 bytes) 000231 1772470861.881758 fallocate 0x02d72f thru 0x03c23b (0xeb0d bytes) 000232 1772470861.889575 trunc from 0x03c23c to 0x021d26 (0x1a515 bytes) 000233 1772470861.914125 mapread 0x01eb75 thru 0x021d26 (0x31b2 bytes) 000234 1772470861.918507 write 0x019ef9 thru 0x0290f6 (0xf1fe bytes) 000235 1772470861.922871 mapwrite 0x009a59 thru 0x00bfdc (0x2584 bytes) 000236 1772470861.972069 trunc from 0x0290f7 to 0x000f59 (0x2819d bytes) 000237 1772470862.002332 read 0x000a35 thru 0x000f59 (0x525 bytes) 000238 1772470862.003049 mapread 0x000c7b thru 0x000f59 (0x2df bytes) 000239 1772470862.004018 mapwrite 0x01728c thru 0x01e922 (0x7697 bytes) 000240 1772470862.086264 MIRROR_RESYNC 2 mirrors 000241 1772470863.019665 trunc from 0x01e923 to 0x033921 (0x14fff bytes) 000242 1772470863.102224 trunc from 0x01a1da to 0x022dfc (0x8c23 bytes) 000243 1772470863.113221 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000244 1772470863.267730 trunc from 0x00a1c6 to 0x015756 (0xb591 bytes) 000246 1772470863.277588 fallocate 0x038c01 thru 0x039bdd (0xfdd bytes) 000247 1772470863.284206 fallocate 0x02f825 thru 0x03d6e5 (0xdec1 bytes) 000248 1772470863.290063 read 0x007d69 thru 0x010b68 (0x8e00 bytes) 000249 1772470863.300972 punch from 0x007682 to 0x0118db (0xa25a bytes) 000250 1772470863.306935 read 0x01e8eb thru 0x020d4e (0x2464 bytes) 000252 1772470863.323094 mapread 0x01719d thru 0x02674f (0xf5b3 bytes) 000254 1772470863.327216 write 0x02c95a thru 0x036e92 (0xa539 bytes) 000255 1772470863.333740 mapwrite 0x00a031 thru 0x0140dc (0xa0ac bytes) 000256 1772470863.361733 punch from 0x012445 to 0x01d467 (0xb023 bytes) 000257 1772470863.368059 mapwrite 0x0044da thru 0x008d9f (0x48c6 bytes) 000258 1772470863.386015 mapread 0x027292 thru 0x02f8c1 (0x8630 bytes) 000259 1772470863.387700 mapwrite 0x024ef8 thru 0x02d105 (0x820e bytes) 000260 1772470863.407598 trunc from 0x036e93 to 0x019f4c (0x1cf46 bytes) 000261 1772470863.425719 trunc from 0x011386 to 0x019f4c (0x8bc7 bytes) 000263 1772470863.442565 trunc from 0x019f4d to 0x017e8b (0x20c1 bytes) 000264 1772470863.478281 mapwrite 0x0234d2 thru 0x0285eb (0x511a bytes) 000265 1772470863.596370 mapread 0x017637 thru 0x0244c5 (0xce8f bytes) 000266 1772470863.735179 punch from 0x01accf to 0x01d63d (0x296f bytes) 000267 1772470863.753197 punch from 0x00ff67 to 0x010e70 (0xf0a bytes) 000268 1772470863.765527 write 0x019a09 thru 0x026d66 (0xd35e bytes) 000269 1772470863.768313 mapread 0x027daa thru 0x0285eb (0x842 bytes) 000270 1772470863.777228 read 0x008f2e thru 0x00d9c5 (0x4a98 bytes) 000272 1772470863.798709 MIRROR_EXTEND to 2 mirrors 000273 1772470864.747050 mapwrite 0x0245d7 thru 0x031891 (0xd2bb bytes) 000274 1772470864.835744 mapread 0x01fdcd thru 0x021a96 (0x1cca bytes) 000275 1772470864.836896 read 0x007177 thru 0x011c3d (0xaac7 bytes) 000276 1772470864.848165 read 0x0226aa thru 0x0226e1 (0x38 bytes) 000277 1772470864.849295 trunc from 0x031892 to 0x003f45 (0x2d94c bytes) 000278 1772470864.878252 punch from 0x000beb to 0x003f45 (0x335b bytes) 000280 1772470864.888935 mapwrite 0x0184dd thru 0x02290c (0xa430 bytes) 000281 1772470864.951045 punch from 0x017991 to 0x01cf5f (0x55cf bytes) 000282 1772470864.957714 fallocate 0x037ba3 thru 0x03ffff (0x845d bytes) 000283 1772470864.964841 mapread 0x002e90 thru 0x003f67 (0x10d8 bytes) 000284 1772470864.972851 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2290d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2290d): Mirror 2: 0x1b6de6a7 (stale) CRC-32 checksum value for chunk [0, 0x2290d): Mirror 1: 0x6d975ea3 Mirror 3: 0x6d975ea3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2290d: skipped 000285 1772470866.099756 punch from 0x004022 to 0x007af1 (0x3ad0 bytes) 000287 1772470866.145890 trunc from 0x011e39 to 0x015133 (0x32fb bytes) 000288 1772470866.151819 mapread 0x011a66 thru 0x01fca6 (0xe241 bytes) 000289 1772470866.182061 mapread 0x01be25 thru 0x02290c (0x6ae8 bytes) 000290 1772470866.183515 mapread 0x01c862 thru 0x020ef4 (0x4693 bytes) 000291 1772470866.185984 fallocate 0x02248e thru 0x02386d (0x13e0 bytes) 000292 1772470866.192576 fallocate 0x008b60 thru 0x01393d (0xadde bytes) 000293 1772470866.199164 mapwrite 0x03e676 thru 0x03ffff (0x198a bytes) 000295 1772470866.283738 mapread 0x02899a thru 0x02fff5 (0x765c bytes) 000296 1772470866.306999 mapread 0x01748a thru 0x01bb11 (0x4688 bytes) 000297 1772470866.314767 mapwrite 0x025d1f thru 0x029199 (0x347b bytes) 000298 1772470866.343916 punch from 0x03be6a to 0x03ffff (0x4196 bytes) 000299 1772470866.351047 punch from 0x0176c3 to 0x0205a9 (0x8ee7 bytes) 000300 1772470866.357437 mapread 0x024a8c thru 0x027634 (0x2ba9 bytes) 000301 1772470866.362845 MIRROR_RESYNC 3 mirrors 000302 1772470867.552128 trunc from 0x040000 to 0x003daa (0x3c255 bytes) 000303 1772470867.623127 punch from 0x002067 to 0x003daa (0x1d44 bytes) 000304 1772470867.630845 read 0x002220 thru 0x003daa (0x1b8b bytes) 000305 1772470867.639846 fallocate 0x0031dc thru 0x013141 (0xff66 bytes) 000306 1772470867.651014 fallocate 0x01bf15 thru 0x028934 (0xca20 bytes) 000307 1772470867.663603 trunc from 0x028935 to 0x00ea6b (0x19ec9 bytes) 000308 1772470867.714736 read 0x0050a3 thru 0x00e90a (0x9868 bytes) 000309 1772470867.726691 mapread 0x0006c5 thru 0x006894 (0x61d0 bytes) 000310 1772470867.728887 read 0x00762c thru 0x00aded (0x37c2 bytes) 000311 1772470867.730128 read 0x00c458 thru 0x00ea6b (0x2614 bytes) 000312 1772470867.735896 write 0x009208 thru 0x01015e (0x6f57 bytes) 000313 1772470867.752760 MIRROR_RESYNC 3 mirrors 000314 1772470868.997185 trunc from 0x00cf8a to 0x01015e (0x31d5 bytes) 000315 1772470869.026003 mapwrite 0x0017d8 thru 0x00bd01 (0xa52a bytes) 000316 1772470869.063531 trunc from 0x0100ac to 0x01015e (0xb3 bytes) 000318 1772470869.069149 write 0x00c54b thru 0x0187fb (0xc2b1 bytes) 000320 1772470869.076101 mapread 0x00086b thru 0x00bc86 (0xb41c bytes) 000321 1772470869.081551 write 0x0011d9 thru 0x00f695 (0xe4bd bytes) 000322 1772470869.083191 read 0x01159a thru 0x0187fb (0x7262 bytes) 000323 1772470869.084159 write 0x012f34 thru 0x019569 (0x6636 bytes) 000324 1772470869.085453 trunc from 0x00f2cb to 0x019569 (0xa29f bytes) 000325 1772470869.101455 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1956a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1956a): Mirror 2: 0x1ed08361 (stale) Mirror 3: 0x1ed08361 (stale) CRC-32 checksum value for chunk [0, 0x1956a): Mirror 1: 0xfcba0080 Mirror 4: 0xfcba0080 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1956a: skipped 000326 1772470869.866102 read 0x0163d5 thru 0x019569 (0x3195 bytes) 000327 1772470869.880870 mapwrite 0x018f67 thru 0x020d1d (0x7db7 bytes) 000328 1772470869.987952 trunc from 0x015ffd to 0x018985 (0x2989 bytes) 000329 1772470869.998761 mapread 0x0182ce thru 0x0199ed (0x1720 bytes) 000330 1772470870.015901 MIRROR_RESYNC 4 mirrors 000331 1772470871.072616 mapread 0x01ebcf thru 0x020d1d (0x214f bytes) 000332 1772470871.123206 mapread 0x01ceba thru 0x020d1d (0x3e64 bytes) 000333 1772470871.125041 mapwrite 0x0030cf thru 0x00dc9b (0xabcd bytes) 000334 1772470871.287978 read 0x020a52 thru 0x020d1d (0x2cc bytes) 000335 1772470871.321732 read 0x019e03 thru 0x01f042 (0x5240 bytes) 000336 1772470871.361707 fallocate 0x03b678 thru 0x03ffff (0x4988 bytes) 000337 1772470871.371050 write 0x015dbc thru 0x0226c5 (0xc90a bytes) 000338 1772470871.395053 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x226c6): Mirror 2: 0x60817f92 (stale) Mirror 3: 0x60817f92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x226c6): Mirror 2: 0x60817f92 (stale) Mirror 3: 0x60817f92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x226c6: skipped 000339 1772470872.042519 mapwrite 0x004150 thru 0x00b12e (0x6fdf bytes) 000340 1772470872.162497 fallocate 0x01e012 thru 0x02ba84 (0xda73 bytes) 000341 1772470872.188497 read 0x0186b3 thru 0x01c613 (0x3f61 bytes) 000342 1772470872.209540 punch from 0x01a02f to 0x0226c5 (0x8697 bytes) 000343 1772470872.227658 fallocate 0x0139f3 thru 0x0227cc (0xedda bytes) 000345 1772470872.276446 read 0x01bae3 thru 0x02005a (0x4578 bytes) 000346 1772470872.348163 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x226c6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x226c6): Mirror 2: 0x60817f92 (stale) Mirror 3: 0x60817f92 (stale) CRC-32 checksum value for chunk [0, 0x226c6): Mirror 1: 0xe26568c4 Mirror 4: 0xe26568c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x226c6: skipped 000347 1772470874.295440 fallocate 0x03b1d3 thru 0x03b615 (0x443 bytes) 000348 1772470874.382759 trunc from 0x0226c6 to 0x011def (0x108d6 bytes) 000349 1772470874.453398 trunc from 0x011df0 to 0x00c25a (0x5b95 bytes) 000351 1772470874.489998 trunc from 0x00c25b to 0x02ded2 (0x21c78 bytes) 000352 1772470874.553740 write 0x006d14 thru 0x01679b (0xfa88 bytes) 000353 1772470874.570756 fallocate 0x01d96a thru 0x02550e (0x7ba5 bytes) 000354 1772470874.591229 trunc from 0x012106 to 0x020b7b (0xea76 bytes) 000355 1772470874.624336 mapread 0x00fd1c thru 0x016902 (0x6be7 bytes) 000357 1772470874.638435 write 0x003564 thru 0x00b4d6 (0x7f73 bytes) 000358 1772470874.653959 write 0x0124b7 thru 0x01366f (0x11b9 bytes) 000359 1772470874.655850 mapwrite 0x001caf thru 0x004102 (0x2454 bytes) 000360 1772470874.676926 trunc from 0x02ded3 to 0x02ade2 (0x30f0 bytes) 000361 1772470874.690875 mapread 0x01acb6 thru 0x01dfc8 (0x3313 bytes) 000362 1772470874.708084 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ade3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ade3): Mirror 2: 0x183335e6 (stale) Mirror 3: 0x183335e6 (stale) Mirror 4: 0xc95c2874 (stale) CRC-32 checksum value for chunk [0, 0x2ade3): Mirror 1: 0xddb9c827 Mirror 5: 0xddb9c827 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ade3: skipped 000363 1772470875.882463 mapwrite 0x015cce thru 0x0167a1 (0xad4 bytes) 000364 1772470876.055749 trunc from 0x025cf6 to 0x02ade2 (0x50ed bytes) 000365 1772470876.063115 punch from 0x00ab14 to 0x00d0cb (0x25b8 bytes) 000366 1772470876.072277 mapwrite 0x0211e3 thru 0x02a74b (0x9569 bytes) 000367 1772470876.122075 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ade3): Mirror 2: 0x183335e6 (stale) Mirror 3: 0x183335e6 (stale) Mirror 4: 0xc95c2874 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ade3): Mirror 2: 0x183335e6 (stale) Mirror 3: 0x183335e6 (stale) Mirror 4: 0xc95c2874 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ade3: skipped 000368 1772470876.822050 read 0x009471 thru 0x0103db (0x6f6b bytes) 000369 1772470876.863293 MIRROR_RESYNC 4 mirrors 000370 1772470878.948669 trunc from 0x02ade3 to 0x02704e (0x3d94 bytes) 000371 1772470879.008654 read 0x02473c thru 0x0253fe (0xcc3 bytes) 000372 1772470879.023153 trunc from 0x02704f to 0x036a1a (0xf9cc bytes) 000373 1772470879.073884 read 0x033c85 thru 0x036a1a (0x2d96 bytes) 000374 1772470879.111569 fallocate 0x026f09 thru 0x030f6c (0xa064 bytes) 000375 1772470879.123273 punch from 0x01689f to 0x017a12 (0x1174 bytes) 000376 1772470879.131710 fallocate 0x02aec7 thru 0x037238 (0xc372 bytes) 000377 1772470879.143620 mapwrite 0x00df75 thru 0x010552 (0x25de bytes) 000378 1772470879.172542 mapread 0x015363 thru 0x023f10 (0xebae bytes) 000380 1772470879.204667 trunc from 0x037239 to 0x02638f (0x10ea9 bytes) 000381 1772470879.231895 write 0x0221ea thru 0x024591 (0x23a8 bytes) 000382 1772470879.233017 trunc from 0x026390 to 0x00e22d (0x18162 bytes) 000383 1772470879.283683 write 0x011646 thru 0x01ea2c (0xd3e7 bytes) 000384 1772470879.291357 read 0x0099b6 thru 0x011076 (0x76c1 bytes) 000385 1772470879.296309 fallocate 0x01ec8d thru 0x02cfc6 (0xe33a bytes) 000386 1772470879.313076 punch from 0x007969 to 0x011534 (0x9bcc bytes) 000387 1772470879.341063 punch from 0x00f2cc to 0x01e579 (0xf2ae bytes) 000388 1772470879.352126 mapwrite 0x037290 thru 0x03ffff (0x8d70 bytes) 000389 1772470879.434088 fallocate 0x02dac1 thru 0x03b218 (0xd758 bytes) 000390 1772470879.444005 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x77904bbc (stale) Mirror 4: 0x77904bbc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x77904bbc (stale) Mirror 4: 0x77904bbc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000391 1772470879.795136 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x77904bbc (stale) Mirror 4: 0x77904bbc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb45ad273 Mirror 5: 0xb45ad273 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000392 1772470880.687730 trunc from 0x01a77c to 0x01c7e5 (0x206a bytes) 000393 1772470880.728977 write 0x01b2c5 thru 0x023c54 (0x8990 bytes) 000394 1772470880.742804 trunc from 0x040000 to 0x028f9d (0x17062 bytes) 000395 1772470880.766628 write 0x000b95 thru 0x007b47 (0x6fb3 bytes) 000396 1772470880.781086 write 0x03f648 thru 0x03ffff (0x9b8 bytes) 000397 1772470880.783100 trunc from 0x003281 to 0x005b0c (0x288c bytes) 000399 1772470880.801059 trunc from 0x040000 to 0x02f9eb (0x10614 bytes) 000400 1772470880.830011 trunc from 0x0002fc to 0x00123e (0xf43 bytes) 000401 1772470880.835739 read 0x0260b5 thru 0x02f9eb (0x9937 bytes) 000402 1772470880.849970 MIRROR_RESYNC 4 mirrors 000403 1772470881.812618 trunc from 0x0049bf to 0x006130 (0x1772 bytes) 000404 1772470881.843670 punch from 0x00a555 to 0x00fa3c (0x54e8 bytes) 000405 1772470881.850015 fallocate 0x02246a thru 0x02a822 (0x83b9 bytes) 000408 1772470881.859142 write 0x00426b thru 0x006947 (0x26dd bytes) 000409 1772470881.869738 write 0x0132b6 thru 0x01f037 (0xbd82 bytes) 000410 1772470881.883904 mapread 0x00c7a0 thru 0x01aedd (0xe73e bytes) 000411 1772470881.885270 mapwrite 0x0081bc thru 0x00d329 (0x516e bytes) 000413 1772470881.916971 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f9ec): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f9ec): Mirror 3: 0xe39d340 (stale) Mirror 4: 0xe39d340 (stale) Mirror 5: 0xe39d340 (stale) CRC-32 checksum value for chunk [0, 0x2f9ec): Mirror 1: 0x9e667d14 Mirror 6: 0x9e667d14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f9ec: skipped 000414 1772470882.813166 punch from 0x0216ab to 0x02668e (0x4fe4 bytes) 000415 1772470882.861738 write 0x01ecba thru 0x02de35 (0xf17c bytes) 000416 1772470882.877035 write 0x01c211 thru 0x0238a9 (0x7699 bytes) 000417 1772470882.886507 write 0x0074bc thru 0x00f284 (0x7dc9 bytes) 000419 1772470882.902433 fallocate 0x015e5a thru 0x01c48c (0x6633 bytes) 000420 1772470882.908269 mapread 0x001512 thru 0x008eba (0x79a9 bytes) 000421 1772470882.913947 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f9ec): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f9ec): Mirror 3: 0xe39d340 (stale) Mirror 4: 0xe39d340 (stale) Mirror 5: 0xe39d340 (stale) Mirror 6: 0x9e667d14 (stale) CRC-32 checksum value for chunk [0, 0x2f9ec): Mirror 1: 0x80ca0f78 Mirror 7: 0x80ca0f78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f9ec: skipped 000422 1772470883.845555 trunc from 0x02f9ec to 0x012fb3 (0x1ca38 bytes) 000423 1772470883.898892 read 0x00df7c thru 0x012fb3 (0x5038 bytes) 000424 1772470883.908819 mapread 0x003da7 thru 0x010e1d (0xd077 bytes) 000425 1772470883.928617 punch from 0x00d935 to 0x00e6e3 (0xdaf bytes) 000426 1772470883.934608 read 0x011b47 thru 0x012fb3 (0x146d bytes) 000427 1772470883.935346 read 0x00199d thru 0x001e65 (0x4c9 bytes) 000428 1772470883.936477 mapread 0x00874f thru 0x00d8b7 (0x5169 bytes) 000429 1772470883.953414 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12fb4): Mirror 4: 0x3abe71bf (stale) Mirror 5: 0x3abe71bf (stale) Mirror 6: 0x2b7d5c42 (stale) Mirror 7: 0x46fea760 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12fb4): Mirror 4: 0x3abe71bf (stale) Mirror 5: 0x3abe71bf (stale) Mirror 6: 0x2b7d5c42 (stale) Mirror 7: 0x46fea760 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12fb4: skipped 000430 1772470884.322762 write 0x036493 thru 0x03ffff (0x9b6d bytes) 000432 1772470884.332553 mapwrite 0x008ed5 thru 0x016efe (0xe02a bytes) 000434 1772470884.394131 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x837456ea (stale) Mirror 6: 0x5c699eec (stale) Mirror 7: 0x6597f1bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x837456ea (stale) Mirror 6: 0x5c699eec (stale) Mirror 7: 0x6597f1bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000435 1772470884.897839 punch from 0x0180e7 to 0x026d61 (0xec7b bytes) 000436 1772470884.903917 fallocate 0x0142ab thru 0x01fbf6 (0xb94c bytes) 000437 1772470884.910755 fallocate 0x03cdd1 thru 0x03ffff (0x322f bytes) 000438 1772470884.917085 trunc from 0x0233b7 to 0x032c4b (0xf895 bytes) 000439 1772470884.923649 trunc from 0x040000 to 0x02cd52 (0x132ad bytes) 000440 1772470884.947577 punch from 0x01d76c to 0x02b148 (0xd9dd bytes) 000441 1772470884.953702 punch from 0x01f11d to 0x020036 (0xf1a bytes) 000442 1772470884.959236 trunc from 0x02cd53 to 0x015e41 (0x16f11 bytes) 000445 1772470884.977933 trunc from 0x015e42 to 0x037c8d (0x21e4c bytes) 000446 1772470885.003996 fallocate 0x02b14c thru 0x03b10c (0xffc1 bytes) 000447 1772470885.010809 write 0x02dad7 thru 0x037620 (0x9b4a bytes) 000448 1772470885.024985 punch from 0x002209 to 0x006c8a (0x4a82 bytes) 000449 1772470885.030320 fallocate 0x015a32 thru 0x01bbfc (0x61cb bytes) 000450 1772470885.038505 mapwrite 0x02128a thru 0x0306d3 (0xf44a bytes) 000451 1772470885.080021 trunc from 0x031bdd to 0x03b10c (0x9530 bytes) 000452 1772470885.086566 trunc from 0x018bcb to 0x021c5c (0x9092 bytes) 000453 1772470885.094658 punch from 0x02cafd to 0x037de1 (0xb2e5 bytes) 000454 1772470885.101852 fallocate 0x023adf thru 0x02c443 (0x8965 bytes) 000455 1772470885.108403 read 0x031e1b thru 0x037e3f (0x6025 bytes) 000456 1772470885.125827 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b10d): Mirror 5: 0x7d0336b8 (stale) Mirror 7: 0xb27ea1f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b10d): Mirror 5: 0x7d0336b8 (stale) Mirror 7: 0xb27ea1f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b10d: skipped 000457 1772470885.490304 fallocate 0x009841 thru 0x01938e (0xfb4e bytes) 000458 1772470885.505619 trunc from 0x03b10d to 0x006ceb (0x34421 bytes) 000459 1772470885.573471 punch from 0x005456 to 0x006ceb (0x1896 bytes) 000460 1772470885.585487 write 0x01f955 thru 0x02dbcb (0xe277 bytes) 000461 1772470885.588132 write 0x0240f3 thru 0x026356 (0x2264 bytes) 000463 1772470885.590947 mapwrite 0x017b87 thru 0x01f8db (0x7d55 bytes) 000464 1772470885.648290 write 0x00cf57 thru 0x01b36d (0xe417 bytes) 000465 1772470885.669996 mapread 0x00e23b thru 0x01d4bc (0xf282 bytes) 000466 1772470885.672119 trunc from 0x0145df to 0x017cb2 (0x36d4 bytes) 000467 1772470885.691868 mapread 0x0236fa thru 0x029c36 (0x653d bytes) 000468 1772470885.693575 mapwrite 0x01c33c thru 0x024db2 (0x8a77 bytes) 000469 1772470885.718943 mapread 0x01e912 thru 0x024c6a (0x6359 bytes) 000470 1772470885.720177 fallocate 0x028f41 thru 0x02eade (0x5b9e bytes) 000471 1772470885.727578 write 0x0197ff thru 0x027399 (0xdb9b bytes) 000472 1772470885.729232 mapread 0x0198a4 thru 0x028ab4 (0xf211 bytes) 000473 1772470885.731884 write 0x01e77b thru 0x027e34 (0x96ba bytes) 000474 1772470885.733707 mapwrite 0x0020c1 thru 0x007be6 (0x5b26 bytes) 000475 1772470885.776988 mapread 0x01e568 thru 0x02b618 (0xd0b1 bytes) 000476 1772470885.779089 trunc from 0x0223d9 to 0x02561b (0x3243 bytes) 000477 1772470885.796046 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2eadf): Mirror 7: 0x82ee5a82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eadf): Mirror 7: 0x82ee5a82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eadf: skipped 000478 1772470886.190057 trunc from 0x02de73 to 0x02eade (0xc6c bytes) 000479 1772470886.197432 mapread 0x02baf9 thru 0x02eade (0x2fe6 bytes) 000480 1772470886.220576 read 0x00d4f7 thru 0x01afc1 (0xdacb bytes) 000481 1772470886.231652 fallocate 0x02374d thru 0x025ab7 (0x236b bytes) 000482 1772470886.238571 read 0x0263fb thru 0x027e19 (0x1a1f bytes) 000483 1772470886.239889 read 0x01664c thru 0x018db7 (0x276c bytes) 000484 1772470886.241089 mapwrite 0x0208c4 thru 0x022d17 (0x2454 bytes) 000485 1772470886.284692 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000486 1772470886.475567 mapwrite 0x03d448 thru 0x03ffff (0x2bb8 bytes) 000487 1772470886.618814 punch from 0x025232 to 0x03211e (0xceed bytes) 000488 1772470886.638307 punch from 0x00de04 to 0x01d062 (0xf25f bytes) 000489 1772470886.648175 read 0x00fb3a thru 0x01866a (0x8b31 bytes) 000490 1772470886.671377 mapwrite 0x034420 thru 0x034e9a (0xa7b bytes) 000492 1772470886.784230 fallocate 0x02ac5d thru 0x031a54 (0x6df8 bytes) 000493 1772470886.794703 trunc from 0x040000 to 0x03340b (0xcbf4 bytes) 000494 1772470886.920547 mapread 0x025b98 thru 0x02c46d (0x68d6 bytes) 000495 1772470886.955505 MIRROR_EXTEND to 2 mirrors 000496 1772470888.503974 trunc from 0x030f3b to 0x03340b (0x24d1 bytes) 000497 1772470888.578271 MIRROR_RESYNC 2 mirrors 000498 1772470889.312026 mapread 0x0291dd thru 0x02cd43 (0x3b67 bytes) 000499 1772470889.332838 punch from 0x030df2 to 0x032735 (0x1944 bytes) 000500 1772470889.374691 trunc from 0x03340c to 0x02af03 (0x8508 bytes) 000502 1772470889.397826 mapwrite 0x006983 thru 0x00bd8a (0x5408 bytes) 000504 1772470889.427418 mapread 0x007912 thru 0x012cf3 (0xb3e2 bytes) 000505 1772470889.437058 fallocate 0x0165e1 thru 0x01e36d (0x7d8d bytes) 000506 1772470889.442316 mapread 0x007c48 thru 0x0144c0 (0xc879 bytes) 000507 1772470889.445691 punch from 0x01e0d4 to 0x024df3 (0x6d20 bytes) 000508 1772470889.451765 punch from 0x0111f1 to 0x01fe79 (0xec89 bytes) 000509 1772470889.456693 read 0x002386 thru 0x004c78 (0x28f3 bytes) 000510 1772470889.457294 trunc from 0x02af04 to 0x03e7d4 (0x138d1 bytes) 000511 1772470889.471993 mapread 0x039296 thru 0x03e7d4 (0x553f bytes) 000512 1772470889.489912 mapread 0x00b91e thru 0x012713 (0x6df6 bytes) 000513 1772470889.501673 read 0x016890 thru 0x0217b4 (0xaf25 bytes) 000515 1772470889.525830 write 0x003bd9 thru 0x0074e3 (0x390b bytes) 000516 1772470889.528321 trunc from 0x037a29 to 0x03e7d4 (0x6dac bytes) 000517 1772470889.533401 mapread 0x004ec5 thru 0x011121 (0xc25d bytes) 000518 1772470889.535198 trunc from 0x013950 to 0x01cb68 (0x9219 bytes) 000520 1772470889.543851 write 0x02100f thru 0x02c842 (0xb834 bytes) 000521 1772470889.546892 write 0x024bff thru 0x02e030 (0x9432 bytes) 000523 1772470889.549602 trunc from 0x03e7d5 to 0x02cdf0 (0x119e4 bytes) 000525 1772470889.568639 fallocate 0x03f4cc thru 0x03ffff (0xb34 bytes) 000526 1772470889.574051 write 0x000eb5 thru 0x00acf0 (0x9e3c bytes) 000528 1772470889.575904 mapread 0x003b91 thru 0x007e5a (0x42ca bytes) 000529 1772470889.576623 mapread 0x01cc34 thru 0x01e90f (0x1cdc bytes) 000530 1772470889.583514 punch from 0x00d226 to 0x00d3a3 (0x17e bytes) 000531 1772470889.589531 punch from 0x021574 to 0x022517 (0xfa4 bytes) 000532 1772470889.606879 mapwrite 0x00293f thru 0x012193 (0xf855 bytes) 000533 1772470889.637939 trunc from 0x013e90 to 0x01e326 (0xa497 bytes) 000536 1772470889.643731 mapwrite 0x03a9a4 thru 0x03ffff (0x565c bytes) 000537 1772470889.708115 read 0x0020d6 thru 0x002570 (0x49b bytes) 000538 1772470889.709640 trunc from 0x040000 to 0x02beee (0x14111 bytes) 000539 1772470889.727845 punch from 0x02bdeb to 0x02beee (0x104 bytes) 000541 1772470889.733481 write 0x033173 thru 0x036923 (0x37b1 bytes) 000542 1772470889.736125 trunc from 0x036924 to 0x010b7e (0x25da5 bytes) 000543 1772470889.754628 read 0x00080d thru 0x00199a (0x118e bytes) 000545 1772470889.755962 trunc from 0x00e774 to 0x00ea36 (0x2c3 bytes) 000546 1772470889.763925 punch from 0x00864b to 0x00a2a0 (0x1c56 bytes) 000547 1772470889.783917 MIRROR_RESYNC 2 mirrors 000548 1772470890.331625 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000549 1772470890.516257 mapwrite 0x025fb5 thru 0x02bd64 (0x5db0 bytes) 000550 1772470890.583599 mapwrite 0x036556 thru 0x03ffff (0x9aaa bytes) 000552 1772470890.666877 punch from 0x006597 to 0x00fa2d (0x9497 bytes) 000553 1772470890.676761 mapread 0x0225a3 thru 0x029004 (0x6a62 bytes) 000554 1772470890.677956 punch from 0x03faa2 to 0x03ffff (0x55e bytes) 000555 1772470890.691336 mapwrite 0x03e592 thru 0x03ffff (0x1a6e bytes) 000556 1772470890.714150 write 0x0099bf thru 0x015154 (0xb796 bytes) 000557 1772470890.733200 read 0x00bb85 thru 0x00c828 (0xca4 bytes) 000559 1772470890.735007 read 0x0123b5 thru 0x017626 (0x5272 bytes) 000561 1772470890.761218 MIRROR_EXTEND to 2 mirrors 000562 1772470891.865552 read 0x0161ed thru 0x02208e (0xbea2 bytes) 000563 1772470891.925015 trunc from 0x040000 to 0x025233 (0x1adcc bytes) 000564 1772470891.990257 punch from 0x024d4b to 0x025233 (0x4e9 bytes) 000565 1772470892.010411 trunc from 0x025234 to 0x021511 (0x3d22 bytes) 000566 1772470892.130677 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x21512): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21512): Mirror 2: 0x2d874496 (stale) CRC-32 checksum value for chunk [0, 0x21512): Mirror 1: 0x2d874496 Mirror 3: 0x2d874496 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21512: skipped 000567 1772470893.224641 trunc from 0x021512 to 0x01553f (0xbfd2 bytes) 000568 1772470893.286683 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x15540): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15540): Mirror 2: 0x9a6d8cf5 (stale) Mirror 3: 0x9a6d8cf5 (stale) CRC-32 checksum value for chunk [0, 0x15540): Mirror 1: 0x9a6d8cf5 Mirror 4: 0x9a6d8cf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15540: skipped 000569 1772470894.501514 write 0x022c3c thru 0x032739 (0xfafe bytes) 000570 1772470894.549712 mapread 0x016a27 thru 0x01d53f (0x6b19 bytes) 000571 1772470894.577401 read 0x01f3be thru 0x0285a1 (0x91e4 bytes) 000572 1772470894.584625 trunc from 0x03273a to 0x0384b3 (0x5d7a bytes) 000573 1772470894.613890 trunc from 0x0384b4 to 0x00a151 (0x2e362 bytes) 000574 1772470894.632556 write 0x01b3fb thru 0x026c5d (0xb863 bytes) 000575 1772470894.636759 punch from 0x00b426 to 0x01235c (0x6f37 bytes) 000576 1772470894.646928 fallocate 0x00cfb4 thru 0x018f52 (0xbf9f bytes) 000577 1772470894.655763 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26c5e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c5e): Mirror 2: 0x40929f98 (stale) Mirror 3: 0xd46989ef (stale) Mirror 4: 0xd46989ef (stale) CRC-32 checksum value for chunk [0, 0x26c5e): Mirror 1: 0x64cbe647 Mirror 5: 0x64cbe647 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c5e: skipped 000578 1772470896.044531 read 0x00462d thru 0x0120a8 (0xda7c bytes) 000579 1772470896.065009 punch from 0x01a89b to 0x0219b2 (0x7118 bytes) 000580 1772470896.111523 trunc from 0x007c68 to 0x013eb9 (0xc252 bytes) 000581 1772470896.117256 trunc from 0x026c5e to 0x009385 (0x1d8d8 bytes) 000582 1772470896.155602 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9386): Mirror 2: 0x50533b20 (stale) Mirror 3: 0x50533b20 (stale) Mirror 5: 0x50533b20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9386): Mirror 2: 0x50533b20 (stale) Mirror 3: 0x50533b20 (stale) Mirror 5: 0x50533b20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9386: skipped 000583 1772470896.478424 mapread 0x005bbd thru 0x0090fd (0x3541 bytes) 000584 1772470896.512261 fallocate 0x029920 thru 0x0361b2 (0xc893 bytes) 000585 1772470896.519362 mapwrite 0x009f8e thru 0x015247 (0xb2ba bytes) 000587 1772470896.569575 fallocate 0x02d3ce thru 0x03b26e (0xdea1 bytes) 000588 1772470896.574886 mapread 0x01d242 thru 0x023e18 (0x6bd7 bytes) 000589 1772470896.585258 punch from 0x00b248 to 0x014e45 (0x9bfe bytes) 000590 1772470896.592601 fallocate 0x00b503 thru 0x01b307 (0xfe05 bytes) 000591 1772470896.613682 punch from 0x0032c6 to 0x004309 (0x1044 bytes) 000592 1772470896.624573 mapread 0x0113dd thru 0x01ab23 (0x9747 bytes) 000593 1772470896.646058 punch from 0x02d8da to 0x0361b2 (0x88d9 bytes) 000594 1772470896.655491 fallocate 0x03846d thru 0x03ca68 (0x45fc bytes) 000595 1772470896.667347 trunc from 0x0361b3 to 0x03c717 (0x6565 bytes) 000596 1772470896.697435 write 0x03eac6 thru 0x03ffff (0x153a bytes) 000597 1772470896.700326 write 0x021724 thru 0x02c7b5 (0xb092 bytes) 000598 1772470896.712074 mapread 0x02b290 thru 0x02ffbe (0x4d2f bytes) 000599 1772470896.724466 trunc from 0x040000 to 0x01d58a (0x22a75 bytes) 000600 1772470896.747483 trunc from 0x014c4a to 0x015acb (0xe82 bytes) 000602 1772470896.755564 punch from 0x013065 to 0x01ccce (0x9c6a bytes) 000603 1772470896.765852 punch from 0x0149d9 to 0x016619 (0x1c41 bytes) 000604 1772470896.772414 punch from 0x015e58 to 0x019841 (0x39ea bytes) 000606 1772470896.779515 fallocate 0x01fecb thru 0x02edd0 (0xef06 bytes) 000607 1772470896.787637 read 0x02b0d2 thru 0x02edd0 (0x3cff bytes) 000608 1772470896.795476 trunc from 0x02edd1 to 0x00c1d4 (0x22bfc bytes) 000609 1772470896.807378 fallocate 0x01b091 thru 0x022fe4 (0x7f54 bytes) 000611 1772470896.813568 fallocate 0x02b661 thru 0x02e19b (0x2b3b bytes) 000612 1772470896.820069 mapwrite 0x029322 thru 0x02a397 (0x1076 bytes) 000613 1772470896.843532 trunc from 0x02e19c to 0x02c818 (0x1983 bytes) 000615 1772470896.860192 mapread 0x0195ac thru 0x026ae9 (0xd53e bytes) 000616 1772470896.880330 punch from 0x01ecde to 0x02b955 (0xcc78 bytes) 000618 1772470896.888442 read 0x00c60b thru 0x0108f9 (0x42ef bytes) 000619 1772470896.900250 fallocate 0x036082 thru 0x03da72 (0x79f1 bytes) 000620 1772470896.905202 mapread 0x01b5d1 thru 0x01b75e (0x18e bytes) 000621 1772470896.906699 mapread 0x01c5d3 thru 0x024f4e (0x897c bytes) 000622 1772470896.909048 trunc from 0x0135ac to 0x01b9e9 (0x843e bytes) 000623 1772470896.915526 read 0x0200ba thru 0x023059 (0x2fa0 bytes) 000625 1772470896.917548 write 0x031d6d thru 0x03fbf7 (0xde8b bytes) 000627 1772470896.921889 trunc from 0x03fbf8 to 0x02d305 (0x128f2 bytes) 000628 1772470896.946971 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d306): Mirror 3: 0x735dd6d4 (stale) Mirror 5: 0xf81c0346 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d306): Mirror 3: 0x735dd6d4 (stale) Mirror 5: 0xf81c0346 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d306: skipped 000629 1772470897.241112 trunc from 0x028979 to 0x02967c (0xd04 bytes) 000631 1772470897.250470 trunc from 0x02d306 to 0x039814 (0xc50f bytes) 000632 1772470897.273909 write 0x00244c thru 0x011996 (0xf54b bytes) 000633 1772470897.285280 trunc from 0x039815 to 0x03cec3 (0x36af bytes) 000634 1772470897.306026 punch from 0x00c2c8 to 0x01523e (0x8f77 bytes) 000635 1772470897.330970 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cec4): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cec4): Mirror 3: 0x9aae72d2 (stale) Mirror 5: 0x8b1a81a4 (stale) CRC-32 checksum value for chunk [0, 0x3cec4): Mirror 1: 0x9917dc11 Mirror 6: 0x9917dc11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cec4: skipped 000636 1772470898.484317 mapread 0x036b02 thru 0x039524 (0x2a23 bytes) 000638 1772470898.563239 trunc from 0x003b7f to 0x008822 (0x4ca4 bytes) 000639 1772470898.677383 write 0x0205ad thru 0x02271c (0x2170 bytes) 000640 1772470898.717229 trunc from 0x03cec4 to 0x023493 (0x19a30 bytes) 000641 1772470898.771508 fallocate 0x0199df thru 0x0238f1 (0x9f13 bytes) 000642 1772470898.778681 punch from 0x014535 to 0x01b3a0 (0x6e6c bytes) 000643 1772470898.786811 punch from 0x00f213 to 0x011467 (0x2255 bytes) 000644 1772470898.792751 mapwrite 0x01e153 thru 0x025776 (0x7624 bytes) 000646 1772470898.840056 fallocate 0x004233 thru 0x00511b (0xee9 bytes) 000648 1772470898.847023 trunc from 0x025777 to 0x00ffc7 (0x157af bytes) 000649 1772470898.859283 mapread 0x00679f thru 0x00ffc7 (0x9829 bytes) 000650 1772470898.871984 trunc from 0x00ffc8 to 0x01e0fc (0xe135 bytes) 000651 1772470898.911308 punch from 0x0196a0 to 0x01e0fc (0x4a5d bytes) 000652 1772470898.950040 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1e0fd): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e0fd): Mirror 3: 0xf507de9e (stale) Mirror 5: 0xbd403687 (stale) Mirror 6: 0x23dd57f2 (stale) CRC-32 checksum value for chunk [0, 0x1e0fd): Mirror 1: 0xb14a33c5 Mirror 7: 0xb14a33c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e0fd: skipped 000653 1772470900.114951 fallocate 0x027638 thru 0x02e57c (0x6f45 bytes) 000654 1772470900.163583 trunc from 0x01e0fd to 0x03e510 (0x20414 bytes) 000655 1772470900.188823 write 0x01df1b thru 0x01f187 (0x126d bytes) 000656 1772470900.200398 punch from 0x03e4bd to 0x03e510 (0x54 bytes) 000657 1772470900.208804 mapwrite 0x02a6bd thru 0x034366 (0x9caa bytes) 000658 1772470900.262340 trunc from 0x02c00d to 0x0351e9 (0x91dd bytes) 000659 1772470900.270637 trunc from 0x03e511 to 0x011fc3 (0x2c54d bytes) 000660 1772470900.286728 trunc from 0x011fc4 to 0x021fb3 (0xfff0 bytes) 000661 1772470900.306702 trunc from 0x021fb4 to 0x014916 (0xd69d bytes) 000662 1772470900.320449 punch from 0x0024c5 to 0x010254 (0xdd90 bytes) 000663 1772470900.326883 write 0x000c3d thru 0x0027f3 (0x1bb7 bytes) 000664 1772470900.337262 punch from 0x001d8c to 0x002eb2 (0x1127 bytes) 000665 1772470900.349276 read 0x003660 thru 0x0078e4 (0x4285 bytes) 000666 1772470900.350179 mapread 0x00b654 thru 0x01458a (0x8f37 bytes) 000667 1772470900.361411 write 0x022b0a thru 0x031543 (0xea3a bytes) 000668 1772470900.364504 punch from 0x00ebd1 to 0x00fc4f (0x107f bytes) 000669 1772470900.368945 fallocate 0x030e89 thru 0x035047 (0x41bf bytes) 000671 1772470900.375338 trunc from 0x035048 to 0x020e78 (0x141cf bytes) 000673 1772470900.391688 trunc from 0x020e79 to 0x02c499 (0xb621 bytes) 000674 1772470900.411304 MIRROR_RESYNC 5 mirrors 000675 1772470901.275934 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000676 1772470901.600454 MIRROR_RESYNC 4 mirrors 000677 1772470901.996644 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000678 1772470902.296028 trunc from 0x00e87d to 0x012b74 (0x42f8 bytes) 000679 1772470902.343499 fallocate 0x005ca8 thru 0x00c37b (0x66d4 bytes) 000680 1772470902.348820 punch from 0x020643 to 0x02c499 (0xbe57 bytes) 000681 1772470902.354456 mapwrite 0x033926 thru 0x03ffff (0xc6da bytes) 000682 1772470902.423168 write 0x00889e thru 0x00b934 (0x3097 bytes) 000683 1772470902.435010 mapread 0x005a97 thru 0x00d825 (0x7d8f bytes) 000684 1772470902.436752 mapwrite 0x0279a5 thru 0x02ab2b (0x3187 bytes) 000685 1772470902.465291 trunc from 0x040000 to 0x01b0c3 (0x24f3c bytes) 000686 1772470902.482898 trunc from 0x00919c to 0x0169fe (0xd863 bytes) 000687 1772470902.489030 mapread 0x012c06 thru 0x0176fb (0x4af6 bytes) 000688 1772470902.499941 trunc from 0x000373 to 0x0042c2 (0x3f50 bytes) 000690 1772470902.506334 trunc from 0x0141f5 to 0x017875 (0x3681 bytes) 000691 1772470902.511983 write 0x00fec3 thru 0x01f67e (0xf7bc bytes) 000692 1772470902.515819 write 0x02247a thru 0x0274af (0x5036 bytes) 000693 1772470902.518394 trunc from 0x0274b0 to 0x03c1aa (0x14cfb bytes) 000694 1772470902.533429 trunc from 0x03c1ab to 0x02edc2 (0xd3e8 bytes) 000695 1772470902.547359 trunc from 0x02edc3 to 0x03fa39 (0x10c77 bytes) 000696 1772470902.569650 fallocate 0x0238c8 thru 0x026339 (0x2a72 bytes) 000697 1772470902.575384 punch from 0x0262d8 to 0x035e00 (0xfb29 bytes) 000698 1772470902.590763 fallocate 0x02b1af thru 0x03640c (0xb25e bytes) 000699 1772470902.596381 trunc from 0x03fa3a to 0x006090 (0x399a9 bytes) 000700 1772470902.612517 trunc from 0x001163 to 0x00543b (0x42d9 bytes) 000701 1772470902.618395 mapwrite 0x02237d thru 0x029fbe (0x7c42 bytes) 000702 1772470902.668538 mapwrite 0x022969 thru 0x025870 (0x2f08 bytes) 000703 1772470902.689014 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29fbf): Mirror 6: 0xb3eb9f39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29fbf): Mirror 6: 0xb3eb9f39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29fbf: skipped 000704 1772470902.965718 trunc from 0x0160d1 to 0x01bb6d (0x5a9d bytes) 000706 1772470902.972421 trunc from 0x029fbf to 0x00adeb (0x1f1d3 bytes) 000707 1772470902.992183 fallocate 0x02d6d3 thru 0x030fe7 (0x3915 bytes) 000708 1772470902.998669 trunc from 0x00adec to 0x03c893 (0x31aa8 bytes) 000709 1772470903.013542 punch from 0x0343bd to 0x03c893 (0x84d7 bytes) 000710 1772470903.020263 mapread 0x01c2da thru 0x022568 (0x628f bytes) 000711 1772470903.040616 mapread 0x02bf3c thru 0x032a70 (0x6b35 bytes) 000712 1772470903.052293 mapwrite 0x03d881 thru 0x03ffff (0x277f bytes) 000713 1772470903.100569 mapwrite 0x036d69 thru 0x03e0b5 (0x734d bytes) 000715 1772470903.143338 MIRROR_RESYNC 2 mirrors 000716 1772470903.972044 trunc from 0x02b5c3 to 0x03ae73 (0xf8b1 bytes) 000717 1772470904.011242 punch from 0x0219f1 to 0x0293e8 (0x79f8 bytes) 000718 1772470904.022263 trunc from 0x03e5d7 to 0x03ffff (0x1a29 bytes) 000719 1772470904.035473 write 0x0316de thru 0x036782 (0x50a5 bytes) 000720 1772470904.057964 mapwrite 0x031f3a thru 0x039635 (0x76fc bytes) 000721 1772470904.135579 fallocate 0x015ab2 thru 0x019604 (0x3b53 bytes) 000722 1772470904.144942 punch from 0x00730b to 0x00aeb9 (0x3baf bytes) 000723 1772470904.152799 mapread 0x02e66d thru 0x0363d7 (0x7d6b bytes) 000724 1772470904.166509 punch from 0x0149d8 to 0x02405d (0xf686 bytes) 000725 1772470904.171971 read 0x0251aa thru 0x033399 (0xe1f0 bytes) 000726 1772470904.182989 punch from 0x01c1ef to 0x029fbd (0xddcf bytes) 000727 1772470904.188756 mapread 0x022c5f thru 0x02cc8c (0xa02e bytes) 000728 1772470904.201221 MIRROR_RESYNC 2 mirrors 000729 1772470905.079396 trunc from 0x030f5c to 0x03a40b (0x94b0 bytes) 000730 1772470905.120863 read 0x03981a thru 0x03ffff (0x67e6 bytes) 000731 1772470905.137156 mapwrite 0x01e7f3 thru 0x01eaab (0x2b9 bytes) 000732 1772470905.155425 read 0x011ef7 thru 0x017845 (0x594f bytes) 000733 1772470905.194347 write 0x008882 thru 0x010e4f (0x85ce bytes) 000734 1772470905.195815 fallocate 0x0226bb thru 0x02d6bd (0xb003 bytes) 000735 1772470905.201751 write 0x023ff0 thru 0x026111 (0x2122 bytes) 000736 1772470905.203892 mapread 0x031724 thru 0x03f1c5 (0xdaa2 bytes) 000737 1772470905.205748 trunc from 0x040000 to 0x006b21 (0x394de bytes) 000738 1772470905.256429 read 0x0050b3 thru 0x006b21 (0x1a6f bytes) 000739 1772470905.257709 mapread 0x00414e thru 0x006b21 (0x29d4 bytes) 000741 1772470905.259945 read 0x0043df thru 0x006b21 (0x2743 bytes) 000742 1772470905.267315 MIRROR_RESYNC 2 mirrors 000743 1772470905.830989 trunc from 0x004c13 to 0x006b21 (0x1f0f bytes) 000744 1772470905.903913 trunc from 0x000b45 to 0x003fc2 (0x347e bytes) 000746 1772470905.911946 write 0x02acfe thru 0x03558e (0xa891 bytes) 000747 1772470905.916710 trunc from 0x01f08f to 0x02895b (0x98cd bytes) 000748 1772470905.934681 trunc from 0x03558f to 0x00bbb8 (0x299d6 bytes) 000749 1772470905.979245 trunc from 0x00bbb9 to 0x01a81c (0xec64 bytes) 000750 1772470906.001746 read 0x00bd7f thru 0x010bdd (0x4e5f bytes) 000752 1772470906.012351 read 0x001c7c thru 0x00ddab (0xc130 bytes) 000753 1772470906.022249 read 0x0085a2 thru 0x0124ae (0x9f0d bytes) 000754 1772470906.022845 write 0x02884f thru 0x0301e2 (0x7994 bytes) 000755 1772470906.024039 punch from 0x00a023 to 0x00d539 (0x3517 bytes) 000756 1772470906.027513 mapwrite 0x0298ee thru 0x039493 (0xfba6 bytes) 000757 1772470906.080566 read 0x02b249 thru 0x039493 (0xe24b bytes) 000759 1772470906.083926 write 0x034e78 thru 0x03ddc5 (0x8f4e bytes) 000760 1772470906.086903 mapread 0x0208bb thru 0x02b7f7 (0xaf3d bytes) 000761 1772470906.089000 write 0x015911 thru 0x01aeb3 (0x55a3 bytes) 000762 1772470906.096844 punch from 0x005f97 to 0x006ee4 (0xf4e bytes) 000763 1772470906.111951 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000764 1772470906.277964 punch from 0x020285 to 0x026e66 (0x6be2 bytes) skipping zero length punch hole 000766 1772470906.289131 trunc from 0x021b3e to 0x02433d (0x2800 bytes) 000767 1772470906.297227 fallocate 0x02b36b thru 0x03366c (0x8302 bytes) 000770 1772470906.312510 write 0x017e21 thru 0x024eba (0xd09a bytes) 000771 1772470906.330814 trunc from 0x03ddc6 to 0x0227fc (0x1b5c9 bytes) 000772 1772470906.353183 write 0x016988 thru 0x017157 (0x7d0 bytes) 000773 1772470906.373368 punch from 0x014a9c to 0x01fc1e (0xb183 bytes) 000775 1772470906.391429 mapread 0x0174eb thru 0x01d9af (0x64c5 bytes) 000776 1772470906.404415 mapread 0x00df0f thru 0x019df9 (0xbeeb bytes) 000778 1772470906.428381 MIRROR_EXTEND to 2 mirrors 000779 1772470907.118172 trunc from 0x0227fd to 0x0221a8 (0x654 bytes) 000780 1772470907.215531 punch from 0x007e1f to 0x01577a (0xd95c bytes) 000781 1772470907.221965 mapwrite 0x00c6bd thru 0x015030 (0x8974 bytes) 000782 1772470907.269845 MIRROR_RESYNC 2 mirrors 000783 1772470907.858466 trunc from 0x0221a9 to 0x021298 (0xf10 bytes) 000784 1772470907.960858 punch from 0x01e050 to 0x021298 (0x3249 bytes) 000785 1772470907.966701 fallocate 0x00f104 thru 0x012540 (0x343d bytes) 000786 1772470907.972677 trunc from 0x018f24 to 0x021298 (0x8375 bytes) 000787 1772470907.977698 read 0x02125a thru 0x021298 (0x3f bytes) 000788 1772470907.982990 read 0x012ccd thru 0x018cf7 (0x602b bytes) 000789 1772470908.009555 punch from 0x00c7ca to 0x00debc (0x16f3 bytes) 000790 1772470908.029532 MIRROR_RESYNC 2 mirrors 000791 1772470908.897047 trunc from 0x000880 to 0x0074c5 (0x6c46 bytes) 000792 1772470908.967640 mapread 0x013706 thru 0x01f0fe (0xb9f9 bytes) 000794 1772470909.018384 punch from 0x007161 to 0x008723 (0x15c3 bytes) 000795 1772470909.024932 fallocate 0x0224a0 thru 0x02b17d (0x8cde bytes) 000796 1772470909.036211 write 0x016ad6 thru 0x02461b (0xdb46 bytes) 000797 1772470909.040591 write 0x007463 thru 0x00b0ea (0x3c88 bytes) 000798 1772470909.057991 trunc from 0x02461c to 0x0146dc (0xff3f bytes) 000799 1772470909.130358 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000801 1772470909.394481 read 0x00f6a9 thru 0x0146dc (0x5034 bytes) 000802 1772470909.407271 mapread 0x007b9f thru 0x00995c (0x1dbe bytes) 000803 1772470909.430677 fallocate 0x0096f6 thru 0x011d0e (0x8619 bytes) 000804 1772470909.438866 mapread 0x0127a9 thru 0x0146dc (0x1f34 bytes) 000808 1772470909.446573 trunc from 0x011fca to 0x0146dc (0x2713 bytes) 000809 1772470909.453764 mapread 0x00f403 thru 0x0146dc (0x52da bytes) 000810 1772470909.461946 mapread 0x01233b thru 0x0146dc (0x23a2 bytes) 000811 1772470909.466438 punch from 0x001a46 to 0x005171 (0x372c bytes) 000812 1772470909.471947 mapwrite 0x01820f thru 0x0218a1 (0x9693 bytes) 000813 1772470909.526501 write 0x03ed76 thru 0x03f5d9 (0x864 bytes) 000815 1772470909.529842 punch from 0x012cc5 to 0x012fbf (0x2fb bytes) 000816 1772470909.536887 write 0x00c077 thru 0x016b09 (0xaa93 bytes) 000817 1772470909.538845 punch from 0x0235ef to 0x029da6 (0x67b8 bytes) 000818 1772470909.544765 punch from 0x006bca to 0x0151b5 (0xe5ec bytes) 000819 1772470909.558668 mapread 0x02e700 thru 0x030963 (0x2264 bytes) 000822 1772470909.570635 trunc from 0x01e802 to 0x01f1ea (0x9e9 bytes) 000823 1772470909.575766 fallocate 0x01d4b8 thru 0x02ab3d (0xd686 bytes) 000824 1772470909.580383 trunc from 0x0374ff to 0x03f5d9 (0x80db bytes) 000825 1772470909.587266 write 0x01cd87 thru 0x02a010 (0xd28a bytes) 000826 1772470909.590253 trunc from 0x028dfd to 0x02cef7 (0x40fb bytes) 000827 1772470909.607424 write 0x00877d thru 0x009525 (0xda9 bytes) 000828 1772470909.629999 punch from 0x03ae5c to 0x03cb7f (0x1d24 bytes) 000831 1772470909.637907 mapread 0x02e777 thru 0x033c92 (0x551c bytes) 000832 1772470909.639513 punch from 0x02e325 to 0x03923e (0xaf1a bytes) 000834 1772470909.646058 mapwrite 0x032cbf thru 0x037ebb (0x51fd bytes) 000835 1772470909.687519 punch from 0x03a960 to 0x03f5d9 (0x4c7a bytes) 000836 1772470909.691953 fallocate 0x00ced8 thru 0x01ad04 (0xde2d bytes) 000837 1772470909.696338 write 0x004369 thru 0x007cc6 (0x395e bytes) 000838 1772470909.697163 mapwrite 0x03bd49 thru 0x03ffff (0x42b7 bytes) 000839 1772470909.787212 write 0x031309 thru 0x0339cb (0x26c3 bytes) 000840 1772470909.788473 trunc from 0x00e18f to 0x014b3a (0x69ac bytes) 000841 1772470909.796708 punch from 0x03de64 to 0x03ffbd (0x215a bytes) 000842 1772470909.804153 trunc from 0x040000 to 0x0014e6 (0x3eb19 bytes) 000843 1772470909.824934 write 0x02b799 thru 0x03005f (0x48c7 bytes) 000844 1772470909.828837 mapread 0x028047 thru 0x02bc71 (0x3c2b bytes) 000845 1772470909.837129 read 0x00e4fb thru 0x0178eb (0x93f1 bytes) 000846 1772470909.850175 mapwrite 0x00e8b9 thru 0x01568d (0x6dd5 bytes) 000847 1772470909.877135 fallocate 0x0375c6 thru 0x03ffff (0x8a3a bytes) 000848 1772470909.885612 trunc from 0x012083 to 0x01dec3 (0xbe41 bytes) 000849 1772470909.892877 trunc from 0x009a70 to 0x00c06d (0x25fe bytes) 000850 1772470909.898723 write 0x017da0 thru 0x01a41d (0x267e bytes) 000852 1772470909.910160 mapwrite 0x026fca thru 0x02c72e (0x5765 bytes) 000853 1772470909.949260 trunc from 0x030060 to 0x026ca7 (0x93b8 bytes) 000854 1772470909.962745 fallocate 0x03fcd1 thru 0x03ffff (0x32f bytes) 000855 1772470909.966716 punch from 0x00a16c to 0x013fd2 (0x9e67 bytes) 000856 1772470909.977141 MIRROR_EXTEND to 2 mirrors 000858 1772470910.780461 punch from 0x007d17 to 0x014e6f (0xd159 bytes) 000859 1772470910.832595 fallocate 0x002b03 thru 0x00ec43 (0xc141 bytes) 000860 1772470910.846179 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26ca8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ca8): Mirror 2: 0xf3bafc46 (stale) CRC-32 checksum value for chunk [0, 0x26ca8): Mirror 1: 0xf3bafc46 Mirror 3: 0xf3bafc46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ca8: skipped 000861 1772470911.892129 MIRROR_RESYNC 3 mirrors 000863 1772470912.673528 mapwrite 0x010da8 thru 0x01ec4b (0xdea4 bytes) 000864 1772470912.807115 trunc from 0x026ca8 to 0x02ef4a (0x82a3 bytes) 000865 1772470912.836150 fallocate 0x010848 thru 0x0111de (0x997 bytes) 000866 1772470912.844156 mapread 0x01e680 thru 0x024c99 (0x661a bytes) 000867 1772470912.861923 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ef4b): Mirror 2: 0xdc274069 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef4b): Mirror 2: 0xdc274069 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef4b: skipped 000868 1772470913.208004 trunc from 0x02ef4b to 0x020774 (0xe7d6 bytes) 000870 1772470913.225013 trunc from 0x00a3ee to 0x00d10c (0x2d1f bytes) 000871 1772470913.229454 trunc from 0x020775 to 0x00a92b (0x15e49 bytes) 000872 1772470913.241265 write 0x02f25b thru 0x03bdba (0xcb60 bytes) 000873 1772470913.243942 trunc from 0x03bdbb to 0x01e03d (0x1dd7d bytes) 000874 1772470913.257510 mapwrite 0x024656 thru 0x03373b (0xf0e6 bytes) 000875 1772470913.403517 read 0x009e22 thru 0x012d03 (0x8ee2 bytes) 000876 1772470913.414972 mapwrite 0x0076c6 thru 0x00e86b (0x71a6 bytes) 000877 1772470913.451364 write 0x02ee05 thru 0x02f3a5 (0x5a1 bytes) 000878 1772470913.452612 trunc from 0x01231e to 0x01d9f6 (0xb6d9 bytes) 000879 1772470913.459727 mapwrite 0x0227f3 thru 0x02673b (0x3f49 bytes) 000880 1772470913.483056 fallocate 0x028006 thru 0x0320ef (0xa0ea bytes) 000881 1772470913.491219 read 0x024566 thru 0x03054f (0xbfea bytes) 000882 1772470913.495707 write 0x0151bd thru 0x0225ed (0xd431 bytes) 000883 1772470913.519939 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3373c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3373c): Mirror 2: 0xee7f44e9 (stale) CRC-32 checksum value for chunk [0, 0x3373c): Mirror 1: 0x8ab71367 Mirror 3: 0x8ab71367 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3373c: skipped 000884 1772470914.548597 mapwrite 0x02a1c9 thru 0x02b7a8 (0x15e0 bytes) 000885 1772470914.674464 mapwrite 0x009552 thru 0x01586e (0xc31d bytes) 000886 1772470914.733693 fallocate 0x01c865 thru 0x026942 (0xa0de bytes) 000887 1772470914.739633 trunc from 0x03373c to 0x0071e8 (0x2c553 bytes) 000888 1772470914.791156 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x71e9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x71e9): Mirror 2: 0xb9587436 (stale) Mirror 3: 0xb9587436 (stale) CRC-32 checksum value for chunk [0, 0x71e9): Mirror 1: 0xb9587436 Mirror 4: 0xb9587436 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x71e9: skipped 000889 1772470915.642201 trunc from 0x0071e9 to 0x03e00c (0x36e24 bytes) 000890 1772470915.720760 MIRROR_RESYNC 4 mirrors 000891 1772470916.798181 trunc from 0x03e00d to 0x0269cc (0x17640 bytes) 000892 1772470916.836904 punch from 0x00962e to 0x011d02 (0x86d5 bytes) 000894 1772470916.843695 read 0x025165 thru 0x0269cc (0x1868 bytes) 000895 1772470916.849859 punch from 0x0108a6 to 0x01ee6a (0xe5c5 bytes) 000896 1772470916.855482 read 0x001417 thru 0x00b9dc (0xa5c6 bytes) 000897 1772470916.872360 trunc from 0x01da6c to 0x01dd3f (0x2d4 bytes) 000898 1772470916.876750 write 0x022088 thru 0x02c547 (0xa4c0 bytes) 000899 1772470916.878994 fallocate 0x039e17 thru 0x03ffff (0x61e9 bytes) 000900 1772470916.884478 trunc from 0x009460 to 0x016847 (0xd3e8 bytes) 000901 1772470916.889546 punch from 0x0333e5 to 0x033a51 (0x66d bytes) 000902 1772470916.894712 trunc from 0x02ea1e to 0x03ba61 (0xd044 bytes) 000903 1772470916.900452 trunc from 0x040000 to 0x003efb (0x3c104 bytes) 000905 1772470916.938144 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3efc): Mirror 3: 0x5337ebea (stale) Mirror 4: 0x5337ebea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efc): Mirror 3: 0x5337ebea (stale) Mirror 4: 0x5337ebea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efc: skipped 000906 1772470917.359246 mapwrite 0x024dee thru 0x0298d7 (0x4aea bytes) 000907 1772470917.463296 mapread 0x029039 thru 0x0298d7 (0x89f bytes) 000908 1772470917.466492 write 0x005c0e thru 0x012ae8 (0xcedb bytes) 000909 1772470917.489924 trunc from 0x015344 to 0x021291 (0xbf4e bytes) 000910 1772470917.497167 write 0x0317c0 thru 0x037c9b (0x64dc bytes) 000912 1772470917.508420 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 4: 0x22be109e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 4: 0x22be109e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c9c: skipped 000913 1772470917.999339 punch from 0x03090b to 0x03106a (0x760 bytes) 000914 1772470918.008757 trunc from 0x02994b to 0x02fed4 (0x658a bytes) 000915 1772470918.016631 mapread 0x022010 thru 0x02df17 (0xbf08 bytes) 000917 1772470918.097678 trunc from 0x014434 to 0x0190cf (0x4c9c bytes) 000918 1772470918.108737 read 0x00e8c5 thru 0x0152b7 (0x69f3 bytes) 000919 1772470918.133953 write 0x0339cc thru 0x039f59 (0x658e bytes) 000920 1772470918.154727 fallocate 0x03e189 thru 0x03ffff (0x1e77 bytes) 000921 1772470918.169975 trunc from 0x040000 to 0x01cf83 (0x2307c bytes) 000922 1772470918.210789 trunc from 0x01cf84 to 0x00b8f2 (0x11691 bytes) 000923 1772470918.241381 mapwrite 0x0043d8 thru 0x008205 (0x3e2e bytes) 000924 1772470918.267652 punch from 0x000ad3 to 0x00b8f2 (0xae20 bytes) 000925 1772470918.272692 trunc from 0x0008cf to 0x00b8f2 (0xb024 bytes) 000927 1772470918.283238 MIRROR_RESYNC 2 mirrors 000928 1772470918.892451 MIRROR_EXTEND to 3 mirrors 000929 1772470920.088621 trunc from 0x00b8f3 to 0x00b9cd (0xdb bytes) 000930 1772470920.207179 fallocate 0x030b11 thru 0x0327c7 (0x1cb7 bytes) 000931 1772470920.216156 trunc from 0x02b118 to 0x0327c7 (0x76b0 bytes) 000932 1772470920.222279 trunc from 0x0327c8 to 0x031a16 (0xdb1 bytes) 000933 1772470920.247653 fallocate 0x02e163 thru 0x033343 (0x51e1 bytes) 000934 1772470920.254439 write 0x0256b0 thru 0x02ed9d (0x96ee bytes) 000935 1772470920.266332 trunc from 0x033344 to 0x01c918 (0x16a2b bytes) 000936 1772470920.283679 read 0x00b891 thru 0x011442 (0x5bb2 bytes) 000937 1772470920.298297 punch from 0x0129d3 to 0x019b0d (0x713b bytes) 000938 1772470920.307596 mapread 0x0125a3 thru 0x018768 (0x61c6 bytes) 000940 1772470920.323830 read 0x01374a thru 0x015e5e (0x2715 bytes) 000942 1772470920.327593 punch from 0x0012e3 to 0x0055a0 (0x42be bytes) 000943 1772470920.332535 fallocate 0x008326 thru 0x00d3cd (0x50a8 bytes) 000944 1772470920.337718 trunc from 0x013865 to 0x01c918 (0x90b4 bytes) 000945 1772470920.348396 MIRROR_RESYNC 3 mirrors 000946 1772470921.163135 punch from 0x000827 to 0x00eb5a (0xe334 bytes) 000947 1772470921.207244 mapread 0x014b5f thru 0x01bcef (0x7191 bytes) 000948 1772470921.234109 mapread 0x0108ef thru 0x01c918 (0xc02a bytes) 000949 1772470921.237098 trunc from 0x00371d to 0x007a6b (0x434f bytes) 000950 1772470921.243038 fallocate 0x021281 thru 0x02d44a (0xc1ca bytes) 000951 1772470921.249284 mapread 0x019a0f thru 0x01a671 (0xc63 bytes) 000952 1772470921.251189 punch from 0x012bbc to 0x01c918 (0x9d5d bytes) 000953 1772470921.256418 trunc from 0x014db4 to 0x01c918 (0x7b65 bytes) 000954 1772470921.262448 trunc from 0x00c5fc to 0x017ba5 (0xb5aa bytes) 000955 1772470921.268031 punch from 0x006dea to 0x015846 (0xea5d bytes) 000956 1772470921.274043 trunc from 0x01c919 to 0x006231 (0x166e7 bytes) 000957 1772470921.326890 read 0x004a3c thru 0x006231 (0x17f6 bytes) 000958 1772470921.335876 trunc from 0x006232 to 0x01066d (0xa43c bytes) 000962 1772470921.361629 fallocate 0x033c9b thru 0x03ffff (0xc365 bytes) 000963 1772470921.375352 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 6: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000965 1772470922.370382 trunc from 0x006b5f to 0x015c68 (0xf10a bytes) 000966 1772470922.493050 fallocate 0x03c695 thru 0x03de15 (0x1781 bytes) 000967 1772470922.503446 mapread 0x0107e7 thru 0x01f462 (0xec7c bytes) 000968 1772470922.584654 read 0x007199 thru 0x0142d3 (0xd13b bytes) 000969 1772470922.607413 punch from 0x0218e6 to 0x027d49 (0x6464 bytes) 000970 1772470922.626888 MIRROR_RESYNC 4 mirrors 000971 1772470923.889721 trunc from 0x017aa0 to 0x01c53b (0x4a9c bytes) 000972 1772470923.949600 trunc from 0x040000 to 0x01f93e (0x206c1 bytes) 000974 1772470924.005023 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f93f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f93f): Mirror 4: 0x7306a4 (stale) Mirror 5: 0x7306a4 (stale) Mirror 6: 0x7306a4 (stale) CRC-32 checksum value for chunk [0, 0x1f93f): Mirror 1: 0x7306a4 Mirror 7: 0x7306a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f93f: skipped 000975 1772470925.130142 punch from 0x00d04b to 0x00e318 (0x12ce bytes) 000976 1772470925.169056 read 0x012562 thru 0x01c5f5 (0xa094 bytes) 000977 1772470925.191609 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1f93f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f93f): Mirror 4: 0x7306a4 (stale) Mirror 5: 0x7306a4 (stale) Mirror 6: 0x7306a4 (stale) Mirror 7: 0x7306a4 (stale) CRC-32 checksum value for chunk [0, 0x1f93f): Mirror 1: 0x7306a4 Mirror 8: 0x7306a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f93f: skipped 000978 1772470926.584048 mapread 0x010206 thru 0x01979d (0x9598 bytes) 000979 1772470926.640435 trunc from 0x01f93f to 0x03d8c1 (0x1df83 bytes) 000980 1772470926.733597 trunc from 0x01bef6 to 0x028465 (0xc570 bytes) 000981 1772470926.742335 read 0x03540b thru 0x03d8c1 (0x84b7 bytes) 000982 1772470926.753715 mapread 0x03bc1c thru 0x03d8c1 (0x1ca6 bytes) 000983 1772470926.761590 MIRROR_RESYNC 6 mirrors 000984 1772470928.427680 mapwrite 0x02f77a thru 0x03b9fe (0xc285 bytes) 000985 1772470928.611623 trunc from 0x038759 to 0x03b522 (0x2dca bytes) 000987 1772470928.625642 write 0x00aea0 thru 0x017533 (0xc694 bytes) 000988 1772470928.709256 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d8c2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8c2): Mirror 4: 0x87700ad7 (stale) Mirror 5: 0x87700ad7 (stale) Mirror 6: 0x87700ad7 (stale) Mirror 7: 0x87700ad7 (stale) Mirror 8: 0x87700ad7 (stale) CRC-32 checksum value for chunk [0, 0x3d8c2): Mirror 1: 0x9900dc23 Mirror 9: 0x9900dc23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8c2: skipped 000989 1772470931.006489 mapread 0x03a392 thru 0x03cc75 (0x28e4 bytes) 000990 1772470931.061300 write 0x024839 thru 0x030846 (0xc00e bytes) 000991 1772470931.145266 mapread 0x032a2e thru 0x03d8c1 (0xae94 bytes) 000992 1772470931.193235 write 0x02c26f thru 0x03688f (0xa621 bytes) 000993 1772470931.198217 punch from 0x00501d to 0x00b749 (0x672d bytes) 000994 1772470931.229561 trunc from 0x03d8c2 to 0x02f793 (0xe12e bytes) 000995 1772470931.292085 trunc from 0x02497a to 0x02a59d (0x5c24 bytes) 000996 1772470931.333277 MIRROR_RESYNC 7 mirrors 000997 1772470934.101146 trunc from 0x02f794 to 0x02968f (0x6104 bytes) 000998 1772470934.157648 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29690): Mirror 4: 0x749a0c08 (stale) Mirror 5: 0x749a0c08 (stale) Mirror 6: 0x749a0c08 (stale) Mirror 7: 0x749a0c08 (stale) Mirror 8: 0x749a0c08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29690): Mirror 4: 0x749a0c08 (stale) Mirror 5: 0x749a0c08 (stale) Mirror 6: 0x749a0c08 (stale) Mirror 7: 0x749a0c08 (stale) Mirror 8: 0x749a0c08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29690: skipped 000999 1772470934.681452 trunc from 0x029690 to 0x03754e (0xdebf bytes) 001000 1772470934.710230 trunc from 0x02733d to 0x030ddd (0x9aa1 bytes) All operations completed A-OK! PASS 70a (99s) == sanity-flr test 71: check mirror extend parent fid ==== 12:02:20 (1772470940) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1ae:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 12:02:33 (1772470953) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189072 s, 11.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129711 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 12:02:42 (1772470962) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre2 Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7410592 .. Extending file size to 5663424 .. done Extending file size to 7840256 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7561184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9063936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8327040 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8599584 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8746464 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59778 59779 59781 59783 59784 done 192.168.203.115@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre2 (opts:) 192.168.203.115@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:04:10 (1772471050) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre2 Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 1631968 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2858784 .. Extending file size to 8517888 .. Extending file size to 6508352 .. Extending file size to 8794080 .. Extending file size to 8945152 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8606240 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61356 61358 61359 61360 61362 failed 192.168.203.115@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre2 (opts:) 192.168.203.115@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:05:35 (1772471135) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1269096 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2156 1269148 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24284 3582736 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18696 3588324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42980 7171060 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1269096 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2156 1269148 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24284 3582736 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18696 3588324 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42980 7171060 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223236 s, 9.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240174 s, 8.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9c3ca3ba (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.115@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:05:57 (1772471157) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100371 s, 20.9 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 12:06:07 (1772471167) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:06:16 (1772471176) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117387 s, 17.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:06:25 (1772471185) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174514 s, 12.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:06:35 (1772471195) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170443 s, 12.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:07:10 (1772471230) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg315-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114563 s, 9.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:08:05 (1772471285) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg315-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10608 s, 19.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:09:01 (1772471341) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg315-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg315-server: Pool lustre.test_204f created Adding targets to pool oleg315-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg315-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169827 s, 12.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg315-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg315-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg315-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:10:22 (1772471422) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:10:32 (1772471432) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:10:40 (1772471440) oleg315-server: Pool lustre.test_206 created Adding targets to pool oleg315-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg315-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg315-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg315-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg315-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:11:10 (1772471470) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114675 s, 9.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102464 s, 10.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb12ca6e3 Mirror 3: 0xb12ca6e3 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:11:23 (1772471483) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:11:27 (1772471487) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:11:30 (1772471490) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0745757 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23c0000 Read this (4 bytes): BBBB 192.168.203.115@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:12:13 (1772471533) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.032994 s, 248 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:12:20 (1772471540) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544933 s, 19.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611269 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 12:12:22.000000000 -0500 Modify: 2026-03-02 12:12:22.000000000 -0500 Change: 2026-03-02 12:12:23.000000000 -0500 Birth: 2026-03-02 12:12:22.000000000 -0500 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:12:29 (1772471549) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055754 s, 18.8 MB/s oleg315-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg315-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg315-server: O/280000401/d30/be: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:12:50 (1772471570) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.200598 s, 204 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181178 s, 226 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 2 12:12 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 2 12:12 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:12:58 (1772471578) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:13:06 (1772471586) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1938 sec =========== 12:13:16 (1772471596) === sanity-flr: start cleanup 12:13:18 (1772471598) === === sanity-flr: finish cleanup 12:13:27 (1772471607) ===