-----============= acceptance-small: sanity-flr ============----- Mon Aug 31 08:21:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg332-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:22:12 (1788178932) === oleg332-client.virtnet: executing check_config_client /mnt/lustre oleg332-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg332-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9592c5064000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9592c5064000.idle_timeout=debug disable quota as required oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:22:46 (1788178966) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8227] == sanity-flr test 0a: lfs mirror create with -N option == 08:22:48 (1788178968) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:23:03 (1788178983) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:23:06 (1788178986) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:23:10 (1788178990) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:23:32 (1788179012) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:23:35 (1788179015) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 08:23:38 (1788179018) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9592c5064000.stats=clear osc.lustre-OST0001-osc-ffff9592c5064000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:23:47 (1788179027) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:23:55 (1788179035) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:24:02 (1788179042) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:24:05 (1788179045) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:24:12 (1788179052) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 08:24:20 (1788179060) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:24:27 (1788179067) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:24:35 (1788179075) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:24:45 (1788179085) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:24:57 (1788179097) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:25:03 (1788179103) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118228 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088184 s, 11.9 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:25:10 (1788179110) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.190636 s, 27.5 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:25:17 (1788179117) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:25:25 (1788179125) Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (63s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:26:28 (1788179188) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.467383 s, 22.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:27:05 (1788179225) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (69s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:28:15 (1788179295) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.86727 s, 18.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 31 08:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.558s user 0m0.000s sys 0m0.154s Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9592c5064000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.140s user 0m0.000s sys 0m0.170s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.995s user 0m0.001s sys 0m0.171s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg332-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9592c5064000.ost_server_uuid in FULL state after 0 sec PASS 33b (82s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:29:37 (1788179377) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:29:39 (1788179379) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:29:42 (1788179382) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 08:29:44 (1788179384) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:29:51 (1788179391) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.641303 s, 19.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:30:04 (1788179404) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344195 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253654 s, 41.3 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:30:11 (1788179411) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:30:19 (1788179419) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000715046 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119722 s, 855 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011812 s, 867 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109763 s, 933 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 08:30:28 (1788179428) create test files with size 15730748 .. cd5926fc4f570e339eb35755ea99cb71 - bec994b514867d8dfc375f2f56391b07 - 9340e7e11857a960f8ff325f61bd2b17 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 08:30:51 (1788179451) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:31:04 (1788179464) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:31:10 (1788179470) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799045 s, 13.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277233 s, 37.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000400:0xd1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 08:31:18 (1788179478) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122349 s, 17.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.214313 s, 19.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102959 s, 20.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 08:31:31 (1788179491) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:31:34 (1788179494) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:31:38 (1788179498) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228289 s, 45.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:31:46 (1788179506) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.224566 s, 14.0 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:31:55 (1788179515) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xda:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:32:04 (1788179524) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.315423 s, 33.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:32:14 (1788179534) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9592c8699000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.36925 s, 28.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9592c42a1000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:32:29 (1788179549) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:33:03 (1788179583) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:33:37 (1788179617) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:33:40 (1788179620) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.240658 s, 13.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.134615 s, 23.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf6174f0b (nosync,stale) Mirror 2: 0xf6174f0b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2108a5883bfe06806d630003e161a467 - mirror 1 checksum: c74cef6fa3b25a2a49436335936795f8 - mirror 2 checksum: 2108a5883bfe06806d630003e161a467 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T08:33:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T08:33:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2108a5883bfe06806d630003e161a467 - mirror 1 checksum: c74cef6fa3b25a2a49436335936795f8 - mirror 2 checksum: c74cef6fa3b25a2a49436335936795f8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T08:33:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T08:33:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:33:49 (1788179629) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643661 s, 16.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:33:56 (1788179636) Failing mds1 on oleg332-server Stopping /mnt/lustre-mds1 (opts:) on oleg332-server 08:34:08 (1788179648) shut down facet: mds1 facet_host: oleg332-server facet_failover_host: oleg332-server Failover mds1 to oleg332-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 08:34:31 (1788179671) targets are mounted 08:34:31 (1788179671) facet_failover done oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212323 s, 49.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:34:49 (1788179689) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0310886 s, 659 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.011575 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:34:58 (1788179698) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0470764 s, 435 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00812944 s, 2.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:35:07 (1788179707) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0135269 s, 1.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:35:15 (1788179715) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0476881 s, 429 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:35:23 (1788179723) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00943456 s, 434 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0204168 s, 201 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:35:30 (1788179730) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:35:50 (1788179750) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:36:16 (1788179776) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:36:40 (1788179800) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:37:03 (1788179823) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 08:37:10 (1788179830) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57536 mirror split pid 57538 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57536 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57538 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 08:38:17 (1788179897) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 9898 skipping zero length zero range skipping zero length punch hole skipping zero size read truncating to largest ever: 0xec20 000005 1788179899.066536 trunc from 00000000 to 0x00ec1f (0xec20 bytes) 000006 1788179899.084859 read 0x00401b thru 0x00d459 (0x943f bytes) 000007 1788179899.099322 write 0x000077 thru 0x00c9ce (0xc958 bytes) 000009 1788179899.107226 mapwrite 0x004158 thru 0x012bef (0xea98 bytes) 000010 1788179899.159428 trunc from 0x009976 to 0x00a47c (0xb07 bytes) 000011 1788179899.164771 mapread 0x0120cb thru 0x012bef (0xb25 bytes) 000012 1788179899.166332 write 0x000452 thru 0x000c90 (0x83f bytes) 000013 1788179899.167974 read 0x0114ec thru 0x012bef (0x1704 bytes) 000014 1788179899.168879 trunc from 0x000248 to 0x0022b4 (0x206d bytes) truncating to largest ever: 0x35fc5 000015 1788179899.181323 trunc from 0x012bf0 to 0x035fc4 (0x233d5 bytes) 000016 1788179899.196293 mapwrite 0x01a3a3 thru 0x01d59c (0x31fa bytes) 000017 1788179899.220945 trunc from 0x035fc5 to 0x00d211 (0x28db3 bytes) fallocating to largest ever: 0x3f53f 000018 1788179899.247524 fallocate 0x03087b thru 0x03f53e (0xecc4 bytes) 000019 1788179899.252302 mapread 0x00c017 thru 0x00d211 (0x11fb bytes) 000020 1788179899.255906 read 0x00a297 thru 0x00d211 (0x2f7b bytes) 000021 1788179899.256995 fallocate 0x01822d thru 0x026706 (0xe4da bytes) 000022 1788179899.261916 read 0x00bc3a thru 0x00c019 (0x3e0 bytes) 000023 1788179899.263255 write 0x008e2a thru 0x00e8ad (0x5a84 bytes) 000025 1788179899.266263 write 0x038e92 thru 0x03d74a (0x48b9 bytes) fallocating to largest ever: 0x40000 000026 1788179899.268311 fallocate 0x0353eb thru 0x03ffff (0xac15 bytes) 000027 1788179899.272830 trunc from 0x008f08 to 0x0154b0 (0xc5a9 bytes) 000031 1788179899.297709 MIRROR_EXTEND to 2 mirrors 000032 1788179900.021994 MIRROR_EXTEND to 3 mirrors 000034 1788179900.752666 mapread 0x02d435 thru 0x034ba6 (0x7772 bytes) 000035 1788179900.793364 trunc from 0x0051bc to 0x0051c4 (0x9 bytes) 000036 1788179900.851763 write 0x01aaa5 thru 0x027c73 (0xd1cf bytes) 000037 1788179900.880912 write 0x007c89 thru 0x016b4e (0xeec6 bytes) 000038 1788179900.900199 write 0x02d394 thru 0x02d605 (0x272 bytes) 000039 1788179900.911399 trunc from 0x012064 to 0x01f844 (0xd7e1 bytes) 000040 1788179900.955506 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xec112f19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xec112f19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000041 1788179901.262743 mapwrite 0x01d2c2 thru 0x023918 (0x6657 bytes) 000042 1788179901.344650 trunc from 0x01ca4d to 0x01f185 (0x2739 bytes) 000043 1788179901.353383 write 0x02d4db thru 0x02f5ae (0x20d4 bytes) 000044 1788179901.376594 mapwrite 0x03c4b9 thru 0x03ffff (0x3b47 bytes) 000045 1788179901.418922 trunc from 0x0306e2 to 0x03b224 (0xab43 bytes) 000046 1788179901.427056 mapwrite 0x012d75 thru 0x016ea7 (0x4133 bytes) 000047 1788179901.458491 read 0x02cc72 thru 0x0351eb (0x857a bytes) 000048 1788179901.471814 trunc from 0x040000 to 0x0368ef (0x9710 bytes) 000049 1788179901.492542 trunc from 0x0368f0 to 0x020be9 (0x15d06 bytes) 000050 1788179901.511903 mapread 0x00dca4 thru 0x015269 (0x75c6 bytes) 000051 1788179901.520150 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20bea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20bea): Mirror 2: 0xe7b8e6c6 (stale) CRC-32 checksum value for chunk [0, 0x20bea): Mirror 1: 0xe99b69a3 Mirror 3: 0xe99b69a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20bea: skipped 000052 1788179902.316601 punch from 0x003345 to 0x004e97 (0x1b53 bytes) 000053 1788179902.377310 trunc from 0x020bea to 0x03eb5b (0x1df72 bytes) 000054 1788179902.439282 write 0x002011 thru 0x00fd28 (0xdd18 bytes) 000055 1788179902.455335 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3eb5c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb5c): Mirror 2: 0x9414d6f6 (stale) Mirror 3: 0x1b59679e (stale) CRC-32 checksum value for chunk [0, 0x3eb5c): Mirror 1: 0xb11b70e3 Mirror 4: 0xb11b70e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb5c: skipped 000056 1788179903.435717 mapread 0x00890b thru 0x01113d (0x8833 bytes) 000057 1788179903.474881 mapread 0x025580 thru 0x027530 (0x1fb1 bytes) 000058 1788179903.509768 trunc from 0x004114 to 0x00ffc0 (0xbead bytes) 000059 1788179903.576310 trunc from 0x00a3ae to 0x0105f5 (0x6248 bytes) 000060 1788179903.586314 fallocate 0x0053e7 thru 0x00e262 (0x8e7c bytes) 000061 1788179903.592986 mapwrite 0x030bae thru 0x03c425 (0xb878 bytes) 000062 1788179903.637424 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb5c): Mirror 2: 0x9414d6f6 (stale) Mirror 4: 0xb11b70e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb5c): Mirror 2: 0x9414d6f6 (stale) Mirror 4: 0xb11b70e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb5c: skipped 000063 1788179903.974296 MIRROR_RESYNC 3 mirrors 000064 1788179905.237210 mapread 0x03ddb2 thru 0x03e85c (0xaab bytes) 000065 1788179905.251180 trunc from 0x010949 to 0x01cbd3 (0xc28b bytes) 000067 1788179905.285589 write 0x008265 thru 0x00e694 (0x6430 bytes) 000068 1788179905.295994 trunc from 0x01b178 to 0x02a5ab (0xf434 bytes) 000069 1788179905.300596 trunc from 0x027b47 to 0x0327cd (0xac87 bytes) 000070 1788179905.305220 read 0x008c07 thru 0x0160f1 (0xd4eb bytes) 000072 1788179905.318474 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb5c): Mirror 4: 0xb282c74b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb5c): Mirror 4: 0xb282c74b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb5c: skipped 000073 1788179905.620871 fallocate 0x028f01 thru 0x030d92 (0x7e92 bytes) 000074 1788179905.628979 write 0x015b69 thru 0x01d032 (0x74ca bytes) 000075 1788179905.655043 MIRROR_RESYNC 2 mirrors 000076 1788179906.627548 trunc from 0x002c60 to 0x010396 (0xd737 bytes) 000077 1788179906.692639 mapread 0x010301 thru 0x0116e2 (0x13e2 bytes) 000078 1788179906.708987 read 0x010de7 thru 0x018c4c (0x7e66 bytes) 000079 1788179906.728952 mapwrite 0x02e423 thru 0x03b9a0 (0xd57e bytes) 000080 1788179906.784150 trunc from 0x03bb36 to 0x03eb5b (0x3026 bytes) 000081 1788179906.789818 mapwrite 0x038a55 thru 0x03e7d9 (0x5d85 bytes) 000082 1788179906.813797 mapwrite 0x03c45d thru 0x03ce8d (0xa31 bytes) 000083 1788179906.824983 trunc from 0x021624 to 0x0236c2 (0x209f bytes) 000084 1788179906.829675 trunc from 0x03eb5c to 0x0013f4 (0x3d767 bytes) 000085 1788179906.853911 trunc from 0x0013f5 to 0x00d064 (0xbc70 bytes) 000086 1788179906.874018 punch from 0x008658 to 0x00a9ef (0x2398 bytes) 000088 1788179906.880115 write 0x00b20a thru 0x0126a9 (0x74a0 bytes) 000089 1788179906.885360 mapread 0x001fef thru 0x00d922 (0xb934 bytes) 000090 1788179906.899612 mapwrite 0x005322 thru 0x00f69d (0xa37c bytes) 000091 1788179906.920611 mapwrite 0x0255e2 thru 0x03105b (0xba7a bytes) 000092 1788179906.973490 read 0x00731f thru 0x00f4f1 (0x81d3 bytes) 000093 1788179906.977065 trunc from 0x02c599 to 0x03105b (0x4ac3 bytes) 000094 1788179906.984030 mapread 0x02edd7 thru 0x03105b (0x2285 bytes) 000095 1788179907.005137 MIRROR_RESYNC 2 mirrors 000096 1788179907.849290 read 0x00e72f thru 0x01a945 (0xc217 bytes) 000097 1788179907.866999 trunc from 0x03105c to 0x02b17a (0x5ee1 bytes) 000098 1788179907.911666 read 0x01487c thru 0x02060e (0xbd93 bytes) 000099 1788179907.930697 mapwrite 0x003f90 thru 0x005b4c (0x1bbd bytes) 000100 1788179907.963918 mapread 0x01b383 thru 0x025c42 (0xa8c0 bytes) 000101 1788179907.968754 trunc from 0x02b17b to 0x037067 (0xbeed bytes) 000102 1788179908.014327 mapread 0x01d0af thru 0x021005 (0x3f57 bytes) 000103 1788179908.015472 mapwrite 0x038a6b thru 0x03ffff (0x7595 bytes) 000104 1788179908.110619 punch from 0x01d234 to 0x01db5f (0x92c bytes) 000106 1788179908.116893 write 0x0124c7 thru 0x021e98 (0xf9d2 bytes) 000107 1788179908.119985 fallocate 0x0086bd thru 0x00dcb3 (0x55f7 bytes) 000108 1788179908.125008 mapread 0x00c2e4 thru 0x0126de (0x63fb bytes) 000109 1788179908.126958 mapread 0x002ef3 thru 0x007c6c (0x4d7a bytes) 000110 1788179908.128624 mapwrite 0x00aacc thru 0x0187ec (0xdd21 bytes) 000111 1788179908.157341 trunc from 0x040000 to 0x03510f (0xaef0 bytes) 000112 1788179908.175361 trunc from 0x01f390 to 0x02aeee (0xbb5f bytes) 000113 1788179908.182366 read 0x02f764 thru 0x02ff06 (0x7a3 bytes) 000114 1788179908.186121 trunc from 0x035110 to 0x029c8a (0xb485 bytes) 000115 1788179908.211565 write 0x009958 thru 0x00e1e5 (0x488e bytes) 000116 1788179908.212749 punch from 0x020d25 to 0x029c8a (0x8f66 bytes) 000117 1788179908.217767 mapwrite 0x012483 thru 0x01565a (0x31d8 bytes) 000118 1788179908.238878 punch from 0x002927 to 0x00f3d3 (0xcaad bytes) 000119 1788179908.246937 mapwrite 0x01f848 thru 0x02375c (0x3f15 bytes) 000120 1788179908.292641 trunc from 0x025ce9 to 0x0292a3 (0x35bb bytes) 000121 1788179908.325494 read 0x011c1d thru 0x01c13e (0xa522 bytes) 000122 1788179908.328185 trunc from 0x011cd4 to 0x0181a3 (0x64d0 bytes) 000123 1788179908.336166 punch from 0x01a154 to 0x0216b3 (0x7560 bytes) 000124 1788179908.344716 trunc from 0x01fa6e to 0x029c8a (0xa21d bytes) 000125 1788179908.352377 mapwrite 0x00af52 thru 0x0118de (0x698d bytes) 000126 1788179908.392903 mapwrite 0x02eb6f thru 0x03ca94 (0xdf26 bytes) 000127 1788179908.469913 punch from 0x023ca0 to 0x027f22 (0x4283 bytes) 000128 1788179908.476318 mapwrite 0x018735 thru 0x02805a (0xf926 bytes) 000129 1788179908.527914 punch from 0x007bd6 to 0x016d9c (0xf1c7 bytes) 000130 1788179908.540066 trunc from 0x03ca95 to 0x03c12a (0x96a bytes) 000131 1788179908.568056 write 0x01d0af thru 0x024c0b (0x7b5d bytes) 000132 1788179908.569632 mapwrite 0x00a471 thru 0x019507 (0xf097 bytes) 000133 1788179908.715137 trunc from 0x032a35 to 0x03c12a (0x96f6 bytes) 000134 1788179908.727510 trunc from 0x034f61 to 0x035ada (0xb7a bytes) 000135 1788179908.735393 punch from 0x01b403 to 0x024979 (0x9577 bytes) 000137 1788179908.745276 punch from 0x009f5a to 0x00f6a5 (0x574c bytes) 000138 1788179908.756181 punch from 0x023f60 to 0x033004 (0xf0a5 bytes) 000139 1788179908.796134 MIRROR_RESYNC 2 mirrors 000140 1788179909.558818 MIRROR_RESYNC 2 mirrors 000141 1788179909.848168 mapwrite 0x02e33c thru 0x036fdc (0x8ca1 bytes) 000142 1788179909.943855 punch from 0x00df49 to 0x011a3a (0x3af2 bytes) 000143 1788179909.949855 read 0x02a996 thru 0x038572 (0xdbdd bytes) 000144 1788179909.955647 write 0x01359c thru 0x01e6f4 (0xb159 bytes) 000145 1788179909.964995 trunc from 0x03c12b to 0x002153 (0x39fd7 bytes) 000146 1788179909.992884 read 0x001ee0 thru 0x002153 (0x274 bytes) 000147 1788179910.003678 mapwrite 0x00b7a4 thru 0x011da4 (0x6601 bytes) 000148 1788179910.063793 trunc from 0x0031d7 to 0x01085d (0xd687 bytes) 000149 1788179910.075514 read 0x009417 thru 0x00fb93 (0x677d bytes) 000150 1788179910.098357 read 0x00fa61 thru 0x011da4 (0x2344 bytes) 000151 1788179910.102428 mapread 0x0048a7 thru 0x00dd35 (0x948f bytes) 000153 1788179910.105204 trunc from 0x011da5 to 0x021c28 (0xfe84 bytes) 000154 1788179910.129746 MIRROR_RESYNC 2 mirrors 000155 1788179910.726290 read 0x0080ab thru 0x017d38 (0xfc8e bytes) 000156 1788179910.751522 mapwrite 0x02376e thru 0x02faa1 (0xc334 bytes) 000157 1788179910.946018 punch from 0x01577e to 0x017a99 (0x231c bytes) 000158 1788179910.952836 fallocate 0x006037 thru 0x00607c (0x46 bytes) 000159 1788179910.958250 mapwrite 0x02a86b thru 0x02a935 (0xcb bytes) 000160 1788179910.973296 punch from 0x01f081 to 0x029a90 (0xaa10 bytes) 000161 1788179910.980318 write 0x02f757 thru 0x03ccfb (0xd5a5 bytes) 000162 1788179910.982395 trunc from 0x03c558 to 0x03ccfb (0x7a4 bytes) 000163 1788179911.012570 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ccfc): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ccfc): Mirror 4: 0x8605a305 (stale) CRC-32 checksum value for chunk [0, 0x3ccfc): Mirror 1: 0x65e34398 Mirror 5: 0x65e34398 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ccfc: skipped 000164 1788179911.895787 fallocate 0x012dad thru 0x01699d (0x3bf1 bytes) 000165 1788179911.959817 write 0x018398 thru 0x01c1aa (0x3e13 bytes) 000166 1788179911.975374 write 0x002c9a thru 0x010da3 (0xe10a bytes) 000167 1788179911.998864 punch from 0x02d39f to 0x034fd4 (0x7c36 bytes) 000168 1788179912.007315 mapread 0x0227f8 thru 0x02b2f6 (0x8aff bytes) 000169 1788179912.034632 read 0x002c7e thru 0x00e03c (0xb3bf bytes) 000170 1788179912.035805 fallocate 0x002599 thru 0x00d94c (0xb3b4 bytes) 000171 1788179912.063218 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ccfc): Mirror 5: 0x65e34398 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ccfc): Mirror 5: 0x65e34398 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ccfc: skipped 000172 1788179912.360265 fallocate 0x03d702 thru 0x03ffff (0x28fe bytes) 000173 1788179912.367685 write 0x01f7ef thru 0x02f520 (0xfd32 bytes) 000174 1788179912.385828 fallocate 0x007dd3 thru 0x00c2c1 (0x44ef bytes) 000175 1788179912.392178 mapread 0x022226 thru 0x0300c5 (0xdea0 bytes) 000177 1788179912.399881 punch from 0x024916 to 0x030f54 (0xc63f bytes) 000178 1788179912.413575 trunc from 0x00d309 to 0x0145a6 (0x729e bytes) 000179 1788179912.420016 read 0x00e651 thru 0x011f61 (0x3911 bytes) 000180 1788179912.426346 write 0x022baf thru 0x02c96e (0x9dc0 bytes) 000181 1788179912.437228 trunc from 0x03ccfc to 0x02847c (0x1487f bytes) 000182 1788179912.463246 MIRROR_RESYNC 2 mirrors 000184 1788179913.332387 fallocate 0x016e55 thru 0x01b3d2 (0x457e bytes) 000185 1788179913.362247 read 0x00b8f3 thru 0x013dc8 (0x84d6 bytes) 000186 1788179913.369467 trunc from 0x00a4ac to 0x00d18f (0x2ce4 bytes) 000187 1788179913.376536 mapwrite 0x03c3e8 thru 0x03ffff (0x3c18 bytes) 000188 1788179913.450457 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xda9de13c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d9c03be Mirror 6: 0x7d9c03be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1788179914.479864 read 0x023e79 thru 0x029bf2 (0x5d7a bytes) 000190 1788179914.493185 punch from 0x01d523 to 0x02367f (0x615d bytes) 000191 1788179914.543383 punch from 0x014a24 to 0x01aa59 (0x6036 bytes) 000192 1788179914.553179 trunc from 0x027cdf to 0x02c7fa (0x4b1c bytes) 000193 1788179914.563949 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xda9de13c (stale) Mirror 6: 0x7d9c03be (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xed381704 Mirror 7: 0xed381704 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000195 1788179915.589263 punch from 0x02187d to 0x023ef9 (0x267d bytes) 000196 1788179915.630962 read 0x00e7fd thru 0x013e18 (0x561c bytes) 000198 1788179915.641895 punch from 0x0251a7 to 0x02708e (0x1ee8 bytes) 000199 1788179915.648542 mapwrite 0x012a49 thru 0x01fe4b (0xd403 bytes) 000200 1788179915.727705 mapread 0x01e9f1 thru 0x02e304 (0xf914 bytes) 000201 1788179915.751666 punch from 0x005341 to 0x008643 (0x3303 bytes) 000202 1788179915.759713 write 0x004885 thru 0x01345e (0xebda bytes) 000203 1788179915.779594 fallocate 0x0164da thru 0x01da72 (0x7599 bytes) 000206 1788179915.803635 trunc from 0x040000 to 0x01d01f (0x22fe0 bytes) 000207 1788179915.832383 trunc from 0x01d020 to 0x034ea4 (0x17e85 bytes) 000208 1788179915.856786 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34ea5): Mirror 5: 0x4eb040ff (stale) Mirror 7: 0x38d9fa1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ea5): Mirror 5: 0x4eb040ff (stale) Mirror 7: 0x38d9fa1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ea5: skipped 000210 1788179916.190661 punch from 0x0018ea to 0x00dad7 (0xc1ee bytes) 000211 1788179916.195350 trunc from 0x034ea5 to 0x0338c1 (0x15e3 bytes) 000212 1788179916.217680 trunc from 0x01f781 to 0x028446 (0x8cc6 bytes) 000213 1788179916.240608 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x338c2): Mirror 5: 0x8f7322f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338c2): Mirror 5: 0x8f7322f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338c2: skipped 000214 1788179916.665029 trunc from 0x0338c2 to 0x01b41e (0x184a3 bytes) 000215 1788179916.696011 mapwrite 0x0109a6 thru 0x01f9ab (0xf006 bytes) 000216 1788179916.832539 write 0x019d88 thru 0x0244bb (0xa734 bytes) 000217 1788179916.834268 write 0x00f98f thru 0x019db6 (0xa428 bytes) 000218 1788179916.838683 trunc from 0x0244bc to 0x03e5f4 (0x1a139 bytes) 000219 1788179916.864075 read 0x027ee8 thru 0x02ff11 (0x802a bytes) 000220 1788179916.882523 trunc from 0x03cb13 to 0x03e5f4 (0x1ae2 bytes) 000221 1788179916.886737 mapwrite 0x00510c thru 0x0095b8 (0x44ad bytes) 000222 1788179916.951266 read 0x03ab0b thru 0x03e5f4 (0x3aea bytes) 000223 1788179916.967163 read 0x031056 thru 0x039f32 (0x8edd bytes) 000224 1788179916.968004 trunc from 0x03e5f5 to 0x008ab6 (0x35b3e bytes) 000225 1788179916.993438 write 0x01139a thru 0x01b30b (0x9f72 bytes) 000226 1788179916.996507 punch from 0x0068c1 to 0x007a24 (0x1164 bytes) 000227 1788179917.004006 write 0x004ccb thru 0x00b668 (0x699e bytes) 000228 1788179917.015412 punch from 0x00e280 to 0x0120f2 (0x3e73 bytes) 000229 1788179917.044160 mapwrite 0x024f34 thru 0x02c142 (0x720f bytes) 000230 1788179917.114001 fallocate 0x0023b9 thru 0x00e0af (0xbcf7 bytes) 000232 1788179917.128044 mapwrite 0x026de9 thru 0x02efc5 (0x81dd bytes) 000234 1788179917.187871 punch from 0x0161e1 to 0x0211bb (0xafdb bytes) 000235 1788179917.196774 read 0x002bd4 thru 0x0064ce (0x38fb bytes) 000237 1788179917.198163 trunc from 0x004596 to 0x0124f2 (0xdf5d bytes) 000239 1788179917.204289 trunc from 0x02efc6 to 0x023fb2 (0xb013 bytes) 000240 1788179917.218281 punch from 0x0033e5 to 0x00b29f (0x7ebb bytes) 000242 1788179917.225746 mapwrite 0x030ae3 thru 0x03f34d (0xe86b bytes) 000243 1788179917.285905 punch from 0x023755 to 0x02fc9a (0xc546 bytes) 000244 1788179917.291256 read 0x020203 thru 0x028696 (0x8494 bytes) 000246 1788179917.303316 trunc from 0x03f34e to 0x00dc37 (0x31716 bytes) 000247 1788179917.326504 fallocate 0x029cff thru 0x035eb2 (0xc1b4 bytes) 000248 1788179917.337231 mapread 0x005b0b thru 0x00bfeb (0x64e1 bytes) 000249 1788179917.342618 write 0x02226c thru 0x024e1e (0x2bb3 bytes) 000250 1788179917.345160 trunc from 0x024e1f to 0x00c8c6 (0x18558 bytes) 000251 1788179917.366324 write 0x03b7dd thru 0x03ffff (0x4823 bytes) 000252 1788179917.367767 trunc from 0x016635 to 0x016b9c (0x568 bytes) 000253 1788179917.378356 trunc from 0x019426 to 0x01af71 (0x1b4c bytes) 000254 1788179917.383286 punch from 0x019dec to 0x0206bf (0x68d4 bytes) 000255 1788179917.393007 trunc from 0x01ba96 to 0x025c68 (0xa1d3 bytes) 000256 1788179917.399038 trunc from 0x0038e1 to 0x00eb34 (0xb254 bytes) 000257 1788179917.403024 fallocate 0x031ba9 thru 0x03fd1f (0xe177 bytes) 000259 1788179917.408778 punch from 0x026ae0 to 0x03269d (0xbbbe bytes) 000260 1788179917.412253 mapwrite 0x025db6 thru 0x03572b (0xf976 bytes) 000262 1788179917.502185 punch from 0x0052a6 to 0x0072bc (0x2017 bytes) 000263 1788179917.508116 read 0x03bb16 thru 0x03ffff (0x44ea bytes) 000264 1788179917.512568 MIRROR_RESYNC 2 mirrors 000265 1788179918.156152 MIRROR_RESYNC 2 mirrors 000266 1788179918.560487 read 0x03e1a4 thru 0x03e57b (0x3d8 bytes) 000267 1788179918.569766 MIRROR_RESYNC 2 mirrors 000268 1788179918.973843 trunc from 0x040000 to 0x00b7f1 (0x3480e bytes) 000269 1788179919.061064 fallocate 0x00ed98 thru 0x01e57c (0xf7e5 bytes) 000270 1788179919.072227 mapwrite 0x00a12a thru 0x017205 (0xd0dc bytes) 000271 1788179919.136071 fallocate 0x024777 thru 0x0275b7 (0x2e41 bytes) 000272 1788179919.141580 read 0x01f126 thru 0x02058f (0x146a bytes) 000273 1788179919.148061 write 0x03bdb9 thru 0x03ffff (0x4247 bytes) 000274 1788179919.155022 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000277 1788179919.349602 read 0x007540 thru 0x00b4b0 (0x3f71 bytes) 000278 1788179919.362003 write 0x037a21 thru 0x03ffff (0x85df bytes) 000279 1788179919.371957 fallocate 0x01aa94 thru 0x029d3b (0xf2a8 bytes) 000280 1788179919.377062 read 0x029b00 thru 0x02ffb4 (0x64b5 bytes) 000281 1788179919.398538 punch from 0x02dfdc to 0x035df2 (0x7e17 bytes) 000282 1788179919.406531 mapwrite 0x0295ea thru 0x038945 (0xf35c bytes) 000283 1788179919.524931 MIRROR_EXTEND to 2 mirrors 000284 1788179920.234419 fallocate 0x01462a thru 0x01b83a (0x7211 bytes) 000285 1788179920.296820 trunc from 0x040000 to 0x000c33 (0x3f3cc bytes) 000286 1788179920.360731 trunc from 0x000c34 to 0x020c39 (0x20006 bytes) 000287 1788179920.381041 mapwrite 0x020d32 thru 0x02ab4f (0x9e1e bytes) 000288 1788179920.454372 mapwrite 0x01fb55 thru 0x029780 (0x9c2c bytes) 000289 1788179920.479874 read 0x0030bf thru 0x00b5a4 (0x84e6 bytes) 000290 1788179920.501560 trunc from 0x02ab50 to 0x02ff8f (0x5440 bytes) 000291 1788179920.527972 mapwrite 0x00c06d thru 0x010c21 (0x4bb5 bytes) 000292 1788179920.571510 fallocate 0x0330fd thru 0x03b29c (0x81a0 bytes) 000293 1788179920.576630 mapwrite 0x03f7c9 thru 0x03ffff (0x837 bytes) 000294 1788179920.612859 read 0x027a87 thru 0x02b949 (0x3ec3 bytes) 000295 1788179920.623909 fallocate 0x01bf7b thru 0x025efe (0x9f84 bytes) 000296 1788179920.629664 write 0x01f6e4 thru 0x0257bf (0x60dc bytes) 000297 1788179920.632195 mapwrite 0x00d07c thru 0x012949 (0x58ce bytes) 000298 1788179920.655405 read 0x036bc4 thru 0x03b9df (0x4e1c bytes) 000299 1788179920.657188 fallocate 0x03f6d5 thru 0x03ffff (0x92b bytes) 000300 1788179920.664742 fallocate 0x02340c thru 0x028ccd (0x58c2 bytes) 000301 1788179920.670186 mapread 0x01069c thru 0x01c76d (0xc0d2 bytes) 000302 1788179920.672573 mapwrite 0x007f05 thru 0x014376 (0xc472 bytes) 000303 1788179920.694352 trunc from 0x03b9d2 to 0x03e58d (0x2bbc bytes) 000304 1788179920.702219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000305 1788179920.901105 trunc from 0x00bba4 to 0x015fbf (0xa41c bytes) 000306 1788179920.907706 write 0x011eeb thru 0x013a9f (0x1bb5 bytes) 000307 1788179920.929353 mapread 0x00e6c9 thru 0x0121b6 (0x3aee bytes) 000308 1788179920.967171 trunc from 0x0308ff to 0x03a5be (0x9cc0 bytes) 000310 1788179920.974220 read 0x0380d6 thru 0x03ffff (0x7f2a bytes) 000311 1788179920.988302 write 0x0325fe thru 0x03ffff (0xda02 bytes) 000312 1788179921.004948 MIRROR_EXTEND to 2 mirrors 000314 1788179921.831809 fallocate 0x00462f thru 0x00aaf6 (0x64c8 bytes) 000315 1788179921.874279 mapread 0x035683 thru 0x03f873 (0xa1f1 bytes) 000316 1788179921.895546 punch from 0x01ec77 to 0x0298d1 (0xac5b bytes) 000317 1788179921.900956 write 0x018c0c thru 0x02584b (0xcc40 bytes) 000318 1788179921.915966 trunc from 0x040000 to 0x00d02f (0x32fd0 bytes) 000319 1788179921.937004 punch from 0x00306d to 0x00d02f (0x9fc3 bytes) 000320 1788179921.943900 fallocate 0x0394a4 thru 0x03a219 (0xd76 bytes) 000322 1788179921.953620 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a21a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a21a): Mirror 2: 0x6c5a1ed5 (stale) CRC-32 checksum value for chunk [0, 0x3a21a): Mirror 1: 0x9ce16aa9 Mirror 3: 0x9ce16aa9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a21a: skipped 000323 1788179922.756808 read 0x0208a9 thru 0x026ad3 (0x622b bytes) 000324 1788179922.777725 fallocate 0x0147b7 thru 0x02446f (0xfcb9 bytes) 000325 1788179922.822959 write 0x009523 thru 0x00c286 (0x2d64 bytes) 000327 1788179922.838822 fallocate 0x029824 thru 0x02baa0 (0x227d bytes) 000328 1788179922.843684 write 0x03d56e thru 0x03fd8b (0x281e bytes) 000329 1788179922.844851 write 0x011135 thru 0x021060 (0xff2c bytes) 000330 1788179922.859008 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd8c): Mirror 3: 0x9761dc10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd8c): Mirror 3: 0x9761dc10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd8c: skipped 000331 1788179923.164958 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd8c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd8c): Mirror 3: 0x9761dc10 (stale) CRC-32 checksum value for chunk [0, 0x3fd8c): Mirror 1: 0xf21e190b Mirror 4: 0xf21e190b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd8c: skipped 000333 1788179924.128980 mapwrite 0x020b11 thru 0x02fa00 (0xeef0 bytes) 000334 1788179924.257948 punch from 0x0062cc to 0x00e1fa (0x7f2f bytes) 000335 1788179924.263891 trunc from 0x016fec to 0x01ba7d (0x4a92 bytes) 000337 1788179924.269948 fallocate 0x033d41 thru 0x0367ec (0x2aac bytes) 000338 1788179924.275414 fallocate 0x00ea11 thru 0x01b62f (0xcc1f bytes) 000340 1788179924.289861 fallocate 0x02f036 thru 0x033f85 (0x4f50 bytes) 000341 1788179924.301046 read 0x02c687 thru 0x03a9b9 (0xe333 bytes) 000342 1788179924.333443 trunc from 0x03663e to 0x03fd8b (0x974e bytes) 000343 1788179924.341975 mapwrite 0x01ffa7 thru 0x02674b (0x67a5 bytes) 000344 1788179924.363846 mapread 0x01c000 thru 0x02b9d4 (0xf9d5 bytes) 000345 1788179924.366210 trunc from 0x009871 to 0x016609 (0xcd99 bytes) 000346 1788179924.371650 mapread 0x03746e thru 0x03fd8b (0x891e bytes) 000347 1788179924.380237 trunc from 0x03fd8c to 0x0311a3 (0xebe8 bytes) 000348 1788179924.407019 punch from 0x001048 to 0x00ec08 (0xdbc1 bytes) 000349 1788179924.415006 trunc from 0x0311a4 to 0x02b182 (0x6021 bytes) 000350 1788179924.429619 mapread 0x0140fd thru 0x01f52c (0xb430 bytes) 000351 1788179924.442939 punch from 0x007f9d to 0x00950c (0x1570 bytes) 000352 1788179924.449669 mapwrite 0x023ca5 thru 0x0332dd (0xf639 bytes) 000353 1788179924.493863 mapread 0x019ddd thru 0x028987 (0xebab bytes) 000354 1788179924.495232 write 0x01c14e thru 0x02bbeb (0xfa9e bytes) 000355 1788179924.497907 fallocate 0x01ddb6 thru 0x02c5c6 (0xe811 bytes) 000356 1788179924.503032 fallocate 0x02dbe5 thru 0x02fa6e (0x1e8a bytes) 000357 1788179924.508075 write 0x024707 thru 0x0265c2 (0x1ebc bytes) 000358 1788179924.509556 trunc from 0x004508 to 0x013bb0 (0xf6a9 bytes) 000359 1788179924.515154 punch from 0x0145ba to 0x020cf7 (0xc73e bytes) 000360 1788179924.556985 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x332de): Mirror 3: 0x75a6e8e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x332de): Mirror 3: 0x75a6e8e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x332de: skipped 000361 1788179924.867045 write 0x02023c thru 0x029fe3 (0x9da8 bytes) 000362 1788179924.884469 trunc from 0x02af1e to 0x0332dd (0x83c0 bytes) 000363 1788179924.890005 mapwrite 0x031c03 thru 0x03f433 (0xd831 bytes) 000364 1788179924.980815 punch from 0x011aac to 0x015c7f (0x41d4 bytes) 000365 1788179924.985279 punch from 0x001238 to 0x0062cd (0x5096 bytes) 000367 1788179924.991189 write 0x03db9a thru 0x03ffff (0x2466 bytes) 000368 1788179924.993343 mapread 0x00aa3f thru 0x00b311 (0x8d3 bytes) 000369 1788179925.014804 write 0x019600 thru 0x01e389 (0x4d8a bytes) 000370 1788179925.034601 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000372 1788179925.204553 trunc from 0x040000 to 0x020edf (0x1f120 bytes) 000373 1788179925.227377 punch from 0x00c6dd to 0x00dc79 (0x159d bytes) 000374 1788179925.237676 mapwrite 0x00d622 thru 0x00d93c (0x31b bytes) 000375 1788179925.274460 mapwrite 0x01821e thru 0x02451f (0xc302 bytes) 000376 1788179925.389790 read 0x01eecd thru 0x02451f (0x5653 bytes) 000377 1788179925.390878 fallocate 0x018259 thru 0x01a040 (0x1de8 bytes) 000380 1788179925.398590 fallocate 0x02d162 thru 0x03705b (0x9efa bytes) 000381 1788179925.404877 trunc from 0x00af3d to 0x00e750 (0x3814 bytes) 000382 1788179925.411970 mapread 0x01b3cf thru 0x01eacd (0x36ff bytes) 000383 1788179925.415823 read 0x011fc8 thru 0x0193f8 (0x7431 bytes) 000385 1788179925.428343 write 0x01eff0 thru 0x02e837 (0xf848 bytes) 000386 1788179925.434273 MIRROR_EXTEND to 2 mirrors 000387 1788179926.171540 trunc from 0x007432 to 0x013ef2 (0xcac1 bytes) 000388 1788179926.222476 read 0x01c8a9 thru 0x0249ee (0x8146 bytes) 000389 1788179926.234834 read 0x0050a2 thru 0x00d5a9 (0x8508 bytes) 000390 1788179926.245266 mapread 0x00fbeb thru 0x01b45f (0xb875 bytes) 000391 1788179926.247630 mapwrite 0x00e616 thru 0x016769 (0x8154 bytes) 000392 1788179926.272024 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000393 1788179926.405036 write 0x0090e6 thru 0x0159d5 (0xc8f0 bytes) 000394 1788179926.420922 trunc from 0x02e838 to 0x018fea (0x1584d bytes) 000395 1788179926.456902 mapwrite 0x00043a thru 0x00face (0xf695 bytes) 000396 1788179926.507718 trunc from 0x00dafd to 0x012f75 (0x5479 bytes) 000397 1788179926.513107 trunc from 0x00c108 to 0x014fcb (0x8ec4 bytes) 000398 1788179926.519148 write 0x00bfc5 thru 0x013537 (0x7573 bytes) 000400 1788179926.525842 fallocate 0x00d685 thru 0x018eb4 (0xb830 bytes) 000401 1788179926.531146 punch from 0x00137b to 0x0091cd (0x7e53 bytes) 000402 1788179926.536674 write 0x00347f thru 0x00a366 (0x6ee8 bytes) 000403 1788179926.544353 punch from 0x00105d to 0x010193 (0xf137 bytes) 000404 1788179926.559184 write 0x023c9c thru 0x024f2e (0x1293 bytes) 000407 1788179926.563375 trunc from 0x024f2f to 0x02b7ac (0x687e bytes) 000408 1788179926.580952 write 0x003d0d thru 0x00543f (0x1733 bytes) 000409 1788179926.591933 trunc from 0x02b7ad to 0x034148 (0x899c bytes) 000410 1788179926.611071 trunc from 0x034149 to 0x010f9d (0x231ab bytes) 000411 1788179926.636066 write 0x011d03 thru 0x018afa (0x6df8 bytes) 000415 1788179926.650020 mapwrite 0x016bc0 thru 0x01f9ea (0x8e2b bytes) 000416 1788179926.698205 punch from 0x00d99d to 0x00e6a1 (0xd05 bytes) 000417 1788179926.705678 fallocate 0x00ccbd thru 0x00ce86 (0x1ca bytes) 000419 1788179926.711309 fallocate 0x02d8b5 thru 0x0308bd (0x3009 bytes) 000421 1788179926.717308 trunc from 0x000830 to 0x00efc0 (0xe791 bytes) 000422 1788179926.724890 mapread 0x025bb7 thru 0x026417 (0x861 bytes) 000424 1788179926.733861 mapwrite 0x015eca thru 0x019ae6 (0x3c1d bytes) 000426 1788179926.750013 fallocate 0x03f16e thru 0x03ffff (0xe92 bytes) 000428 1788179926.755781 write 0x03c497 thru 0x03d3fa (0xf64 bytes) 000429 1788179926.757648 trunc from 0x020afb to 0x024603 (0x3b09 bytes) 000430 1788179926.762067 mapread 0x0366a3 thru 0x037ca1 (0x15ff bytes) 000431 1788179926.769890 fallocate 0x00a2d1 thru 0x0114c3 (0x71f3 bytes) 000432 1788179926.774935 punch from 0x021127 to 0x02ecef (0xdbc9 bytes) 000433 1788179926.779675 write 0x01773b thru 0x022447 (0xad0d bytes) 000434 1788179926.785763 read 0x004503 thru 0x013818 (0xf316 bytes) 000435 1788179926.795510 mapread 0x01c22a thru 0x01e819 (0x25f0 bytes) 000436 1788179926.796529 fallocate 0x016042 thru 0x020ee8 (0xaea7 bytes) 000437 1788179926.801196 mapwrite 0x019e2b thru 0x01b250 (0x1426 bytes) 000438 1788179926.817000 write 0x03228a thru 0x03d72f (0xb4a6 bytes) 000439 1788179926.821124 punch from 0x039abc to 0x03d72f (0x3c74 bytes) 000440 1788179926.843749 mapwrite 0x01ad71 thru 0x02a853 (0xfae3 bytes) 000441 1788179926.904044 read 0x014d7f thru 0x015234 (0x4b6 bytes) 000442 1788179926.905147 read 0x0222d1 thru 0x02db1e (0xb84e bytes) 000443 1788179926.917828 trunc from 0x03d730 to 0x0138b3 (0x29e7c bytes) 000444 1788179926.940003 trunc from 0x001320 to 0x00e204 (0xcee5 bytes) 000445 1788179926.947304 trunc from 0x0138b4 to 0x00a2dc (0x95d7 bytes) 000446 1788179926.969533 fallocate 0x0203f7 thru 0x0281f9 (0x7e03 bytes) 000447 1788179926.979800 mapwrite 0x035f87 thru 0x0394b5 (0x352f bytes) 000449 1788179927.037917 punch from 0x021a83 to 0x02aa42 (0x8fc0 bytes) 000450 1788179927.042780 fallocate 0x03c8d6 thru 0x03ffff (0x372a bytes) 000451 1788179927.049275 trunc from 0x0394b6 to 0x018bd5 (0x208e0 bytes) 000452 1788179927.068714 trunc from 0x018bd6 to 0x03e060 (0x2548b bytes) 000453 1788179927.085599 write 0x03877b thru 0x03ffff (0x7885 bytes) 000455 1788179927.097124 trunc from 0x009098 to 0x0176c1 (0xe62a bytes) 000456 1788179927.102062 write 0x02bd06 thru 0x0385ee (0xc8e9 bytes) 000457 1788179927.108005 fallocate 0x0258ed thru 0x02fdd9 (0xa4ed bytes) 000458 1788179927.112744 trunc from 0x013237 to 0x014c7f (0x1a49 bytes) 000459 1788179927.117458 fallocate 0x01d313 thru 0x02d2c0 (0xffae bytes) 000460 1788179927.121540 write 0x028dcf thru 0x035a2c (0xcc5e bytes) 000461 1788179927.127489 read 0x0076b6 thru 0x008151 (0xa9c bytes) 000462 1788179927.136998 write 0x01662e thru 0x019e10 (0x37e3 bytes) 000463 1788179927.138747 fallocate 0x00c3b3 thru 0x01b5fa (0xf248 bytes) 000464 1788179927.143776 mapread 0x0343ef thru 0x03e45b (0xa06d bytes) 000465 1788179927.145669 mapread 0x016a3e thru 0x01ee7d (0x8440 bytes) 000466 1788179927.147179 mapwrite 0x037020 thru 0x03ffff (0x8fe0 bytes) 000467 1788179927.174778 trunc from 0x040000 to 0x025dce (0x1a231 bytes) 000468 1788179927.190074 write 0x027547 thru 0x029c3e (0x26f8 bytes) 000469 1788179927.191294 mapread 0x003750 thru 0x007a35 (0x42e6 bytes) 000470 1788179927.192328 fallocate 0x004df5 thru 0x0091b9 (0x43c5 bytes) 000471 1788179927.197625 trunc from 0x004ecb to 0x01017c (0xb2b2 bytes) 000472 1788179927.202876 punch from 0x015582 to 0x017465 (0x1ee4 bytes) 000473 1788179927.208013 trunc from 0x029c3f to 0x022e7a (0x6dc4 bytes) 000474 1788179927.223376 write 0x011df1 thru 0x017940 (0x5b50 bytes) 000475 1788179927.227925 read 0x016066 thru 0x01ff1e (0x9eb9 bytes) 000476 1788179927.228726 write 0x02304b thru 0x02fbc5 (0xcb7b bytes) 000478 1788179927.232620 trunc from 0x01df36 to 0x02b39c (0xd467 bytes) 000479 1788179927.246909 read 0x02a1fc thru 0x02fbc5 (0x59ca bytes) 000480 1788179927.256963 trunc from 0x014e81 to 0x0205cb (0xb74b bytes) 000481 1788179927.261829 read 0x02e78b thru 0x02fbc5 (0x143b bytes) 000482 1788179927.262648 fallocate 0x007cfd thru 0x01282a (0xab2e bytes) 000483 1788179927.266987 punch from 0x003ecf to 0x00b7ab (0x78dd bytes) 000484 1788179927.271203 read 0x003099 thru 0x003a54 (0x9bc bytes) 000485 1788179927.283891 punch from 0x01ca79 to 0x0208a5 (0x3e2d bytes) 000486 1788179927.292432 MIRROR_EXTEND to 2 mirrors 000487 1788179927.938145 fallocate 0x03600b thru 0x036471 (0x467 bytes) 000488 1788179928.018198 read 0x021d9a thru 0x02b760 (0x99c7 bytes) 000489 1788179928.035003 read 0x0295bd thru 0x02fbc5 (0x6609 bytes) 000490 1788179928.053190 write 0x0104ee thru 0x01a214 (0x9d27 bytes) 000491 1788179928.059012 mapread 0x0195d5 thru 0x026345 (0xcd71 bytes) 000492 1788179928.060959 punch from 0x012437 to 0x01a705 (0x82cf bytes) 000493 1788179928.075132 trunc from 0x02fbc6 to 0x039400 (0x983b bytes) 000494 1788179928.114694 mapwrite 0x00df43 thru 0x014f61 (0x701f bytes) 000495 1788179928.149876 fallocate 0x02ad13 thru 0x038705 (0xd9f3 bytes) 000496 1788179928.155786 write 0x011047 thru 0x01de25 (0xcddf bytes) 000497 1788179928.158550 punch from 0x01435d to 0x01c6bb (0x835f bytes) 000499 1788179928.189125 mapread 0x02244d thru 0x025a2e (0x35e2 bytes) 000500 1788179928.192114 trunc from 0x039401 to 0x015fc5 (0x2343b bytes) 000501 1788179928.207935 fallocate 0x00f330 thru 0x018ab8 (0x9789 bytes) 000502 1788179928.218042 punch from 0x005546 to 0x00c1a5 (0x6c60 bytes) 000503 1788179928.222977 trunc from 0x018ab9 to 0x00eb10 (0x9fa8 bytes) 000504 1788179928.246536 mapread 0x00053c thru 0x00eb10 (0xe5d5 bytes) 000505 1788179928.300832 fallocate 0x02ffe5 thru 0x03974e (0x976a bytes) 000506 1788179928.305938 punch from 0x0326f1 to 0x03974e (0x705e bytes) 000507 1788179928.310882 read 0x016aea thru 0x025954 (0xee6b bytes) 000508 1788179928.322765 read 0x039299 thru 0x03974e (0x4b6 bytes) 000509 1788179928.324189 fallocate 0x0169b7 thru 0x024e2f (0xe479 bytes) 000510 1788179928.330688 fallocate 0x02f2bc thru 0x032a83 (0x37c8 bytes) 000511 1788179928.343291 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3974f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3974f): Mirror 2: 0x147ec073 (stale) CRC-32 checksum value for chunk [0, 0x3974f): Mirror 1: 0x8523cb3f Mirror 3: 0x8523cb3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3974f: skipped 000513 1788179929.265196 mapwrite 0x00b2f0 thru 0x0100df (0x4df0 bytes) 000514 1788179929.393906 fallocate 0x01c64a thru 0x0261b9 (0x9b70 bytes) 000517 1788179929.411050 read 0x01e7c6 thru 0x01fcfa (0x1535 bytes) 000518 1788179929.416682 mapwrite 0x03bd7e thru 0x03ffff (0x4282 bytes) 000519 1788179929.502195 MIRROR_RESYNC 3 mirrors 000521 1788179930.197938 write 0x031407 thru 0x039887 (0x8481 bytes) 000522 1788179930.242102 read 0x02aebb thru 0x0303bf (0x5505 bytes) 000523 1788179930.259003 trunc from 0x021b83 to 0x023bae (0x202c bytes) 000524 1788179930.268898 trunc from 0x01b934 to 0x01cde8 (0x14b5 bytes) 000525 1788179930.291380 write 0x00d446 thru 0x016f61 (0x9b1c bytes) 000526 1788179930.315438 mapread 0x001f4a thru 0x00c96a (0xaa21 bytes) 000527 1788179930.376170 trunc from 0x036b94 to 0x03e375 (0x77e2 bytes) 000528 1788179930.408023 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x50c947dc (stale) Mirror 3: 0x50c947dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd1e480aa Mirror 4: 0xd1e480aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000529 1788179931.357872 mapwrite 0x03ad13 thru 0x03ffff (0x52ed bytes) 000530 1788179931.508587 mapwrite 0x013670 thru 0x02114c (0xdadd bytes) 000531 1788179931.599898 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x50c947dc (stale) Mirror 3: 0x50c947dc (stale) Mirror 4: 0xd1e480aa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x63c6da48 Mirror 5: 0x63c6da48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000533 1788179932.627692 trunc from 0x019d6a to 0x024007 (0xa29e bytes) 000535 1788179932.672692 read 0x00b2eb thru 0x0197ef (0xe505 bytes) 000536 1788179932.685567 trunc from 0x040000 to 0x027410 (0x18bef bytes) 000537 1788179932.729952 trunc from 0x027411 to 0x01d817 (0x9bf9 bytes) 000538 1788179932.775112 trunc from 0x01d818 to 0x0076bd (0x1615a bytes) 000539 1788179932.802635 fallocate 0x01b798 thru 0x021de2 (0x664b bytes) 000540 1788179932.810537 read 0x002848 thru 0x0076bd (0x4e76 bytes) 000541 1788179932.832057 read 0x001d31 thru 0x0076bd (0x598d bytes) 000542 1788179932.842747 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x76be): Mirror 3: 0xdb94c5fd (stale) Mirror 4: 0xdb94c5fd (stale) Mirror 5: 0xdb94c5fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x76be): Mirror 3: 0xdb94c5fd (stale) Mirror 4: 0xdb94c5fd (stale) Mirror 5: 0xdb94c5fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x76be: skipped 000543 1788179933.401844 fallocate 0x02ffbc thru 0x03d044 (0xd089 bytes) 000544 1788179933.416728 read 0x00d801 thru 0x00e153 (0x953 bytes) 000545 1788179933.443336 trunc from 0x00afd2 to 0x0113e5 (0x6414 bytes) 000546 1788179933.451924 trunc from 0x000bd5 to 0x009836 (0x8c62 bytes) 000547 1788179933.462167 mapread 0x021fc2 thru 0x02cfee (0xb02d bytes) 000548 1788179933.547501 trunc from 0x03d045 to 0x01c0df (0x20f65 bytes) 000549 1788179933.587706 fallocate 0x035f5f thru 0x03ffff (0xa0a1 bytes) 000550 1788179933.615415 trunc from 0x0308b9 to 0x0356f0 (0x4e38 bytes) 000551 1788179933.631265 trunc from 0x03edfd to 0x03ffff (0x1203 bytes) 000552 1788179933.647687 punch from 0x02b88e to 0x03147e (0x5bf1 bytes) 000553 1788179933.655365 trunc from 0x040000 to 0x02d1ed (0x12e12 bytes) 000554 1788179933.681870 write 0x0109e8 thru 0x01adc5 (0xa3de bytes) 000555 1788179933.700834 trunc from 0x018667 to 0x021402 (0x8d9c bytes) 000556 1788179933.738773 fallocate 0x01c4e3 thru 0x01d9ea (0x1508 bytes) 000557 1788179933.746505 write 0x01e60c thru 0x029e20 (0xb815 bytes) 000558 1788179933.759290 punch from 0x011d81 to 0x0122ad (0x52d bytes) 000559 1788179933.765635 write 0x01f09b thru 0x02c8e2 (0xd848 bytes) 000560 1788179933.773926 read 0x00ad21 thru 0x014b3d (0x9e1d bytes) 000561 1788179933.782803 mapread 0x00107d thru 0x010e9d (0xfe21 bytes) 000562 1788179933.785131 trunc from 0x02d1ee to 0x03669a (0x94ad bytes) 000563 1788179933.810057 mapwrite 0x0120fc thru 0x013f91 (0x1e96 bytes) 000564 1788179933.830572 punch from 0x001df8 to 0x00a94a (0x8b53 bytes) 000565 1788179933.842358 trunc from 0x03669b to 0x00d3d4 (0x292c6 bytes) 000566 1788179933.870005 mapwrite 0x003355 thru 0x006a5a (0x3706 bytes) 000567 1788179933.930774 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xd3d5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd3d5): Mirror 3: 0x9642cf7b (stale) Mirror 4: 0x9642cf7b (stale) Mirror 5: 0x9642cf7b (stale) CRC-32 checksum value for chunk [0, 0xd3d5): Mirror 1: 0xbd091a1e Mirror 6: 0xbd091a1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd3d5: skipped 000568 1788179934.959941 fallocate 0x022578 thru 0x030130 (0xdbb9 bytes) 000569 1788179935.009927 trunc from 0x025678 to 0x030130 (0xaab9 bytes) 000570 1788179935.018656 read 0x006d63 thru 0x00a64a (0x38e8 bytes) 000571 1788179935.031991 trunc from 0x023581 to 0x02ffd4 (0xca54 bytes) 000572 1788179935.041065 MIRROR_RESYNC 5 mirrors 000574 1788179936.301908 read 0x0176e8 thru 0x0263da (0xecf3 bytes) 000575 1788179936.324836 trunc from 0x030131 to 0x0151f2 (0x1af3e bytes) 000576 1788179936.424426 punch from 0x014a02 to 0x0151f2 (0x7f1 bytes) 000577 1788179936.438433 mapwrite 0x037f32 thru 0x03cdd9 (0x4ea8 bytes) 000578 1788179936.541467 trunc from 0x00be45 to 0x00e5e6 (0x27a2 bytes) 000579 1788179936.545948 mapwrite 0x0297bc thru 0x0329ed (0x9232 bytes) 000580 1788179936.572937 write 0x010a7d thru 0x0137ab (0x2d2f bytes) 000581 1788179936.584474 write 0x007862 thru 0x00c21e (0x49bd bytes) 000582 1788179936.598878 write 0x03283f thru 0x03ffff (0xd7c1 bytes) 000583 1788179936.605507 punch from 0x02baae to 0x02ed82 (0x32d5 bytes) 000584 1788179936.614150 trunc from 0x0034fc to 0x00d83d (0xa342 bytes) 000585 1788179936.636850 punch from 0x00054b to 0x005dff (0x58b5 bytes) 000586 1788179936.642585 mapread 0x03b081 thru 0x03cbfe (0x1b7e bytes) 000587 1788179936.643782 read 0x03be53 thru 0x03ffff (0x41ad bytes) 000588 1788179936.645154 read 0x014cc2 thru 0x01784f (0x2b8e bytes) 000589 1788179936.662710 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6d3b944a (stale) Mirror 4: 0x6d3b944a (stale) Mirror 5: 0x6d3b944a (stale) Mirror 6: 0x6d3b944a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x220d5ade Mirror 7: 0x220d5ade lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000590 1788179937.709875 punch from 0x024cd1 to 0x034a48 (0xfd78 bytes) 000592 1788179937.797043 punch from 0x01e739 to 0x0248bc (0x6184 bytes) 000593 1788179937.806208 punch from 0x0244c6 to 0x0275b7 (0x30f2 bytes) 000594 1788179937.813057 trunc from 0x03e328 to 0x03ffff (0x1cd8 bytes) 000595 1788179937.822450 trunc from 0x000faa to 0x003371 (0x23c8 bytes) 000596 1788179937.831274 mapread 0x0062ab thru 0x00fbec (0x9942 bytes) 000597 1788179937.862593 trunc from 0x040000 to 0x0303fd (0xfc02 bytes) 000598 1788179937.898691 trunc from 0x005ca8 to 0x0080c6 (0x241f bytes) 000599 1788179937.908340 mapwrite 0x003b88 thru 0x00ceaa (0x9323 bytes) 000600 1788179937.951447 mapread 0x02b9ba thru 0x0303fd (0x4a44 bytes) 000601 1788179937.972686 trunc from 0x0303fe to 0x02fd6d (0x690 bytes) 000604 1788179938.010108 punch from 0x00202a to 0x010eaa (0xee81 bytes) 000605 1788179938.017729 read 0x018da7 thru 0x0258bf (0xcb19 bytes) 000606 1788179938.041804 fallocate 0x018900 thru 0x019848 (0xf49 bytes) 000607 1788179938.049953 mapwrite 0x03f8dc thru 0x03ffff (0x724 bytes) 000608 1788179938.092646 trunc from 0x040000 to 0x01ce28 (0x231d7 bytes) 000609 1788179938.113593 MIRROR_RESYNC 6 mirrors 000610 1788179939.703076 read 0x00acac thru 0x018ed4 (0xe229 bytes) 000612 1788179939.712834 punch from 0x015f3b to 0x01ce28 (0x6eee bytes) 000614 1788179939.749767 fallocate 0x03410a thru 0x039d87 (0x5c7e bytes) 000615 1788179939.755121 trunc from 0x0185bd to 0x01a90c (0x2350 bytes) 000616 1788179939.761554 fallocate 0x02a8bf thru 0x038a05 (0xe147 bytes) 000617 1788179939.766527 punch from 0x021eea to 0x028a1f (0x6b36 bytes) 000618 1788179939.771418 write 0x01976d thru 0x01a3ed (0xc81 bytes) 000619 1788179939.783984 mapread 0x0231d7 thru 0x02eadf (0xb909 bytes) 000620 1788179939.811921 trunc from 0x038a06 to 0x008aac (0x2ff59 bytes) 000621 1788179939.829698 read 0x0008b4 thru 0x00384b (0x2f98 bytes) 000622 1788179939.839798 fallocate 0x03e685 thru 0x03ffff (0x197b bytes) 000623 1788179939.851147 write 0x00c52d thru 0x00fc67 (0x373b bytes) 000624 1788179939.852826 trunc from 0x002a3e to 0x00fc67 (0xd22a bytes) 000625 1788179939.863225 punch from 0x00c5ff to 0x00fc67 (0x3669 bytes) 000626 1788179939.868043 trunc from 0x00fc68 to 0x0125ef (0x2988 bytes) 000627 1788179939.886179 write 0x030f14 thru 0x03499e (0x3a8b bytes) 000628 1788179939.889786 read 0x009eb4 thru 0x017df8 (0xdf45 bytes) 000629 1788179939.911002 mapread 0x02083f thru 0x02e618 (0xddda bytes) 000630 1788179939.913772 fallocate 0x019f8f thru 0x021621 (0x7693 bytes) 000631 1788179939.918811 fallocate 0x028d29 thru 0x02e984 (0x5c5c bytes) 000632 1788179939.924821 fallocate 0x01ee03 thru 0x028305 (0x9503 bytes) 000634 1788179939.931690 mapread 0x00b4cf thru 0x0139e3 (0x8515 bytes) 000635 1788179939.933919 mapwrite 0x0036da thru 0x0123f9 (0xed20 bytes) 000636 1788179939.959875 mapread 0x012e92 thru 0x01a3d1 (0x7540 bytes) 000637 1788179939.961518 punch from 0x009571 to 0x016d92 (0xd822 bytes) 000638 1788179939.968663 trunc from 0x03499f to 0x02bc65 (0x8d39 bytes) 000639 1788179939.992529 trunc from 0x006f92 to 0x0099c4 (0x2a33 bytes) 000640 1788179940.003977 write 0x01ec92 thru 0x0250a7 (0x6416 bytes) 000641 1788179940.007530 mapread 0x01579a thru 0x015fdf (0x846 bytes) 000642 1788179940.013998 trunc from 0x0065cf to 0x00af43 (0x4975 bytes) 000644 1788179940.024811 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bc66): Mirror 3: 0x6b7785c0 (stale) Mirror 5: 0x6b7785c0 (stale) Mirror 6: 0x6b7785c0 (stale) Mirror 7: 0x6b7785c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc66): Mirror 3: 0x6b7785c0 (stale) Mirror 5: 0x6b7785c0 (stale) Mirror 6: 0x6b7785c0 (stale) Mirror 7: 0x6b7785c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc66: skipped 000646 1788179940.563252 read 0x0010df thru 0x004e1f (0x3d41 bytes) 000647 1788179940.578852 mapread 0x00ac4d thru 0x019b04 (0xeeb8 bytes) 000648 1788179940.629352 mapwrite 0x03b5a1 thru 0x03e5fb (0x305b bytes) 000649 1788179940.726914 write 0x01eda6 thru 0x026591 (0x77ec bytes) 000650 1788179940.759413 trunc from 0x023e4f to 0x02a8a4 (0x6a56 bytes) 000651 1788179940.775459 punch from 0x005abe to 0x0089b9 (0x2efc bytes) 000652 1788179940.781326 trunc from 0x03e5fc to 0x03ac27 (0x39d4 bytes) 000653 1788179940.798049 trunc from 0x026746 to 0x02d484 (0x6d3f bytes) 000654 1788179940.805808 punch from 0x029819 to 0x02a71b (0xf03 bytes) 000655 1788179940.810978 write 0x02c5f9 thru 0x0309d3 (0x43db bytes) 000657 1788179940.823918 mapwrite 0x0182e9 thru 0x022f6e (0xac86 bytes) 000658 1788179940.848647 trunc from 0x01279b to 0x01898c (0x61f2 bytes) 000659 1788179940.854587 fallocate 0x024c34 thru 0x02c21b (0x75e8 bytes) 000660 1788179940.860066 trunc from 0x03ac28 to 0x01dd8a (0x1ce9d bytes) 000661 1788179940.904915 trunc from 0x00002e to 0x0030a8 (0x307b bytes) 000662 1788179940.921362 punch from 0x004e0e to 0x008656 (0x3849 bytes) 000665 1788179940.931454 trunc from 0x00f510 to 0x011137 (0x1c28 bytes) 000666 1788179940.950026 mapread 0x006608 thru 0x0089e1 (0x23da bytes) 000667 1788179940.978539 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dd8b): Mirror 3: 0x704b8b (stale) Mirror 6: 0x704b8b (stale) Mirror 7: 0x704b8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dd8b): Mirror 3: 0x704b8b (stale) Mirror 6: 0x704b8b (stale) Mirror 7: 0x704b8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dd8b: skipped 000669 1788179941.338849 fallocate 0x02ea99 thru 0x03dd77 (0xf2df bytes) 000670 1788179941.346769 mapwrite 0x0034f1 thru 0x00f1d2 (0xbce2 bytes) 000672 1788179941.397166 trunc from 0x038d57 to 0x03dd77 (0x5021 bytes) 000673 1788179941.401894 mapread 0x02fbd3 thru 0x03c579 (0xc9a7 bytes) 000674 1788179941.427763 trunc from 0x0037d4 to 0x00fb09 (0xc336 bytes) 000675 1788179941.434052 mapwrite 0x022389 thru 0x023bcb (0x1843 bytes) 000676 1788179941.454613 punch from 0x030ff2 to 0x03dd77 (0xcd86 bytes) 000678 1788179941.461050 mapwrite 0x029297 thru 0x02ddae (0x4b18 bytes) 000679 1788179941.483916 trunc from 0x03dd78 to 0x02a803 (0x13574 bytes) 000680 1788179941.503156 read 0x009ad3 thru 0x00a333 (0x861 bytes) 000681 1788179941.508602 punch from 0x00518d to 0x00bdb5 (0x6c29 bytes) 000682 1788179941.512446 trunc from 0x02a804 to 0x0369be (0xc1bb bytes) 000683 1788179941.528468 fallocate 0x012e87 thru 0x0190ec (0x6266 bytes) 000684 1788179941.534411 trunc from 0x00ffe0 to 0x01fa7f (0xfaa0 bytes) 000685 1788179941.540016 trunc from 0x0202e9 to 0x02369a (0x33b2 bytes) 000686 1788179941.544862 write 0x028f6f thru 0x029bf4 (0xc86 bytes) 000687 1788179941.546345 write 0x034f96 thru 0x03ffff (0xb06a bytes) 000688 1788179941.556943 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba71d997 (stale) Mirror 6: 0xba71d997 (stale) Mirror 7: 0xba71d997 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa6dd3b1e Mirror 8: 0xa6dd3b1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000689 1788179942.588461 mapread 0x015e65 thru 0x0187b5 (0x2951 bytes) 000690 1788179942.641014 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba71d997 (stale) Mirror 6: 0xba71d997 (stale) Mirror 7: 0xba71d997 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba71d997 (stale) Mirror 6: 0xba71d997 (stale) Mirror 7: 0xba71d997 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000691 1788179942.996777 mapwrite 0x00d26a thru 0x017ff9 (0xad90 bytes) 000692 1788179943.043771 mapwrite 0x014bc5 thru 0x021d52 (0xd18e bytes) 000693 1788179943.077652 trunc from 0x040000 to 0x022700 (0x1d8ff bytes) 000694 1788179943.098394 write 0x00d4a5 thru 0x01a5c9 (0xd125 bytes) 000695 1788179943.099969 trunc from 0x022701 to 0x00d659 (0x150a7 bytes) 000696 1788179943.122433 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd65a): Mirror 3: 0x775e6bfe (stale) Mirror 7: 0x775e6bfe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd65a): Mirror 3: 0x775e6bfe (stale) Mirror 7: 0x775e6bfe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd65a: skipped 000697 1788179943.482853 trunc from 0x00d65a to 0x01b758 (0xe0ff bytes) 000698 1788179943.517283 trunc from 0x00cd19 to 0x00f898 (0x2b80 bytes) 000699 1788179943.528010 read 0x002d38 thru 0x00853a (0x5803 bytes) 000700 1788179943.543714 read 0x00afb2 thru 0x016d3b (0xbd8a bytes) 000701 1788179943.554295 mapwrite 0x036d10 thru 0x03e70c (0x79fd bytes) 000702 1788179943.625698 fallocate 0x01b712 thru 0x023cd3 (0x85c2 bytes) 000703 1788179943.633576 trunc from 0x03e70d to 0x014649 (0x2a0c3 bytes) 000704 1788179943.688828 fallocate 0x02b36a thru 0x038368 (0xcfff bytes) 000705 1788179943.696296 mapread 0x00a273 thru 0x0132ed (0x907b bytes) 000706 1788179943.705170 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1464a): Mirror 7: 0xe7835e44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1464a): Mirror 7: 0xe7835e44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1464a: skipped 000707 1788179944.037530 punch from 0x002bef to 0x0116c3 (0xead5 bytes) 000708 1788179944.042538 punch from 0x0019bd to 0x007cb7 (0x62fb bytes) 000709 1788179944.051231 write 0x00b24a thru 0x01645f (0xb216 bytes) 000711 1788179944.061596 trunc from 0x00bc4e to 0x01296c (0x6d1f bytes) 000712 1788179944.089569 punch from 0x0155f6 to 0x01645f (0xe6a bytes) 000713 1788179944.098612 read 0x0020de thru 0x00222d (0x150 bytes) 000714 1788179944.105703 mapread 0x0058cf thru 0x00852e (0x2c60 bytes) 000715 1788179944.131062 fallocate 0x0144f0 thru 0x01d67b (0x918c bytes) 000716 1788179944.137689 trunc from 0x016460 to 0x000600 (0x15e5f bytes) 000717 1788179944.159507 write 0x00e041 thru 0x010470 (0x2430 bytes) 000718 1788179944.161225 trunc from 0x00c532 to 0x010470 (0x3f3f bytes) 000719 1788179944.178178 trunc from 0x010471 to 0x01f6c9 (0xf259 bytes) 000720 1788179944.193641 trunc from 0x01f6ca to 0x027046 (0x797d bytes) 000721 1788179944.212644 trunc from 0x00364d to 0x007e26 (0x47da bytes) 000722 1788179944.217240 trunc from 0x027047 to 0x01cb91 (0xa4b5 bytes) 000723 1788179944.232889 fallocate 0x03d313 thru 0x03ffff (0x2ced bytes) 000724 1788179944.241855 trunc from 0x01cb92 to 0x00a00e (0x12b83 bytes) 000726 1788179944.262685 trunc from 0x00a00f to 0x007bb1 (0x245d bytes) 000727 1788179944.281771 write 0x008bd1 thru 0x017e73 (0xf2a3 bytes) 000729 1788179944.284462 mapwrite 0x018951 thru 0x025c5c (0xd30c bytes) 000730 1788179944.345527 mapwrite 0x033c6a thru 0x03a671 (0x6a08 bytes) 000731 1788179944.428562 mapread 0x019fbf thru 0x0258cd (0xb90f bytes) 000732 1788179944.433508 fallocate 0x037218 thru 0x03f23a (0x8023 bytes) 000733 1788179944.441867 trunc from 0x03a672 to 0x008e86 (0x317eb bytes) 000734 1788179944.459757 trunc from 0x0070a4 to 0x008e86 (0x1de3 bytes) 000735 1788179944.468519 mapread 0x004489 thru 0x008e86 (0x49fe bytes) 000736 1788179944.482433 punch from 0x003609 to 0x008e86 (0x587e bytes) 000737 1788179944.489944 trunc from 0x0083e4 to 0x008e86 (0xaa3 bytes) 000738 1788179944.494476 fallocate 0x0078ac thru 0x0137e4 (0xbf39 bytes) 000739 1788179944.499987 mapwrite 0x00b172 thru 0x01846c (0xd2fb bytes) 000740 1788179944.568253 trunc from 0x01846d to 0x03a701 (0x22295 bytes) 000741 1788179944.599067 fallocate 0x037cff thru 0x03ffff (0x8301 bytes) 000742 1788179944.608567 write 0x00d052 thru 0x00dfc4 (0xf73 bytes) 000744 1788179944.614493 trunc from 0x015399 to 0x01c2fe (0x6f66 bytes) 000745 1788179944.621427 mapwrite 0x0371bb thru 0x03ffff (0x8e45 bytes) 000746 1788179944.664394 trunc from 0x040000 to 0x02264a (0x1d9b5 bytes) 000747 1788179944.681586 mapread 0x006a60 thru 0x00e9a7 (0x7f48 bytes) 000748 1788179944.683211 mapread 0x0016ed thru 0x0056cb (0x3fdf bytes) 000749 1788179944.688781 write 0x00de7b thru 0x018dcb (0xaf51 bytes) 000750 1788179944.702799 MIRROR_RESYNC 2 mirrors 000751 1788179945.370593 fallocate 0x03e8c2 thru 0x03f12c (0x86b bytes) 000752 1788179945.406791 trunc from 0x02264b to 0x0108aa (0x11da0 bytes) 000753 1788179945.447056 MIRROR_RESYNC 2 mirrors 000754 1788179946.106070 fallocate 0x01d7ba thru 0x023010 (0x5857 bytes) 000755 1788179946.163850 mapread 0x00571f thru 0x00cdb6 (0x7698 bytes) 000756 1788179946.187472 trunc from 0x023011 to 0x01821d (0xadf3 bytes) 000757 1788179946.216841 trunc from 0x00c719 to 0x012fd1 (0x68b9 bytes) 000758 1788179946.223269 read 0x0072a0 thru 0x00df69 (0x6cca bytes) 000759 1788179946.232790 MIRROR_RESYNC 2 mirrors 000760 1788179946.780987 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000761 1788179946.953729 punch from 0x00a6f5 to 0x00bc93 (0x159f bytes) 000763 1788179946.995070 mapwrite 0x01389a thru 0x02210c (0xe873 bytes) 000764 1788179947.133914 mapread 0x00e4cf thru 0x0108d9 (0x240b bytes) 000765 1788179947.135939 mapwrite 0x01a755 thru 0x01a977 (0x223 bytes) 000766 1788179947.147631 read 0x016a31 thru 0x02210c (0xb6dc bytes) 000767 1788179947.148421 fallocate 0x023d86 thru 0x02ed13 (0xaf8e bytes) 000768 1788179947.153405 mapread 0x01f97f thru 0x02210c (0x278e bytes) 000769 1788179947.154759 mapread 0x02048b thru 0x02210c (0x1c82 bytes) 000770 1788179947.155834 mapwrite 0x010da7 thru 0x01f3b1 (0xe60b bytes) 000771 1788179947.175390 trunc from 0x008777 to 0x00bfa1 (0x382b bytes) 000772 1788179947.180362 trunc from 0x02210d to 0x03cd5c (0x1ac50 bytes) 000773 1788179947.214432 fallocate 0x01624b thru 0x01e98c (0x8742 bytes) 000774 1788179947.221121 mapread 0x029c76 thru 0x02e453 (0x47de bytes) 000775 1788179947.259188 MIRROR_EXTEND to 2 mirrors 000776 1788179947.905688 read 0x029464 thru 0x02efd3 (0x5b70 bytes) 000777 1788179947.915043 write 0x009ec6 thru 0x011f77 (0x80b2 bytes) 000778 1788179947.952008 write 0x009932 thru 0x00b18f (0x185e bytes) 000779 1788179947.955623 fallocate 0x00a8ad thru 0x00ae0f (0x563 bytes) 000780 1788179947.960008 fallocate 0x02a2be thru 0x0311f8 (0x6f3b bytes) 000781 1788179947.965455 mapread 0x01ddd9 thru 0x025754 (0x797c bytes) 000782 1788179947.984244 read 0x016a6e thru 0x023d02 (0xd295 bytes) 000783 1788179947.995610 read 0x023e41 thru 0x02ae49 (0x7009 bytes) 000784 1788179947.996719 mapread 0x038f8d thru 0x03a942 (0x19b6 bytes) 000786 1788179948.000490 trunc from 0x03cd5d to 0x02b6b6 (0x116a6 bytes) 000787 1788179948.067540 fallocate 0x025eed thru 0x02f51d (0x9631 bytes) 000788 1788179948.072987 punch from 0x0127b5 to 0x018fcb (0x6817 bytes) 000789 1788179948.081733 write 0x016506 thru 0x02042e (0x9f29 bytes) 000791 1788179948.097074 trunc from 0x02b6b7 to 0x03ffbb (0x14905 bytes) 000792 1788179948.121878 read 0x00c55b thru 0x00ed17 (0x27bd bytes) 000793 1788179948.122644 read 0x008752 thru 0x010e81 (0x8730 bytes) 000794 1788179948.123311 fallocate 0x02c14d thru 0x03a546 (0xe3fa bytes) 000795 1788179948.134954 mapread 0x02932d thru 0x034f5e (0xbc32 bytes) 000796 1788179948.143627 mapread 0x016036 thru 0x01d3a4 (0x736f bytes) 000797 1788179948.145575 fallocate 0x02f412 thru 0x02fe26 (0xa15 bytes) 000798 1788179948.159429 mapread 0x0272de thru 0x02e563 (0x7286 bytes) 000799 1788179948.161678 trunc from 0x03ffbc to 0x03b071 (0x4f4a bytes) 000800 1788179948.176406 mapread 0x006771 thru 0x007c2a (0x14ba bytes) 000801 1788179948.178747 mapread 0x0072af thru 0x0078c0 (0x612 bytes) 000802 1788179948.181330 punch from 0x024187 to 0x02d318 (0x9192 bytes) 000803 1788179948.189502 punch from 0x010f83 to 0x013c1f (0x2c9d bytes) 000804 1788179948.203365 trunc from 0x02cc42 to 0x02da8f (0xe4e bytes) 000805 1788179948.208212 fallocate 0x01b822 thru 0x022686 (0x6e65 bytes) 000806 1788179948.213650 punch from 0x009d17 to 0x014157 (0xa441 bytes) 000807 1788179948.223600 read 0x028209 thru 0x02a972 (0x276a bytes) 000808 1788179948.234788 fallocate 0x01aaa1 thru 0x01d251 (0x27b1 bytes) 000809 1788179948.238996 punch from 0x021c82 to 0x03015f (0xe4de bytes) 000810 1788179948.246911 punch from 0x026fc3 to 0x03445c (0xd49a bytes) 000811 1788179948.251887 mapread 0x02aa09 thru 0x0372e5 (0xc8dd bytes) 000812 1788179948.313997 trunc from 0x03b072 to 0x03cb08 (0x1a97 bytes) 000814 1788179948.328357 fallocate 0x00bfba thru 0x014785 (0x87cc bytes) 000815 1788179948.339723 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000816 1788179948.465479 fallocate 0x005e08 thru 0x015b85 (0xfd7e bytes) 000818 1788179948.476852 MIRROR_EXTEND to 2 mirrors 000819 1788179949.197241 fallocate 0x0243d2 thru 0x02b31f (0x6f4e bytes) 000820 1788179949.263163 read 0x017745 thru 0x01ec9e (0x755a bytes) 000821 1788179949.285768 trunc from 0x00ccd9 to 0x015fe0 (0x9308 bytes) 000822 1788179949.293855 fallocate 0x0074cd thru 0x0150ba (0xdbee bytes) 000823 1788179949.309371 trunc from 0x03cb09 to 0x038709 (0x43ff bytes) 000824 1788179949.358694 punch from 0x00d916 to 0x01acce (0xd3b9 bytes) 000825 1788179949.365873 punch from 0x02bf81 to 0x038709 (0xc789 bytes) 000826 1788179949.372054 write 0x02ef0c thru 0x0322bd (0x33b2 bytes) 000827 1788179949.384632 MIRROR_RESYNC 2 mirrors 000828 1788179950.014173 trunc from 0x03870a to 0x038a43 (0x33a bytes) 000830 1788179950.099738 punch from 0x016b48 to 0x0212f8 (0xa7b1 bytes) 000831 1788179950.106006 punch from 0x008ecb to 0x00e4a6 (0x55dc bytes) 000832 1788179950.119480 MIRROR_RESYNC 2 mirrors 000834 1788179950.885914 fallocate 0x02bb11 thru 0x035318 (0x9808 bytes) 000835 1788179950.941852 read 0x000ff1 thru 0x00c77f (0xb78f bytes) 000836 1788179950.958633 write 0x03d376 thru 0x03ee69 (0x1af4 bytes) 000837 1788179950.960341 trunc from 0x0234d3 to 0x03349d (0xffcb bytes) 000838 1788179950.966320 trunc from 0x03ee6a to 0x020d8f (0x1e0da bytes) 000840 1788179950.990987 fallocate 0x021798 thru 0x028222 (0x6a8b bytes) 000841 1788179950.997900 write 0x031b51 thru 0x034d67 (0x3217 bytes) 000842 1788179951.000888 mapread 0x005d39 thru 0x00d6b0 (0x7978 bytes) 000843 1788179951.015746 punch from 0x027926 to 0x02d5dd (0x5cb8 bytes) 000844 1788179951.022547 trunc from 0x009a63 to 0x0101e0 (0x677e bytes) 000845 1788179951.027915 punch from 0x0006a0 to 0x00366e (0x2fcf bytes) 000846 1788179951.036561 read 0x01be7c thru 0x027d08 (0xbe8d bytes) 000847 1788179951.063266 trunc from 0x034d68 to 0x02499b (0x103cc bytes) 000848 1788179951.075922 fallocate 0x01495c thru 0x01f06e (0xa713 bytes) 000849 1788179951.083470 fallocate 0x0379fb thru 0x03c088 (0x468e bytes) 000850 1788179951.088410 mapread 0x02016e thru 0x02499b (0x482e bytes) 000851 1788179951.090822 trunc from 0x02499c to 0x013dd0 (0x10bcb bytes) 000852 1788179951.105680 trunc from 0x013dd1 to 0x02cd59 (0x18f89 bytes) 000853 1788179951.125075 read 0x010d1d thru 0x0163de (0x56c2 bytes) 000855 1788179951.136926 fallocate 0x025406 thru 0x027aa4 (0x269f bytes) 000856 1788179951.144016 read 0x00ea9b thru 0x016a42 (0x7fa8 bytes) 000857 1788179951.144754 trunc from 0x02cd5a to 0x023be0 (0x9179 bytes) 000858 1788179951.159953 mapread 0x00fbaf thru 0x01b2e6 (0xb738 bytes) 000859 1788179951.161459 trunc from 0x023be1 to 0x0046aa (0x1f536 bytes) 000861 1788179951.181072 mapread 0x0002c0 thru 0x0046aa (0x43eb bytes) 000862 1788179951.183518 fallocate 0x00fab9 thru 0x015ec6 (0x640e bytes) 000863 1788179951.188397 punch from 0x001587 to 0x0046aa (0x3124 bytes) 000865 1788179951.194470 trunc from 0x0046ab to 0x0226bd (0x1e013 bytes) 000866 1788179951.211573 write 0x01f8da thru 0x024f24 (0x564b bytes) 000867 1788179951.221963 fallocate 0x019d61 thru 0x02420c (0xa4ac bytes) 000869 1788179951.228057 punch from 0x00d427 to 0x00e754 (0x132e bytes) 000870 1788179951.232583 punch from 0x01b775 to 0x024f24 (0x97b0 bytes) 000872 1788179951.246609 trunc from 0x01bcfd to 0x01d39c (0x16a0 bytes) 000873 1788179951.253718 mapwrite 0x02fe71 thru 0x031400 (0x1590 bytes) 000874 1788179951.325683 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000875 1788179951.445889 mapread 0x024ffd thru 0x02d375 (0x8379 bytes) 000876 1788179951.473580 punch from 0x02cf6e to 0x031400 (0x4493 bytes) 000877 1788179951.478448 mapread 0x00ae02 thru 0x019b5b (0xed5a bytes) 000878 1788179951.505383 trunc from 0x0125e8 to 0x01f883 (0xd29c bytes) 000879 1788179951.511994 mapread 0x016ec9 thru 0x01c624 (0x575c bytes) 000880 1788179951.527953 mapwrite 0x003230 thru 0x0033f2 (0x1c3 bytes) 000883 1788179951.554343 read 0x0013a2 thru 0x0058fe (0x455d bytes) 000884 1788179951.569678 fallocate 0x0243c4 thru 0x03195c (0xd599 bytes) 000887 1788179951.588281 mapwrite 0x014f27 thru 0x024532 (0xf60c bytes) 000888 1788179951.624556 trunc from 0x03195d to 0x0263f5 (0xb567 bytes) 000889 1788179951.644899 trunc from 0x01af77 to 0x01bea0 (0xf2a bytes) 000890 1788179951.656472 write 0x036a44 thru 0x0375ce (0xb8b bytes) 000891 1788179951.657741 mapwrite 0x01b688 thru 0x01efd5 (0x394e bytes) 000892 1788179951.691442 mapread 0x0335bb thru 0x034647 (0x108d bytes) 000893 1788179951.699515 mapwrite 0x036e04 thru 0x03ffff (0x91fc bytes) 000895 1788179951.742187 trunc from 0x040000 to 0x0398b8 (0x6747 bytes) 000896 1788179951.761050 trunc from 0x016a10 to 0x017c25 (0x1216 bytes) 000897 1788179951.766650 punch from 0x030172 to 0x038925 (0x87b4 bytes) 000898 1788179951.771608 read 0x022b6e thru 0x02483a (0x1ccd bytes) 000899 1788179951.772600 trunc from 0x00ebc5 to 0x01b959 (0xcd95 bytes) 000901 1788179951.778507 fallocate 0x006714 thru 0x0092d5 (0x2bc2 bytes) 000902 1788179951.782824 trunc from 0x02e936 to 0x0398b8 (0xaf83 bytes) 000903 1788179951.787506 mapwrite 0x015d93 thru 0x020d34 (0xafa2 bytes) 000906 1788179951.817823 trunc from 0x027d6f to 0x02965d (0x18ef bytes) 000907 1788179951.826480 MIRROR_EXTEND to 2 mirrors 000908 1788179952.459384 MIRROR_RESYNC 2 mirrors 000909 1788179952.884706 write 0x00c12e thru 0x01b975 (0xf848 bytes) 000910 1788179952.929842 read 0x00e31a thru 0x017eae (0x9b95 bytes) 000911 1788179952.932016 read 0x02133d thru 0x02e98e (0xd652 bytes) 000912 1788179952.948843 mapwrite 0x0051b9 thru 0x00cf6d (0x7db5 bytes) 000913 1788179952.975494 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) 000914 1788179953.116011 trunc from 0x0398b9 to 0x02cbc3 (0xccf5 bytes) 000915 1788179953.137617 fallocate 0x03153b thru 0x03afb4 (0x9a7a bytes) 000916 1788179953.143183 write 0x0121a6 thru 0x016a33 (0x488e bytes) 000917 1788179953.152477 trunc from 0x03afb5 to 0x01345d (0x27b57 bytes) 000918 1788179953.178191 write 0x03418b thru 0x0352b4 (0x112a bytes) 000919 1788179953.179442 read 0x01a07c thru 0x026b5a (0xcadf bytes) 000920 1788179953.190654 mapread 0x0074b0 thru 0x013b6f (0xc6c0 bytes) 000921 1788179953.209552 read 0x033685 thru 0x0352b4 (0x1c30 bytes) 000922 1788179953.216611 fallocate 0x000e2c thru 0x00a58c (0x9761 bytes) 000923 1788179953.224732 MIRROR_EXTEND to 2 mirrors 000924 1788179953.872234 trunc from 0x0352b5 to 0x03e77e (0x94ca bytes) 000925 1788179954.035751 mapread 0x021e8d thru 0x030b08 (0xec7c bytes) 000926 1788179954.083687 punch from 0x015c92 to 0x01900a (0x3379 bytes) 000927 1788179954.087955 punch from 0x02b532 to 0x02bc76 (0x745 bytes) 000928 1788179954.094300 fallocate 0x03d5cb thru 0x03ffff (0x2a35 bytes) 000929 1788179954.112958 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 2: 0x8cf9d1e4 (stale) CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 1: 0x8cf9d1e4 Mirror 3: 0x8cf9d1e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e77f: skipped 000930 1788179955.267419 read 0x02866e thru 0x034970 (0xc303 bytes) 000931 1788179955.283879 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 2: 0x8cf9d1e4 (stale) CRC-32 checksum value for chunk [0, 0x3e77f): Mirror 1: 0x8cf9d1e4 Mirror 3: 0x8cf9d1e4 Mirror 4: 0x8cf9d1e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e77f: skipped 000932 1788179956.379660 punch from 0x0261f7 to 0x02bcee (0x5af8 bytes) 000933 1788179956.417558 write 0x03c826 thru 0x03ffff (0x37da bytes) 000934 1788179956.429606 write 0x011ca5 thru 0x014bad (0x2f09 bytes) 000935 1788179956.438654 write 0x00b3c9 thru 0x00be3f (0xa77 bytes) 000936 1788179956.443555 trunc from 0x040000 to 0x00d569 (0x32a96 bytes) 000937 1788179956.470989 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xd56a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd56a): Mirror 2: 0x63ee5104 (stale) Mirror 3: 0x63ee5104 (stale) Mirror 4: 0x63ee5104 (stale) CRC-32 checksum value for chunk [0, 0xd56a): Mirror 1: 0x707091aa Mirror 5: 0x707091aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd56a: skipped 000939 1788179957.379996 trunc from 0x00d56a to 0x01bc1b (0xe6b2 bytes) 000940 1788179957.425387 read 0x00453e thru 0x00d728 (0x91eb bytes) 000941 1788179957.434385 trunc from 0x01bc1c to 0x0351a9 (0x1958e bytes) 000942 1788179957.469573 mapwrite 0x016f3d thru 0x02679f (0xf863 bytes) 000943 1788179957.556627 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1c7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x351aa): Mirror 3: 0xfeca7aa4 (stale) Mirror 4: 0xfeca7aa4 (stale) Mirror 5: 0x5928ca40 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351aa): Mirror 3: 0xfeca7aa4 (stale) Mirror 4: 0xfeca7aa4 (stale) Mirror 5: 0x5928ca40 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351aa: skipped 000944 1788179957.954060 write 0x024cb8 thru 0x02d666 (0x89af bytes) 000945 1788179957.976407 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x351aa): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351aa): Mirror 3: 0xfeca7aa4 (stale) Mirror 4: 0xfeca7aa4 (stale) Mirror 5: 0x5928ca40 (stale) CRC-32 checksum value for chunk [0, 0x351aa): Mirror 1: 0x6f82bf53 Mirror 6: 0x6f82bf53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351aa: skipped 000946 1788179958.864963 mapread 0x0011be thru 0x00ac3d (0x9a80 bytes) 000947 1788179958.906747 mapwrite 0x00e11b thru 0x018d46 (0xac2c bytes) 000948 1788179959.020192 read 0x0287e6 thru 0x032852 (0xa06d bytes) 000949 1788179959.030498 trunc from 0x02a606 to 0x0351a9 (0xaba4 bytes) 000950 1788179959.042047 mapwrite 0x017083 thru 0x021677 (0xa5f5 bytes) 000951 1788179959.093239 trunc from 0x01a780 to 0x0299a1 (0xf222 bytes) 000952 1788179959.103656 trunc from 0x0351aa to 0x026651 (0xeb58 bytes) 000953 1788179959.138774 write 0x036576 thru 0x03da51 (0x74dc bytes) 000954 1788179959.146651 fallocate 0x013bbd thru 0x013ce5 (0x129 bytes) 000958 1788179959.160627 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3da52): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da52): Mirror 3: 0x532225c7 (stale) Mirror 4: 0x532225c7 (stale) Mirror 5: 0xe897c0fa (stale) Mirror 6: 0x15bd936b (stale) CRC-32 checksum value for chunk [0, 0x3da52): Mirror 1: 0x40e159f1 Mirror 7: 0x40e159f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da52: skipped 000959 1788179960.185013 trunc from 0x03da52 to 0x03ed55 (0x1304 bytes) 000961 1788179960.269873 trunc from 0x03ed56 to 0x031665 (0xd6f0 bytes) 000962 1788179960.295974 write 0x0257bc thru 0x029233 (0x3a78 bytes) 000963 1788179960.308230 fallocate 0x036086 thru 0x03d9f0 (0x796b bytes) 000965 1788179960.313917 write 0x03fab4 thru 0x03ffff (0x54c bytes) 000966 1788179960.314940 mapwrite 0x0186c6 thru 0x023e25 (0xb760 bytes) 000968 1788179960.378242 punch from 0x01a214 to 0x022595 (0x8382 bytes) 000969 1788179960.384066 mapread 0x01c01d thru 0x024e88 (0x8e6c bytes) 000970 1788179960.403075 write 0x01d758 thru 0x02be42 (0xe6eb bytes) 000971 1788179960.406926 trunc from 0x019a35 to 0x020a1a (0x6fe6 bytes) 000973 1788179960.423491 read 0x0091ba thru 0x00f418 (0x625f bytes) 000974 1788179960.434228 punch from 0x039226 to 0x03ffff (0x6dda bytes) 000976 1788179960.440954 mapwrite 0x014741 thru 0x01f09b (0xa95b bytes) 000977 1788179960.475492 mapread 0x025a2b thru 0x0327f0 (0xcdc6 bytes) 000978 1788179960.485267 trunc from 0x006aa7 to 0x01127b (0xa7d5 bytes) 000979 1788179960.491449 read 0x01fd65 thru 0x02ec5c (0xeef8 bytes) 000980 1788179960.492210 read 0x002a1d thru 0x00b92a (0x8f0e bytes) 000981 1788179960.504695 mapread 0x018476 thru 0x01b20d (0x2d98 bytes) 000982 1788179960.505755 read 0x01d0d4 thru 0x0230e2 (0x600f bytes) 000983 1788179960.506871 trunc from 0x040000 to 0x00961c (0x369e3 bytes) 000984 1788179960.526732 fallocate 0x019530 thru 0x027195 (0xdc66 bytes) 000985 1788179960.531854 mapread 0x013720 thru 0x01945b (0x5d3c bytes) 000986 1788179960.557576 MIRROR_RESYNC 6 mirrors 000987 1788179962.169884 fallocate 0x03505e thru 0x03ffff (0xafa2 bytes) 000988 1788179962.211959 fallocate 0x0375ac thru 0x03df77 (0x69cc bytes) 000989 1788179962.218314 mapwrite 0x00cd3a thru 0x0108e6 (0x3bad bytes) 000990 1788179962.260076 write 0x01d314 thru 0x028a38 (0xb725 bytes) 000991 1788179962.271763 mapwrite 0x0296b2 thru 0x033977 (0xa2c6 bytes) 000992 1788179962.409599 punch from 0x000928 to 0x00d5c3 (0xcc9c bytes) 000993 1788179962.420190 trunc from 0x033978 to 0x037aeb (0x4174 bytes) 000994 1788179962.472639 trunc from 0x037aec to 0x019183 (0x1e968 bytes) 000995 1788179962.502687 trunc from 0x00a5c1 to 0x00e930 (0x4370 bytes) 000996 1788179962.509579 write 0x0061f2 thru 0x0083cb (0x21da bytes) 000997 1788179962.531958 trunc from 0x019184 to 0x02ee33 (0x15cb0 bytes) 000998 1788179962.556582 fallocate 0x027c1c thru 0x029976 (0x1d5b bytes) 000999 1788179962.574792 MIRROR_RESYNC 6 mirrors 001000 1788179964.232759 mapwrite 0x01fc6f thru 0x02f8ac (0xfc3e bytes) All operations completed A-OK! PASS 70a (71s) == sanity-flr test 71: check mirror extend parent fid ==== 08:39:28 (1788179968) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1ff:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 08:39:40 (1788179980) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163796 s, 12.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981577 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 08:39:49 (1788179989) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4669216 .. Extending file size to 4696160 .. Extending file size to 8442784 .. Extending file size to 4718368 .. Extending file size to 6178496 .. Extending file size to 8888992 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8534592 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8624288 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9051840 .. Waiting 61450 61451 61452 61455 61456 failed 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) 192.168.203.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:41:09 (1788180069) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4308608 .. Extending file size to 3660544 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7905664 .. Extending file size to 4889568 .. failed Extending file size to 7534912 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8117504 .. Extending file size to 8684864 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8297984 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8896960 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62987 62988 62990 62991 62993 done 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) 192.168.203.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:42:31 (1788180151) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2176 1269128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23992 3583028 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18560 3588460 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42552 7171488 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2176 1269128 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23992 3583028 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18560 3588460 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42552 7171488 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16674 s, 12.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201893 s, 10.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9a8b7583 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:42:48 (1788180168) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0869703 s, 24.1 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 08:42:55 (1788180175) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000400:0x12b:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:43:02 (1788180182) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0731088 s, 28.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x12c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:43:10 (1788180190) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100946 s, 20.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x12f:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:43:17 (1788180197) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0964771 s, 21.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x131:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:43:50 (1788180230) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104267 s, 10.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x288:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x288:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x288:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x288:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x289:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:44:42 (1788180282) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.097719 s, 21.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x133:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:45:29 (1788180329) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg332-server: Pool lustre.test_204f created Adding targets to pool oleg332-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg332-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106844 s, 19.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg332-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg332-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg332-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:46:42 (1788180402) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:46:49 (1788180409) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:46:56 (1788180416) oleg332-server: Pool lustre.test_206 created Adding targets to pool oleg332-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg332-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg332-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg332-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg332-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:47:23 (1788180443) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875699 s, 12.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245072 s, 42.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8d11c6ab Mirror 3: 0x8d11c6ab PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:47:33 (1788180453) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:47:36 (1788180456) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:47:39 (1788180459) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0760423 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf90000 Read this (4 bytes): BBBB 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:48:15 (1788180495) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0158164 s, 518 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:48:21 (1788180501) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560011 s, 18.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x13e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 08:48:22.000000000 -0400 Modify: 2026-08-31 08:48:22.000000000 -0400 Change: 2026-08-31 08:48:23.000000000 -0400 Birth: 2026-08-31 08:48:22.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:48:28 (1788180508) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759743 s, 13.8 MB/s oleg332-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg332-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg332-server: O/280000400/d31/13f: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:48:53 (1788180533) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.239993 s, 171 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0204538 s, 200 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 31 08:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 31 08:48 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:49:01 (1788180541) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:49:07 (1788180547) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1637 sec =========== 08:49:14 (1788180554) === sanity-flr: start cleanup 08:49:16 (1788180556) === === sanity-flr: finish cleanup 08:49:25 (1788180565) ===