-----============= acceptance-small: sanity-flr ============----- Tue Jul 28 02:24:44 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg410-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:25:03 (1785219903) === oleg410-client.virtnet: executing check_config_client /mnt/lustre oleg410-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg410-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d5d8b8af000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d5d8b8af000.idle_timeout=debug disable quota as required oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:25:43 (1785219943) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8226] == sanity-flr test 0a: lfs mirror create with -N option == 02:25:46 (1785219946) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:26:01 (1785219961) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:26:05 (1785219965) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:26:08 (1785219968) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:26:32 (1785219992) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:26:35 (1785219995) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 02:26:39 (1785219999) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d5d8b8af000.stats=clear osc.lustre-OST0001-osc-ffff8d5d8b8af000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:26:47 (1785220007) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:26:57 (1785220017) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:27:05 (1785220025) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:27:09 (1785220029) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:27:17 (1785220037) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 02:27:27 (1785220047) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:27:35 (1785220055) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:27:43 (1785220063) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:27:52 (1785220072) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:28:04 (1785220084) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:28:12 (1785220092) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.809292 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562956 s, 18.6 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:28:23 (1785220103) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.592911 s, 28.3 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:28:33 (1785220113) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:28:45 (1785220125) Stopping /mnt/lustre-ost1 (opts:) on oleg410-server oleg410-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg410-server: oleg410-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg410-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg410-server oleg410-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg410-server: oleg410-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg410-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (80s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:30:05 (1785220205) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.210923 s, 19.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg410-server oleg410-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg410-server: oleg410-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg410-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:30:47 (1785220247) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg410-server oleg410-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg410-server: oleg410-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg410-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg410-server oleg410-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg410-server: oleg410-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg410-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:32:09 (1785220329) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.33 s, 15.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 28 02:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.027s user 0m0.001s sys 0m0.158s Stopping /mnt/lustre-ost1 (opts:) on oleg410-server oleg410-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d5d8b8af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.439s user 0m0.001s sys 0m0.214s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg410-server: oleg410-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg410-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg410-server oleg410-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.225s user 0m0.001s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg410-server: oleg410-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg410-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg410-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d5d8b8af000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:33:46 (1785220426) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:33:49 (1785220429) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:33:52 (1785220432) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:33:56 (1785220436) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 02:34:03 (1785220443) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.635851 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:34:14 (1785220454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300943 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028992 s, 36.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:34:21 (1785220461) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:34:29 (1785220469) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000763665 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00227387 s, 450 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102396 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00230331 s, 445 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 02:34:37 (1785220477) create test files with size 15744457 .. 7bb042420e5e2ebc2b30599fdcc447ba - 321ebc9932a3521867e35935fb5f3802 - 7407b79c1e6c12db316d321f528614c0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg410-client.virtnet /mnt/lustre (opts:) Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 02:34:57 (1785220497) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:35:08 (1785220508) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:35:14 (1785220514) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691349 s, 15.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491064 s, 21.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 02:35:23 (1785220523) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0563723 s, 37.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.232282 s, 18.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180666 s, 11.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 02:35:38 (1785220538) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 02:35:42 (1785220542) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:35:46 (1785220546) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383678 s, 27.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 02:35:53 (1785220553) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.351848 s, 8.9 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:36:01 (1785220561) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:36:10 (1785220570) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.260884 s, 40.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg410-client.virtnet /mnt/lustre (opts:) Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:36:19 (1785220579) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8d5d8a90f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.337911 s, 31.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg410-client.virtnet /mnt/lustre (opts:) Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8d5dc0d8e800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:36:34 (1785220594) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:37:11 (1785220631) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:37:46 (1785220666) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 02:37:49 (1785220669) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.198973 s, 15.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.104127 s, 30.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7db3f807 (nosync,stale) Mirror 2: 0x7db3f807 (nosync,stale) ** verify mirror 2 doesn't change original checksum: b751c0bc473d3722b5d64f3d6c406f32 - mirror 1 checksum: a356852f12f3f13079a56edafa69b7b8 - mirror 2 checksum: b751c0bc473d3722b5d64f3d6c406f32 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-28T02:37:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-28T02:37:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b751c0bc473d3722b5d64f3d6c406f32 - mirror 1 checksum: a356852f12f3f13079a56edafa69b7b8 - mirror 2 checksum: a356852f12f3f13079a56edafa69b7b8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-28T02:37:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-28T02:37:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:37:57 (1785220677) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196861 s, 53.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:38:04 (1785220684) Failing mds1 on oleg410-server Stopping /mnt/lustre-mds1 (opts:) on oleg410-server 02:38:15 (1785220695) shut down facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server Failover mds1 to oleg410-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 02:38:37 (1785220717) targets are mounted 02:38:37 (1785220717) facet_failover done oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291298 s, 36.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:38:53 (1785220733) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0451945 s, 453 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0331405 s, 618 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:39:01 (1785220741) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.072911 s, 281 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0106927 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:39:11 (1785220751) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00985088 s, 1.7 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:39:19 (1785220759) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0645166 s, 317 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:39:27 (1785220767) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00820126 s, 499 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0189743 s, 216 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:39:33 (1785220773) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:39:52 (1785220792) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:40:18 (1785220818) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:40:41 (1785220841) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:41:03 (1785220863) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 02:41:10 (1785220870) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57546 mirror split pid 57547 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57546 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57547 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 02:42:17 (1785220937) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 939 skipping zero length zero range skipping zero size read truncating to largest ever: 0x70b 000004 1785220939.809354 trunc from 00000000 to 0x00070a (0x70b bytes) 000005 1785220939.821847 mapread 0x0002a1 thru 0x00070a (0x46a bytes) 000006 1785220939.827840 read 0x000591 thru 0x00070a (0x17a bytes) fallocating to largest ever: 0x40000 000007 1785220939.828973 fallocate 0x03c3c9 thru 0x03ffff (0x3c37 bytes) 000008 1785220939.833232 mapwrite 0x00e8e5 thru 0x01cca8 (0xe3c4 bytes) 000009 1785220939.892990 mapwrite 0x0161a1 thru 0x0249bc (0xe81c bytes) 000010 1785220939.937880 trunc from 0x024e18 to 0x0309bf (0xbba8 bytes) 000011 1785220939.945565 write 0x0110e2 thru 0x01c8ef (0xb80e bytes) 000012 1785220939.957981 MIRROR_EXTEND to 2 mirrors 000013 1785220940.638818 write 0x031023 thru 0x03fde6 (0xedc4 bytes) 000014 1785220940.695787 trunc from 0x040000 to 0x024252 (0x1bdad bytes) 000015 1785220940.736820 mapread 0x01a22f thru 0x024252 (0xa024 bytes) 000016 1785220940.785460 fallocate 0x011e12 thru 0x014fc8 (0x31b7 bytes) 000017 1785220940.791753 mapwrite 0x00f960 thru 0x01aacd (0xb16e bytes) 000018 1785220940.865378 read 0x00e614 thru 0x01c7b9 (0xe1a6 bytes) 000019 1785220940.866249 punch from 0x01ffe1 to 0x024252 (0x4272 bytes) 000020 1785220940.879969 trunc from 0x0005e6 to 0x00ce2c (0xc847 bytes) 000021 1785220940.887119 read 0x00bf16 thru 0x018437 (0xc522 bytes) 000022 1785220940.898938 trunc from 0x024253 to 0x033f23 (0xfcd1 bytes) 000023 1785220940.915659 write 0x038c47 thru 0x03ffff (0x73b9 bytes) 000024 1785220940.918162 trunc from 0x040000 to 0x035ba7 (0xa458 bytes) 000025 1785220940.950924 MIRROR_RESYNC 2 mirrors 000026 1785220941.571120 read 0x003456 thru 0x00eb7f (0xb72a bytes) 000027 1785220941.581298 write 0x036f9a thru 0x03ffff (0x9066 bytes) 000028 1785220941.603481 write 0x03d3d8 thru 0x03ffff (0x2c28 bytes) 000029 1785220941.609740 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000030 1785220941.796585 fallocate 0x01a688 thru 0x02a42d (0xfda6 bytes) 000031 1785220941.802065 fallocate 0x00daba thru 0x01d4e2 (0xfa29 bytes) 000033 1785220941.808853 write 0x01ea16 thru 0x023428 (0x4a13 bytes) 000034 1785220941.828452 read 0x020bf0 thru 0x024cf0 (0x4101 bytes) 000035 1785220941.832909 trunc from 0x040000 to 0x010a28 (0x2f5d7 bytes) 000036 1785220941.847721 trunc from 0x010a29 to 0x024920 (0x13ef8 bytes) 000038 1785220941.859996 fallocate 0x001db4 thru 0x00b08c (0x92d9 bytes) 000039 1785220941.865063 trunc from 0x024921 to 0x01e037 (0x68e9 bytes) 000040 1785220941.881784 trunc from 0x01e038 to 0x026a9a (0x8a63 bytes) 000041 1785220941.901645 punch from 0x007db1 to 0x011fea (0xa23a bytes) 000042 1785220941.911102 read 0x009078 thru 0x00f460 (0x63e9 bytes) 000043 1785220941.926438 write 0x01e044 thru 0x02ac7a (0xcc37 bytes) 000044 1785220941.933334 trunc from 0x0175ca to 0x01bd66 (0x479d bytes) 000045 1785220941.946033 write 0x003876 thru 0x008ae1 (0x526c bytes) 000047 1785220941.952307 fallocate 0x003ea6 thru 0x00cf1d (0x9078 bytes) 000048 1785220941.959996 punch from 0x023d9f to 0x028d2d (0x4f8f bytes) 000049 1785220941.990385 trunc from 0x02ac7b to 0x0129f1 (0x18289 bytes) 000050 1785220942.006012 trunc from 0x00398f to 0x00f4c1 (0xbb33 bytes) 000051 1785220942.022779 MIRROR_EXTEND to 2 mirrors 000052 1785220942.554134 mapread 0x00e361 thru 0x0129f1 (0x4691 bytes) 000053 1785220942.586245 write 0x0325f7 thru 0x03fdf4 (0xd7fe bytes) 000054 1785220942.623767 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000055 1785220942.790638 MIRROR_EXTEND to 2 mirrors 000056 1785220943.494842 trunc from 0x03fdf5 to 0x03223f (0xdbb5 bytes) 000057 1785220943.555707 write 0x03fd28 thru 0x03ffff (0x2d8 bytes) 000058 1785220943.557433 mapread 0x024e88 thru 0x0301b2 (0xb32b bytes) 000059 1785220943.614878 read 0x024823 thru 0x034695 (0xfe73 bytes) 000060 1785220943.617569 mapwrite 0x02a224 thru 0x02a8b9 (0x696 bytes) 000061 1785220943.662750 read 0x00ce84 thru 0x014425 (0x75a2 bytes) 000062 1785220943.701417 fallocate 0x02b96a thru 0x02ed7a (0x3411 bytes) 000063 1785220943.718535 punch from 0x020c76 to 0x0229dd (0x1d68 bytes) 000064 1785220943.739400 MIRROR_RESYNC 2 mirrors 000065 1785220944.864925 trunc from 0x040000 to 0x007f44 (0x380bb bytes) 000066 1785220944.927119 mapwrite 0x02c96c thru 0x03bc78 (0xf30d bytes) 000067 1785220945.075060 trunc from 0x03bc79 to 0x01ada4 (0x20ed4 bytes) 000068 1785220945.099402 trunc from 0x0145cc to 0x01ada4 (0x67d9 bytes) 000069 1785220945.107571 fallocate 0x01928d thru 0x01a60c (0x1380 bytes) 000071 1785220945.118604 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ada5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ada5): Mirror 2: 0xaa07ac45 (stale) CRC-32 checksum value for chunk [0, 0x1ada5): Mirror 1: 0xaa07ac45 Mirror 3: 0xaa07ac45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ada5: skipped 000072 1785220945.881407 mapread 0x007a58 thru 0x00ccc2 (0x526b bytes) 000073 1785220945.930123 write 0x00ec88 thru 0x00f827 (0xba0 bytes) 000074 1785220945.982962 write 0x03c8c0 thru 0x03f48e (0x2bcf bytes) 000075 1785220945.984705 read 0x0072ee thru 0x012978 (0xb68b bytes) 000076 1785220945.996609 punch from 0x003a61 to 0x00e6a7 (0xac47 bytes) 000077 1785220946.010690 fallocate 0x02bd2e thru 0x030aaf (0x4d82 bytes) 000078 1785220946.016234 punch from 0x003d28 to 0x00b367 (0x7640 bytes) 000079 1785220946.022465 trunc from 0x025df2 to 0x026e27 (0x1036 bytes) 000080 1785220946.027215 trunc from 0x03c099 to 0x03c2f5 (0x25d bytes) 000081 1785220946.032810 mapwrite 0x00aae6 thru 0x00c256 (0x1771 bytes) 000082 1785220946.052962 write 0x030b3c thru 0x0340a1 (0x3566 bytes) 000083 1785220946.063271 write 0x00b77c thru 0x016c1a (0xb49f bytes) 000084 1785220946.068449 trunc from 0x03f48f to 0x01151b (0x2df73 bytes) 000085 1785220946.094576 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1151c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1151c): Mirror 2: 0x2f38af29 (stale) Mirror 3: 0x2f38af29 (stale) CRC-32 checksum value for chunk [0, 0x1151c): Mirror 1: 0x7cf659a1 Mirror 4: 0x7cf659a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1151c: skipped 000086 1785220946.960921 punch from 0x008d3b to 0x009fa8 (0x126e bytes) 000087 1785220946.994407 trunc from 0x0075d2 to 0x0097b1 (0x21e0 bytes) 000088 1785220946.999166 write 0x02eb89 thru 0x039779 (0xabf1 bytes) 000089 1785220947.011437 mapwrite 0x02a6cd thru 0x030ec9 (0x67fd bytes) 000090 1785220947.058677 trunc from 0x03977a to 0x01c8e1 (0x1ce98 bytes) 000091 1785220947.126455 mapread 0x00ad24 thru 0x016c46 (0xbf23 bytes) 000092 1785220947.170965 trunc from 0x016e8d to 0x01c8e1 (0x5a55 bytes) 000093 1785220947.185266 fallocate 0x02405f thru 0x026734 (0x26d6 bytes) 000094 1785220947.195041 mapread 0x0174e0 thru 0x01c375 (0x4e96 bytes) 000095 1785220947.224176 fallocate 0x016fc0 thru 0x0266f4 (0xf735 bytes) 000096 1785220947.232498 mapread 0x018e33 thru 0x022120 (0x92ee bytes) 000097 1785220947.263289 mapwrite 0x00e16f thru 0x00e832 (0x6c4 bytes) 000098 1785220947.312552 punch from 0x001689 to 0x005349 (0x3cc1 bytes) 000099 1785220947.345055 MIRROR_RESYNC 4 mirrors 000100 1785220948.342486 read 0x008425 thru 0x015a83 (0xd65f bytes) 000101 1785220948.365644 fallocate 0x01c634 thru 0x02042d (0x3dfa bytes) 000102 1785220948.409106 mapread 0x00cdbd thru 0x0177fe (0xaa42 bytes) 000103 1785220948.447325 mapread 0x01d9d3 thru 0x026734 (0x8d62 bytes) 000104 1785220948.463030 punch from 0x0158aa to 0x01e156 (0x88ad bytes) 000105 1785220948.471651 punch from 0x01bdf3 to 0x01cb11 (0xd1f bytes) 000106 1785220948.479569 read 0x0026ac thru 0x00af2e (0x8883 bytes) 000107 1785220948.491521 read 0x017861 thru 0x01bac5 (0x4265 bytes) 000109 1785220948.493424 fallocate 0x0285fa thru 0x03683e (0xe245 bytes) 000110 1785220948.500492 read 0x029cc4 thru 0x0352c6 (0xb603 bytes) 000112 1785220948.518708 mapread 0x00ca9d thru 0x019eb3 (0xd417 bytes) 000113 1785220948.521256 punch from 0x02fa46 to 0x03683e (0x6df9 bytes) 000114 1785220948.526938 read 0x02118e thru 0x0213f9 (0x26c bytes) 000115 1785220948.528900 trunc from 0x03683f to 0x02a1c8 (0xc676 bytes) 000117 1785220948.583532 read 0x022d3a thru 0x02a1c8 (0x748f bytes) 000118 1785220948.588831 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a1c9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1c9): Mirror 2: 0x1dce1443 (stale) Mirror 3: 0x1dce1443 (stale) Mirror 4: 0x1dce1443 (stale) CRC-32 checksum value for chunk [0, 0x2a1c9): Mirror 1: 0x1dce1443 Mirror 5: 0x1dce1443 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1c9: skipped 000119 1785220949.359971 fallocate 0x00c9d9 thru 0x0183d0 (0xb9f8 bytes) 000120 1785220949.494579 fallocate 0x008cc6 thru 0x0170a3 (0xe3de bytes) 000121 1785220949.499833 fallocate 0x02de81 thru 0x0320e6 (0x4266 bytes) 000122 1785220949.507423 punch from 0x0189e3 to 0x021589 (0x8ba7 bytes) 000123 1785220949.517735 fallocate 0x024ea6 thru 0x027e55 (0x2fb0 bytes) 000124 1785220949.527886 read 0x009d22 thru 0x013b0b (0x9dea bytes) 000125 1785220949.537527 write 0x0261ed thru 0x032cde (0xcaf2 bytes) 000126 1785220949.557382 mapread 0x02df61 thru 0x032cde (0x4d7e bytes) 000129 1785220949.564453 read 0x006d80 thru 0x008cfd (0x1f7e bytes) 000130 1785220949.578843 trunc from 0x016e5d to 0x0198d8 (0x2a7c bytes) 000131 1785220949.584337 write 0x015263 thru 0x01d49d (0x823b bytes) 000132 1785220949.588827 punch from 0x003ffc to 0x011ac2 (0xdac7 bytes) 000134 1785220949.602494 punch from 0x00408a to 0x00dead (0x9e24 bytes) 000136 1785220949.624837 mapread 0x023720 thru 0x026d0c (0x35ed bytes) 000137 1785220949.627717 trunc from 0x032cdf to 0x01e4ec (0x147f2 bytes) 000138 1785220949.667547 trunc from 0x01e4ed to 0x003f25 (0x1a5c7 bytes) 000139 1785220949.698649 trunc from 0x003f26 to 0x00c9b2 (0x8a8d bytes) 000140 1785220949.718678 read 0x002fb4 thru 0x0068a9 (0x38f6 bytes) 000141 1785220949.732726 trunc from 0x00c9b3 to 0x0136a1 (0x6cef bytes) 000143 1785220949.761438 fallocate 0x01cbea thru 0x01d41b (0x832 bytes) 000144 1785220949.771753 read 0x0123a8 thru 0x0136a1 (0x12fa bytes) 000145 1785220949.783273 read 0x00904f thru 0x0136a1 (0xa653 bytes) 000146 1785220949.784109 punch from 0x00a21f to 0x0136a1 (0x9483 bytes) 000147 1785220949.794334 fallocate 0x02ac35 thru 0x036c87 (0xc053 bytes) 000148 1785220949.807176 read 0x012b5f thru 0x0133ab (0x84d bytes) 000149 1785220949.823306 trunc from 0x0036db to 0x00be79 (0x879f bytes) 000150 1785220949.832796 write 0x03437d thru 0x03e261 (0x9ee5 bytes) 000151 1785220949.844362 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e262): Mirror 2: 0xf4f01d81 (stale) Mirror 4: 0xf4f01d81 (stale) Mirror 5: 0xf4f01d81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e262): Mirror 2: 0xf4f01d81 (stale) Mirror 4: 0xf4f01d81 (stale) Mirror 5: 0xf4f01d81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e262: skipped 000152 1785220950.285109 punch from 0x02d717 to 0x03768f (0x9f79 bytes) 000153 1785220950.291348 trunc from 0x03e262 to 0x037da4 (0x64bd bytes) 000154 1785220950.334311 mapwrite 0x026bae thru 0x02cfd1 (0x6424 bytes) 000155 1785220950.381729 trunc from 0x037da5 to 0x00d2fc (0x2aaa8 bytes) 000157 1785220950.405333 trunc from 0x00d2fd to 0x038a6a (0x2b76e bytes) 000158 1785220950.425593 fallocate 0x00e7bd thru 0x0142b3 (0x5af7 bytes) 000159 1785220950.432476 fallocate 0x0201aa thru 0x02f05c (0xeeb3 bytes) 000160 1785220950.441805 MIRROR_RESYNC 4 mirrors 000161 1785220951.071597 trunc from 0x011b95 to 0x01959c (0x7a08 bytes) 000162 1785220951.100494 trunc from 0x01455d to 0x01fe6b (0xb90f bytes) 000163 1785220951.105310 trunc from 0x038a6b to 0x02016e (0x188fc bytes) 000164 1785220951.128531 mapread 0x017b30 thru 0x01e0d9 (0x65aa bytes) 000165 1785220951.149403 MIRROR_RESYNC 4 mirrors 000166 1785220951.811950 read 0x011c65 thru 0x0157d9 (0x3b75 bytes) 000167 1785220951.819257 punch from 0x01cb00 to 0x02016e (0x366f bytes) 000168 1785220951.855790 write 0x019f13 thru 0x01a944 (0xa32 bytes) 000169 1785220951.865073 read 0x00ec6e thru 0x010618 (0x19ab bytes) 000170 1785220951.878739 MIRROR_RESYNC 4 mirrors 000171 1785220952.837010 read 0x00527a thru 0x012737 (0xd4be bytes) 000172 1785220952.845451 punch from 0x00d465 to 0x00d76d (0x309 bytes) 000173 1785220952.901185 punch from 0x01913f to 0x01965d (0x51f bytes) 000174 1785220952.918170 mapread 0x018621 thru 0x01c9af (0x438f bytes) 000175 1785220952.940375 mapread 0x01b0c1 thru 0x02016e (0x50ae bytes) 000176 1785220952.942220 fallocate 0x0042c9 thru 0x0101d2 (0xbf0a bytes) 000177 1785220952.948344 write 0x00e736 thru 0x019a5f (0xb32a bytes) 000178 1785220952.955539 trunc from 0x00d4ed to 0x01d167 (0xfc7b bytes) 000180 1785220952.972133 trunc from 0x02016f to 0x01681d (0x9951 bytes) 000181 1785220952.997084 mapwrite 0x02cd47 thru 0x03886a (0xbb24 bytes) 000182 1785220953.058481 trunc from 0x0109b1 to 0x013376 (0x29c6 bytes) 000183 1785220953.064479 trunc from 0x021e05 to 0x0306ba (0xe8b6 bytes) 000184 1785220953.070180 write 0x01b3df thru 0x026d4a (0xb96c bytes) 000185 1785220953.082603 mapread 0x019fd7 thru 0x028e56 (0xee80 bytes) 000186 1785220953.102741 trunc from 0x02e9d4 to 0x0378be (0x8eeb bytes) 000187 1785220953.109563 fallocate 0x006e3c thru 0x00a20a (0x33cf bytes) 000189 1785220953.117937 fallocate 0x0189f6 thru 0x0254b7 (0xcac2 bytes) 000190 1785220953.133498 mapread 0x02098c thru 0x022a6a (0x20df bytes) 000191 1785220953.134536 trunc from 0x00c233 to 0x01201a (0x5de8 bytes) 000192 1785220953.162850 MIRROR_RESYNC 4 mirrors 000193 1785220954.495269 mapread 0x00daa3 thru 0x0180df (0xa63d bytes) 000194 1785220954.599421 write 0x012ad3 thru 0x01c721 (0x9c4f bytes) 000195 1785220954.707655 mapwrite 0x024121 thru 0x030a75 (0xc955 bytes) 000196 1785220954.803579 trunc from 0x006ec4 to 0x00ae61 (0x3f9e bytes) 000197 1785220954.809687 fallocate 0x012a2b thru 0x01fbd3 (0xd1a9 bytes) 000198 1785220954.824943 MIRROR_RESYNC 4 mirrors 000199 1785220955.867463 mapread 0x00569e thru 0x00c47c (0x6ddf bytes) 000200 1785220955.891279 read 0x00b434 thru 0x019866 (0xe433 bytes) 000202 1785220955.900737 punch from 0x033c1e to 0x03886a (0x4c4d bytes) 000203 1785220955.934816 read 0x030238 thru 0x03886a (0x8633 bytes) 000205 1785220955.947469 fallocate 0x03621b thru 0x03f810 (0x95f6 bytes) 000206 1785220955.953061 fallocate 0x01de64 thru 0x022a66 (0x4c03 bytes) 000207 1785220955.958526 mapread 0x0264bd thru 0x02f9a0 (0x94e4 bytes) 000208 1785220955.960546 mapread 0x00a462 thru 0x00b16a (0xd09 bytes) 000209 1785220955.962116 mapread 0x010fe3 thru 0x01e098 (0xd0b6 bytes) 000210 1785220955.964038 trunc from 0x024ab2 to 0x032241 (0xd790 bytes) 000211 1785220955.970258 punch from 0x032bcf to 0x03886a (0x5c9c bytes) 000212 1785220955.975109 mapwrite 0x023a10 thru 0x02b45a (0x7a4b bytes) 000213 1785220956.060578 write 0x0322b4 thru 0x03fedf (0xdc2c bytes) 000214 1785220956.069855 trunc from 0x00f316 to 0x0165a2 (0x728d bytes) 000215 1785220956.088115 punch from 0x0116e0 to 0x017616 (0x5f37 bytes) 000216 1785220956.099707 mapread 0x008c25 thru 0x00e93c (0x5d18 bytes) 000217 1785220956.104581 fallocate 0x034f05 thru 0x03eb4f (0x9c4b bytes) 000218 1785220956.133827 mapwrite 0x029d82 thru 0x034c2c (0xaeab bytes) 000219 1785220956.258191 read 0x0171b6 thru 0x01f119 (0x7f64 bytes) 000221 1785220956.271393 punch from 0x027e47 to 0x02acf7 (0x2eb1 bytes) 000222 1785220956.284785 read 0x01ebe0 thru 0x02283c (0x3c5d bytes) 000223 1785220956.285785 mapwrite 0x036d03 thru 0x03ffff (0x92fd bytes) 000224 1785220956.336240 mapwrite 0x01973d thru 0x0276c0 (0xdf84 bytes) 000225 1785220956.394350 mapwrite 0x00e32a thru 0x0115db (0x32b2 bytes) 000226 1785220956.406805 trunc from 0x040000 to 0x0268b4 (0x1974b bytes) 000227 1785220956.429260 read 0x00537c thru 0x00bb28 (0x67ad bytes) 000228 1785220956.430366 trunc from 0x016827 to 0x0234c2 (0xcc9c bytes) 000229 1785220956.438217 mapread 0x01dd23 thru 0x0268b4 (0x8b92 bytes) 000230 1785220956.458913 write 0x0068e4 thru 0x007152 (0x86f bytes) 000231 1785220956.468801 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x268b5): Mirror 4: 0xb90313f5 (stale) Mirror 5: 0xb90313f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268b5): Mirror 4: 0xb90313f5 (stale) Mirror 5: 0xb90313f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268b5: skipped 000232 1785220956.842016 trunc from 0x01983e to 0x020a49 (0x720c bytes) 000233 1785220956.846792 punch from 0x009517 to 0x0193b0 (0xfe9a bytes) 000234 1785220956.853293 write 0x009a23 thru 0x00f94f (0x5f2d bytes) 000235 1785220956.883055 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x268b5): Mirror 5: 0xb90313f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268b5): Mirror 5: 0xb90313f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268b5: skipped 000237 1785220957.242795 fallocate 0x01e105 thru 0x023565 (0x5461 bytes) 000238 1785220957.248577 read 0x025579 thru 0x0268b4 (0x133c bytes) 000239 1785220957.257412 write 0x021a9a thru 0x021d86 (0x2ed bytes) 000240 1785220957.269529 trunc from 0x0268b5 to 0x03d673 (0x16dbf bytes) 000241 1785220957.289339 trunc from 0x011b52 to 0x01ff4e (0xe3fd bytes) 000242 1785220957.297143 mapread 0x021c66 thru 0x02d855 (0xbbf0 bytes) 000244 1785220957.331034 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d674): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d674): Mirror 5: 0x69c3d4f8 (stale) CRC-32 checksum value for chunk [0, 0x3d674): Mirror 1: 0x2e1dd35a Mirror 6: 0x2e1dd35a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d674: skipped 000245 1785220958.365415 mapread 0x03264d thru 0x03ceed (0xa8a1 bytes) 000246 1785220958.412939 write 0x02b853 thru 0x02e7a4 (0x2f52 bytes) 000248 1785220958.493925 fallocate 0x01fded thru 0x02d4eb (0xd6ff bytes) 000249 1785220958.502714 punch from 0x00242a to 0x004120 (0x1cf7 bytes) 000253 1785220958.513394 trunc from 0x03d674 to 0x016803 (0x26e70 bytes) 000254 1785220958.587252 read 0x00d193 thru 0x00d46d (0x2db bytes) 000255 1785220958.596606 mapread 0x012078 thru 0x01235a (0x2e3 bytes) 000256 1785220958.606083 write 0x02b2be thru 0x02cb05 (0x1848 bytes) 000258 1785220958.607808 mapread 0x013af9 thru 0x0186a1 (0x4ba9 bytes) 000259 1785220958.621578 trunc from 0x02cb06 to 0x01c90b (0x101fa bytes) 000261 1785220958.647490 fallocate 0x0233ee thru 0x029dbc (0x69cf bytes) 000262 1785220958.656518 punch from 0x01ab9e to 0x0236bb (0x8b1e bytes) 000263 1785220958.666433 trunc from 0x029dbd to 0x025c7d (0x413f bytes) 000264 1785220958.682052 mapwrite 0x030bb3 thru 0x031d4f (0x119d bytes) 000265 1785220958.720787 write 0x012bb0 thru 0x020523 (0xd974 bytes) 000266 1785220958.728106 mapwrite 0x0067f5 thru 0x00c45a (0x5c66 bytes) 000267 1785220958.754993 punch from 0x021d7d to 0x02543f (0x36c3 bytes) 000268 1785220958.760156 mapwrite 0x015515 thru 0x022186 (0xcc72 bytes) 000269 1785220958.804508 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d50): Mirror 6: 0x14583fc7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d50): Mirror 6: 0x14583fc7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d50: skipped 000270 1785220959.139570 fallocate 0x016a2b thru 0x02270e (0xbce4 bytes) 000271 1785220959.151199 punch from 0x0168e8 to 0x017ef5 (0x160e bytes) 000272 1785220959.161844 write 0x03b2f4 thru 0x03ffff (0x4d0c bytes) 000273 1785220959.182950 punch from 0x02d680 to 0x03ae04 (0xd785 bytes) 000274 1785220959.194042 read 0x0380de thru 0x03ffff (0x7f22 bytes) 000275 1785220959.213521 trunc from 0x040000 to 0x02f187 (0x10e78 bytes) 000276 1785220959.253591 fallocate 0x010375 thru 0x01975b (0x93e7 bytes) 000278 1785220959.274466 trunc from 0x02f188 to 0x02462c (0xab5b bytes) 000280 1785220959.332151 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2462d): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2462d): Mirror 6: 0x2a362cb1 (stale) CRC-32 checksum value for chunk [0, 0x2462d): Mirror 1: 0x3c815a80 Mirror 7: 0x3c815a80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2462d: skipped 000281 1785220960.322045 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000283 1785220960.829941 MIRROR_EXTEND to 3 mirrors 000284 1785220961.633365 read 0x019567 thru 0x01fa6b (0x6505 bytes) 000285 1785220961.649376 trunc from 0x003832 to 0x00c9ec (0x91bb bytes) 000286 1785220961.712871 mapwrite 0x016f4e thru 0x0198aa (0x295d bytes) 000287 1785220961.743460 read 0x01080c thru 0x01c26b (0xba60 bytes) 000288 1785220961.768333 read 0x00a627 thru 0x00f13b (0x4b15 bytes) 000289 1785220961.769165 trunc from 0x000262 to 0x000d16 (0xab5 bytes) 000290 1785220961.774016 write 0x0361a5 thru 0x03cd26 (0x6b82 bytes) 000293 1785220961.790747 trunc from 0x03cd27 to 0x03a689 (0x269d bytes) 000294 1785220961.861545 MIRROR_RESYNC 3 mirrors 000296 1785220962.742021 fallocate 0x01dd73 thru 0x02c21c (0xe4aa bytes) 000297 1785220962.780161 fallocate 0x026934 thru 0x02b415 (0x4ae2 bytes) 000298 1785220962.785481 write 0x01c470 thru 0x02aaeb (0xe67c bytes) 000299 1785220962.816601 punch from 0x02ed11 to 0x03067c (0x196c bytes) 000301 1785220962.827729 mapread 0x00bc96 thru 0x012a7b (0x6de6 bytes) 000302 1785220962.849144 mapwrite 0x03db14 thru 0x03ffff (0x24ec bytes) 000303 1785220962.897239 trunc from 0x020ce6 to 0x02ba56 (0xad71 bytes) 000304 1785220962.902416 read 0x02d32c thru 0x038d03 (0xb9d8 bytes) 000305 1785220962.909422 punch from 0x024d02 to 0x02674b (0x1a4a bytes) 000306 1785220962.914453 trunc from 0x040000 to 0x00cda7 (0x33258 bytes) 000307 1785220962.931033 write 0x0209bf thru 0x02f656 (0xec98 bytes) 000308 1785220962.934733 write 0x02aa24 thru 0x02bc80 (0x125d bytes) 000309 1785220962.936197 write 0x022c57 thru 0x028941 (0x5ceb bytes) 000310 1785220962.938307 trunc from 0x0212eb to 0x02be4c (0xab62 bytes) 000311 1785220962.955253 mapwrite 0x03536d thru 0x03d9aa (0x863e bytes) 000312 1785220963.019102 fallocate 0x03e17b thru 0x03ffff (0x1e85 bytes) 000313 1785220963.028805 mapread 0x00b89d thru 0x01372e (0x7e92 bytes) 000314 1785220963.055433 mapwrite 0x03425a thru 0x035cbd (0x1a64 bytes) 000315 1785220963.074300 trunc from 0x026593 to 0x026f06 (0x974 bytes) 000316 1785220963.090472 write 0x03cbf0 thru 0x03d635 (0xa46 bytes) 000317 1785220963.092198 write 0x03c295 thru 0x03e2a0 (0x200c bytes) 000318 1785220963.099364 read 0x012ac9 thru 0x013411 (0x949 bytes) 000319 1785220963.104991 punch from 0x03a060 to 0x03d8d2 (0x3873 bytes) 000320 1785220963.130984 punch from 0x029f11 to 0x0324df (0x85cf bytes) 000322 1785220963.136033 trunc from 0x00b050 to 0x00cc71 (0x1c22 bytes) 000324 1785220963.141049 fallocate 0x006575 thru 0x011336 (0xadc2 bytes) 000325 1785220963.146282 mapread 0x0012c2 thru 0x009909 (0x8648 bytes) 000326 1785220963.166059 trunc from 0x009fec to 0x0178dc (0xd8f1 bytes) 000327 1785220963.177487 trunc from 0x03e2a1 to 0x01866e (0x25c32 bytes) 000328 1785220963.199481 punch from 0x0112ad to 0x0160da (0x4e2e bytes) 000329 1785220963.211013 punch from 0x008190 to 0x014fc8 (0xce39 bytes) 000330 1785220963.219303 trunc from 0x001534 to 0x007428 (0x5ef5 bytes) 000331 1785220963.227198 fallocate 0x013079 thru 0x0143ae (0x1336 bytes) 000332 1785220963.234543 punch from 0x009ed1 to 0x013e9b (0x9fcb bytes) 000333 1785220963.250923 trunc from 0x016182 to 0x01866e (0x24ed bytes) 000334 1785220963.256750 trunc from 0x01866f to 0x027a2f (0xf3c1 bytes) 000335 1785220963.276552 punch from 0x00d6f8 to 0x016cd1 (0x95da bytes) 000337 1785220963.286581 read 0x01bfcc thru 0x024238 (0x826d bytes) 000338 1785220963.307327 trunc from 0x027a30 to 0x02416a (0x38c5 bytes) 000339 1785220963.327532 write 0x014f2d thru 0x01ca2d (0x7b01 bytes) 000340 1785220963.329147 trunc from 0x02001e to 0x02416a (0x414d bytes) 000341 1785220963.336446 write 0x022284 thru 0x0295b0 (0x732d bytes) 000342 1785220963.342094 trunc from 0x0295b1 to 0x02bfa5 (0x29f5 bytes) 000343 1785220963.360112 trunc from 0x02bfa6 to 0x00acc2 (0x212e3 bytes) 000344 1785220963.394344 write 0x00bd1f thru 0x00be9b (0x17d bytes) 000345 1785220963.409750 MIRROR_RESYNC 3 mirrors 000346 1785220964.273908 read 0x00a1bd thru 0x00be9b (0x1cdf bytes) 000347 1785220964.278415 trunc from 0x00be9c to 0x03a410 (0x2e575 bytes) 000348 1785220964.347076 punch from 0x02210d to 0x02a27c (0x8170 bytes) 000349 1785220964.351997 punch from 0x03a0d8 to 0x03a410 (0x339 bytes) 000350 1785220964.356332 punch from 0x01d15c to 0x02940a (0xc2af bytes) 000351 1785220964.360422 trunc from 0x03a411 to 0x01c71f (0x1dcf1 bytes) 000352 1785220964.384446 fallocate 0x01c055 thru 0x0292e8 (0xd294 bytes) 000353 1785220964.388869 trunc from 0x013a9a to 0x0183a8 (0x490f bytes) 000354 1785220964.394263 mapwrite 0x01a24d thru 0x021add (0x7891 bytes) 000355 1785220964.456616 write 0x004395 thru 0x01428a (0xfef6 bytes) 000357 1785220964.469037 trunc from 0x021ade to 0x024bd0 (0x30f3 bytes) 000358 1785220964.484076 read 0x002dcc thru 0x0116df (0xe914 bytes) 000359 1785220964.497042 write 0x009356 thru 0x011249 (0x7ef4 bytes) 000360 1785220964.499965 trunc from 0x00786c to 0x0127dd (0xaf72 bytes) 000361 1785220964.513539 read 0x010f99 thru 0x018fb8 (0x8020 bytes) 000362 1785220964.523426 trunc from 0x024bd1 to 0x0216d2 (0x34fe bytes) 000363 1785220964.539395 trunc from 0x0216d3 to 0x03713d (0x15a6b bytes) 000364 1785220964.555813 mapwrite 0x03123d thru 0x03e9a1 (0xd765 bytes) 000365 1785220964.619504 mapwrite 0x00d7fb thru 0x01658a (0x8d90 bytes) 000366 1785220964.675907 trunc from 0x03e9a2 to 0x02065b (0x1e346 bytes) 000367 1785220964.700362 mapwrite 0x037a9e thru 0x03ffff (0x8562 bytes) 000368 1785220964.775390 write 0x00e851 thru 0x00ece5 (0x495 bytes) 000369 1785220964.776823 punch from 0x0180a2 to 0x01cbd4 (0x4b33 bytes) 000370 1785220964.783376 punch from 0x006b52 to 0x00f89f (0x8d4e bytes) 000371 1785220964.792255 fallocate 0x00bd85 thru 0x00c146 (0x3c2 bytes) 000372 1785220964.796932 trunc from 0x040000 to 0x00f347 (0x30cb8 bytes) 000373 1785220964.811263 write 0x032d5a thru 0x03ffaf (0xd256 bytes) 000374 1785220964.815072 punch from 0x023288 to 0x023a2f (0x7a8 bytes) 000375 1785220964.819696 read 0x03b0a1 thru 0x03ffaf (0x4f0f bytes) 000376 1785220964.820452 mapwrite 0x00621a thru 0x010f86 (0xad6d bytes) 000377 1785220964.858428 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ffb0): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffb0): Mirror 7: 0xcd8ee9b6 (stale) Mirror 8: 0xcd8ee9b6 (stale) CRC-32 checksum value for chunk [0, 0x3ffb0): Mirror 1: 0x866a78cf Mirror 9: 0x866a78cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffb0: skipped 000378 1785220965.607125 fallocate 0x01b7a1 thru 0x02a095 (0xe8f5 bytes) 000379 1785220965.660121 write 0x02fc2b thru 0x03ee58 (0xf22e bytes) 000380 1785220965.680912 write 0x039b3c thru 0x03ffff (0x64c4 bytes) 000381 1785220965.684586 mapwrite 0x039b73 thru 0x03ffff (0x648d bytes) 000382 1785220965.707010 trunc from 0x023456 to 0x031210 (0xddbb bytes) 000383 1785220965.713456 punch from 0x0120be to 0x01268e (0x5d1 bytes) 000384 1785220965.719127 mapread 0x00a103 thru 0x00dc44 (0x3b42 bytes) 000385 1785220965.743583 fallocate 0x036aba thru 0x03ffff (0x9546 bytes) 000387 1785220965.752202 trunc from 0x00112e to 0x010b55 (0xfa28 bytes) 000388 1785220965.758036 fallocate 0x00f69b thru 0x017300 (0x7c66 bytes) 000389 1785220965.764014 write 0x01a920 thru 0x01ba76 (0x1157 bytes) 000390 1785220965.773021 trunc from 0x040000 to 0x028eff (0x17100 bytes) 000391 1785220965.799938 trunc from 0x00a34f to 0x00e428 (0x40da bytes) 000392 1785220965.804982 mapread 0x00f5db thru 0x01cedf (0xd905 bytes) 000393 1785220965.824527 write 0x00fb4b thru 0x01c95d (0xce13 bytes) 000394 1785220965.836963 trunc from 0x028f00 to 0x00e047 (0x1aeb8 bytes) 000395 1785220965.855566 trunc from 0x00e048 to 0x027064 (0x1901d bytes) 000396 1785220965.875301 read 0x021394 thru 0x022f8e (0x1bfb bytes) 000397 1785220965.884532 punch from 0x023444 to 0x025b5a (0x2717 bytes) 000398 1785220965.893165 fallocate 0x0073c0 thru 0x014424 (0xd065 bytes) 000399 1785220965.909420 mapwrite 0x02844b thru 0x0376bd (0xf273 bytes) 000400 1785220965.994229 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376be): Mirror 7: 0xa8a17fd1 (stale) Mirror 9: 0xf9f944cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376be): Mirror 7: 0xa8a17fd1 (stale) Mirror 9: 0xf9f944cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376be: skipped 000401 1785220966.352104 trunc from 0x0376be to 0x01008b (0x27632 bytes) 000402 1785220966.375447 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1008c): Mirror 9: 0xa8c6fd1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1008c): Mirror 9: 0xa8c6fd1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1008c: skipped 000403 1785220966.700889 fallocate 0x006aaf thru 0x015281 (0xe7d3 bytes) 000404 1785220966.706824 read 0x00dfcd thru 0x015281 (0x72b5 bytes) 000405 1785220966.715088 punch from 0x009249 to 0x015281 (0xc039 bytes) 000406 1785220966.719500 mapread 0x005cd5 thru 0x00c4c1 (0x67ed bytes) 000408 1785220966.743186 trunc from 0x015282 to 0x025e83 (0x10c02 bytes) 000409 1785220966.764600 mapwrite 0x026fa8 thru 0x035f94 (0xefed bytes) 000410 1785220966.819527 write 0x03aa48 thru 0x03df14 (0x34cd bytes) 000411 1785220966.822125 trunc from 0x027bea to 0x030edd (0x92f4 bytes) 000412 1785220966.827196 mapwrite 0x03ab20 thru 0x03ffff (0x54e0 bytes) 000413 1785220966.872353 mapread 0x03a10b thru 0x03ffff (0x5ef5 bytes) 000414 1785220966.873790 fallocate 0x02ad6b thru 0x03ad32 (0xffc8 bytes) 000415 1785220966.879883 fallocate 0x031937 thru 0x03d2b2 (0xb97c bytes) 000416 1785220966.888119 write 0x01f7e1 thru 0x02a611 (0xae31 bytes) 000417 1785220966.901511 mapread 0x039b96 thru 0x03ffff (0x646a bytes) 000418 1785220966.902758 punch from 0x0096fe to 0x012f6f (0x9872 bytes) 000419 1785220966.915047 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000420 1785220967.059404 trunc from 0x00f1c7 to 0x01316f (0x3fa9 bytes) 000421 1785220967.064870 fallocate 0x03cd62 thru 0x03d1be (0x45d bytes) 000424 1785220967.085017 write 0x0283e1 thru 0x02f3a5 (0x6fc5 bytes) 000425 1785220967.100805 write 0x01c291 thru 0x0221e1 (0x5f51 bytes) 000426 1785220967.110205 mapread 0x028c3b thru 0x037ac6 (0xee8c bytes) 000427 1785220967.127558 trunc from 0x030623 to 0x03bf87 (0xb965 bytes) 000428 1785220967.133075 mapwrite 0x0146e4 thru 0x01f30c (0xac29 bytes) 000429 1785220967.162277 trunc from 0x040000 to 0x00f599 (0x30a66 bytes) 000430 1785220967.184853 trunc from 0x00f59a to 0x01a4e6 (0xaf4d bytes) 000431 1785220967.209464 write 0x00e508 thru 0x01d356 (0xee4f bytes) 000432 1785220967.213429 mapread 0x012611 thru 0x0128b7 (0x2a7 bytes) 000433 1785220967.215053 mapwrite 0x0139c0 thru 0x014cd2 (0x1313 bytes) 000434 1785220967.246120 trunc from 0x011249 to 0x018bb9 (0x7971 bytes) 000435 1785220967.252269 fallocate 0x01f137 thru 0x020506 (0x13d0 bytes) 000436 1785220967.258970 trunc from 0x020507 to 0x01b59a (0x4f6c bytes) 000437 1785220967.279115 read 0x00a8f3 thru 0x00b2e9 (0x9f7 bytes) 000438 1785220967.286075 punch from 0x010210 to 0x0118a7 (0x1698 bytes) 000440 1785220967.292278 mapwrite 0x02ded6 thru 0x031cf7 (0x3e22 bytes) 000441 1785220967.335790 punch from 0x01f9c8 to 0x02c16e (0xc7a7 bytes) 000442 1785220967.340901 mapread 0x00c707 thru 0x012b04 (0x63fe bytes) 000443 1785220967.354236 punch from 0x028365 to 0x02d22e (0x4eca bytes) 000444 1785220967.360617 read 0x006694 thru 0x00bd52 (0x56bf bytes) 000445 1785220967.361889 read 0x0024c1 thru 0x00d6d2 (0xb212 bytes) 000446 1785220967.375999 mapread 0x016b6d thru 0x022db2 (0xc246 bytes) 000447 1785220967.379135 punch from 0x01d40d to 0x02b842 (0xe436 bytes) 000448 1785220967.384941 trunc from 0x028bef to 0x02be24 (0x3236 bytes) 000449 1785220967.390531 mapread 0x005e78 thru 0x005ff5 (0x17e bytes) 000450 1785220967.394242 fallocate 0x03f15f thru 0x03ffff (0xea1 bytes) 000451 1785220967.403307 read 0x023284 thru 0x02a098 (0x6e15 bytes) 000452 1785220967.416132 punch from 0x02e2f5 to 0x031cf7 (0x3a03 bytes) 000453 1785220967.422229 trunc from 0x031cf8 to 0x01631e (0x1b9d9 bytes) 000454 1785220967.450024 write 0x019acc thru 0x01e52f (0x4a64 bytes) 000455 1785220967.453560 mapread 0x00ab3a thru 0x014dea (0xa2b1 bytes) 000456 1785220967.455319 trunc from 0x01e530 to 0x022278 (0x3d49 bytes) 000457 1785220967.496566 trunc from 0x022279 to 0x02ac4b (0x89d3 bytes) 000458 1785220967.515076 mapwrite 0x01f615 thru 0x0232e5 (0x3cd1 bytes) 000459 1785220967.554895 read 0x021267 thru 0x0213fc (0x196 bytes) 000460 1785220967.555708 trunc from 0x026c05 to 0x02a8da (0x3cd6 bytes) 000461 1785220967.562392 punch from 0x011163 to 0x019332 (0x81d0 bytes) 000463 1785220967.570082 mapwrite 0x02be85 thru 0x031fb4 (0x6130 bytes) 000464 1785220967.614213 fallocate 0x010560 thru 0x01f53f (0xefe0 bytes) 000465 1785220967.619375 write 0x017c20 thru 0x01f293 (0x7674 bytes) 000467 1785220967.623109 mapwrite 0x003294 thru 0x003be1 (0x94e bytes) 000468 1785220967.636456 trunc from 0x031fb5 to 0x03ddb2 (0xbdfe bytes) 000469 1785220967.663151 punch from 0x031843 to 0x038cd2 (0x7490 bytes) 000470 1785220967.669146 write 0x02549d thru 0x031270 (0xbdd4 bytes) 000471 1785220967.683206 trunc from 0x01cd75 to 0x0295e9 (0xc875 bytes) 000473 1785220967.695262 trunc from 0x03ddb3 to 0x03bcb7 (0x20fb bytes) 000474 1785220967.710554 mapread 0x01fe69 thru 0x0276fe (0x7896 bytes) 000475 1785220967.762043 MIRROR_EXTEND to 2 mirrors 000476 1785220968.494505 MIRROR_RESYNC 2 mirrors 000477 1785220968.862315 punch from 0x035f03 to 0x03bcb7 (0x5db5 bytes) 000478 1785220968.932421 trunc from 0x008b00 to 0x016ed9 (0xe3da bytes) 000479 1785220968.938687 read 0x010673 thru 0x0123e4 (0x1d72 bytes) 000480 1785220968.945585 read 0x03660b thru 0x0376de (0x10d4 bytes) 000481 1785220968.958217 mapread 0x013e39 thru 0x017814 (0x39dc bytes) 000482 1785220968.959880 trunc from 0x03bcb8 to 0x022c98 (0x1901f bytes) 000483 1785220968.999523 read 0x01ea32 thru 0x020ea1 (0x2470 bytes) 000484 1785220969.001091 fallocate 0x02e5af thru 0x03c640 (0xe092 bytes) 000485 1785220969.008211 mapread 0x00edda thru 0x013b92 (0x4db9 bytes) 000486 1785220969.009585 read 0x00f84c thru 0x01bad4 (0xc289 bytes) 000487 1785220969.011464 mapread 0x01ce24 thru 0x02289e (0x5a7b bytes) 000488 1785220969.012934 mapread 0x0048fc thru 0x008405 (0x3b0a bytes) 000490 1785220969.016349 trunc from 0x022a4a to 0x022c98 (0x24f bytes) 000491 1785220969.029748 mapread 0x0105f3 thru 0x01acb5 (0xa6c3 bytes) 000492 1785220969.031010 mapwrite 0x038ee3 thru 0x039c1a (0xd38 bytes) 000493 1785220969.068481 mapwrite 0x016314 thru 0x022df0 (0xcadd bytes) 000494 1785220969.097359 trunc from 0x00cf80 to 0x018782 (0xb803 bytes) 000495 1785220969.102149 fallocate 0x000c79 thru 0x005539 (0x48c1 bytes) 000498 1785220969.108330 write 0x025e21 thru 0x034712 (0xe8f2 bytes) 000499 1785220969.110781 write 0x008029 thru 0x010703 (0x86db bytes) 000500 1785220969.117111 fallocate 0x01b98a thru 0x024dbb (0x9432 bytes) 000501 1785220969.122054 mapread 0x0299ee thru 0x02d950 (0x3f63 bytes) 000502 1785220969.123598 write 0x008453 thru 0x01836b (0xff19 bytes) 000503 1785220969.133866 mapwrite 0x03d5c7 thru 0x03ffff (0x2a39 bytes) 000504 1785220969.182220 write 0x021d87 thru 0x02ba6c (0x9ce6 bytes) 000505 1785220969.192227 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000506 1785220969.356236 read 0x00fb49 thru 0x0111f7 (0x16af bytes) 000507 1785220969.363053 trunc from 0x040000 to 0x00e86a (0x31795 bytes) 000508 1785220969.387340 punch from 0x003379 to 0x007657 (0x42df bytes) 000509 1785220969.392364 fallocate 0x038540 thru 0x03abe8 (0x26a9 bytes) 000510 1785220969.396516 trunc from 0x00e86b to 0x031b84 (0x2331a bytes) 000511 1785220969.412065 mapread 0x013338 thru 0x01a42c (0x70f5 bytes) 000512 1785220969.431416 trunc from 0x031b85 to 0x01955d (0x18627 bytes) 000514 1785220969.450449 trunc from 0x000291 to 0x008795 (0x8505 bytes) 000515 1785220969.455527 write 0x0294fd thru 0x0311c9 (0x7ccd bytes) 000517 1785220969.459477 punch from 0x022f69 to 0x028534 (0x55cc bytes) 000518 1785220969.464514 mapwrite 0x0033e7 thru 0x00d48f (0xa0a9 bytes) 000520 1785220969.497321 mapwrite 0x02b170 thru 0x037c35 (0xcac6 bytes) 000521 1785220969.566477 MIRROR_EXTEND to 2 mirrors 000522 1785220970.180913 fallocate 0x000b28 thru 0x006dce (0x62a7 bytes) 000523 1785220970.241406 mapwrite 0x02c427 thru 0x037fdf (0xbbb9 bytes) 000524 1785220970.326645 punch from 0x023460 to 0x032c5d (0xf7fe bytes) 000525 1785220970.338186 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 2: 0x8365b332 (stale) CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 1: 0xd26093a3 Mirror 3: 0xd26093a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fe0: skipped 000526 1785220971.184024 fallocate 0x0110d9 thru 0x01cfe0 (0xbf08 bytes) 000527 1785220971.223248 mapread 0x00e266 thru 0x0102ed (0x2088 bytes) 000528 1785220971.258148 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 2: 0x8365b332 (stale) Mirror 3: 0xd26093a3 (stale) CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 1: 0xd26093a3 Mirror 4: 0xd26093a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fe0: skipped 000529 1785220972.197167 write 0x02299c thru 0x030d7f (0xe3e4 bytes) 000530 1785220972.252051 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 2: 0x8365b332 (stale) Mirror 3: 0xd26093a3 (stale) Mirror 4: 0xd26093a3 (stale) CRC-32 checksum value for chunk [0, 0x37fe0): Mirror 1: 0x90efd1ff Mirror 5: 0x90efd1ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fe0: skipped 000531 1785220973.361219 write 0x02b210 thru 0x02ef69 (0x3d5a bytes) 000532 1785220973.406170 trunc from 0x037fe0 to 0x002ffe (0x34fe1 bytes) 000533 1785220973.442440 punch from 0x001602 to 0x002ffe (0x19fd bytes) 000534 1785220973.448335 fallocate 0x0259c0 thru 0x02a467 (0x4aa8 bytes) 000535 1785220973.461258 trunc from 0x002675 to 0x002ffe (0x98a bytes) 000536 1785220973.467089 mapwrite 0x034b22 thru 0x03fa32 (0xaf11 bytes) 000537 1785220973.531384 trunc from 0x01045f to 0x01fa9e (0xf640 bytes) 000538 1785220973.542287 write 0x01476f thru 0x021832 (0xd0c4 bytes) 000539 1785220973.559907 read 0x012fb6 thru 0x01f974 (0xc9bf bytes) 000540 1785220973.570365 MIRROR_RESYNC 5 mirrors 000541 1785220974.881956 trunc from 0x03fa33 to 0x00ef58 (0x30ada bytes) 000542 1785220974.962258 trunc from 0x000f27 to 0x0088ca (0x79a4 bytes) 000543 1785220974.973267 mapwrite 0x0354c2 thru 0x03ffff (0xab3e bytes) 000544 1785220975.073617 punch from 0x02ee59 to 0x03edc6 (0xff6e bytes) 000546 1785220975.083011 read 0x025264 thru 0x0284d7 (0x3274 bytes) 000547 1785220975.091482 mapwrite 0x03d629 thru 0x03ffff (0x29d7 bytes) 000548 1785220975.127806 trunc from 0x040000 to 0x03373b (0xc8c4 bytes) 000549 1785220975.154359 mapread 0x008a0f thru 0x013b6f (0xb161 bytes) 000550 1785220975.191907 trunc from 0x03373c to 0x00a924 (0x28e17 bytes) 000551 1785220975.212971 fallocate 0x03c3d3 thru 0x03ffff (0x3c2d bytes) 000553 1785220975.221203 trunc from 0x040000 to 0x018551 (0x27aae bytes) 000554 1785220975.236579 read 0x00a2ec thru 0x0113ad (0x70c2 bytes) 000555 1785220975.251526 write 0x01234d thru 0x01567a (0x332e bytes) 000556 1785220975.252747 mapread 0x017c11 thru 0x018551 (0x941 bytes) 000557 1785220975.254984 punch from 0x001409 to 0x00c5b3 (0xb1ab bytes) 000558 1785220975.264570 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x18552): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18552): Mirror 2: 0x5c997445 (stale) Mirror 3: 0x5c997445 (stale) Mirror 4: 0x5c997445 (stale) Mirror 5: 0x5c997445 (stale) CRC-32 checksum value for chunk [0, 0x18552): Mirror 1: 0x259f615d Mirror 6: 0x259f615d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18552: skipped 000559 1785220976.131014 MIRROR_RESYNC 6 mirrors 000560 1785220977.180776 punch from 0x0071e9 to 0x00c55f (0x5377 bytes) 000562 1785220977.228923 trunc from 0x018552 to 0x01aae5 (0x2594 bytes) 000563 1785220977.274163 read 0x016176 thru 0x01aae5 (0x4970 bytes) 000565 1785220977.281583 fallocate 0x008bbe thru 0x011f85 (0x93c8 bytes) 000566 1785220977.286863 mapread 0x0001fa thru 0x009705 (0x950c bytes) 000567 1785220977.320725 punch from 0x015030 to 0x01aae5 (0x5ab6 bytes) 000568 1785220977.325812 mapread 0x00cd94 thru 0x01a17f (0xd3ec bytes) 000569 1785220977.342099 mapread 0x0182a4 thru 0x01aae5 (0x2842 bytes) 000570 1785220977.343173 trunc from 0x01aae6 to 0x02a03e (0xf559 bytes) 000571 1785220977.364133 mapwrite 0x023d47 thru 0x02c21f (0x84d9 bytes) 000572 1785220977.433795 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c220): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c220): Mirror 2: 0xe9e54d74 (stale) Mirror 3: 0xe9e54d74 (stale) Mirror 4: 0xe9e54d74 (stale) Mirror 5: 0xe9e54d74 (stale) Mirror 6: 0xe9e54d74 (stale) CRC-32 checksum value for chunk [0, 0x2c220): Mirror 1: 0xeffdecdc Mirror 7: 0xeffdecdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c220: skipped 000573 1785220978.243254 read 0x026e7c thru 0x02c21f (0x53a4 bytes) 000574 1785220978.261025 read 0x000f2b thru 0x0105bc (0xf692 bytes) 000575 1785220978.284999 punch from 0x002356 to 0x00bee5 (0x9b90 bytes) 000576 1785220978.328847 fallocate 0x00fe97 thru 0x01e213 (0xe37d bytes) 000577 1785220978.334570 mapwrite 0x0121ec thru 0x015112 (0x2f27 bytes) 000578 1785220978.365110 trunc from 0x02c220 to 0x03ff8b (0x13d6c bytes) 000579 1785220978.444065 mapread 0x02402e thru 0x03084f (0xc822 bytes) 000580 1785220978.501815 fallocate 0x024e4d thru 0x0347d1 (0xf985 bytes) 000581 1785220978.518533 read 0x0395bc thru 0x03ff8b (0x69d0 bytes) 000582 1785220978.520160 mapwrite 0x033c31 thru 0x039edd (0x62ad bytes) 000583 1785220978.558023 trunc from 0x007d90 to 0x017a1e (0xfc8f bytes) 000584 1785220978.573190 MIRROR_RESYNC 7 mirrors 000585 1785220980.356159 mapread 0x017231 thru 0x01d5fa (0x63ca bytes) 000587 1785220980.379606 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) 000588 1785220980.830698 punch from 0x036e74 to 0x03ff8b (0x9118 bytes) 000589 1785220980.873924 write 0x018d0e thru 0x023574 (0xa867 bytes) 000590 1785220980.888732 trunc from 0x03ff8c to 0x015ce6 (0x2a2a5 bytes) 000591 1785220980.909806 fallocate 0x00dad7 thru 0x01bdf1 (0xe31b bytes) 000592 1785220980.915891 write 0x0315a6 thru 0x031e79 (0x8d4 bytes) 000593 1785220980.917752 trunc from 0x00e707 to 0x01ab1e (0xc418 bytes) 000594 1785220980.922817 punch from 0x030add to 0x031e79 (0x139d bytes) 000595 1785220980.932980 trunc from 0x031e7a to 0x0357bc (0x3943 bytes) 000596 1785220980.947046 read 0x023945 thru 0x029af2 (0x61ae bytes) 000597 1785220980.966040 mapwrite 0x01a28f thru 0x025535 (0xb2a7 bytes) 000598 1785220981.015228 trunc from 0x017aaf to 0x026c06 (0xf158 bytes) 000599 1785220981.022859 punch from 0x02fb7f to 0x0357bc (0x5c3e bytes) 000600 1785220981.033121 MIRROR_RESYNC 6 mirrors 000601 1785220981.889870 trunc from 0x00173b to 0x004a79 (0x333f bytes) 000602 1785220981.923718 mapwrite 0x0058dd thru 0x00ebbb (0x92df bytes) 000603 1785220981.968200 mapwrite 0x0339e4 thru 0x039147 (0x5764 bytes) 000604 1785220982.061137 punch from 0x00dff5 to 0x01aaaa (0xcab6 bytes) 000605 1785220982.075707 mapread 0x00c08f thru 0x012aa9 (0x6a1b bytes) 000606 1785220982.085361 mapwrite 0x03ba33 thru 0x03d157 (0x1725 bytes) 000607 1785220982.132545 fallocate 0x002ad5 thru 0x004a01 (0x1f2d bytes) 000608 1785220982.140523 fallocate 0x014094 thru 0x01945a (0x53c7 bytes) 000609 1785220982.153805 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d158): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d158): Mirror 3: 0x2904f839 (stale) Mirror 4: 0x2904f839 (stale) Mirror 5: 0x2904f839 (stale) Mirror 6: 0x2904f839 (stale) Mirror 7: 0x2904f839 (stale) CRC-32 checksum value for chunk [0, 0x3d158): Mirror 1: 0x4f8f212e Mirror 8: 0x4f8f212e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d158: skipped 000610 1785220983.170900 trunc from 0x01add9 to 0x01c27d (0x14a5 bytes) 000611 1785220983.248172 write 0x0118e4 thru 0x014bcd (0x32ea bytes) 000612 1785220983.285166 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d158): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d158): Mirror 3: 0x2904f839 (stale) Mirror 4: 0x2904f839 (stale) Mirror 5: 0x2904f839 (stale) Mirror 6: 0x2904f839 (stale) Mirror 7: 0x2904f839 (stale) Mirror 8: 0x4f8f212e (stale) CRC-32 checksum value for chunk [0, 0x3d158): Mirror 1: 0x478e1a9b Mirror 9: 0x478e1a9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d158: skipped 000613 1785220984.248468 mapread 0x000947 thru 0x00d244 (0xc8fe bytes) 000614 1785220984.313029 fallocate 0x011095 thru 0x015a76 (0x49e2 bytes) 000615 1785220984.357393 punch from 0x002b49 to 0x0069de (0x3e96 bytes) 000616 1785220984.366407 mapread 0x010dea thru 0x016ede (0x60f5 bytes) 000617 1785220984.383723 mapread 0x007c5f thru 0x010051 (0x83f3 bytes) 000618 1785220984.397288 trunc from 0x03d158 to 0x030251 (0xcf06 bytes) 000620 1785220984.437265 mapwrite 0x02c00d thru 0x02d5a1 (0x1595 bytes) 000621 1785220984.465715 fallocate 0x023ffe thru 0x02e007 (0xa00a bytes) 000623 1785220984.475045 write 0x03749b thru 0x038c79 (0x17df bytes) 000624 1785220984.480070 punch from 0x036bfa to 0x038c79 (0x2080 bytes) 000625 1785220984.496006 trunc from 0x038c7a to 0x02cbe6 (0xc093 bytes) 000626 1785220984.546675 MIRROR_RESYNC 8 mirrors 000627 1785220986.489947 trunc from 0x01521b to 0x0230e7 (0xdecd bytes) 000628 1785220986.535901 fallocate 0x00987d thru 0x011eef (0x8673 bytes) 000629 1785220986.543890 trunc from 0x015063 to 0x023e12 (0xedb0 bytes) 000631 1785220986.550693 mapwrite 0x02f9eb thru 0x02fc9d (0x2b3 bytes) 000632 1785220986.591944 fallocate 0x01a4f8 thru 0x01fbf2 (0x56fb bytes) 000633 1785220986.602617 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fc9e): Mirror 4: 0x411757f8 (stale) Mirror 5: 0x411757f8 (stale) Mirror 6: 0x411757f8 (stale) Mirror 7: 0x411757f8 (stale) Mirror 8: 0x411757f8 (stale) Mirror 9: 0x411757f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fc9e): Mirror 4: 0x411757f8 (stale) Mirror 5: 0x411757f8 (stale) Mirror 6: 0x411757f8 (stale) Mirror 7: 0x411757f8 (stale) Mirror 8: 0x411757f8 (stale) Mirror 9: 0x411757f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fc9e: skipped 000634 1785220987.058389 punch from 0x023f24 to 0x02fc9d (0xbd7a bytes) 000636 1785220987.082822 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fc9e): Mirror 4: 0x411757f8 (stale) Mirror 5: 0x411757f8 (stale) Mirror 6: 0x411757f8 (stale) Mirror 7: 0x411757f8 (stale) Mirror 8: 0x411757f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fc9e): Mirror 4: 0x411757f8 (stale) Mirror 5: 0x411757f8 (stale) Mirror 6: 0x411757f8 (stale) Mirror 7: 0x411757f8 (stale) Mirror 8: 0x411757f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fc9e: skipped 000637 1785220987.552191 punch from 0x0130b5 to 0x014112 (0x105e bytes) 000638 1785220987.560920 trunc from 0x02fc9e to 0x038f07 (0x926a bytes) 000640 1785220987.589288 mapwrite 0x02ac6d thru 0x03a05b (0xf3ef bytes) 000641 1785220987.733093 fallocate 0x011444 thru 0x014c63 (0x3820 bytes) 000642 1785220987.740043 fallocate 0x01c170 thru 0x029762 (0xd5f3 bytes) 000643 1785220987.745774 write 0x024e91 thru 0x034871 (0xf9e1 bytes) 000645 1785220987.755071 trunc from 0x03a05c to 0x008847 (0x31814 bytes) 000646 1785220987.774662 mapread 0x002a44 thru 0x008847 (0x5e04 bytes) 000647 1785220987.785856 punch from 0x00850e to 0x008847 (0x33a bytes) 000648 1785220987.798900 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x8848): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8848): Mirror 4: 0x2536476a (stale) Mirror 5: 0x2536476a (stale) Mirror 6: 0x2536476a (stale) Mirror 7: 0x2536476a (stale) Mirror 8: 0x2536476a (stale) CRC-32 checksum value for chunk [0, 0x8848): Mirror 1: 0xba232366 Mirror 9: 0xba232366 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8848: skipped 000649 1785220988.524238 mapread 0x008277 thru 0x008847 (0x5d1 bytes) 000650 1785220988.568486 punch from 0x0002c7 to 0x001fc3 (0x1cfd bytes) 000651 1785220988.609869 mapwrite 0x039835 thru 0x03ffff (0x67cb bytes) 000652 1785220988.691748 fallocate 0x027f3d thru 0x02d882 (0x5946 bytes) 000653 1785220988.705798 punch from 0x02b276 to 0x038495 (0xd220 bytes) 000654 1785220988.715774 punch from 0x008fb7 to 0x00cd10 (0x3d5a bytes) 000655 1785220988.728593 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf7253885 (stale) Mirror 5: 0xf7253885 (stale) Mirror 7: 0xf7253885 (stale) Mirror 8: 0xf7253885 (stale) Mirror 9: 0x60fbc13d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf7253885 (stale) Mirror 5: 0xf7253885 (stale) Mirror 7: 0xf7253885 (stale) Mirror 8: 0xf7253885 (stale) Mirror 9: 0x60fbc13d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000656 1785220989.221890 mapwrite 0x03587d thru 0x03ba02 (0x6186 bytes) 000657 1785220989.295854 mapwrite 0x000b45 thru 0x0065b2 (0x5a6e bytes) 000658 1785220989.334440 punch from 0x00b661 to 0x018ef4 (0xd894 bytes) 000659 1785220989.338923 mapwrite 0x02240e thru 0x031fdb (0xfbce bytes) 000660 1785220989.375596 fallocate 0x0223c9 thru 0x02ca67 (0xa69f bytes) 000661 1785220989.385078 MIRROR_RESYNC 6 mirrors 000662 1785220990.693076 mapread 0x028dac thru 0x0369bd (0xdc12 bytes) 000663 1785220990.729984 punch from 0x012d1c to 0x01c853 (0x9b38 bytes) 000664 1785220990.789953 trunc from 0x040000 to 0x00805e (0x37fa1 bytes) 000665 1785220990.835187 fallocate 0x019d6c thru 0x01f079 (0x530e bytes) 000666 1785220990.839803 trunc from 0x00805f to 0x01807d (0x1001f bytes) 000667 1785220990.882791 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1807e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1807e): Mirror 4: 0x93a99f82 (stale) Mirror 5: 0x93a99f82 (stale) Mirror 7: 0x93a99f82 (stale) Mirror 8: 0x93a99f82 (stale) Mirror 9: 0x93a99f82 (stale) CRC-32 checksum value for chunk [0, 0x1807e): Mirror 1: 0x6eaaa97a Mirror 10: 0x6eaaa97a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1807e: skipped 000668 1785220991.687654 read 0x0134b1 thru 0x01807d (0x4bcd bytes) 000669 1785220991.710512 trunc from 0x01807e to 0x00b072 (0xd00b bytes) 000671 1785220991.774030 fallocate 0x03e7c9 thru 0x03ffff (0x1837 bytes) 000672 1785220991.780240 read 0x039b4b thru 0x03a83a (0xcf0 bytes) 000673 1785220991.808395 fallocate 0x005f3f thru 0x00956c (0x362e bytes) 000674 1785220991.816797 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x548eed5d (stale) Mirror 5: 0x548eed5d (stale) Mirror 7: 0x548eed5d (stale) Mirror 8: 0x548eed5d (stale) Mirror 9: 0x548eed5d (stale) Mirror 10: 0x5048adb2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5048adb2 Mirror 11: 0x5048adb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000677 1785220992.553740 punch from 0x032305 to 0x03c1cb (0x9ec7 bytes) 000679 1785220992.594176 trunc from 0x040000 to 0x01d646 (0x229b9 bytes) 000680 1785220992.621997 fallocate 0x00c61e thru 0x018c70 (0xc653 bytes) 000681 1785220992.626992 trunc from 0x01d647 to 0x00e7d4 (0xee72 bytes) 000682 1785220992.642947 read 0x00cebb thru 0x00e7d4 (0x191a bytes) 000683 1785220992.652659 punch from 0x00008e to 0x00d6f3 (0xd666 bytes) 000684 1785220992.662647 trunc from 0x00e7d5 to 0x00383a (0xaf9a bytes) 000685 1785220992.675649 trunc from 0x00383b to 0x02d61d (0x29de3 bytes) 000686 1785220992.689754 mapread 0x003b11 thru 0x00b777 (0x7c67 bytes) 000687 1785220992.706563 trunc from 0x02d61e to 0x01be6b (0x117b2 bytes) 000688 1785220992.721943 trunc from 0x01be6c to 0x003b61 (0x1830a bytes) 000689 1785220992.734731 trunc from 0x003b62 to 0x0063b1 (0x2850 bytes) 000690 1785220992.748111 trunc from 0x0063b2 to 0x014cde (0xe92d bytes) 000691 1785220992.761793 mapread 0x009731 thru 0x014cde (0xb5ae bytes) 000692 1785220992.776863 trunc from 0x005ed1 to 0x00baef (0x5c1f bytes) 000693 1785220992.780912 mapwrite 0x01f3aa thru 0x02df6f (0xebc6 bytes) 000694 1785220992.854900 punch from 0x0189d8 to 0x0265bb (0xdbe4 bytes) 000695 1785220992.860072 mapread 0x01ab4d thru 0x021ba2 (0x7056 bytes) 000696 1785220992.875168 mapread 0x02312b thru 0x02bad9 (0x89af bytes) 000697 1785220992.888060 mapwrite 0x01ecf7 thru 0x025681 (0x698b bytes) 000698 1785220992.914878 trunc from 0x0163ad to 0x01f6ee (0x9342 bytes) 000699 1785220992.920760 write 0x0053d6 thru 0x009587 (0x41b2 bytes) 000700 1785220992.941921 write 0x03939f thru 0x03ffff (0x6c61 bytes) 000701 1785220992.943258 punch from 0x03ad78 to 0x03ffff (0x5288 bytes) 000702 1785220992.955847 write 0x01ae91 thru 0x026386 (0xb4f6 bytes) 000703 1785220992.961545 trunc from 0x01e4a4 to 0x021997 (0x34f4 bytes) 000704 1785220992.978401 mapread 0x015afa thru 0x01e854 (0x8d5b bytes) 000706 1785220992.987399 trunc from 0x040000 to 0x02867c (0x17983 bytes) 000707 1785220993.004361 trunc from 0x02867d to 0x01a0b7 (0xe5c5 bytes) 000708 1785220993.019789 read 0x002125 thru 0x00c346 (0xa222 bytes) 000709 1785220993.027082 trunc from 0x00cdf0 to 0x01a008 (0xd219 bytes) 000710 1785220993.035140 mapwrite 0x0190f1 thru 0x020686 (0x7596 bytes) 000711 1785220993.094993 mapwrite 0x025376 thru 0x02c488 (0x7113 bytes) 000712 1785220993.149598 trunc from 0x01f029 to 0x02b61a (0xc5f2 bytes) 000713 1785220993.155813 read 0x00fe1c thru 0x01674b (0x6930 bytes) 000714 1785220993.164837 mapwrite 0x026204 thru 0x033c7c (0xda79 bytes) 000716 1785220993.206761 fallocate 0x022436 thru 0x029f64 (0x7b2f bytes) 000717 1785220993.211548 fallocate 0x016fe1 thru 0x01d573 (0x6593 bytes) 000718 1785220993.215559 trunc from 0x003481 to 0x01043b (0xcfbb bytes) 000719 1785220993.220397 punch from 0x0155ee to 0x01cc66 (0x7679 bytes) 000720 1785220993.225534 mapread 0x0019bc thru 0x006ac3 (0x5108 bytes) 000721 1785220993.243720 mapread 0x02c9f3 thru 0x033c7c (0x728a bytes) 000722 1785220993.245515 mapread 0x02342d thru 0x02a138 (0x6d0c bytes) 000723 1785220993.250777 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33c7d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c7d): Mirror 4: 0x1e4eee6a (stale) Mirror 5: 0x1e4eee6a (stale) Mirror 7: 0x1e4eee6a (stale) Mirror 8: 0x1e4eee6a (stale) Mirror 9: 0x1e4eee6a (stale) Mirror 10: 0xbfcd3494 (stale) Mirror 11: 0xbfcd3494 (stale) CRC-32 checksum value for chunk [0, 0x33c7d): Mirror 1: 0x2c518fa7 Mirror 12: 0x2c518fa7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c7d: skipped 000724 1785220994.012434 write 0x035662 thru 0x0377dd (0x217c bytes) 000725 1785220994.060230 fallocate 0x034f36 thru 0x0386dd (0x37a8 bytes) 000726 1785220994.067984 mapwrite 0x01ebc0 thru 0x0295e5 (0xaa26 bytes) 000727 1785220994.136558 mapwrite 0x03c43f thru 0x03fa0d (0x35cf bytes) 000728 1785220994.216088 fallocate 0x00d005 thru 0x010aa7 (0x3aa3 bytes) 000730 1785220994.222006 read 0x02e045 thru 0x0324e6 (0x44a2 bytes) 000731 1785220994.226611 trunc from 0x03fa0e to 0x03635d (0x96b0 bytes) 000732 1785220994.239297 mapwrite 0x0141dd thru 0x01c8a2 (0x86c6 bytes) 000733 1785220994.277853 mapwrite 0x032fdf thru 0x034d24 (0x1d46 bytes) 000734 1785220994.298874 fallocate 0x00dee9 thru 0x00efa8 (0x10c0 bytes) 000735 1785220994.303472 trunc from 0x03635e to 0x0119ca (0x24993 bytes) 000736 1785220994.327019 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x119cb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x119cb): Mirror 4: 0x8d1d6a6d (stale) Mirror 5: 0x8d1d6a6d (stale) Mirror 7: 0x8d1d6a6d (stale) Mirror 8: 0x8d1d6a6d (stale) Mirror 9: 0x8d1d6a6d (stale) Mirror 10: 0x2de9dce3 (stale) Mirror 11: 0x2de9dce3 (stale) Mirror 12: 0xc554b9e9 (stale) CRC-32 checksum value for chunk [0, 0x119cb): Mirror 1: 0xc554b9e9 Mirror 13: 0xc554b9e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x119cb: skipped 000737 1785220995.220596 trunc from 0x008a23 to 0x0119ca (0x8fa8 bytes) 000738 1785220995.269820 mapwrite 0x006ad2 thru 0x00ceef (0x641e bytes) 000739 1785220995.355215 write 0x030045 thru 0x031d8c (0x1d48 bytes) 000740 1785220995.368804 write 0x00aec7 thru 0x012a47 (0x7b81 bytes) 000741 1785220995.408893 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d8d): Mirror 4: 0x2326f94d (stale) Mirror 5: 0x2326f94d (stale) Mirror 7: 0x2326f94d (stale) Mirror 8: 0x2326f94d (stale) Mirror 10: 0xfdcc8fc5 (stale) Mirror 11: 0xfdcc8fc5 (stale) Mirror 12: 0x26a512bb (stale) Mirror 13: 0xb7d773c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d8d): Mirror 4: 0x2326f94d (stale) Mirror 5: 0x2326f94d (stale) Mirror 7: 0x2326f94d (stale) Mirror 8: 0x2326f94d (stale) Mirror 10: 0xfdcc8fc5 (stale) Mirror 11: 0xfdcc8fc5 (stale) Mirror 12: 0x26a512bb (stale) Mirror 13: 0xb7d773c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d8d: skipped 000743 1785220995.984525 fallocate 0x01a00d thru 0x027ff4 (0xdfe8 bytes) 000744 1785220995.991742 write 0x002fa1 thru 0x00fb8f (0xcbef bytes) 000745 1785220996.005705 trunc from 0x00153f to 0x00cd59 (0xb81b bytes) 000746 1785220996.025067 mapwrite 0x03f46e thru 0x03ffff (0xb92 bytes) 000747 1785220996.094767 punch from 0x017a41 to 0x02089a (0x8e5a bytes) 000748 1785220996.099433 mapwrite 0x034efa thru 0x03ada2 (0x5ea9 bytes) 000749 1785220996.149550 punch from 0x023210 to 0x0328fb (0xf6ec bytes) 000750 1785220996.160148 mapwrite 0x02a974 thru 0x034631 (0x9cbe bytes) 000751 1785220996.185707 mapread 0x03ffdf thru 0x03ffff (0x21 bytes) 000752 1785220996.187255 write 0x038fac thru 0x03b0a9 (0x20fe bytes) 000753 1785220996.188968 trunc from 0x00bf1e to 0x0130d2 (0x71b5 bytes) 000754 1785220996.194524 mapread 0x03a99f thru 0x03eb1a (0x417c bytes) 000755 1785220996.199387 punch from 0x01aad9 to 0x0253cb (0xa8f3 bytes) 000756 1785220996.204076 trunc from 0x00eddd to 0x01012d (0x1351 bytes) 000757 1785220996.208456 write 0x026751 thru 0x02decd (0x777d bytes) 000759 1785220996.210998 read 0x03f601 thru 0x03ffff (0x9ff bytes) 000760 1785220996.212030 read 0x01bdc7 thru 0x01cba6 (0xde0 bytes) 000761 1785220996.221746 punch from 0x0399c7 to 0x03cdbf (0x33f9 bytes) 000763 1785220996.234966 mapread 0x019753 thru 0x01f1c3 (0x5a71 bytes) 000764 1785220996.241049 MIRROR_RESYNC 9 mirrors 000765 1785220998.330111 fallocate 0x00df65 thru 0x01a130 (0xc1cc bytes) 000766 1785220998.377341 read 0x03e412 thru 0x03ffff (0x1bee bytes) 000768 1785220998.383489 trunc from 0x039ba3 to 0x03ffff (0x645d bytes) 000769 1785220998.395786 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa24236e3 (stale) Mirror 5: 0xa24236e3 (stale) Mirror 7: 0xa24236e3 (stale) Mirror 8: 0xa24236e3 (stale) Mirror 11: 0xa24236e3 (stale) Mirror 12: 0xa24236e3 (stale) Mirror 13: 0xa24236e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa24236e3 (stale) Mirror 5: 0xa24236e3 (stale) Mirror 7: 0xa24236e3 (stale) Mirror 8: 0xa24236e3 (stale) Mirror 11: 0xa24236e3 (stale) Mirror 12: 0xa24236e3 (stale) Mirror 13: 0xa24236e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000770 1785220998.944860 write 0x01f869 thru 0x023390 (0x3b28 bytes) 000771 1785220998.970164 trunc from 0x02b465 to 0x037c85 (0xc821 bytes) 000772 1785220998.976076 trunc from 0x01bf1f to 0x01cd95 (0xe77 bytes) 000774 1785220998.984024 punch from 0x030ab3 to 0x0392af (0x87fd bytes) 000775 1785220998.996580 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa24236e3 (stale) Mirror 5: 0xa24236e3 (stale) Mirror 7: 0xa24236e3 (stale) Mirror 8: 0xa24236e3 (stale) Mirror 11: 0xa24236e3 (stale) Mirror 12: 0xa24236e3 (stale) Mirror 13: 0xa24236e3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x37bb40de Mirror 14: 0x37bb40de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000776 1785221000.303105 mapread 0x0140c4 thru 0x0178c6 (0x3803 bytes) 000777 1785221000.345236 mapwrite 0x024d57 thru 0x0266c5 (0x196f bytes) 000778 1785221000.470315 mapread 0x034c79 thru 0x03ffff (0xb387 bytes) 000779 1785221000.490942 mapwrite 0x014133 thru 0x019e40 (0x5d0e bytes) 000781 1785221000.533807 write 0x021c03 thru 0x02b1b9 (0x95b7 bytes) 000782 1785221000.554888 trunc from 0x011790 to 0x01d918 (0xc189 bytes) 000783 1785221000.564910 read 0x009c25 thru 0x00b28d (0x1669 bytes) 000784 1785221000.578613 mapwrite 0x02ba25 thru 0x03a594 (0xeb70 bytes) 000785 1785221000.606919 read 0x007893 thru 0x008c41 (0x13af bytes) 000786 1785221000.618611 punch from 0x00d7a0 to 0x018672 (0xaed3 bytes) 000787 1785221000.624884 mapwrite 0x014faf thru 0x017579 (0x25cb bytes) 000789 1785221000.680502 trunc from 0x035d9d to 0x03616a (0x3ce bytes) 000790 1785221000.689269 mapwrite 0x011aa2 thru 0x01a375 (0x88d4 bytes) 000791 1785221000.724991 trunc from 0x01006e to 0x017b99 (0x7b2c bytes) 000792 1785221000.730024 read 0x01e048 thru 0x02b2b5 (0xd26e bytes) 000793 1785221000.734245 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa24236e3 (stale) Mirror 5: 0xa24236e3 (stale) Mirror 7: 0xa24236e3 (stale) Mirror 8: 0xa24236e3 (stale) Mirror 11: 0xa24236e3 (stale) Mirror 13: 0xa24236e3 (stale) Mirror 14: 0x37bb40de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa24236e3 (stale) Mirror 5: 0xa24236e3 (stale) Mirror 7: 0xa24236e3 (stale) Mirror 8: 0xa24236e3 (stale) Mirror 11: 0xa24236e3 (stale) Mirror 13: 0xa24236e3 (stale) Mirror 14: 0x37bb40de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000795 1785221001.152053 punch from 0x01a882 to 0x02515a (0xa8d9 bytes) 000796 1785221001.164207 trunc from 0x035f99 to 0x03ce43 (0x6eab bytes) 000797 1785221001.173809 trunc from 0x040000 to 0x007089 (0x38f76 bytes) 000799 1785221001.199743 mapread 0x0032be thru 0x007089 (0x3dcc bytes) 000800 1785221001.218257 write 0x02cf87 thru 0x0367bf (0x9839 bytes) 000801 1785221001.220551 trunc from 0x02251a to 0x024dd8 (0x28bf bytes) 000802 1785221001.225466 trunc from 0x0367c0 to 0x0056d0 (0x310ef bytes) 000803 1785221001.244114 trunc from 0x004f8f to 0x0056d0 (0x742 bytes) 000804 1785221001.248922 mapwrite 0x0274c5 thru 0x035437 (0xdf73 bytes) 000805 1785221001.298468 punch from 0x01313f to 0x022d2c (0xfbee bytes) 000806 1785221001.303428 trunc from 0x00640f to 0x015320 (0xef12 bytes) 000807 1785221001.308598 punch from 0x001150 to 0x004735 (0x35e6 bytes) 000808 1785221001.313427 mapwrite 0x0391bf thru 0x03bff6 (0x2e38 bytes) 000809 1785221001.347572 read 0x02d55d thru 0x03bce8 (0xe78c bytes) 000810 1785221001.348972 mapread 0x02ec1d thru 0x03597f (0x6d63 bytes) 000811 1785221001.351344 fallocate 0x023319 thru 0x02429c (0xf84 bytes) 000812 1785221001.360501 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 4: 0xc024a0aa (stale) Mirror 5: 0xc024a0aa (stale) Mirror 7: 0xc024a0aa (stale) Mirror 8: 0xc024a0aa (stale) Mirror 11: 0xc024a0aa (stale) Mirror 13: 0xc024a0aa (stale) Mirror 14: 0x962f218a (stale) CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 1: 0x86753097 Mirror 15: 0x86753097 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bff7: skipped 000813 1785221002.178908 mapread 0x0019f1 thru 0x00ba1c (0xa02c bytes) 000814 1785221002.206997 punch from 0x021aba to 0x026281 (0x47c8 bytes) 000815 1785221002.263567 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 4: 0xc024a0aa (stale) Mirror 5: 0xc024a0aa (stale) Mirror 7: 0xc024a0aa (stale) Mirror 8: 0xc024a0aa (stale) Mirror 11: 0xc024a0aa (stale) Mirror 13: 0xc024a0aa (stale) Mirror 15: 0x86753097 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 4: 0xc024a0aa (stale) Mirror 5: 0xc024a0aa (stale) Mirror 7: 0xc024a0aa (stale) Mirror 8: 0xc024a0aa (stale) Mirror 11: 0xc024a0aa (stale) Mirror 13: 0xc024a0aa (stale) Mirror 15: 0x86753097 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bff7: skipped 000816 1785221002.684024 mapwrite 0x01b809 thru 0x02997a (0xe172 bytes) 000817 1785221002.738747 read 0x024283 thru 0x02f10a (0xae88 bytes) 000819 1785221002.750093 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 4: 0xc024a0aa (stale) Mirror 5: 0xc024a0aa (stale) Mirror 7: 0xc024a0aa (stale) Mirror 8: 0xc024a0aa (stale) Mirror 11: 0xc024a0aa (stale) Mirror 13: 0xc024a0aa (stale) Mirror 15: 0x86753097 (stale) CRC-32 checksum value for chunk [0, 0x3bff7): Mirror 1: 0x41ac86d7 Mirror 16: 0x41ac86d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bff7: skipped 000820 1785221003.778946 read 0x00c9cf thru 0x01c7dc (0xfe0e bytes) 000821 1785221003.805711 fallocate 0x023952 thru 0x02a962 (0x7011 bytes) 000823 1785221003.897284 read 0x01ef96 thru 0x02759a (0x8605 bytes) 000824 1785221003.909828 read 0x037d22 thru 0x03bff6 (0x42d5 bytes) 000825 1785221003.913783 trunc from 0x00c020 to 0x0119f1 (0x59d2 bytes) 000826 1785221003.918823 mapread 0x03580a thru 0x037ee7 (0x26de bytes) 000827 1785221003.921522 mapread 0x00ef83 thru 0x016927 (0x79a5 bytes) 000828 1785221003.940253 punch from 0x0270a1 to 0x029c00 (0x2b60 bytes) 000829 1785221003.957741 MIRROR_RESYNC 9 mirrors 000831 1785221006.320946 trunc from 0x0267ae to 0x02c033 (0x5886 bytes) 000832 1785221006.363436 mapwrite 0x020f2c thru 0x02ab09 (0x9bde bytes) 000833 1785221006.399824 read 0x000c05 thru 0x00cffc (0xc3f8 bytes) 000834 1785221006.408579 trunc from 0x03bff7 to 0x010579 (0x2ba7d bytes) 000835 1785221006.429486 mapwrite 0x02c3ce thru 0x03b267 (0xee9a bytes) 000836 1785221006.508954 punch from 0x0174c6 to 0x020469 (0x8fa4 bytes) 000837 1785221006.515817 punch from 0x01e331 to 0x02d5e7 (0xf2b7 bytes) 000838 1785221006.530103 read 0x00c65a thru 0x012953 (0x62fa bytes) 000839 1785221006.544065 punch from 0x0355a9 to 0x036932 (0x138a bytes) 000841 1785221006.551128 fallocate 0x01d268 thru 0x01de05 (0xb9e bytes) 000842 1785221006.562656 punch from 0x001e4c to 0x007027 (0x51dc bytes) 000843 1785221006.569611 punch from 0x00b0dc to 0x013cf6 (0x8c1b bytes) 000844 1785221006.580095 read 0x00d9bb thru 0x00fb44 (0x218a bytes) 000845 1785221006.592742 trunc from 0x03b268 to 0x021762 (0x19b05 bytes) 000846 1785221006.611083 write 0x029477 thru 0x037961 (0xe4eb bytes) 000847 1785221006.616506 read 0x008ab2 thru 0x009cd5 (0x1224 bytes) 000848 1785221006.618184 mapwrite 0x01fbee thru 0x0242ad (0x46c0 bytes) 000849 1785221006.652723 read 0x022782 thru 0x024687 (0x1f06 bytes) 000850 1785221006.653425 mapread 0x031019 thru 0x031640 (0x628 bytes) 000851 1785221006.654390 fallocate 0x0295e6 thru 0x032e76 (0x9891 bytes) 000852 1785221006.659590 trunc from 0x02b090 to 0x033044 (0x7fb5 bytes) 000853 1785221006.665083 read 0x028e60 thru 0x037961 (0xeb02 bytes) 000854 1785221006.673276 read 0x019291 thru 0x021675 (0x83e5 bytes) 000855 1785221006.674288 mapwrite 0x01993f thru 0x02674f (0xce11 bytes) 000856 1785221006.691748 fallocate 0x00f4b1 thru 0x01ca94 (0xd5e4 bytes) 000857 1785221006.697109 write 0x00643f thru 0x00b184 (0x4d46 bytes) 000858 1785221006.698595 punch from 0x0032f7 to 0x008293 (0x4f9d bytes) 000859 1785221006.712221 trunc from 0x00beb8 to 0x01bd83 (0xfecc bytes) 000860 1785221006.723106 fallocate 0x007713 thru 0x00dcd7 (0x65c5 bytes) 000861 1785221006.731544 read 0x02222a thru 0x02dd7a (0xbb51 bytes) 000863 1785221006.733839 mapread 0x022d7c thru 0x03271b (0xf9a0 bytes) 000864 1785221006.735670 fallocate 0x00a49a thru 0x00e259 (0x3dc0 bytes) 000865 1785221006.741363 punch from 0x02da7c to 0x037961 (0x9ee6 bytes) 000866 1785221006.748673 punch from 0x01322c to 0x020a0e (0xd7e3 bytes) 000867 1785221006.771983 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x37962): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37962): Mirror 4: 0xfc0bce0 (stale) Mirror 5: 0xfc0bce0 (stale) Mirror 7: 0xfc0bce0 (stale) Mirror 8: 0xfc0bce0 (stale) Mirror 11: 0xfc0bce0 (stale) Mirror 13: 0xfc0bce0 (stale) Mirror 15: 0xfc0bce0 (stale) Mirror 16: 0xfc0bce0 (stale) CRC-32 checksum value for chunk [0, 0x37962): Mirror 1: 0x70c408c0 Mirror 17: 0x70c408c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37962: skipped 000868 1785221007.741274 trunc from 0x037962 to 0x0126b4 (0x252ad bytes) 000869 1785221007.804416 punch from 0x004deb to 0x012239 (0xd44f bytes) 000870 1785221007.812139 mapread 0x0104ad thru 0x0126b4 (0x2208 bytes) 000871 1785221007.829724 mapread 0x011f2c thru 0x0126b4 (0x789 bytes) 000872 1785221007.830917 fallocate 0x003eb7 thru 0x007b33 (0x3c7d bytes) 000873 1785221007.838542 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x126b5): Mirror 4: 0xcafafbef (stale) Mirror 5: 0xcafafbef (stale) Mirror 7: 0xcafafbef (stale) Mirror 8: 0xcafafbef (stale) Mirror 11: 0xcafafbef (stale) Mirror 15: 0xcafafbef (stale) Mirror 16: 0xcafafbef (stale) Mirror 17: 0xac69e923 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x126b5): Mirror 4: 0xcafafbef (stale) Mirror 5: 0xcafafbef (stale) Mirror 7: 0xcafafbef (stale) Mirror 8: 0xcafafbef (stale) Mirror 11: 0xcafafbef (stale) Mirror 15: 0xcafafbef (stale) Mirror 16: 0xcafafbef (stale) Mirror 17: 0xac69e923 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x126b5: skipped 000874 1785221008.315911 mapread 0x002d0c thru 0x00b072 (0x8367 bytes) 000877 1785221008.345575 write 0x0010db thru 0x0097a3 (0x86c9 bytes) 000879 1785221008.359974 mapread 0x004a41 thru 0x011ded (0xd3ad bytes) 000880 1785221008.370421 trunc from 0x00aecb to 0x011ef7 (0x702d bytes) 000881 1785221008.376144 trunc from 0x00d95c to 0x0126b4 (0x4d59 bytes) 000883 1785221008.383448 mapread 0x00dade thru 0x0126b4 (0x4bd7 bytes) 000884 1785221008.394290 mapwrite 0x03d686 thru 0x03ffff (0x297a bytes) 000885 1785221008.468419 fallocate 0x03af63 thru 0x03ffff (0x509d bytes) 000886 1785221008.499038 MIRROR_RESYNC 9 mirrors 000887 1785221009.954995 trunc from 0x01e27f to 0x01ef3b (0xcbd bytes) 000888 1785221010.000834 read 0x00177a thru 0x00c45f (0xace6 bytes) 000889 1785221010.008766 trunc from 0x039948 to 0x03ffff (0x66b8 bytes) 000890 1785221010.013915 read 0x00964b thru 0x01436f (0xad25 bytes) 000891 1785221010.033148 fallocate 0x006e80 thru 0x014b54 (0xdcd5 bytes) 000892 1785221010.037477 fallocate 0x018d60 thru 0x027bb6 (0xee57 bytes) 000893 1785221010.042806 mapwrite 0x00ea41 thru 0x019741 (0xad01 bytes) 000894 1785221010.087514 write 0x0014bd thru 0x006a15 (0x5559 bytes) 000895 1785221010.089264 trunc from 0x02a08b to 0x03161a (0x7590 bytes) 000896 1785221010.102562 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc680367e (stale) Mirror 5: 0xc680367e (stale) Mirror 7: 0xc680367e (stale) Mirror 8: 0xc680367e (stale) Mirror 11: 0xc680367e (stale) Mirror 15: 0xc680367e (stale) Mirror 16: 0xc680367e (stale) Mirror 17: 0xc680367e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe90e5246 Mirror 18: 0xe90e5246 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000897 1785221010.988036 read 0x0124af thru 0x0193d5 (0x6f27 bytes) 000898 1785221011.007289 write 0x032bff thru 0x03438b (0x178d bytes) 000899 1785221011.051067 read 0x01b3c3 thru 0x021758 (0x6396 bytes) 000900 1785221011.072421 read 0x035518 thru 0x03d3d8 (0x7ec1 bytes) 000901 1785221011.074506 mapread 0x02590c thru 0x033626 (0xdd1b bytes) 000902 1785221011.077343 mapread 0x02b50c thru 0x033017 (0x7b0c bytes) 000904 1785221011.079757 trunc from 0x040000 to 0x023ec2 (0x1c13d bytes) 000905 1785221011.129156 write 0x009cba thru 0x017da5 (0xe0ec bytes) 000906 1785221011.147854 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23ec3): Mirror 4: 0x7eb7be3c (stale) Mirror 7: 0x7eb7be3c (stale) Mirror 8: 0x7eb7be3c (stale) Mirror 11: 0x7eb7be3c (stale) Mirror 15: 0x7eb7be3c (stale) Mirror 16: 0x7eb7be3c (stale) Mirror 17: 0x7eb7be3c (stale) Mirror 18: 0xb5269b91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23ec3): Mirror 4: 0x7eb7be3c (stale) Mirror 7: 0x7eb7be3c (stale) Mirror 8: 0x7eb7be3c (stale) Mirror 11: 0x7eb7be3c (stale) Mirror 15: 0x7eb7be3c (stale) Mirror 16: 0x7eb7be3c (stale) Mirror 17: 0x7eb7be3c (stale) Mirror 18: 0xb5269b91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23ec3: skipped 000907 1785221011.730297 MIRROR_RESYNC 9 mirrors 000908 1785221013.060458 punch from 0x01ff14 to 0x023ec2 (0x3faf bytes) 000909 1785221013.103868 trunc from 0x00fa98 to 0x01e1e3 (0xe74c bytes) 000911 1785221013.111535 trunc from 0x021abb to 0x023ec2 (0x2408 bytes) 000913 1785221013.117985 write 0x03d63c thru 0x03ffff (0x29c4 bytes) 000914 1785221013.119901 mapwrite 0x009050 thru 0x0148c7 (0xb878 bytes) 000915 1785221013.189821 trunc from 0x0184a6 to 0x01eb0b (0x6666 bytes) 000916 1785221013.197343 write 0x01ad9d thru 0x029afc (0xed60 bytes) 000917 1785221013.225598 mapread 0x038cdb thru 0x03ffff (0x7325 bytes) 000918 1785221013.244562 punch from 0x0156d8 to 0x0254eb (0xfe14 bytes) 000919 1785221013.275866 write 0x023e2b thru 0x02f2d9 (0xb4af bytes) 000920 1785221013.285811 fallocate 0x001e22 thru 0x00bd5d (0x9f3c bytes) 000921 1785221013.304670 read 0x03532b thru 0x03bdb3 (0x6a89 bytes) 000922 1785221013.313777 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x15a521e2 (stale) Mirror 7: 0x15a521e2 (stale) Mirror 8: 0x15a521e2 (stale) Mirror 11: 0x15a521e2 (stale) Mirror 16: 0x15a521e2 (stale) Mirror 17: 0x15a521e2 (stale) Mirror 18: 0x15a521e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x15a521e2 (stale) Mirror 7: 0x15a521e2 (stale) Mirror 8: 0x15a521e2 (stale) Mirror 11: 0x15a521e2 (stale) Mirror 16: 0x15a521e2 (stale) Mirror 17: 0x15a521e2 (stale) Mirror 18: 0x15a521e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1785221013.858346 write 0x0314af thru 0x0339c1 (0x2513 bytes) 000925 1785221013.878127 trunc from 0x015cf3 to 0x01e4a3 (0x87b1 bytes) 000926 1785221013.884505 write 0x02c8c2 thru 0x03342a (0x6b69 bytes) 000927 1785221013.894207 trunc from 0x035a76 to 0x035f14 (0x49f bytes) 000928 1785221013.901478 mapread 0x005072 thru 0x007e7c (0x2e0b bytes) 000929 1785221013.919468 mapread 0x03e207 thru 0x03f64f (0x1449 bytes) 000930 1785221013.934999 fallocate 0x01a731 thru 0x01b971 (0x1241 bytes) 000931 1785221013.942366 trunc from 0x00db92 to 0x010cff (0x316e bytes) 000932 1785221013.947715 fallocate 0x00aac1 thru 0x0172b5 (0xc7f5 bytes) 000933 1785221013.953325 write 0x001167 thru 0x00a2db (0x9175 bytes) 000934 1785221013.955042 trunc from 0x040000 to 0x038de3 (0x721c bytes) 000935 1785221013.973469 punch from 0x007187 to 0x00e049 (0x6ec3 bytes) 000937 1785221014.003729 mapread 0x01a3dd thru 0x024362 (0x9f86 bytes) 000938 1785221014.021707 write 0x01f819 thru 0x0288b3 (0x909b bytes) 000939 1785221014.023255 read 0x003be0 thru 0x00f7b7 (0xbbd8 bytes) 000940 1785221014.031163 punch from 0x0211ff to 0x02d4c6 (0xc2c8 bytes) 000941 1785221014.046908 trunc from 0x038de4 to 0x005e8d (0x32f56 bytes) 000942 1785221014.062391 fallocate 0x03e071 thru 0x03ffff (0x1f8f bytes) 000943 1785221014.066329 trunc from 0x03b878 to 0x03ffff (0x4788 bytes) 000945 1785221014.075207 mapwrite 0x03de49 thru 0x03ffff (0x21b7 bytes) 000947 1785221014.101136 write 0x006666 thru 0x012384 (0xbd1f bytes) 000948 1785221014.114531 mapwrite 0x026ea8 thru 0x028478 (0x15d1 bytes) 000949 1785221014.137768 trunc from 0x004e7d to 0x014810 (0xf994 bytes) 000950 1785221014.151383 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x15a521e2 (stale) Mirror 7: 0x15a521e2 (stale) Mirror 8: 0x15a521e2 (stale) Mirror 11: 0x15a521e2 (stale) Mirror 17: 0x15a521e2 (stale) Mirror 18: 0x15a521e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x15a521e2 (stale) Mirror 7: 0x15a521e2 (stale) Mirror 8: 0x15a521e2 (stale) Mirror 11: 0x15a521e2 (stale) Mirror 17: 0x15a521e2 (stale) Mirror 18: 0x15a521e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000951 1785221014.615601 read 0x0330c6 thru 0x0399cb (0x6906 bytes) 000952 1785221014.628539 trunc from 0x040000 to 0x0042c6 (0x3bd39 bytes) 000953 1785221014.658756 trunc from 0x0042c7 to 0x0275a0 (0x232da bytes) 000954 1785221014.677704 mapwrite 0x024bb7 thru 0x030f3a (0xc384 bytes) 000955 1785221014.742968 trunc from 0x030f3b to 0x02c656 (0x48e4 bytes) 000956 1785221014.770788 punch from 0x01f03a to 0x01fa69 (0xa30 bytes) 000957 1785221014.777406 trunc from 0x016576 to 0x01f5bc (0x9047 bytes) 000958 1785221014.782586 punch from 0x0247ff to 0x02c656 (0x7e58 bytes) 000959 1785221014.790141 write 0x0149fa thru 0x01d0dc (0x86e3 bytes) 000960 1785221014.807045 read 0x017749 thru 0x01f4fe (0x7db6 bytes) 000961 1785221014.819918 fallocate 0x022f1c thru 0x02ba2f (0x8b14 bytes) 000962 1785221014.825290 punch from 0x001a4e to 0x006e81 (0x5434 bytes) 000963 1785221014.834550 read 0x000921 thru 0x008a9e (0x817e bytes) 000964 1785221014.849109 trunc from 0x0197e9 to 0x02083e (0x7056 bytes) 000965 1785221014.870931 fallocate 0x010263 thru 0x01ed0f (0xeaad bytes) 000966 1785221014.901797 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c657): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c657): Mirror 4: 0xb5e15401 (stale) Mirror 7: 0xb5e15401 (stale) Mirror 8: 0xb5e15401 (stale) Mirror 11: 0xb5e15401 (stale) Mirror 17: 0xb5e15401 (stale) Mirror 18: 0xb5e15401 (stale) CRC-32 checksum value for chunk [0, 0x2c657): Mirror 1: 0x2165aba5 Mirror 19: 0x2165aba5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c657: skipped 000967 1785221015.940570 mapread 0x01b14b thru 0x01cc13 (0x1ac9 bytes) 000968 1785221015.965198 trunc from 0x02c657 to 0x03aee0 (0xe88a bytes) 000969 1785221016.032075 mapread 0x02f758 thru 0x032b85 (0x342e bytes) 000970 1785221016.048451 write 0x0024ce thru 0x00800d (0x5b40 bytes) 000971 1785221016.061069 trunc from 0x022f69 to 0x02e944 (0xb9dc bytes) 000972 1785221016.067339 mapread 0x00de3e thru 0x0173a5 (0x9568 bytes) 000973 1785221016.084214 fallocate 0x019062 thru 0x0217b2 (0x8751 bytes) 000974 1785221016.092900 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aee1): Mirror 7: 0x837a176d (stale) Mirror 8: 0x837a176d (stale) Mirror 11: 0x837a176d (stale) Mirror 17: 0x837a176d (stale) Mirror 18: 0x837a176d (stale) Mirror 19: 0x723a88da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aee1): Mirror 7: 0x837a176d (stale) Mirror 8: 0x837a176d (stale) Mirror 11: 0x837a176d (stale) Mirror 17: 0x837a176d (stale) Mirror 18: 0x837a176d (stale) Mirror 19: 0x723a88da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aee1: skipped 000975 1785221016.463008 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3aee1): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aee1): Mirror 7: 0x837a176d (stale) Mirror 8: 0x837a176d (stale) Mirror 11: 0x837a176d (stale) Mirror 17: 0x837a176d (stale) Mirror 18: 0x837a176d (stale) Mirror 19: 0x723a88da (stale) CRC-32 checksum value for chunk [0, 0x3aee1): Mirror 1: 0xd47dd242 Mirror 20: 0xd47dd242 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aee1: skipped 000976 1785221017.526281 write 0x016552 thru 0x02224c (0xbcfb bytes) 000977 1785221017.583888 mapread 0x036b82 thru 0x03aee0 (0x435f bytes) 000978 1785221017.610096 mapwrite 0x025812 thru 0x02d047 (0x7836 bytes) 000979 1785221017.646279 write 0x03a872 thru 0x03ffff (0x578e bytes) 000980 1785221017.649125 write 0x0266fb thru 0x033227 (0xcb2d bytes) 000981 1785221017.653971 punch from 0x037b15 to 0x03985c (0x1d48 bytes) 000982 1785221017.662459 trunc from 0x00161d to 0x00a114 (0x8af8 bytes) 000983 1785221017.667088 mapread 0x01b3b1 thru 0x022b74 (0x77c4 bytes) 000984 1785221017.668364 fallocate 0x03ced8 thru 0x03ffff (0x3128 bytes) 000985 1785221017.676147 read 0x008710 thru 0x010bde (0x84cf bytes) 000987 1785221017.691543 write 0x03e238 thru 0x03ffff (0x1dc8 bytes) 000988 1785221017.694231 read 0x001c5b thru 0x0047fb (0x2ba1 bytes) 000989 1785221017.704507 trunc from 0x024fd5 to 0x032f56 (0xdf82 bytes) 000990 1785221017.721389 mapread 0x03a94d thru 0x03ffff (0x56b3 bytes) 000991 1785221017.722653 trunc from 0x040000 to 0x016e18 (0x291e7 bytes) 000993 1785221017.747219 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x16e19): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16e19): Mirror 7: 0xc3b00898 (stale) Mirror 8: 0xc3b00898 (stale) Mirror 11: 0xc3b00898 (stale) Mirror 17: 0xc3b00898 (stale) Mirror 18: 0xc3b00898 (stale) Mirror 19: 0x29404350 (stale) Mirror 20: 0x36e9a34d (stale) CRC-32 checksum value for chunk [0, 0x16e19): Mirror 1: 0xaa966a26 Mirror 21: 0xaa966a26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16e19: skipped 000994 1785221018.638546 read 0x00a413 thru 0x016e18 (0xca06 bytes) 000995 1785221018.652385 mapread 0x00a572 thru 0x0122a1 (0x7d30 bytes) 000996 1785221018.671188 MIRROR_RESYNC 9 mirrors 000997 1785221020.029156 fallocate 0x03412d thru 0x03ffff (0xbed3 bytes) 000998 1785221020.070313 mapread 0x02e1b1 thru 0x031413 (0x3263 bytes) 000999 1785221020.095812 MIRROR_RESYNC 9 mirrors 001000 1785221021.798542 write 0x0098f0 thru 0x013029 (0x973a bytes) All operations completed A-OK! PASS 70a (89s) == sanity-flr test 71: check mirror extend parent fid ==== 02:43:46 (1785221026) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x57:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 02:43:54 (1785221034) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191834 s, 10.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111805 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 02:44:01 (1785221041) Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre2 Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2931264 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6421856 .. Extending file size to 8191616 .. Extending file size to 6970816 .. Extending file size to 7539200 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7631424 .. Extending file size to 8483872 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8504704 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8539296 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9028224 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8805984 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61311 61313 61315 61318 61319 192.168.204.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg410-client.virtnet /mnt/lustre2 (opts:) 192.168.204.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg410-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:45:20 (1785221120) Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre2 Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8484576 .. Extending file size to 3297696 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4934848 .. Extending file size to 7988736 .. Extending file size to 8487200 .. Extending file size to 8814176 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9156256 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8852544 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62929 62930 62932 62934 62936 192.168.204.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg410-client.virtnet /mnt/lustre2 (opts:) 192.168.204.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg410-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:46:34 (1785221194) Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1268800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24664 3582356 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18780 3588240 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43444 7170596 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1268800 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24664 3582356 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18780 3588240 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43444 7170596 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143948 s, 14.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162269 s, 12.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x61f57e0d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg410-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:46:51 (1785221211) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0902976 s, 23.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 02:46:57 (1785221217) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1da:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x311:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:47:02 (1785221222) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0847844 s, 24.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:47:08 (1785221228) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0538332 s, 39.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:47:14 (1785221234) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.088034 s, 23.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x313:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:47:44 (1785221264) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg410-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077581 s, 13.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:48:29 (1785221309) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg410-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100411 s, 20.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:49:12 (1785221352) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg410-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg410-server: Pool lustre.test_204f created Adding targets to pool oleg410-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg410-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0781663 s, 26.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg410-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg410-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg410-server: Pool lustre.test_204f destroyed PASS 204f (62s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:50:14 (1785221414) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:50:20 (1785221420) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:50:25 (1785221425) oleg410-server: Pool lustre.test_206 created Adding targets to pool oleg410-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg410-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg410-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg410-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg410-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:50:48 (1785221448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720983 s, 14.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452404 s, 23.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xba0a5a92 Mirror 3: 0xba0a5a92 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:50:55 (1785221455) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:50:57 (1785221457) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:51:00 (1785221460) Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0511658 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xe40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x22e0000 Read this (4 bytes): BBBB 192.168.204.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg410-client.virtnet /mnt/lustre2 (opts:) PASS 209a (30s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:51:30 (1785221490) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0313343 s, 261 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:51:35 (1785221495) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334198 s, 31.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f9 lmm_fid: [0x200000405:0x1f9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f9 lmm_fid: [0x200000405:0x1f9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x302:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611641 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-28 02:51:37.000000000 -0400 Modify: 2026-07-28 02:51:37.000000000 -0400 Change: 2026-07-28 02:51:38.000000000 -0400 Birth: 2026-07-28 02:51:36.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:51:41 (1785221501) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283293 s, 37.0 MB/s oleg410-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg410-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg410-server: O/280000401/d1/321: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:51:57 (1785221517) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.161914 s, 253 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0285602 s, 143 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 28 02:51 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1fe:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 28 02:51 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:52:02 (1785221522) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:52:07 (1785221527) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1647 sec =========== 02:52:13 (1785221533) === sanity-flr: start cleanup 02:52:14 (1785221534) === === sanity-flr: finish cleanup 02:52:21 (1785221541) ===