-----============= acceptance-small: sanity-flr ============----- Fri Jun 26 09:32:24 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg103-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:32:46 (1782480766) === oleg103-client.virtnet: executing check_config_client /mnt/lustre oleg103-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg103-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff927bc9fcb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff927bc9fcb800.idle_timeout=debug disable quota as required oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:33:24 (1782480804) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7628] == sanity-flr test 0a: lfs mirror create with -N option == 09:33:27 (1782480807) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:33:45 (1782480825) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:33:49 (1782480829) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:33:53 (1782480833) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:34:18 (1782480858) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:34:22 (1782480862) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:34:26 (1782480866) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff927bc9fcb800.stats=clear osc.lustre-OST0001-osc-ffff927bc9fcb800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:34:34 (1782480874) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:34:47 (1782480887) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:34:56 (1782480896) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:35:00 (1782480900) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:35:10 (1782480910) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:35:20 (1782480920) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:35:28 (1782480928) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:35:31 (1782480931) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:35:39 (1782480939) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:35:52 (1782480952) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:36:00 (1782480960) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.595793 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936874 s, 11.2 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:36:11 (1782480971) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.228517 s, 32.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:36:20 (1782480980) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:36:31 (1782480991) Stopping /mnt/lustre-ost1 (opts:) on oleg103-server oleg103-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg103-server: oleg103-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg103-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg103-server oleg103-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg103-server: oleg103-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg103-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:37:46 (1782481066) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.295022 s, 14.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg103-server oleg103-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg103-server: oleg103-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg103-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:38:31 (1782481111) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg103-server oleg103-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg103-server: oleg103-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg103-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg103-server oleg103-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg103-server: oleg103-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg103-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (73s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:39:44 (1782481184) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.03282 s, 13.0 MB/s -rw-r--r-- 1 root root 52428800 Jun 26 09:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.383s user 0m0.001s sys 0m0.202s Stopping /mnt/lustre-ost1 (opts:) on oleg103-server oleg103-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927bc9fcb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.293s user 0m0.000s sys 0m0.162s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg103-server: oleg103-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg103-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg103-server oleg103-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.891s user 0m0.000s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg103-server: oleg103-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg103-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927bc9fcb800.ost_server_uuid in FULL state after 0 sec PASS 33b (85s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:41:10 (1782481270) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:41:13 (1782481273) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:41:16 (1782481276) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:41:18 (1782481278) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 09:41:24 (1782481284) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.634103 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:41:35 (1782481295) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212488 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310661 s, 33.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:41:42 (1782481302) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:41:50 (1782481310) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00145115 s, 706 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00196505 s, 521 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00496057 s, 206 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000998048 s, 1.0 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 09:41:59 (1782481319) create test files with size 15731710 .. acfb3968cb62ecf8c9e16f0c830573f9 - 635a68a77760329989e8acf239bddc06 - c7c03e5f1a9e5354c58dc9571dee8c69 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg103-client.virtnet /mnt/lustre (opts:) Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 09:42:19 (1782481339) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:42:31 (1782481351) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:42:36 (1782481356) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125001 s, 8.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238054 s, 44.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 09:42:44 (1782481364) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136834 s, 15.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.297427 s, 14.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201876 s, 10.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 09:42:57 (1782481377) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:43:00 (1782481380) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:43:03 (1782481383) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171278 s, 61.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 09:43:10 (1782481390) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:43:13 (1782481393) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:43:21 (1782481401) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.241021 s, 43.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg103-client.virtnet /mnt/lustre (opts:) Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:43:31 (1782481411) llite.lustre-ffff927bc30e9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300801 s, 34.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg103-client.virtnet /mnt/lustre (opts:) Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff927bc7044800.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:43:42 (1782481422) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (29s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:44:11 (1782481451) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:44:41 (1782481481) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:44:44 (1782481484) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.203548 s, 15.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.129002 s, 24.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5fea6f3d (nosync,stale) Mirror 2: 0x5fea6f3d (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7427287ff55fc2e9dfcc4d43043952ff - mirror 1 checksum: d9cc138707da9117dd27e7379e9722ca - mirror 2 checksum: 7427287ff55fc2e9dfcc4d43043952ff - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:44:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:44:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7427287ff55fc2e9dfcc4d43043952ff - mirror 1 checksum: d9cc138707da9117dd27e7379e9722ca - mirror 2 checksum: d9cc138707da9117dd27e7379e9722ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:44:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:44:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:44:54 (1782481494) Failing mds1 on oleg103-server Stopping /mnt/lustre-mds1 (opts:) on oleg103-server 09:44:59 (1782481499) shut down facet: mds1 facet_host: oleg103-server facet_failover_host: oleg103-server Failover mds1 to oleg103-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-MDT0000 09:45:19 (1782481519) targets are mounted 09:45:19 (1782481519) facet_failover done oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200116 s, 52.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 50A (39s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:45:33 (1782481533) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:45:35 (1782481535) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:45:38 (1782481538) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.004328 s, 3.8 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:45:44 (1782481544) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:45:47 (1782481547) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:45:49 (1782481549) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:46:07 (1782481567) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (25s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:46:32 (1782481592) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:46:54 (1782481614) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:47:15 (1782481635) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 09:47:21 (1782481641) mirror create pid 53755 mirror split pid 53756 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7768: 53756 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53755 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 09:48:27 (1782481707) Seed set to 1709 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x34206 000004 1782481709.469659 trunc from 00000000 to 0x034205 (0x34206 bytes) 000005 1782481709.483800 mapread 0x00b190 thru 0x0138ac (0x871d bytes) 000009 1782481709.507093 mapread 0x003a56 thru 0x00dc6f (0xa21a bytes) 000014 1782481709.519349 read 0x0071b8 thru 0x0079be (0x807 bytes) 000015 1782481709.520097 mapwrite 0x001136 thru 0x00657f (0x544a bytes) 000016 1782481709.560853 mapread 0x01b068 thru 0x02accf (0xfc68 bytes) 000017 1782481709.580364 read 0x018467 thru 0x023f70 (0xbb0a bytes) 000019 1782481709.581705 write 0x014f08 thru 0x01791e (0x2a17 bytes) 000020 1782481709.583757 mapread 0x00e222 thru 0x01b14d (0xcf2c bytes) 000021 1782481709.585187 mapwrite 0x000b3e thru 0x00ce87 (0xc34a bytes) 000024 1782481709.653764 read 0x02780b thru 0x027b3d (0x333 bytes) 000027 1782481709.655008 mapread 0x00129e thru 0x00c8f8 (0xb65b bytes) 000029 1782481709.657705 write 0x004892 thru 0x012528 (0xdc97 bytes) 000031 1782481709.659361 write 0x019607 thru 0x025fe8 (0xc9e2 bytes) 000032 1782481709.660372 read 0x025724 thru 0x030e94 (0xb771 bytes) 000033 1782481709.661470 mapwrite 0x01d580 thru 0x025b2e (0x85af bytes) 000034 1782481709.717800 read 0x0147d5 thru 0x015747 (0xf73 bytes) 000036 1782481709.719022 write 0x03fad4 thru 0x03fb35 (0x62 bytes) 000037 1782481709.720043 read 0x024f00 thru 0x0344cc (0xf5cd bytes) 000038 1782481709.721449 write 0x00862c thru 0x00a609 (0x1fde bytes) 000040 1782481709.723323 read 0x00dec8 thru 0x019c2c (0xbd65 bytes) 000041 1782481709.724150 read 0x028e74 thru 0x02f6b1 (0x683e bytes) 000042 1782481709.724920 write 0x031fb4 thru 0x03ffff (0xe04c bytes) 000043 1782481709.726828 write 0x02dfe9 thru 0x0356e5 (0x76fd bytes) 000047 1782481709.730091 write 0x01cb4c thru 0x02a1b8 (0xd66d bytes) 000048 1782481709.733192 mapread 0x02cc62 thru 0x03492f (0x7cce bytes) 000049 1782481709.735071 mapwrite 0x011025 thru 0x0149a1 (0x397d bytes) 000052 1782481709.800954 write 0x01422f thru 0x01d67f (0x9451 bytes) 000053 1782481709.803157 read 0x00295b thru 0x00a4ee (0x7b94 bytes) 000054 1782481709.804622 trunc from 0x040000 to 0x023643 (0x1c9bc bytes) 000055 1782481709.821629 read 0x004fde thru 0x0066be (0x16e1 bytes) 000058 1782481709.825881 write 0x000011 thru 0x00434f (0x433f bytes) 000063 1782481709.830928 mapread 0x01e78a thru 0x023643 (0x4eba bytes) 000064 1782481709.832199 write 0x015aba thru 0x01ce94 (0x73db bytes) 000066 1782481709.834975 read 0x0088dc thru 0x00c8ae (0x3fd3 bytes) 000069 1782481709.842325 MIRROR_EXTEND to 2 mirrors 000070 1782481710.344603 write 0x035eab thru 0x03ffff (0xa155 bytes) 000072 1782481710.421453 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000073 1782481710.563101 write 0x009b79 thru 0x00ff4f (0x63d7 bytes) 000076 1782481710.576565 mapwrite 0x02a463 thru 0x02f694 (0x5232 bytes) 000077 1782481710.626487 mapread 0x0101bd thru 0x01791c (0x7760 bytes) 000079 1782481710.637070 read 0x01313a thru 0x017522 (0x43e9 bytes) 000081 1782481710.638765 read 0x011866 thru 0x01e1ea (0xc985 bytes) 000082 1782481710.660527 mapread 0x01708c thru 0x01be06 (0x4d7b bytes) 000084 1782481710.662913 trunc from 0x040000 to 0x00740a (0x38bf5 bytes) 000085 1782481710.678904 write 0x0006a1 thru 0x0043f8 (0x3d58 bytes) 000087 1782481710.681804 mapwrite 0x015b18 thru 0x02078b (0xac74 bytes) 000088 1782481710.809933 mapwrite 0x03a254 thru 0x03dccf (0x3a7c bytes) 000089 1782481710.904194 read 0x00a4ea thru 0x01a1fc (0xfd13 bytes) 000090 1782481710.913952 write 0x00e555 thru 0x016c13 (0x86bf bytes) 000092 1782481710.915493 read 0x033399 thru 0x0345ec (0x1254 bytes) 000093 1782481710.916472 read 0x03514a thru 0x03dccf (0x8b86 bytes) 000096 1782481710.918192 mapread 0x022510 thru 0x023f30 (0x1a21 bytes) 000099 1782481710.920551 mapread 0x03b64f thru 0x03dccf (0x2681 bytes) 000100 1782481710.921557 mapwrite 0x0029d3 thru 0x00f1a7 (0xc7d5 bytes) 000101 1782481710.952309 write 0x01bf87 thru 0x01c5b9 (0x633 bytes) 000102 1782481710.953842 mapread 0x001918 thru 0x003ed0 (0x25b9 bytes) 000104 1782481710.955851 mapread 0x02b56d thru 0x030f96 (0x5a2a bytes) 000105 1782481710.956791 read 0x02fdad thru 0x037e57 (0x80ab bytes) 000106 1782481710.957526 mapread 0x01c2bb thru 0x025d34 (0x9a7a bytes) 000110 1782481710.959488 mapwrite 0x03d3ba thru 0x03ffff (0x2c46 bytes) 000112 1782481711.025908 mapwrite 0x03b28f thru 0x03ffff (0x4d71 bytes) 000113 1782481711.070367 write 0x005c80 thru 0x00d0e8 (0x7469 bytes) 000114 1782481711.071886 write 0x01163b thru 0x01708d (0x5a53 bytes) 000115 1782481711.073062 mapwrite 0x036075 thru 0x03a0e4 (0x4070 bytes) 000117 1782481711.125548 write 0x006c37 thru 0x009b36 (0x2f00 bytes) 000119 1782481711.130674 trunc from 0x040000 to 0x01b6b5 (0x2494a bytes) 000121 1782481711.161689 mapread 0x010ae6 thru 0x0131a4 (0x26bf bytes) 000124 1782481711.163859 mapwrite 0x03b8c0 thru 0x03ffff (0x4740 bytes) 000129 1782481711.312372 mapwrite 0x0092fe thru 0x01122d (0x7f30 bytes) 000131 1782481711.361899 trunc from 0x040000 to 0x0331a1 (0xce5e bytes) 000132 1782481711.379394 write 0x01159f thru 0x012f8c (0x19ee bytes) 000138 1782481711.385159 trunc from 0x0331a2 to 0x028c4b (0xa556 bytes) 000145 1782481711.409327 mapread 0x00ccba thru 0x00f60d (0x2954 bytes) 000147 1782481711.411534 read 0x008a03 thru 0x00d0bb (0x46b9 bytes) 000148 1782481711.412651 write 0x01cbfa thru 0x02c32d (0xf734 bytes) 000150 1782481711.426869 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3d286 000154 1782481711.993248 trunc from 0x02c32e to 0x03d285 (0x10f58 bytes) 000155 1782481712.150399 trunc from 0x03d286 to 0x009b9e (0x336e7 bytes) 000156 1782481712.185845 write 0x0140ca thru 0x01d2d0 (0x9207 bytes) 000157 1782481712.196616 mapread 0x01b8a5 thru 0x01d2d0 (0x1a2c bytes) 000159 1782481712.200561 read 0x007f3b thru 0x012d9b (0xae61 bytes) 000162 1782481712.231588 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000165 1782481712.385833 read 0x0000f6 thru 0x000ae4 (0x9ef bytes) 000166 1782481712.391533 mapread 0x003483 thru 0x00a2c2 (0x6e40 bytes) 000168 1782481712.411119 trunc from 0x01d2d1 to 0x02bbe3 (0xe913 bytes) 000171 1782481712.431446 read 0x00c3d0 thru 0x00f5b8 (0x31e9 bytes) 000173 1782481712.432451 trunc from 0x02bbe4 to 0x036b31 (0xaf4e bytes) 000175 1782481712.442982 read 0x019784 thru 0x02698e (0xd20b bytes) 000176 1782481712.448778 mapwrite 0x01e7c3 thru 0x02a414 (0xbc52 bytes) 000178 1782481712.507330 MIRROR_EXTEND to 2 mirrors 000179 1782481713.178031 read 0x02d80c thru 0x02e0b5 (0x8aa bytes) 000180 1782481713.182842 mapwrite 0x028f36 thru 0x02d459 (0x4524 bytes) 000181 1782481713.358398 write 0x00f48d thru 0x015222 (0x5d96 bytes) 000182 1782481713.373098 mapwrite 0x03fdfe thru 0x03ffff (0x202 bytes) 000183 1782481713.485898 trunc from 0x040000 to 0x001faf (0x3e050 bytes) 000184 1782481713.498491 mapwrite 0x015d68 thru 0x019180 (0x3419 bytes) 000185 1782481713.541828 trunc from 0x019181 to 0x00845f (0x10d21 bytes) 000186 1782481713.562887 trunc from 0x008460 to 0x01d867 (0x15408 bytes) 000187 1782481713.587418 write 0x033408 thru 0x03ffff (0xcbf8 bytes) 000188 1782481713.590538 read 0x03e722 thru 0x03ffff (0x18de bytes) 000190 1782481713.593400 mapread 0x031199 thru 0x038423 (0x728b bytes) 000192 1782481713.612299 mapwrite 0x015315 thru 0x015542 (0x22e bytes) 000193 1782481713.677563 mapwrite 0x0305f0 thru 0x03d483 (0xce94 bytes) 000194 1782481713.760009 read 0x034c10 thru 0x0397b9 (0x4baa bytes) 000196 1782481713.784750 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000201 1782481713.970500 MIRROR_EXTEND to 2 mirrors 000202 1782481714.668032 mapread 0x02de18 thru 0x03a27e (0xc467 bytes) 000204 1782481714.704665 mapread 0x029c9c thru 0x03006c (0x63d1 bytes) 000205 1782481714.710605 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000208 1782481714.855535 MIRROR_EXTEND to 2 mirrors 000210 1782481715.531961 mapwrite 0x004dd6 thru 0x00d9da (0x8c05 bytes) 000212 1782481715.741633 trunc from 0x040000 to 0x0391e3 (0x6e1c bytes) 000213 1782481715.768234 trunc from 0x0391e4 to 0x00cdf5 (0x2c3ee bytes) 000215 1782481715.785992 mapread 0x0080de thru 0x00cdf5 (0x4d18 bytes) 000217 1782481715.797386 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xcdf6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcdf6): Mirror 2: 0x75ef83c7 (stale) CRC-32 checksum value for chunk [0, 0xcdf6): Mirror 1: 0xa0c2d326 Mirror 3: 0xa0c2d326 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcdf6: skipped 000219 1782481716.585411 MIRROR_RESYNC 3 mirrors 000220 1782481717.347296 trunc from 0x00cdf6 to 0x02cafb (0x1fd06 bytes) 000223 1782481717.429622 read 0x02acd7 thru 0x02cafb (0x1e25 bytes) 000225 1782481717.439370 MIRROR_RESYNC 3 mirrors 000226 1782481718.087901 mapread 0x00bf06 thru 0x016cf6 (0xadf1 bytes) 000229 1782481718.126301 mapwrite 0x03abc1 thru 0x03f5eb (0x4a2b bytes) 000230 1782481718.301245 trunc from 0x03f5ec to 0x0359e6 (0x9c05 bytes) 000231 1782481718.324144 mapwrite 0x02f601 thru 0x0369c7 (0x73c7 bytes) 000232 1782481718.387810 mapwrite 0x00bef3 thru 0x00c582 (0x690 bytes) 000233 1782481718.433689 mapwrite 0x0152c4 thru 0x01bf47 (0x6c84 bytes) 000234 1782481718.486308 read 0x02fd9e thru 0x0369c7 (0x6c2a bytes) 000235 1782481718.487326 read 0x01df48 thru 0x02b476 (0xd52f bytes) 000237 1782481718.500912 mapread 0x012f13 thru 0x01fe21 (0xcf0f bytes) 000238 1782481718.502333 trunc from 0x0369c8 to 0x015ccf (0x20cf8 bytes) 000240 1782481718.523521 read 0x013218 thru 0x015ccf (0x2ab8 bytes) 000242 1782481718.525416 mapread 0x0087a4 thru 0x00a62f (0x1e8c bytes) 000243 1782481718.527117 trunc from 0x015cd0 to 0x02394a (0xdc7b bytes) 000244 1782481718.547974 trunc from 0x02394b to 0x01bf6f (0x79db bytes) 000245 1782481718.573285 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bf70): Mirror 2: 0x3c4772d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bf70): Mirror 2: 0x3c4772d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bf70: skipped 000246 1782481718.798247 mapwrite 0x01e923 thru 0x0234b3 (0x4b91 bytes) 000247 1782481718.908974 write 0x01dfba thru 0x02a682 (0xc6c9 bytes) 000250 1782481718.913328 read 0x008140 thru 0x011123 (0x8fe4 bytes) 000251 1782481718.927163 MIRROR_RESYNC 2 mirrors 000252 1782481719.625512 mapwrite 0x026eef thru 0x032470 (0xb582 bytes) 000253 1782481719.867936 read 0x003714 thru 0x00a759 (0x7046 bytes) 000255 1782481719.882455 trunc from 0x032471 to 0x021978 (0x10af8 bytes) 000256 1782481719.919568 read 0x01e367 thru 0x021978 (0x3612 bytes) 000257 1782481719.938277 mapwrite 0x025591 thru 0x02d882 (0x82f2 bytes) 000261 1782481720.011165 trunc from 0x02d883 to 0x02caaf (0xdd3 bytes) 000262 1782481720.028762 write 0x01747a thru 0x01a2bf (0x2e46 bytes) 000264 1782481720.030662 mapread 0x018149 thru 0x02605b (0xdf13 bytes) 000265 1782481720.033450 mapwrite 0x009257 thru 0x00e333 (0x50dd bytes) 000266 1782481720.081922 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000267 1782481720.260535 mapwrite 0x022221 thru 0x02e0ea (0xbeca bytes) 000269 1782481720.374734 mapwrite 0x01cbfb thru 0x023a1e (0x6e24 bytes) 000270 1782481720.428128 mapwrite 0x020eeb thru 0x02d8f9 (0xca0f bytes) 000273 1782481720.483117 write 0x0253ef thru 0x02584a (0x45c bytes) 000275 1782481720.485494 mapwrite 0x035e12 thru 0x038d88 (0x2f77 bytes) 000276 1782481720.564950 trunc from 0x038d89 to 0x032950 (0x6438 bytes) 000278 1782481720.583112 trunc from 0x032951 to 0x03c98f (0xa03f bytes) 000279 1782481720.594291 trunc from 0x03c990 to 0x01bfcc (0x209c3 bytes) 000280 1782481720.605763 mapwrite 0x01e5f2 thru 0x02ada6 (0xc7b5 bytes) 000281 1782481720.719703 read 0x01bb19 thru 0x0213de (0x58c6 bytes) 000284 1782481720.724461 trunc from 0x02ada7 to 0x01924c (0x11b5a bytes) 000285 1782481720.746776 write 0x02bdeb thru 0x034fea (0x9200 bytes) 000289 1782481720.753067 read 0x023f61 thru 0x0280f9 (0x4199 bytes) 000290 1782481720.763344 mapread 0x011cd1 thru 0x01d0c6 (0xb3f6 bytes) 000292 1782481720.765546 mapread 0x00779f thru 0x00f1ae (0x7a10 bytes) 000293 1782481720.768580 read 0x00b49b thru 0x010606 (0x516c bytes) 000297 1782481720.772402 mapwrite 0x02709f thru 0x0294a7 (0x2409 bytes) 000298 1782481720.847501 write 0x006221 thru 0x01466d (0xe44d bytes) 000301 1782481720.851460 mapwrite 0x007dfc thru 0x012ceb (0xaef0 bytes) 000302 1782481720.892739 read 0x028121 thru 0x02a7b9 (0x2699 bytes) 000303 1782481720.893296 trunc from 0x034feb to 0x02d18e (0x7e5c bytes) 000304 1782481720.903611 write 0x03b3f0 thru 0x03cad7 (0x16e8 bytes) 000307 1782481720.906232 trunc from 0x03cad8 to 0x02f851 (0xd286 bytes) 000308 1782481720.919365 trunc from 0x02f852 to 0x01717d (0x186d4 bytes) 000310 1782481720.936245 trunc from 0x01717e to 0x03599a (0x1e81d bytes) 000312 1782481720.958612 mapread 0x021e4a thru 0x02b797 (0x994e bytes) 000313 1782481720.979438 trunc from 0x03599b to 0x02fb44 (0x5e56 bytes) 000316 1782481721.016481 read 0x012f59 thru 0x013a8c (0xb34 bytes) 000317 1782481721.017187 trunc from 0x02fb45 to 0x02a853 (0x52f1 bytes) 000318 1782481721.033762 write 0x018b9f thru 0x024be0 (0xc042 bytes) 000320 1782481721.037168 mapread 0x006762 thru 0x0111d6 (0xaa75 bytes) 000321 1782481721.038895 read 0x00e972 thru 0x01df67 (0xf5f6 bytes) 000322 1782481721.039875 write 0x0279d7 thru 0x036c84 (0xf2ae bytes) 000324 1782481721.043170 write 0x018bc8 thru 0x01c8ff (0x3d38 bytes) 000326 1782481721.045137 mapwrite 0x005293 thru 0x00f8f6 (0xa664 bytes) 000327 1782481721.090987 mapwrite 0x01782e thru 0x01c917 (0x50ea bytes) 000329 1782481721.138196 write 0x01978d thru 0x01d9a1 (0x4215 bytes) 000332 1782481721.141988 write 0x01079b thru 0x012c0b (0x2471 bytes) 000333 1782481721.143491 write 0x01e565 thru 0x02551d (0x6fb9 bytes) 000334 1782481721.145546 read 0x0298c0 thru 0x0358ef (0xc030 bytes) 000335 1782481721.146516 read 0x032a09 thru 0x036c84 (0x427c bytes) 000337 1782481721.148577 trunc from 0x036c85 to 0x018dd8 (0x1deac bytes) 000343 1782481721.176509 trunc from 0x018dd9 to 0x01d601 (0x4829 bytes) 000346 1782481721.200695 MIRROR_EXTEND to 2 mirrors 000348 1782481721.837526 read 0x007429 thru 0x014343 (0xcf1b bytes) 000349 1782481721.852516 write 0x00741c thru 0x00af4a (0x3b2f bytes) 000352 1782481721.930112 mapwrite 0x027993 thru 0x02863a (0xca8 bytes) 000356 1782481722.004548 read 0x0140ce thru 0x02164c (0xd57f bytes) 000357 1782481722.013500 mapwrite 0x03a140 thru 0x03ffff (0x5ec0 bytes) 000359 1782481722.107551 write 0x026007 thru 0x033da6 (0xdda0 bytes) 000361 1782481722.116447 MIRROR_RESYNC 2 mirrors 000362 1782481722.658241 MIRROR_EXTEND to 3 mirrors 000363 1782481723.156702 trunc from 0x040000 to 0x03a233 (0x5dcc bytes) 000366 1782481723.241288 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a234): Mirror 2: 0x7875bfe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a234): Mirror 2: 0x7875bfe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a234: skipped 000367 1782481723.594550 write 0x032947 thru 0x037a55 (0x510f bytes) 000369 1782481723.615138 read 0x014ba1 thru 0x023a27 (0xee87 bytes) 000370 1782481723.636085 MIRROR_RESYNC 2 mirrors 000371 1782481724.272346 mapwrite 0x001c04 thru 0x0089ab (0x6da8 bytes) 000372 1782481724.439346 mapread 0x031f4a thru 0x03a233 (0x82ea bytes) 000376 1782481724.458585 mapwrite 0x0112a9 thru 0x01d12c (0xbe84 bytes) 000379 1782481724.519330 trunc from 0x03a234 to 0x01b32c (0x1ef07 bytes) 000382 1782481724.546960 read 0x016a1c thru 0x01b32c (0x4911 bytes) 000384 1782481724.548938 write 0x0171dd thru 0x024087 (0xceab bytes) 000385 1782481724.550199 mapwrite 0x02470c thru 0x0271b0 (0x2aa5 bytes) 000386 1782481724.649598 mapwrite 0x03971e thru 0x03ffff (0x68e2 bytes) 000387 1782481724.717474 mapread 0x0199f0 thru 0x023296 (0x98a7 bytes) 000388 1782481724.718857 read 0x018c28 thru 0x01c75e (0x3b37 bytes) 000391 1782481724.721656 write 0x004fe7 thru 0x00d769 (0x8783 bytes) 000393 1782481724.724520 mapwrite 0x0004af thru 0x00101a (0xb6c bytes) 000394 1782481724.769906 trunc from 0x040000 to 0x029273 (0x16d8c bytes) 000395 1782481724.787678 read 0x01a850 thru 0x029273 (0xea24 bytes) 000396 1782481724.788415 trunc from 0x029274 to 0x01335b (0x15f18 bytes) 000399 1782481724.810531 MIRROR_RESYNC 2 mirrors 000400 1782481725.256913 read 0x004f72 thru 0x008da3 (0x3e32 bytes) 000401 1782481725.268130 MIRROR_RESYNC 2 mirrors 000404 1782481725.540480 read 0x006076 thru 0x00bd7f (0x5d0a bytes) 000406 1782481725.551116 MIRROR_RESYNC 2 mirrors 000407 1782481725.806093 read 0x012628 thru 0x01335b (0xd34 bytes) 000409 1782481725.811868 mapread 0x004d67 thru 0x006ee2 (0x217c bytes) 000410 1782481725.812675 read 0x002925 thru 0x0069a2 (0x407e bytes) 000411 1782481725.821584 read 0x00dd94 thru 0x01335b (0x55c8 bytes) 000413 1782481725.825943 mapwrite 0x03303d thru 0x03ef10 (0xbed4 bytes) 000414 1782481725.978319 MIRROR_RESYNC 2 mirrors 000416 1782481726.453849 mapwrite 0x037333 thru 0x03ffff (0x8ccd bytes) 000417 1782481726.581614 mapwrite 0x03a3ef thru 0x03ffff (0x5c11 bytes) 000419 1782481726.627367 mapwrite 0x03afd1 thru 0x03ffff (0x502f bytes) 000420 1782481726.665069 mapread 0x035600 thru 0x03ffff (0xaa00 bytes) 000421 1782481726.667316 trunc from 0x040000 to 0x007062 (0x38f9d bytes) 000423 1782481726.687188 read 0x00229d thru 0x006f12 (0x4c76 bytes) 000424 1782481726.698434 trunc from 0x007063 to 0x02c95e (0x258fc bytes) 000426 1782481726.711354 mapwrite 0x03b61a thru 0x03ffff (0x49e6 bytes) 000428 1782481726.773045 read 0x01190e thru 0x01d00b (0xb6fe bytes) 000430 1782481726.785840 mapread 0x005d9c thru 0x010b35 (0xad9a bytes) 000431 1782481726.787468 write 0x01655d thru 0x019224 (0x2cc8 bytes) 000432 1782481726.789123 trunc from 0x040000 to 0x003010 (0x3cfef bytes) 000434 1782481726.811482 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000436 1782481726.943631 mapwrite 0x00864e thru 0x013cf7 (0xb6aa bytes) truncating to largest ever: 0x3ff37 000439 1782481727.021133 trunc from 0x013cf8 to 0x03ff36 (0x2c23f bytes) 000440 1782481727.040035 mapwrite 0x014c78 thru 0x022eb7 (0xe240 bytes) 000441 1782481727.108966 trunc from 0x03ff37 to 0x030f74 (0xefc2 bytes) 000447 1782481727.147664 write 0x035341 thru 0x035907 (0x5c7 bytes) 000452 1782481727.166539 read 0x00a86e thru 0x0103ef (0x5b82 bytes) 000453 1782481727.168489 trunc from 0x035908 to 0x032766 (0x31a1 bytes) 000454 1782481727.192200 read 0x01a81b thru 0x01fa36 (0x521c bytes) 000458 1782481727.198300 read 0x014165 thru 0x0159b9 (0x1855 bytes) 000459 1782481727.199421 write 0x03fb4c thru 0x03ffff (0x4b4 bytes) 000461 1782481727.205308 trunc from 0x040000 to 0x003379 (0x3cc86 bytes) 000463 1782481727.268831 read 0x0019f7 thru 0x003379 (0x1983 bytes) 000465 1782481727.284191 mapwrite 0x032567 thru 0x034317 (0x1db1 bytes) 000468 1782481727.385460 read 0x02ba20 thru 0x034317 (0x88f8 bytes) 000469 1782481727.386422 read 0x003980 thru 0x00fdd8 (0xc459 bytes) 000471 1782481727.398787 write 0x032bde thru 0x03e520 (0xb943 bytes) 000476 1782481727.413354 read 0x019368 thru 0x0210d7 (0x7d70 bytes) 000477 1782481727.414345 trunc from 0x03e521 to 0x02f76c (0xedb4 bytes) 000478 1782481727.448546 write 0x02a12f thru 0x0391f9 (0xf0cb bytes) 000479 1782481727.452074 mapread 0x01f0a8 thru 0x028b82 (0x9adb bytes) 000480 1782481727.465827 mapread 0x015e56 thru 0x01f500 (0x96ab bytes) 000483 1782481727.473217 mapread 0x01c008 thru 0x0248f5 (0x88ee bytes) 000486 1782481727.484279 mapwrite 0x010c02 thru 0x01ec92 (0xe091 bytes) 000488 1782481727.555983 mapread 0x00439e thru 0x010e38 (0xca9b bytes) 000489 1782481727.558535 read 0x011dde thru 0x01b6a7 (0x98ca bytes) 000490 1782481727.560479 mapread 0x0276f1 thru 0x03561f (0xdf2f bytes) 000492 1782481727.562938 mapwrite 0x01356c thru 0x0148b6 (0x134b bytes) 000494 1782481727.625815 mapread 0x0011ea thru 0x008c36 (0x7a4d bytes) 000495 1782481727.631493 write 0x038dfa thru 0x03b8e6 (0x2aed bytes) 000496 1782481727.648471 MIRROR_EXTEND to 2 mirrors 000497 1782481728.140079 mapwrite 0x032b16 thru 0x03d19f (0xa68a bytes) 000499 1782481728.349670 read 0x018652 thru 0x022bbe (0xa56d bytes) 000501 1782481728.370812 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000502 1782481728.490565 mapread 0x01f3ce thru 0x021f64 (0x2b97 bytes) 000503 1782481728.501629 mapread 0x014f86 thru 0x024e80 (0xfefb bytes) 000504 1782481728.510322 mapread 0x037c5a thru 0x03af94 (0x333b bytes) 000506 1782481728.527097 MIRROR_EXTEND to 2 mirrors 000507 1782481728.953182 mapread 0x027807 thru 0x0315a9 (0x9da3 bytes) 000508 1782481728.972174 read 0x02cc70 thru 0x03996d (0xccfe bytes) 000509 1782481728.978444 mapwrite 0x032cc3 thru 0x03edf0 (0xc12e bytes) 000510 1782481729.175903 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3edf1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3edf1): Mirror 2: 0x997c726f (stale) CRC-32 checksum value for chunk [0, 0x3edf1): Mirror 1: 0xda49c58a Mirror 3: 0xda49c58a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3edf1: skipped 000511 1782481729.954679 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000513 1782481730.295563 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000517 1782481730.461480 mapread 0x0205a8 thru 0x0253f4 (0x4e4d bytes) 000520 1782481730.488597 trunc from 0x03edf1 to 0x01a63f (0x247b1 bytes) 000522 1782481730.582314 MIRROR_EXTEND to 2 mirrors 000523 1782481731.255466 trunc from 0x01a640 to 0x00a110 (0x1052f bytes) 000524 1782481731.316111 trunc from 0x00a111 to 0x00e3c4 (0x42b4 bytes) 000526 1782481731.335299 trunc from 0x00e3c5 to 0x00cef6 (0x14ce bytes) 000527 1782481731.351269 mapwrite 0x03a436 thru 0x03ffff (0x5bca bytes) 000528 1782481731.446307 read 0x037450 thru 0x03abb3 (0x3764 bytes) 000529 1782481731.447338 write 0x00724a thru 0x00ac45 (0x39fc bytes) 000530 1782481731.461127 mapread 0x01fc66 thru 0x02b728 (0xbac3 bytes) 000531 1782481731.480280 trunc from 0x040000 to 0x01b8db (0x24724 bytes) 000532 1782481731.501264 read 0x0151ec thru 0x01b8db (0x66f0 bytes) 000533 1782481731.514549 trunc from 0x01b8dc to 0x00e81b (0xd0c0 bytes) 000534 1782481731.542538 MIRROR_RESYNC 2 mirrors 000535 1782481732.078364 read 0x009059 thru 0x00e81b (0x57c3 bytes) 000536 1782481732.089942 mapread 0x005493 thru 0x00bc8b (0x67f9 bytes) 000537 1782481732.109193 trunc from 0x00e81c to 0x02ff58 (0x2173d bytes) 000538 1782481732.215793 write 0x02a795 thru 0x03127a (0x6ae6 bytes) 000540 1782481732.223614 write 0x0241d0 thru 0x027872 (0x36a3 bytes) 000541 1782481732.242315 mapread 0x00e3ef thru 0x00ffc1 (0x1bd3 bytes) 000544 1782481732.260167 write 0x02c064 thru 0x039e06 (0xdda3 bytes) 000546 1782481732.262650 mapread 0x034d9a thru 0x03924a (0x44b1 bytes) 000547 1782481732.263568 read 0x039137 thru 0x039e06 (0xcd0 bytes) 000549 1782481732.265266 read 0x025929 thru 0x028a01 (0x30d9 bytes) 000550 1782481732.279002 mapread 0x003dd8 thru 0x00848f (0x46b8 bytes) 000551 1782481732.280206 read 0x007f3c thru 0x00e8a3 (0x6968 bytes) 000553 1782481732.281696 mapread 0x00ea23 thru 0x01d4d7 (0xeab5 bytes) 000555 1782481732.283514 read 0x02649c thru 0x03390c (0xd471 bytes) 000556 1782481732.284265 mapwrite 0x01a299 thru 0x02160f (0x7377 bytes) 000557 1782481732.341330 read 0x00f8b6 thru 0x014f04 (0x564f bytes) 000558 1782481732.343781 mapwrite 0x033224 thru 0x0358cb (0x26a8 bytes) 000561 1782481732.397065 read 0x0263a6 thru 0x028943 (0x259e bytes) 000562 1782481732.399077 mapread 0x038ff3 thru 0x0392e6 (0x2f4 bytes) 000564 1782481732.402608 read 0x0230d0 thru 0x033017 (0xff48 bytes) 000565 1782481732.404099 trunc from 0x039e07 to 0x0032c1 (0x36b45 bytes) 000567 1782481732.436633 write 0x0083bd thru 0x00d264 (0x4ea8 bytes) 000568 1782481732.439142 write 0x00055d thru 0x005b8f (0x5633 bytes) 000569 1782481732.459243 trunc from 0x00d265 to 0x0004d4 (0xcd90 bytes) 000570 1782481732.492098 mapwrite 0x03271f thru 0x03597a (0x325c bytes) 000571 1782481732.606946 write 0x02dd4b thru 0x036475 (0x872b bytes) 000572 1782481732.609187 trunc from 0x036476 to 0x02a38d (0xc0e8 bytes) 000573 1782481732.640356 read 0x01f2e9 thru 0x025c8b (0x69a3 bytes) 000574 1782481732.647156 read 0x017e8b thru 0x01d204 (0x537a bytes) 000575 1782481732.649032 read 0x01817e thru 0x01e4a4 (0x6327 bytes) 000578 1782481732.666165 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a38e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a38e): Mirror 2: 0x623c5320 (stale) CRC-32 checksum value for chunk [0, 0x2a38e): Mirror 1: 0xc29b637d Mirror 3: 0xc29b637d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a38e: skipped 000580 1782481733.450014 trunc from 0x02a38e to 0x015e2f (0x1455e bytes) 000582 1782481733.646575 trunc from 0x015e30 to 0x002a65 (0x133ca bytes) 000583 1782481733.702395 mapread 0x001206 thru 0x0016d9 (0x4d4 bytes) 000584 1782481733.708436 mapwrite 0x03488d thru 0x03e204 (0x9978 bytes) 000587 1782481733.832529 read 0x00d2d1 thru 0x015fda (0x8d0a bytes) 000588 1782481733.848213 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e205): Mirror 3: 0x31e3ff79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e205): Mirror 3: 0x31e3ff79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e205: skipped 000591 1782481734.132590 trunc from 0x03e205 to 0x027fcc (0x16238 bytes) 000592 1782481734.156383 MIRROR_RESYNC 2 mirrors 000593 1782481734.777003 trunc from 0x027fcd to 0x00b529 (0x1caa3 bytes) 000594 1782481734.871295 write 0x02053f thru 0x025133 (0x4bf5 bytes) 000595 1782481734.874347 read 0x00de7c thru 0x01529d (0x7422 bytes) 000597 1782481734.887477 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) 000598 1782481735.052548 write 0x0334ac thru 0x03ffff (0xcb54 bytes) 000599 1782481735.054265 write 0x03941d thru 0x03a21e (0xe02 bytes) 000600 1782481735.056886 write 0x03f869 thru 0x03ffff (0x797 bytes) 000601 1782481735.057682 write 0x037191 thru 0x03750d (0x37d bytes) 000602 1782481735.058937 trunc from 0x040000 to 0x0286ae (0x17951 bytes) 000605 1782481735.110102 read 0x00ac1c thru 0x011629 (0x6a0e bytes) 000606 1782481735.127165 mapwrite 0x03b71f thru 0x03ffff (0x48e1 bytes) 000607 1782481735.226255 mapwrite 0x029d7d thru 0x03718b (0xd40f bytes) 000611 1782481735.297554 mapwrite 0x02ab42 thru 0x02cedb (0x239a bytes) 000614 1782481735.355945 trunc from 0x040000 to 0x016469 (0x29b96 bytes) 000615 1782481735.374436 mapread 0x00156e thru 0x009ea0 (0x8933 bytes) 000617 1782481735.384510 mapread 0x00337d thru 0x00b581 (0x8205 bytes) 000618 1782481735.386407 mapread 0x012cbf thru 0x016469 (0x37ab bytes) 000619 1782481735.394344 mapwrite 0x0247da thru 0x027b36 (0x335d bytes) 000620 1782481735.464594 mapwrite 0x035050 thru 0x03a91e (0x58cf bytes) 000621 1782481735.525673 read 0x0127c8 thru 0x021af9 (0xf332 bytes) 000623 1782481735.535180 mapwrite 0x028189 thru 0x02a7b3 (0x262b bytes) 000624 1782481735.567398 read 0x00db0b thru 0x01234c (0x4842 bytes) 000625 1782481735.568348 trunc from 0x03a91f to 0x0058c3 (0x3505b bytes) 000626 1782481735.590027 write 0x032eeb thru 0x03be27 (0x8f3d bytes) 000627 1782481735.595917 MIRROR_EXTEND to 2 mirrors 000628 1782481736.124898 mapwrite 0x03c4d2 thru 0x03ffff (0x3b2e bytes) 000629 1782481736.276580 mapwrite 0x004ebc thru 0x014699 (0xf7de bytes) 000630 1782481736.333912 mapwrite 0x01a390 thru 0x020cc0 (0x6931 bytes) 000633 1782481736.382130 mapwrite 0x00b672 thru 0x017363 (0xbcf2 bytes) 000634 1782481736.469241 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x16d8f56a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6813731d Mirror 3: 0x6813731d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1782481737.195248 trunc from 0x040000 to 0x026e73 (0x1918c bytes) 000636 1782481737.323098 MIRROR_RESYNC 3 mirrors 000638 1782481737.944758 write 0x03f16a thru 0x03ffff (0xe96 bytes) 000639 1782481737.996917 trunc from 0x040000 to 0x0294dd (0x16b22 bytes) 000640 1782481738.011534 write 0x01b191 thru 0x026772 (0xb5e2 bytes) 000642 1782481738.041943 write 0x004f66 thru 0x00d73e (0x87d9 bytes) 000643 1782481738.052212 read 0x0116cd thru 0x020d58 (0xf68c bytes) 000644 1782481738.063435 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x294de): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x294de): Mirror 2: 0x5ad834d3 (stale) Mirror 3: 0x5ad834d3 (stale) CRC-32 checksum value for chunk [0, 0x294de): Mirror 1: 0x58e96b0c Mirror 4: 0x58e96b0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x294de: skipped 000645 1782481738.771354 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x294de): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x294de): Mirror 2: 0x5ad834d3 (stale) Mirror 3: 0x5ad834d3 (stale) CRC-32 checksum value for chunk [0, 0x294de): Mirror 1: 0x58e96b0c Mirror 4: 0x58e96b0c Mirror 5: 0x58e96b0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x294de: skipped 000646 1782481739.496231 mapwrite 0x028aa9 thru 0x0374b6 (0xea0e bytes) 000648 1782481739.680926 read 0x034f99 thru 0x0374b6 (0x251e bytes) 000649 1782481739.681615 mapwrite 0x01c6e3 thru 0x0222f4 (0x5c12 bytes) 000652 1782481739.735983 read 0x007326 thru 0x01340b (0xc0e6 bytes) 000654 1782481739.747199 mapread 0x018452 thru 0x022ddd (0xa98c bytes) 000655 1782481739.753066 MIRROR_RESYNC 5 mirrors 000656 1782481740.521767 mapwrite 0x03a7a8 thru 0x03ffff (0x5858 bytes) 000658 1782481740.692336 write 0x01add2 thru 0x01ae16 (0x45 bytes) 000659 1782481740.697212 mapread 0x027d06 thru 0x0348a9 (0xcba4 bytes) 000660 1782481740.718770 mapwrite 0x028039 thru 0x02a1c8 (0x2190 bytes) 000661 1782481740.763015 read 0x0114e3 thru 0x0143c7 (0x2ee5 bytes) 000662 1782481740.768342 read 0x02b68c thru 0x0320d2 (0x6a47 bytes) 000664 1782481740.770227 mapwrite 0x02d33d thru 0x02d526 (0x1ea bytes) 000666 1782481740.808695 trunc from 0x040000 to 0x0155ac (0x2aa53 bytes) 000668 1782481740.832252 mapread 0x006409 thru 0x0141b4 (0xddac bytes) 000669 1782481740.842125 mapwrite 0x010f9c thru 0x01cef8 (0xbf5d bytes) 000670 1782481740.925225 mapread 0x008891 thru 0x00ed9b (0x650b bytes) 000671 1782481740.931141 MIRROR_RESYNC 5 mirrors 000674 1782481741.706403 trunc from 0x01cef9 to 0x006e06 (0x160f2 bytes) 000675 1782481741.761040 read 0x00655e thru 0x006e06 (0x8a9 bytes) 000676 1782481741.767395 read 0x006d79 thru 0x006e06 (0x8e bytes) 000678 1782481741.769423 write 0x01962d thru 0x023e99 (0xa86d bytes) 000679 1782481741.770628 write 0x00e497 thru 0x01c58c (0xe0f6 bytes) 000681 1782481741.784074 trunc from 0x023e9a to 0x03828c (0x143f3 bytes) 000682 1782481741.810409 write 0x01df2b thru 0x01e6ad (0x783 bytes) 000684 1782481741.813514 read 0x01443e thru 0x0181d6 (0x3d99 bytes) 000686 1782481741.815803 read 0x00f027 thru 0x019674 (0xa64e bytes) 000689 1782481741.818181 mapread 0x037a7a thru 0x03828c (0x813 bytes) 000692 1782481741.829246 write 0x013178 thru 0x018aa9 (0x5932 bytes) 000693 1782481741.830545 trunc from 0x03828d to 0x03cd06 (0x4a7a bytes) 000694 1782481741.848833 mapread 0x02e4b5 thru 0x03a287 (0xbdd3 bytes) 000696 1782481741.857921 mapread 0x00101d thru 0x008af9 (0x7add bytes) 000697 1782481741.859105 trunc from 0x03cd07 to 0x006688 (0x3667e bytes) 000699 1782481741.876849 mapwrite 0x00eec2 thru 0x01a72c (0xb86b bytes) 000702 1782481741.998007 write 0x03626e thru 0x03ffff (0x9d92 bytes) 000703 1782481742.000291 write 0x00774a thru 0x015143 (0xd9fa bytes) 000704 1782481742.010026 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9cf0070f (stale) Mirror 4: 0x9cf0070f (stale) Mirror 5: 0x9cf0070f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9cf0070f (stale) Mirror 4: 0x9cf0070f (stale) Mirror 5: 0x9cf0070f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000706 1782481742.499864 write 0x034f9d thru 0x03a775 (0x57d9 bytes) 000708 1782481742.520958 mapwrite 0x00a099 thru 0x016137 (0xc09f bytes) 000711 1782481742.599838 mapwrite 0x00c11a thru 0x00d0b8 (0xf9f bytes) 000712 1782481742.635976 mapread 0x032dde thru 0x0389f6 (0x5c19 bytes) 000714 1782481742.651085 mapread 0x03cf57 thru 0x03ffff (0x30a9 bytes) 000715 1782481742.658502 mapread 0x032947 thru 0x03a487 (0x7b41 bytes) 000716 1782481742.660091 mapwrite 0x0267ea thru 0x0268b2 (0xc9 bytes) 000717 1782481742.699146 mapread 0x00b309 thru 0x00dc25 (0x291d bytes) 000718 1782481742.701075 trunc from 0x040000 to 0x00e2dc (0x31d23 bytes) 000719 1782481742.714847 mapread 0x00e1cc thru 0x00e2dc (0x111 bytes) 000722 1782481742.718956 read 0x002527 thru 0x00ce0c (0xa8e6 bytes) 000723 1782481742.725071 trunc from 0x00e2dd to 0x02f7af (0x214d3 bytes) 000725 1782481742.761132 mapwrite 0x016217 thru 0x0237d8 (0xd5c2 bytes) 000726 1782481742.826107 read 0x008522 thru 0x00e513 (0x5ff2 bytes) 000727 1782481742.826866 write 0x03aa59 thru 0x03ffff (0x55a7 bytes) 000728 1782481742.828164 read 0x0240cb thru 0x02e4e9 (0xa41f bytes) 000729 1782481742.850769 mapread 0x010d54 thru 0x01271b (0x19c8 bytes) 000732 1782481742.857853 read 0x01fedf thru 0x022662 (0x2784 bytes) 000739 1782481742.870832 write 0x02840c thru 0x02db95 (0x578a bytes) 000740 1782481742.872252 write 0x020de7 thru 0x02198e (0xba8 bytes) 000744 1782481742.880219 read 0x0298a7 thru 0x02fe90 (0x65ea bytes) 000745 1782481742.881099 trunc from 0x040000 to 0x0166f4 (0x2990b bytes) 000748 1782481742.902962 mapwrite 0x035aed thru 0x03df4c (0x8460 bytes) 000749 1782481743.014828 write 0x03cf66 thru 0x03ffff (0x309a bytes) 000750 1782481743.017116 write 0x017be0 thru 0x0195e2 (0x1a03 bytes) 000751 1782481743.018967 trunc from 0x040000 to 0x033294 (0xcd6b bytes) 000752 1782481743.035161 write 0x024030 thru 0x031f67 (0xdf38 bytes) 000753 1782481743.037568 trunc from 0x033295 to 0x01416a (0x1f12a bytes) 000754 1782481743.065800 mapwrite 0x0061d1 thru 0x011451 (0xb281 bytes) 000755 1782481743.173197 trunc from 0x01416b to 0x00be17 (0x8353 bytes) 000756 1782481743.189944 read 0x0069d3 thru 0x009fa3 (0x35d1 bytes) 000757 1782481743.191125 mapread 0x00347b thru 0x00a6a0 (0x7226 bytes) 000758 1782481743.193156 read 0x007ceb thru 0x00be17 (0x412d bytes) 000762 1782481743.195936 trunc from 0x00be18 to 0x00e54d (0x2736 bytes) 000763 1782481743.208530 write 0x0296b1 thru 0x035fdc (0xc92c bytes) 000764 1782481743.211172 write 0x013d2b thru 0x01aeb9 (0x718f bytes) 000765 1782481743.219074 read 0x002ffb thru 0x0043fe (0x1404 bytes) 000767 1782481743.226564 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35fdd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fdd): Mirror 3: 0xfdbd19a9 (stale) Mirror 4: 0xfdbd19a9 (stale) Mirror 5: 0xfdbd19a9 (stale) CRC-32 checksum value for chunk [0, 0x35fdd): Mirror 1: 0xbbe6e509 Mirror 6: 0xbbe6e509 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fdd: skipped 000768 1782481743.998084 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35fdd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fdd): Mirror 3: 0xfdbd19a9 (stale) Mirror 4: 0xfdbd19a9 (stale) Mirror 5: 0xfdbd19a9 (stale) CRC-32 checksum value for chunk [0, 0x35fdd): Mirror 1: 0xbbe6e509 Mirror 6: 0xbbe6e509 Mirror 7: 0xbbe6e509 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fdd: skipped 000771 1782481744.799762 read 0x01fec4 thru 0x02a725 (0xa862 bytes) 000773 1782481744.815906 trunc from 0x035fdd to 0x000ff0 (0x34fec bytes) 000774 1782481744.947203 trunc from 0x000ff1 to 0x02bd9e (0x2adae bytes) 000775 1782481744.973622 read 0x022e43 thru 0x024afa (0x1cb8 bytes) 000777 1782481744.994062 mapwrite 0x01f9b5 thru 0x02e7bb (0xee07 bytes) 000779 1782481745.066015 write 0x03150c thru 0x0316e4 (0x1d9 bytes) 000781 1782481745.072061 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x316e5): Mirror 3: 0x87dc8922 (stale) Mirror 4: 0x87dc8922 (stale) Mirror 5: 0x87dc8922 (stale) Mirror 7: 0xfaaa4b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x316e5): Mirror 3: 0x87dc8922 (stale) Mirror 4: 0x87dc8922 (stale) Mirror 5: 0x87dc8922 (stale) Mirror 7: 0xfaaa4b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x316e5: skipped 000785 1782481745.518395 mapread 0x026ab5 thru 0x02cc92 (0x61de bytes) 000786 1782481745.547575 mapwrite 0x02d981 thru 0x02e6b2 (0xd32 bytes) 000787 1782481745.623632 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x316e5): Mirror 4: 0x87dc8922 (stale) Mirror 5: 0x87dc8922 (stale) Mirror 7: 0xfaaa4b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x316e5): Mirror 4: 0x87dc8922 (stale) Mirror 5: 0x87dc8922 (stale) Mirror 7: 0xfaaa4b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x316e5: skipped 000788 1782481745.890456 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x316e5): Mirror 5: 0x87dc8922 (stale) Mirror 7: 0xfaaa4b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x316e5): Mirror 5: 0x87dc8922 (stale) Mirror 7: 0xfaaa4b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x316e5: skipped 000789 1782481746.231497 mapwrite 0x0340f9 thru 0x03bd16 (0x7c1e bytes) 000791 1782481746.309199 read 0x034b0d thru 0x03bd16 (0x720a bytes) 000793 1782481746.311216 mapread 0x015a09 thru 0x01e8c2 (0x8eba bytes) 000794 1782481746.333098 write 0x02aa18 thru 0x031bfd (0x71e6 bytes) 000795 1782481746.344422 mapread 0x01b3e1 thru 0x0236de (0x82fe bytes) 000796 1782481746.346604 write 0x0115a2 thru 0x01b202 (0x9c61 bytes) 000797 1782481746.348526 trunc from 0x03bd17 to 0x01c43e (0x1f8d8 bytes) 000798 1782481746.367167 mapread 0x012f9a thru 0x01c43e (0x94a5 bytes) 000799 1782481746.372058 mapwrite 0x035b24 thru 0x03ffff (0xa4dc bytes) 000802 1782481746.466955 mapwrite 0x032bb1 thru 0x03f948 (0xcd98 bytes) 000803 1782481746.518184 mapwrite 0x0370a3 thru 0x03f45d (0x83bb bytes) 000805 1782481746.579704 mapwrite 0x03a64a thru 0x03c352 (0x1d09 bytes) 000807 1782481746.632531 write 0x00ff95 thru 0x01edfd (0xee69 bytes) 000814 1782481746.643444 mapread 0x0235b1 thru 0x027cba (0x470a bytes) 000815 1782481746.655830 MIRROR_RESYNC 3 mirrors 000817 1782481747.653421 mapread 0x0344fe thru 0x03909e (0x4ba1 bytes) 000819 1782481747.670297 write 0x003546 thru 0x00d3a1 (0x9e5c bytes) 000820 1782481747.731121 mapread 0x01718e thru 0x025509 (0xe37c bytes) 000821 1782481747.754424 mapwrite 0x0370fc thru 0x037bc6 (0xacb bytes) 000823 1782481747.822474 trunc from 0x040000 to 0x033f6a (0xc095 bytes) 000824 1782481747.853925 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33f6b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f6b): Mirror 5: 0x3d516528 (stale) Mirror 7: 0x3d516528 (stale) CRC-32 checksum value for chunk [0, 0x33f6b): Mirror 1: 0xa410e9b6 Mirror 8: 0xa410e9b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f6b: skipped 000825 1782481748.658251 write 0x03d1ce thru 0x03d7d0 (0x603 bytes) 000828 1782481748.727145 read 0x031a48 thru 0x035cb5 (0x426e bytes) 000831 1782481748.735033 write 0x026806 thru 0x026dbd (0x5b8 bytes) 000834 1782481748.745859 MIRROR_RESYNC 4 mirrors 000835 1782481749.369478 MIRROR_EXTEND to 5 mirrors 000836 1782481750.014470 read 0x0234a1 thru 0x0234d4 (0x34 bytes) 000838 1782481750.033331 MIRROR_EXTEND to 6 mirrors 000839 1782481750.613877 MIRROR_RESYNC 6 mirrors 000840 1782481750.954920 write 0x01770e thru 0x023009 (0xb8fc bytes) 000842 1782481751.027787 MIRROR_RESYNC 6 mirrors 000843 1782481751.894432 read 0x022074 thru 0x027696 (0x5623 bytes) 000844 1782481751.905421 mapwrite 0x031d43 thru 0x03bced (0x9fab bytes) 000849 1782481752.054007 read 0x0328e0 thru 0x0368f0 (0x4011 bytes) 000850 1782481752.054878 read 0x0167e4 thru 0x01b9e9 (0x5206 bytes) 000851 1782481752.063477 mapread 0x034ab7 thru 0x03b01a (0x6564 bytes) 000852 1782481752.069114 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d7d1): Mirror 5: 0x59bb4316 (stale) Mirror 7: 0x59bb4316 (stale) Mirror 8: 0x59bb4316 (stale) Mirror 9: 0x59bb4316 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7d1): Mirror 5: 0x59bb4316 (stale) Mirror 7: 0x59bb4316 (stale) Mirror 8: 0x59bb4316 (stale) Mirror 9: 0x59bb4316 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7d1: skipped 000853 1782481752.380386 mapread 0x0251a7 thru 0x03469b (0xf4f5 bytes) 000854 1782481752.406996 write 0x00b6fb thru 0x01b14f (0xfa55 bytes) 000858 1782481752.420042 read 0x03200c thru 0x036a7b (0x4a70 bytes) 000859 1782481752.420785 read 0x01e16b thru 0x020eef (0x2d85 bytes) 000861 1782481752.433522 read 0x01c3c9 thru 0x025e1a (0x9a52 bytes) 000862 1782481752.434368 write 0x026d12 thru 0x02ac94 (0x3f83 bytes) 000864 1782481752.436860 read 0x01ff40 thru 0x02ace6 (0xada7 bytes) 000865 1782481752.441627 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d7d1): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7d1): Mirror 5: 0x59bb4316 (stale) Mirror 7: 0x59bb4316 (stale) Mirror 8: 0x59bb4316 (stale) Mirror 9: 0x59bb4316 (stale) CRC-32 checksum value for chunk [0, 0x3d7d1): Mirror 1: 0x32e76d54 Mirror 10: 0x32e76d54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7d1: skipped 000867 1782481753.252892 mapwrite 0x01e54a thru 0x0200a9 (0x1b60 bytes) 000868 1782481753.386078 mapread 0x01a6dd thru 0x02662f (0xbf53 bytes) 000869 1782481753.387826 mapwrite 0x01a89f thru 0x028345 (0xdaa7 bytes) 000870 1782481753.431957 write 0x004ede thru 0x0107e9 (0xb90c bytes) 000871 1782481753.450058 trunc from 0x03d7d1 to 0x01fab1 (0x1dd1f bytes) 000872 1782481753.525372 write 0x02d499 thru 0x038fd3 (0xbb3b bytes) 000873 1782481753.532669 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38fd4): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fd4): Mirror 5: 0x441f68d0 (stale) Mirror 7: 0x441f68d0 (stale) Mirror 8: 0x441f68d0 (stale) Mirror 9: 0x441f68d0 (stale) Mirror 10: 0xa73dd632 (stale) CRC-32 checksum value for chunk [0, 0x38fd4): Mirror 1: 0x1305e3be Mirror 11: 0x1305e3be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fd4: skipped 000874 1782481754.518647 mapread 0x02fedc thru 0x0306a6 (0x7cb bytes) 000875 1782481754.561344 mapwrite 0x020793 thru 0x029824 (0x9092 bytes) 000876 1782481754.801849 trunc from 0x038fd4 to 0x013e74 (0x2515f bytes) 000878 1782481754.847925 write 0x02af70 thru 0x038f40 (0xdfd1 bytes) 000879 1782481754.858691 MIRROR_RESYNC 7 mirrors 000880 1782481755.983148 trunc from 0x038f41 to 0x00d79f (0x2b7a1 bytes) 000882 1782481756.068071 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xd7a0): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd7a0): Mirror 5: 0xd4219e50 (stale) Mirror 7: 0xd4219e50 (stale) Mirror 8: 0xd4219e50 (stale) Mirror 9: 0xd4219e50 (stale) Mirror 10: 0xd4219e50 (stale) Mirror 11: 0xd4219e50 (stale) CRC-32 checksum value for chunk [0, 0xd7a0): Mirror 1: 0xd4219e50 Mirror 12: 0xd4219e50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd7a0: skipped 000883 1782481756.858369 read 0x004d8e thru 0x00d79f (0x8a12 bytes) 000884 1782481756.876188 trunc from 0x00d7a0 to 0x000049 (0xd756 bytes) 000886 1782481756.982875 read 0x00001e thru 0x000049 (0x2c bytes) 000887 1782481756.987419 trunc from 0x00004a to 0x0181fd (0x181b4 bytes) 000888 1782481757.005139 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x181fe): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x181fe): Mirror 5: 0x37ac436 (stale) Mirror 7: 0x37ac436 (stale) Mirror 8: 0x37ac436 (stale) Mirror 9: 0x37ac436 (stale) Mirror 10: 0x37ac436 (stale) Mirror 11: 0x37ac436 (stale) Mirror 12: 0x4efe3fdf (stale) CRC-32 checksum value for chunk [0, 0x181fe): Mirror 1: 0x44ef3a15 Mirror 13: 0x44ef3a15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x181fe: skipped 000889 1782481757.812171 write 0x008a4d thru 0x0169df (0xdf93 bytes) 000891 1782481757.910204 mapread 0x013869 thru 0x0179d7 (0x416f bytes) 000893 1782481757.915929 read 0x00e149 thru 0x0124e3 (0x439b bytes) 000894 1782481757.916972 write 0x006e8a thru 0x01640e (0xf585 bytes) 000895 1782481757.923081 trunc from 0x0181fe to 0x01f17f (0x6f82 bytes) 000896 1782481757.954007 write 0x037b7c thru 0x03ffff (0x8484 bytes) 000898 1782481757.960307 MIRROR_RESYNC 9 mirrors 000902 1782481759.056604 trunc from 0x040000 to 0x0107e0 (0x2f81f bytes) 000903 1782481759.114691 mapwrite 0x0397eb thru 0x03ffff (0x6815 bytes) 000907 1782481759.236529 mapread 0x019952 thru 0x01f9c8 (0x6077 bytes) 000908 1782481759.249205 mapread 0x009d04 thru 0x00fc90 (0x5f8d bytes) 000909 1782481759.261559 mapread 0x000cc3 thru 0x000e3f (0x17d bytes) 000910 1782481759.286080 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x98066309 (stale) Mirror 7: 0x98066309 (stale) Mirror 8: 0x98066309 (stale) Mirror 9: 0x98066309 (stale) Mirror 10: 0x98066309 (stale) Mirror 11: 0x98066309 (stale) Mirror 12: 0x98066309 (stale) Mirror 13: 0x98066309 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfca29fc4 Mirror 14: 0xfca29fc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000912 1782481760.023912 mapread 0x034d21 thru 0x03ffff (0xb2df bytes) 000914 1782481760.058964 write 0x03fbe5 thru 0x03ffff (0x41b bytes) 000916 1782481760.116050 mapwrite 0x00ca7e thru 0x012f52 (0x64d5 bytes) 000917 1782481760.179134 trunc from 0x040000 to 0x0301f8 (0xfe07 bytes) 000919 1782481760.226734 mapread 0x000add thru 0x00ece4 (0xe208 bytes) 000922 1782481760.237933 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x301f9): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301f9): Mirror 5: 0x4a492cb3 (stale) Mirror 7: 0x4a492cb3 (stale) Mirror 8: 0x4a492cb3 (stale) Mirror 9: 0x4a492cb3 (stale) Mirror 10: 0x4a492cb3 (stale) Mirror 11: 0x4a492cb3 (stale) Mirror 12: 0x4a492cb3 (stale) Mirror 13: 0x4a492cb3 (stale) Mirror 14: 0x2a3af435 (stale) CRC-32 checksum value for chunk [0, 0x301f9): Mirror 1: 0x96a08313 Mirror 15: 0x96a08313 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301f9: skipped 000923 1782481761.095145 read 0x01f8cb thru 0x02de97 (0xe5cd bytes) 000925 1782481761.121228 mapread 0x02fae1 thru 0x0301f8 (0x718 bytes) 000928 1782481761.169545 mapwrite 0x024bc5 thru 0x02566f (0xaab bytes) 000930 1782481761.335199 mapwrite 0x02d9e1 thru 0x0393e2 (0xba02 bytes) 000932 1782481761.400627 read 0x01d2ba thru 0x02a903 (0xd64a bytes) 000933 1782481761.404350 MIRROR_RESYNC 11 mirrors 000934 1782481762.646563 mapwrite 0x020df8 thru 0x0280cc (0x72d5 bytes) 000937 1782481762.796276 read 0x0165c1 thru 0x020960 (0xa3a0 bytes) 000938 1782481762.810921 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x393e3): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x393e3): Mirror 5: 0xd86d0fae (stale) Mirror 7: 0xd86d0fae (stale) Mirror 8: 0xd86d0fae (stale) Mirror 9: 0xd86d0fae (stale) Mirror 10: 0xd86d0fae (stale) Mirror 11: 0xd86d0fae (stale) Mirror 12: 0xd86d0fae (stale) Mirror 13: 0xd86d0fae (stale) Mirror 14: 0xd86d0fae (stale) Mirror 15: 0xd86d0fae (stale) CRC-32 checksum value for chunk [0, 0x393e3): Mirror 1: 0x1838e6e2 Mirror 16: 0x1838e6e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x393e3: skipped 000941 1782481763.831437 read 0x0342a6 thru 0x03866d (0x43c8 bytes) 000942 1782481763.864681 read 0x0086d6 thru 0x00c064 (0x398f bytes) 000945 1782481763.914483 write 0x02f2be thru 0x0378e3 (0x8626 bytes) 000946 1782481764.025244 write 0x038254 thru 0x03951d (0x12ca bytes) 000948 1782481764.045235 MIRROR_RESYNC 12 mirrors 000949 1782481765.412665 mapwrite 0x03788e thru 0x03ffff (0x8772 bytes) 000950 1782481765.523199 MIRROR_RESYNC 12 mirrors 000953 1782481766.981445 trunc from 0x040000 to 0x028e0c (0x171f3 bytes) 000955 1782481767.084571 MIRROR_RESYNC 12 mirrors 000957 1782481768.384212 mapread 0x016012 thru 0x016303 (0x2f2 bytes) 000959 1782481768.389517 mapwrite 0x003b3f thru 0x007a80 (0x3f42 bytes) 000960 1782481768.539827 MIRROR_RESYNC 12 mirrors 000963 1782481770.009462 trunc from 0x028e0d to 0x039a2c (0x10c20 bytes) 000967 1782481770.073672 read 0x00e467 thru 0x01166e (0x3208 bytes) 000969 1782481770.080828 trunc from 0x039a2d to 0x03027d (0x97af bytes) 000971 1782481770.099980 trunc from 0x03027e to 0x027017 (0x9266 bytes) 000972 1782481770.119341 trunc from 0x027018 to 0x030622 (0x960b bytes) 000974 1782481770.134150 write 0x02325e thru 0x030271 (0xd014 bytes) 000976 1782481770.145019 mapwrite 0x00c42b thru 0x010ace (0x46a4 bytes) 000977 1782481770.206116 read 0x00019a thru 0x00a202 (0xa069 bytes) 000978 1782481770.215692 trunc from 0x030623 to 0x039ac6 (0x94a4 bytes) 000981 1782481770.234564 read 0x001bb7 thru 0x00892a (0x6d74 bytes) 000982 1782481770.235474 mapread 0x0053cb thru 0x012f60 (0xdb96 bytes) 000983 1782481770.237436 read 0x0193b9 thru 0x0225d9 (0x9221 bytes) 000984 1782481770.238637 read 0x027549 thru 0x029142 (0x1bfa bytes) 000985 1782481770.240540 mapwrite 0x03d016 thru 0x03ffff (0x2fea bytes) 000988 1782481770.293807 trunc from 0x040000 to 0x01d348 (0x22cb7 bytes) 000989 1782481770.313272 write 0x0219fc thru 0x027f25 (0x652a bytes) 000991 1782481770.315255 mapwrite 0x02e1db thru 0x03b85e (0xd684 bytes) 000992 1782481770.448291 mapread 0x025f0a thru 0x02a0e0 (0x41d7 bytes) 000993 1782481770.464339 mapread 0x010db3 thru 0x01e068 (0xd2b6 bytes) 000994 1782481770.470015 mapread 0x01da7e thru 0x01faea (0x206d bytes) 000996 1782481770.476264 mapwrite 0x030d44 thru 0x037913 (0x6bd0 bytes) 001000 1782481770.527104 read 0x00d746 thru 0x01c237 (0xeaf2 bytes) All operations completed A-OK! PASS 70a (67s) == sanity-flr test 71: check mirror extend parent fid ==== 09:49:34 (1782481774) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 09:49:37 (1782481777) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188083 s, 11.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146334 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x330:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 09:49:43 (1782481783) Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre2 Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1296000 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6495104 .. failed Extending file size to 3333728 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8691712 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5171840 .. failed Extending file size to 7482368 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8645792 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8990016 .. Extending file size to 8646592 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8974016 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9179872 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57011 57013 57014 57015 57019 done 192.168.201.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg103-client.virtnet /mnt/lustre2 (opts:) 192.168.201.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg103-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:51:00 (1782481860) Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre2 Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6745536 .. Extending file size to 4693216 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5372672 .. done Extending file size to 6590880 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7340128 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8777408 .. Extending file size to 8864416 .. Extending file size to 8899040 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8874528 .. Extending file size to 8986560 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9382912 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58616 58618 58619 58621 58622 done 192.168.201.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg103-client.virtnet /mnt/lustre2 (opts:) 192.168.201.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg103-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:52:20 (1782481940) Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0925354 s, 22.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235839 s, 8.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x55dbfb5b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg103-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:52:35 (1782481955) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0907845 s, 23.1 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 09:52:41 (1782481961) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x225:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x361:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x361:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x362:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:52:46 (1782481966) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142797 s, 14.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x363:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x346:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:52:52 (1782481972) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133753 s, 15.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x365:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x365:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:52:59 (1782481979) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206867 s, 10.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:53:29 (1782482009) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg103-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650482 s, 16.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x367:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x367:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x368:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:54:21 (1782482061) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg103-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x369:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131651 s, 15.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x369:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x369:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:55:12 (1782482112) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg103-server mds-ost sync done. Creating new pool test_204f oleg103-server: Pool lustre.test_204f created Adding targets to pool oleg103-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg103-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18317 s, 11.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg103-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg103-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg103-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (67s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:56:19 (1782482179) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:56:25 (1782482185) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:56:30 (1782482190) oleg103-server: Pool lustre.test_206 created Adding targets to pool oleg103-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg103-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg103-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg103-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg103-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:56:52 (1782482212) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280979 s, 37.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527247 s, 19.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x31e7e600 Mirror 3: 0x31e7e600 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:56:59 (1782482219) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:57:02 (1782482222) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:57:04 (1782482224) Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.111317 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xfb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x19d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x19d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x19d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23d0000 Read this (4 bytes): BBBB 192.168.201.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg103-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:57:32 (1782482252) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0208 s, 394 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:57:38 (1782482258) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397511 s, 26.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24a lmm_fid: [0x200000404:0x24a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24a lmm_fid: [0x200000404:0x24a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x359:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834506 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 09:57:39.000000000 -0400 Modify: 2026-06-26 09:57:39.000000000 -0400 Change: 2026-06-26 09:57:40.000000000 -0400 Birth: 2026-06-26 09:57:40.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:57:44 (1782482264) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:57:46 (1782482266) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.433201 s, 94.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0301724 s, 136 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 26 09:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x24c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 26 09:57 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:57:53 (1782482273) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:57:58 (1782482278) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1539 sec =========== 09:58:05 (1782482285) === sanity-flr: start cleanup 09:58:06 (1782482286) === === sanity-flr: finish cleanup 09:58:11 (1782482291) ===