-----============= acceptance-small: sanity-flr ============----- Tue Jun 23 08:26:08 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg310-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:26:28 (1782217588) === oleg310-client.virtnet: executing check_config_client /mnt/lustre oleg310-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg310-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d2588864800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d2588864800.idle_timeout=debug disable quota as required oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:27:00 (1782217620) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7595] == sanity-flr test 0a: lfs mirror create with -N option == 08:27:03 (1782217623) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:27:20 (1782217640) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:27:23 (1782217643) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:27:27 (1782217647) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:27:52 (1782217672) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:27:55 (1782217675) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:27:59 (1782217679) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d2588864800.stats=clear osc.lustre-OST0001-osc-ffff8d2588864800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:28:08 (1782217688) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:28:19 (1782217699) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:28:27 (1782217707) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:28:31 (1782217711) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:28:38 (1782217718) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 08:28:46 (1782217726) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:28:52 (1782217732) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:28:55 (1782217735) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:29:02 (1782217742) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:29:12 (1782217752) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:29:18 (1782217758) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.5445 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150777 s, 7.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:29:28 (1782217768) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.39908 s, 39.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:29:36 (1782217776) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:29:45 (1782217785) Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg310-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:30:57 (1782217857) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.333475 s, 22.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:31:33 (1782217893) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg310-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (69s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:32:42 (1782217962) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.47701 s, 11.7 MB/s -rw-r--r-- 1 root root 52428800 Jun 23 08:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.203s user 0m0.001s sys 0m0.177s Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d2588864800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.017s user 0m0.000s sys 0m0.197s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.696s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg310-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg310-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d2588864800.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:34:17 (1782218057) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:34:21 (1782218061) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:34:24 (1782218064) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:34:27 (1782218067) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 08:34:35 (1782218075) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.637376 s, 19.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:34:48 (1782218088) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375428 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502171 s, 20.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:34:56 (1782218096) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:35:05 (1782218105) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00177465 s, 577 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00255449 s, 401 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00183888 s, 557 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00213666 s, 479 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 08:35:16 (1782218116) create test files with size 15741445 .. 8bb089cfd10e64157233347bee7566db - 00477f573d1038873ca68e7e8622fb1f - bc03c5129851711aca93a4df16173073 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg310-client.virtnet /mnt/lustre (opts:) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 08:35:44 (1782218144) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:36:00 (1782218160) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:36:08 (1782218168) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192699 s, 5.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545788 s, 19.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 08:36:19 (1782218179) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177862 s, 11.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.363552 s, 11.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292579 s, 7.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 08:36:38 (1782218198) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 08:36:43 (1782218203) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:36:47 (1782218207) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383699 s, 27.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:36:56 (1782218216) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:37:00 (1782218220) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:37:11 (1782218231) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.336971 s, 31.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg310-client.virtnet /mnt/lustre (opts:) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:37:24 (1782218244) llite.lustre-ffff8d25880bd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.254942 s, 41.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg310-client.virtnet /mnt/lustre (opts:) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d2582e86800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:37:38 (1782218258) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:38:20 (1782218300) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:39:04 (1782218344) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:39:08 (1782218348) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241327 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14216 s, 22.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x2e59d144 (nosync,stale) Mirror 2: 0x2e59d144 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 25e9bd1ae3aa5c1b1f1a8e5f2f274c65 - mirror 1 checksum: 2e773da6d00375231c4f1091bd997b24 - mirror 2 checksum: 25e9bd1ae3aa5c1b1f1a8e5f2f274c65 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T08:39:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T08:39:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 25e9bd1ae3aa5c1b1f1a8e5f2f274c65 - mirror 1 checksum: 2e773da6d00375231c4f1091bd997b24 - mirror 2 checksum: 2e773da6d00375231c4f1091bd997b24 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T08:39:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T08:39:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:39:20 (1782218360) Failing mds1 on oleg310-server Stopping /mnt/lustre-mds1 (opts:) on oleg310-server 08:39:27 (1782218367) shut down facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server Failover mds1 to oleg310-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 08:39:50 (1782218390) targets are mounted 08:39:50 (1782218390) facet_failover done oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262767 s, 39.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:40:06 (1782218406) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:40:10 (1782218410) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:40:13 (1782218413) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0102025 s, 1.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:40:22 (1782218422) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:40:27 (1782218427) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:40:31 (1782218431) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:40:58 (1782218458) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:41:26 (1782218486) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:41:51 (1782218511) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:42:16 (1782218536) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 08:42:26 (1782218546) mirror create pid 53743 mirror split pid 53745 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53743 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53745 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 08:43:34 (1782218614) Seed set to 8616 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1782218616.895479 write 0x021024 thru 0x02c1b5 (0xb192 bytes) 000002 1782218616.923809 write 0x03c819 thru 0x03ffff (0x37e7 bytes) 000003 1782218616.925140 mapwrite 0x03528f thru 0x036732 (0x14a4 bytes) 000004 1782218617.053693 mapwrite 0x00b6ba thru 0x0124a7 (0x6dee bytes) 000005 1782218617.122692 mapwrite 0x007f36 thru 0x01786d (0xf938 bytes) 000006 1782218617.190742 mapwrite 0x02d674 thru 0x035a9c (0x8429 bytes) 000008 1782218617.244822 write 0x02c4c4 thru 0x035b49 (0x9686 bytes) 000010 1782218617.250945 mapwrite 0x01a2fb thru 0x025721 (0xb427 bytes) 000014 1782218617.306975 mapwrite 0x039d70 thru 0x039db7 (0x48 bytes) truncating to largest ever: 0x1c2d0 000020 1782218617.371176 trunc from 0x040000 to 0x01c2cf (0x23d30 bytes) 000021 1782218617.394733 read 0x019618 thru 0x01c2cf (0x2cb8 bytes) truncating to largest ever: 0x3174b 000024 1782218617.402813 trunc from 0x01c2d0 to 0x03174a (0x1547b bytes) 000027 1782218617.444913 mapwrite 0x0033a4 thru 0x004200 (0xe5d bytes) 000028 1782218617.530784 read 0x023bcb thru 0x03174a (0xdb80 bytes) 000030 1782218617.554742 mapread 0x01218c thru 0x0181e0 (0x6055 bytes) 000031 1782218617.556812 mapwrite 0x013886 thru 0x01b234 (0x79af bytes) 000032 1782218617.648041 mapread 0x00a9ae thru 0x00b7dd (0xe30 bytes) 000033 1782218617.653196 mapread 0x018803 thru 0x01d2eb (0x4ae9 bytes) 000035 1782218617.663131 mapread 0x007fe9 thru 0x009e78 (0x1e90 bytes) 000039 1782218617.688153 MIRROR_EXTEND to 2 mirrors 000040 1782218618.649145 trunc from 0x03174b to 0x01fafd (0x11c4d bytes) 000042 1782218618.786877 mapread 0x00de0e thru 0x0159bf (0x7bb2 bytes) 000044 1782218618.852375 mapwrite 0x00361d thru 0x008f06 (0x58ea bytes) 000045 1782218618.970381 trunc from 0x01fafe to 0x00fa56 (0x100a7 bytes) 000047 1782218619.017827 trunc from 0x00fa57 to 0x02060b (0x10bb5 bytes) 000048 1782218619.037264 mapread 0x00702a thru 0x00e5b3 (0x758a bytes) 000049 1782218619.048752 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) 000050 1782218619.274431 mapread 0x019497 thru 0x02060b (0x7175 bytes) 000051 1782218619.308669 MIRROR_EXTEND to 2 mirrors 000053 1782218620.272343 mapwrite 0x0009fb thru 0x009e0c (0x9412 bytes) 000055 1782218620.543812 write 0x03f8ea thru 0x03ffff (0x716 bytes) 000056 1782218620.551643 mapread 0x0062f6 thru 0x013cdc (0xd9e7 bytes) 000057 1782218620.570769 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4626ec87 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x81337ac6 Mirror 3: 0x81337ac6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000062 1782218621.714043 write 0x00d947 thru 0x01b3a2 (0xda5c bytes) 000063 1782218621.797840 write 0x002a16 thru 0x007d0d (0x52f8 bytes) 000064 1782218621.813227 mapwrite 0x000ee8 thru 0x00f970 (0xea89 bytes) 000065 1782218621.968299 MIRROR_RESYNC 3 mirrors 000066 1782218623.082689 MIRROR_EXTEND to 4 mirrors 000067 1782218623.996375 mapread 0x014fc0 thru 0x015093 (0xd4 bytes) 000068 1782218624.010686 MIRROR_EXTEND to 5 mirrors 000069 1782218625.164463 mapread 0x03ad16 thru 0x03c58b (0x1876 bytes) 000070 1782218625.197728 read 0x01b1b9 thru 0x01f0e2 (0x3f2a bytes) 000071 1782218625.219633 mapread 0x0158da thru 0x021e8c (0xc5b3 bytes) 000072 1782218625.266295 mapread 0x0391ed thru 0x03c2f8 (0x310c bytes) 000073 1782218625.269214 read 0x005239 thru 0x008fd2 (0x3d9a bytes) 000074 1782218625.286767 mapread 0x025dec thru 0x03108d (0xb2a2 bytes) 000076 1782218625.288521 mapwrite 0x022dda thru 0x032848 (0xfa6f bytes) 000079 1782218625.545696 write 0x01de0a thru 0x020e95 (0x308c bytes) 000080 1782218625.557399 trunc from 0x040000 to 0x01a0ec (0x25f13 bytes) 000082 1782218625.590339 write 0x015485 thru 0x02507e (0xfbfa bytes) 000083 1782218625.602436 mapwrite 0x00fc94 thru 0x01f1c1 (0xf52e bytes) 000085 1782218625.770872 mapwrite 0x000e48 thru 0x001e30 (0xfe9 bytes) 000086 1782218625.811590 write 0x01b78d thru 0x02841f (0xcc93 bytes) 000087 1782218625.813234 mapwrite 0x018487 thru 0x020b1b (0x8695 bytes) 000088 1782218625.890173 trunc from 0x028420 to 0x022ed4 (0x554b bytes) 000090 1782218625.941260 write 0x03fac7 thru 0x03ffff (0x539 bytes) 000092 1782218625.944903 mapread 0x01493c thru 0x020950 (0xc015 bytes) 000093 1782218625.946771 write 0x0220f5 thru 0x02428d (0x2199 bytes) 000094 1782218625.953588 mapwrite 0x0004f3 thru 0x002b6c (0x267a bytes) 000095 1782218626.021364 trunc from 0x040000 to 0x00127b (0x3ed84 bytes) 000096 1782218626.056881 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x127c): Mirror 2: 0x5beac1f8 (stale) Mirror 3: 0x5beac1f8 (stale) Mirror 4: 0x5beac1f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x127c): Mirror 2: 0x5beac1f8 (stale) Mirror 3: 0x5beac1f8 (stale) Mirror 4: 0x5beac1f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x127c: skipped 000097 1782218626.414281 read 0x000e1b thru 0x00127b (0x461 bytes) 000099 1782218626.428224 write 0x0111cc thru 0x01ed21 (0xdb56 bytes) 000102 1782218626.431840 mapwrite 0x00a29d thru 0x00b304 (0x1068 bytes) 000104 1782218626.527414 read 0x0023e6 thru 0x00b44b (0x9066 bytes) 000107 1782218626.552270 trunc from 0x01ed22 to 0x018f90 (0x5d91 bytes) 000108 1782218626.577277 trunc from 0x018f91 to 0x023afd (0xab6d bytes) 000109 1782218626.609447 read 0x0047e9 thru 0x00a617 (0x5e2f bytes) 000110 1782218626.616473 write 0x025708 thru 0x03552d (0xfe26 bytes) 000112 1782218626.623456 trunc from 0x03552e to 0x023955 (0x11bd8 bytes) 000114 1782218626.669716 mapwrite 0x00b775 thru 0x01ad59 (0xf5e5 bytes) 000115 1782218626.871679 trunc from 0x023956 to 0x02e7b5 (0xae60 bytes) 000117 1782218626.913381 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e7b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7b6): Mirror 2: 0xf575b0ea (stale) Mirror 3: 0xf575b0ea (stale) Mirror 4: 0xf575b0ea (stale) CRC-32 checksum value for chunk [0, 0x2e7b6): Mirror 1: 0x1d835847 Mirror 5: 0x1d835847 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7b6: skipped 000120 1782218628.190099 read 0x010ec6 thru 0x01785d (0x6998 bytes) 000121 1782218628.205834 MIRROR_RESYNC 5 mirrors 000122 1782218629.439874 write 0x01ba02 thru 0x0227c1 (0x6dc0 bytes) 000125 1782218629.602346 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e7b6): Mirror 2: 0x1d835847 (stale) Mirror 3: 0x1d835847 (stale) Mirror 5: 0x1d835847 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7b6): Mirror 2: 0x1d835847 (stale) Mirror 3: 0x1d835847 (stale) Mirror 5: 0x1d835847 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7b6: skipped 000129 1782218630.189851 write 0x024127 thru 0x031409 (0xd2e3 bytes) 000133 1782218630.216820 read 0x011ed5 thru 0x01ccb6 (0xade2 bytes) 000134 1782218630.256890 write 0x032599 thru 0x038271 (0x5cd9 bytes) 000135 1782218630.263578 mapwrite 0x03f340 thru 0x03ffff (0xcc0 bytes) 000136 1782218630.448165 read 0x013fec thru 0x02257d (0xe592 bytes) 000137 1782218630.473937 mapwrite 0x004444 thru 0x010567 (0xc124 bytes) 000138 1782218630.543300 trunc from 0x040000 to 0x0123c9 (0x2dc36 bytes) 000139 1782218630.572263 mapread 0x0045b7 thru 0x011400 (0xce4a bytes) 000141 1782218630.575868 write 0x00171b thru 0x004a8e (0x3374 bytes) 000142 1782218630.582812 write 0x02143e thru 0x02a395 (0x8f58 bytes) 000144 1782218630.593254 mapread 0x02a38c thru 0x02a395 (0xa bytes) 000145 1782218630.594614 mapwrite 0x019654 thru 0x0206da (0x7087 bytes) 000146 1782218630.818358 trunc from 0x02a396 to 0x028f6c (0x1429 bytes) 000147 1782218630.850471 mapwrite 0x039ced thru 0x03e946 (0x4c5a bytes) 000148 1782218631.154854 write 0x0070ae thru 0x00b9bb (0x490e bytes) 000149 1782218631.165231 mapread 0x021734 thru 0x028962 (0x722f bytes) 000150 1782218631.173557 read 0x0101eb thru 0x017576 (0x738c bytes) 000151 1782218631.218800 MIRROR_RESYNC 4 mirrors 000152 1782218632.942770 mapwrite 0x035b7b thru 0x035f71 (0x3f7 bytes) 000153 1782218633.278560 mapread 0x0266ac thru 0x030890 (0xa1e5 bytes) 000154 1782218633.324466 mapread 0x011a3e thru 0x017a4f (0x6012 bytes) 000157 1782218633.359970 trunc from 0x03e947 to 0x01bdda (0x22b6c bytes) 000158 1782218633.410572 write 0x017435 thru 0x018ef9 (0x1ac5 bytes) 000159 1782218633.423694 trunc from 0x01bddb to 0x02d5f5 (0x1181b bytes) 000160 1782218633.473959 MIRROR_RESYNC 4 mirrors 000161 1782218634.570456 mapwrite 0x026983 thru 0x02cc0b (0x6289 bytes) 000162 1782218634.769626 MIRROR_RESYNC 4 mirrors 000163 1782218635.793396 MIRROR_EXTEND to 5 mirrors 000164 1782218636.777464 read 0x009580 thru 0x01211d (0x8b9e bytes) truncating to largest ever: 0x3b249 000165 1782218636.788279 trunc from 0x02d5f6 to 0x03b248 (0xdc53 bytes) 000166 1782218636.877103 read 0x02fd89 thru 0x03b248 (0xb4c0 bytes) 000170 1782218636.911301 trunc from 0x03b249 to 0x004ac1 (0x36787 bytes) 000172 1782218636.948871 mapread 0x003a34 thru 0x004ac1 (0x108e bytes) 000174 1782218636.967754 read 0x000f4e thru 0x002fbc (0x206f bytes) 000176 1782218636.970363 mapwrite 0x013a88 thru 0x01d76d (0x9ce6 bytes) 000177 1782218637.111183 write 0x009bc2 thru 0x00b1b4 (0x15f3 bytes) 000178 1782218637.112441 mapread 0x014c05 thru 0x01767d (0x2a79 bytes) 000179 1782218637.120061 mapwrite 0x031301 thru 0x03aa51 (0x9751 bytes) 000180 1782218637.257083 write 0x016fa7 thru 0x019c1d (0x2c77 bytes) 000183 1782218637.260910 mapwrite 0x030be8 thru 0x0397d5 (0x8bee bytes) 000184 1782218637.331344 trunc from 0x03aa52 to 0x019c7c (0x20dd5 bytes) 000187 1782218637.360050 write 0x027d6e thru 0x037170 (0xf403 bytes) 000190 1782218637.368788 write 0x00d1df thru 0x01afe6 (0xde08 bytes) 000191 1782218637.379891 mapread 0x025aea thru 0x034cc6 (0xf1dd bytes) 000192 1782218637.393143 write 0x0022cf thru 0x0062c2 (0x3ff4 bytes) 000194 1782218637.395310 mapread 0x00433d thru 0x009856 (0x551a bytes) 000195 1782218637.401691 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37171): Mirror 2: 0x44d0c7c4 (stale) Mirror 5: 0x44d0c7c4 (stale) Mirror 6: 0x44d0c7c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37171): Mirror 2: 0x44d0c7c4 (stale) Mirror 5: 0x44d0c7c4 (stale) Mirror 6: 0x44d0c7c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37171: skipped 000196 1782218637.844540 write 0x02a90b thru 0x030708 (0x5dfe bytes) 000197 1782218637.855475 read 0x020562 thru 0x0231fe (0x2c9d bytes) 000198 1782218637.861872 mapwrite 0x027e2a thru 0x033212 (0xb3e9 bytes) truncating to largest ever: 0x3cd00 000201 1782218637.933964 trunc from 0x037171 to 0x03ccff (0x5b8f bytes) 000202 1782218637.952154 trunc from 0x03cd00 to 0x02c022 (0x10cdd bytes) 000203 1782218637.967808 mapread 0x0205fc thru 0x027439 (0x6e3e bytes) 000204 1782218637.970111 read 0x004e45 thru 0x010d37 (0xbef3 bytes) 000205 1782218637.980005 trunc from 0x02c023 to 0x01ddff (0xe223 bytes) 000206 1782218638.000743 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1de00): Mirror 2: 0xd5fdd559 (stale) Mirror 5: 0xd5fdd559 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1de00): Mirror 2: 0xd5fdd559 (stale) Mirror 5: 0xd5fdd559 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1de00: skipped 000207 1782218638.278269 read 0x00131b thru 0x00a4db (0x91c1 bytes) 000209 1782218638.286401 trunc from 0x01de00 to 0x02b238 (0xd439 bytes) 000210 1782218638.311595 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b239): Mirror 2: 0x479755e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b239): Mirror 2: 0x479755e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b239: skipped 000211 1782218638.840750 read 0x00e718 thru 0x00ede5 (0x6ce bytes) 000213 1782218638.852996 mapread 0x01daf4 thru 0x027c37 (0xa144 bytes) 000214 1782218638.878259 read 0x00c2e4 thru 0x0194f8 (0xd215 bytes) 000217 1782218638.910265 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b239): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b239): Mirror 2: 0x479755e1 (stale) CRC-32 checksum value for chunk [0, 0x2b239): Mirror 1: 0xe3fd910a Mirror 3: 0xe3fd910a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b239: skipped 000218 1782218640.163480 mapread 0x00dd03 thru 0x01da26 (0xfd24 bytes) 000219 1782218640.253560 write 0x019bb3 thru 0x01ab5b (0xfa9 bytes) 000220 1782218640.381374 trunc from 0x02b239 to 0x019a52 (0x117e6 bytes) 000221 1782218640.413082 read 0x0082c8 thru 0x0180d6 (0xfe0f bytes) 000227 1782218640.438445 write 0x0363a4 thru 0x037459 (0x10b6 bytes) 000228 1782218640.440091 mapread 0x024ebc thru 0x02b57e (0x66c3 bytes) 000231 1782218640.459283 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3745a): Mirror 3: 0x1a763ce1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3745a): Mirror 3: 0x1a763ce1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3745a: skipped 000232 1782218640.929170 read 0x00be4c thru 0x012834 (0x69e9 bytes) 000234 1782218640.953437 mapread 0x01cab7 thru 0x0289fb (0xbf45 bytes) 000235 1782218641.022140 write 0x03ed51 thru 0x03ffff (0x12af bytes) 000236 1782218641.023382 write 0x01501e thru 0x01a289 (0x526c bytes) 000237 1782218641.048460 mapwrite 0x00bac7 thru 0x00cfa4 (0x14de bytes) 000239 1782218641.166537 mapread 0x038a69 thru 0x03ffff (0x7597 bytes) 000240 1782218641.195374 write 0x0104a1 thru 0x01771f (0x727f bytes) 000241 1782218641.196801 read 0x00df82 thru 0x014610 (0x668f bytes) 000243 1782218641.202567 read 0x003fe7 thru 0x011066 (0xd080 bytes) 000244 1782218641.223295 write 0x02ce5a thru 0x03698d (0x9b34 bytes) 000246 1782218641.237698 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57667d23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4518a096 Mirror 4: 0x4518a096 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000247 1782218642.284752 read 0x020b1c thru 0x0268cb (0x5db0 bytes) 000248 1782218642.334981 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57667d23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4518a096 Mirror 4: 0x4518a096 Mirror 5: 0x4518a096 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000250 1782218643.204009 mapwrite 0x01ddbe thru 0x021660 (0x38a3 bytes) 000253 1782218643.378200 read 0x009081 thru 0x014dc5 (0xbd45 bytes) 000257 1782218643.405855 mapwrite 0x014e5b thru 0x0152b2 (0x458 bytes) 000261 1782218643.507168 MIRROR_RESYNC 4 mirrors 000263 1782218644.778595 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) 000264 1782218645.243956 write 0x0290e0 thru 0x02eb93 (0x5ab4 bytes) 000266 1782218645.324102 read 0x027705 thru 0x029024 (0x1920 bytes) 000267 1782218645.329962 mapwrite 0x018994 thru 0x026487 (0xdaf4 bytes) 000268 1782218645.404175 trunc from 0x040000 to 0x00b708 (0x348f7 bytes) 000269 1782218645.431090 mapwrite 0x03e8d5 thru 0x03ffff (0x172b bytes) 000270 1782218645.555948 MIRROR_RESYNC 3 mirrors 000272 1782218646.649094 MIRROR_EXTEND to 4 mirrors 000274 1782218647.337951 mapwrite 0x031087 thru 0x03ffff (0xef79 bytes) 000275 1782218647.537700 mapwrite 0x00d980 thru 0x019c5d (0xc2de bytes) 000279 1782218647.695345 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x261cc587 (stale) Mirror 6: 0x261cc587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x261cc587 (stale) Mirror 6: 0x261cc587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1782218648.094105 trunc from 0x040000 to 0x008ea1 (0x3715e bytes) 000281 1782218648.112479 mapwrite 0x036d51 thru 0x03dfad (0x725d bytes) 000284 1782218648.294652 mapwrite 0x002473 thru 0x0072c6 (0x4e54 bytes) 000285 1782218648.414626 mapwrite 0x020224 thru 0x02c70e (0xc4eb bytes) 000286 1782218648.663239 mapwrite 0x03d3cf thru 0x03ffff (0x2c31 bytes) 000287 1782218648.796240 read 0x0194c9 thru 0x025335 (0xbe6d bytes) 000289 1782218648.801067 mapread 0x0281e1 thru 0x03684c (0xe66c bytes) 000290 1782218648.803055 mapwrite 0x015305 thru 0x01d17f (0x7e7b bytes) 000291 1782218648.848079 trunc from 0x040000 to 0x0065e2 (0x39a1d bytes) 000293 1782218648.869837 write 0x039760 thru 0x03ffff (0x68a0 bytes) 000295 1782218648.873103 write 0x0092e7 thru 0x0100f1 (0x6e0b bytes) 000298 1782218648.883363 read 0x011a3c thru 0x01eecb (0xd490 bytes) 000299 1782218648.892258 mapread 0x005371 thru 0x0079e8 (0x2678 bytes) 000300 1782218648.898309 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x261cc587 (stale) Mirror 6: 0x261cc587 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x705b578a Mirror 7: 0x705b578a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1782218650.022015 mapread 0x03c40b thru 0x03ce87 (0xa7d bytes) 000306 1782218650.044711 trunc from 0x040000 to 0x01dbdd (0x22422 bytes) 000307 1782218650.226079 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dbde): Mirror 5: 0xe18d4c6d (stale) Mirror 7: 0xf7754692 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbde): Mirror 5: 0xe18d4c6d (stale) Mirror 7: 0xf7754692 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbde: skipped 000308 1782218650.612780 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1dbde): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbde): Mirror 5: 0xe18d4c6d (stale) Mirror 7: 0xf7754692 (stale) CRC-32 checksum value for chunk [0, 0x1dbde): Mirror 1: 0xf7754692 Mirror 8: 0xf7754692 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbde: skipped 000309 1782218651.739692 write 0x02423e thru 0x0335c0 (0xf383 bytes) 000310 1782218651.872898 MIRROR_RESYNC 4 mirrors 000312 1782218653.266200 trunc from 0x0335c1 to 0x01df27 (0x15699 bytes) 000313 1782218653.327204 mapread 0x017cd2 thru 0x01df27 (0x6256 bytes) 000315 1782218653.351360 write 0x00c4da thru 0x01badb (0xf602 bytes) 000317 1782218653.363986 trunc from 0x01df28 to 0x003844 (0x1a6e3 bytes) 000318 1782218653.408755 trunc from 0x003845 to 0x00a8f1 (0x70ad bytes) 000320 1782218653.432018 read 0x00969e thru 0x00a8f1 (0x1254 bytes) 000321 1782218653.448265 MIRROR_RESYNC 4 mirrors 000322 1782218654.683672 trunc from 0x00a8f2 to 0x002181 (0x8770 bytes) 000323 1782218654.886896 trunc from 0x002182 to 0x0137f6 (0x11675 bytes) 000324 1782218654.931470 read 0x013425 thru 0x0137f6 (0x3d2 bytes) 000325 1782218654.938259 read 0x00e4be thru 0x012082 (0x3bc5 bytes) 000326 1782218654.952076 trunc from 0x0137f7 to 0x01cc0f (0x9419 bytes) 000327 1782218654.980623 mapwrite 0x02d8ec thru 0x03c6b1 (0xedc6 bytes) 000331 1782218655.127840 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c6b2): Mirror 5: 0xd1b5a54a (stale) Mirror 7: 0xd1b5a54a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6b2): Mirror 5: 0xd1b5a54a (stale) Mirror 7: 0xd1b5a54a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6b2: skipped 000333 1782218655.754710 write 0x03dec8 thru 0x03ffff (0x2138 bytes) 000336 1782218655.768380 write 0x035b28 thru 0x03dbd1 (0x80aa bytes) 000337 1782218655.776225 read 0x02ce4e thru 0x03ca37 (0xfbea bytes) 000339 1782218655.809051 mapwrite 0x031d0d thru 0x0337d0 (0x1ac4 bytes) 000341 1782218655.905306 MIRROR_RESYNC 3 mirrors 000342 1782218656.894039 write 0x017ac5 thru 0x025a85 (0xdfc1 bytes) 000343 1782218656.967059 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xae24dd74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xae24dd74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000345 1782218657.394696 read 0x0395d0 thru 0x03ffff (0x6a30 bytes) 000351 1782218657.441930 trunc from 0x040000 to 0x0223a6 (0x1dc59 bytes) 000352 1782218657.482637 mapwrite 0x01528c thru 0x018987 (0x36fc bytes) 000353 1782218657.574057 mapread 0x015e57 thru 0x01f624 (0x97ce bytes) 000355 1782218657.584331 trunc from 0x0223a7 to 0x01430d (0xe099 bytes) 000356 1782218657.607374 mapwrite 0x03d3e3 thru 0x03ffff (0x2c1d bytes) 000358 1782218657.730064 write 0x035a50 thru 0x03a0f7 (0x46a8 bytes) 000361 1782218657.746039 write 0x0115a8 thru 0x01cb13 (0xb56c bytes) 000362 1782218657.775575 read 0x00293f thru 0x00eea1 (0xc563 bytes) 000363 1782218657.810603 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xae24dd74 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9251b1cd Mirror 8: 0x9251b1cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000364 1782218659.188632 write 0x00fdb9 thru 0x01815a (0x83a2 bytes) 000366 1782218659.320034 write 0x013a08 thru 0x01ffc7 (0xc5c0 bytes) 000367 1782218659.328332 mapread 0x03128b thru 0x03ffff (0xed75 bytes) 000369 1782218659.394657 mapwrite 0x0164d2 thru 0x01d608 (0x7137 bytes) 000371 1782218659.462577 read 0x016155 thru 0x0184be (0x236a bytes) 000372 1782218659.463365 mapwrite 0x01b818 thru 0x02b0e9 (0xf8d2 bytes) 000373 1782218659.524049 read 0x023d38 thru 0x02799f (0x3c68 bytes) 000374 1782218659.527877 mapread 0x00eb0d thru 0x010000 (0x14f4 bytes) 000376 1782218659.541322 trunc from 0x040000 to 0x01bdfc (0x24203 bytes) truncating to largest ever: 0x3f61c 000379 1782218659.576178 trunc from 0x01bdfd to 0x03f61b (0x2381f bytes) 000380 1782218659.607309 MIRROR_RESYNC 3 mirrors 000381 1782218660.630882 write 0x034298 thru 0x03cd9b (0x8b04 bytes) 000386 1782218660.732185 trunc from 0x03f61c to 0x02b43d (0x141de bytes) 000387 1782218660.767459 trunc from 0x02b43e to 0x01890c (0x12b31 bytes) 000388 1782218660.789171 mapread 0x016c9f thru 0x01890c (0x1c6e bytes) 000390 1782218660.841765 trunc from 0x01890d to 0x01a39a (0x1a8e bytes) 000391 1782218660.883315 trunc from 0x01a39b to 0x028074 (0xdcda bytes) 000393 1782218660.925211 write 0x01efbb thru 0x01f654 (0x69a bytes) 000397 1782218660.938020 read 0x005267 thru 0x012901 (0xd69b bytes) 000399 1782218660.949874 mapwrite 0x033368 thru 0x03f28d (0xbf26 bytes) 000400 1782218661.031906 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f28e): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f28e): Mirror 7: 0xaa9d95c4 (stale) Mirror 8: 0xaa9d95c4 (stale) CRC-32 checksum value for chunk [0, 0x3f28e): Mirror 1: 0x8babb7aa Mirror 9: 0x8babb7aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f28e: skipped 000401 1782218662.446921 mapread 0x0232d4 thru 0x031a21 (0xe74e bytes) 000402 1782218662.487202 trunc from 0x03f28e to 0x0374ae (0x7ddf bytes) 000403 1782218662.603323 write 0x02081c thru 0x027ad0 (0x72b5 bytes) 000404 1782218662.616377 write 0x02b1b4 thru 0x033b21 (0x896e bytes) 000407 1782218662.634249 read 0x00b641 thru 0x012b68 (0x7528 bytes) 000410 1782218662.658351 write 0x034228 thru 0x03c6f2 (0x84cb bytes) 000411 1782218662.672594 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c6f3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6f3): Mirror 7: 0xb3c1f511 (stale) Mirror 8: 0xb3c1f511 (stale) Mirror 9: 0xba51e58 (stale) CRC-32 checksum value for chunk [0, 0x3c6f3): Mirror 1: 0x91030fd9 Mirror 10: 0x91030fd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6f3: skipped 000412 1782218663.779917 mapwrite 0x02e17a thru 0x033782 (0x5609 bytes) 000413 1782218663.945302 mapwrite 0x02c132 thru 0x0384cc (0xc39b bytes) 000415 1782218664.021249 mapread 0x008a81 thru 0x00becc (0x344c bytes) 000416 1782218664.032864 read 0x01e0f6 thru 0x02a5a1 (0xc4ac bytes) 000419 1782218664.047705 write 0x026301 thru 0x02c689 (0x6389 bytes) 000421 1782218664.050723 trunc from 0x03c6f3 to 0x03007d (0xc675 bytes) 000423 1782218664.080996 write 0x01420a thru 0x01586c (0x1663 bytes) 000424 1782218664.100162 write 0x03a215 thru 0x03b4d7 (0x12c3 bytes) 000429 1782218664.103821 trunc from 0x03b4d8 to 0x02e72b (0xcdac bytes) 000432 1782218664.122020 mapread 0x009447 thru 0x00bd27 (0x28e1 bytes) 000433 1782218664.123231 mapread 0x013d7b thru 0x01786f (0x3af5 bytes) 000435 1782218664.136942 mapwrite 0x031523 thru 0x0341e0 (0x2cbe bytes) 000436 1782218664.291926 trunc from 0x0341e1 to 0x007e0d (0x2c3d3 bytes) 000439 1782218664.340195 trunc from 0x007e0e to 0x03e974 (0x36b67 bytes) 000440 1782218664.375364 mapread 0x02d8ed thru 0x03ba61 (0xe175 bytes) 000442 1782218664.411912 read 0x02447f thru 0x0332fc (0xee7e bytes) 000443 1782218664.430195 read 0x03cc14 thru 0x03e974 (0x1d61 bytes) 000448 1782218664.450854 mapwrite 0x03dd71 thru 0x03ffff (0x228f bytes) 000450 1782218664.633153 read 0x01e542 thru 0x0256c3 (0x7182 bytes) 000451 1782218664.636070 write 0x00bd24 thru 0x01991a (0xdbf7 bytes) 000457 1782218664.661203 mapread 0x000228 thru 0x00cb38 (0xc911 bytes) 000458 1782218664.670328 write 0x028e64 thru 0x02fde1 (0x6f7e bytes) 000459 1782218664.671682 read 0x010470 thru 0x017554 (0x70e5 bytes) 000464 1782218664.683616 read 0x000b51 thru 0x0098ee (0x8d9e bytes) 000468 1782218664.686458 trunc from 0x040000 to 0x02b49a (0x14b65 bytes) 000469 1782218664.741472 mapread 0x01a6bf thru 0x01e348 (0x3c8a bytes) 000472 1782218664.750273 mapread 0x006802 thru 0x0102f8 (0x9af7 bytes) 000473 1782218664.752379 read 0x016081 thru 0x023f9c (0xdf1c bytes) 000474 1782218664.761121 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 7: 0xb1f6c91 (stale) Mirror 8: 0xb1f6c91 (stale) Mirror 9: 0x52c97668 (stale) Mirror 10: 0xac500d60 (stale) CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 1: 0xe5d8df15 Mirror 11: 0xe5d8df15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b49b: skipped 000475 1782218665.853991 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 7: 0xb1f6c91 (stale) Mirror 9: 0x52c97668 (stale) Mirror 10: 0xac500d60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 7: 0xb1f6c91 (stale) Mirror 9: 0x52c97668 (stale) Mirror 10: 0xac500d60 (stale) CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 1: 0xe5d8df15 Mirror 11: 0xe5d8df15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b49b: skipped 000479 1782218666.541795 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 7: 0xb1f6c91 (stale) Mirror 10: 0xac500d60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 7: 0xb1f6c91 (stale) Mirror 10: 0xac500d60 (stale) CRC-32 checksum value for chunk [0, 0x2b49b): Mirror 1: 0xe5d8df15 Mirror 11: 0xe5d8df15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b49b: skipped 000480 1782218667.509144 read 0x01702d thru 0x021e6a (0xae3e bytes) 000481 1782218667.525706 read 0x0262b1 thru 0x02701c (0xd6c bytes) 000482 1782218667.542251 read 0x00b9c1 thru 0x013254 (0x7894 bytes) 000483 1782218667.543241 write 0x005d3c thru 0x014ff9 (0xf2be bytes) 000484 1782218667.616800 mapwrite 0x03afb3 thru 0x03f10b (0x4159 bytes) 000485 1782218667.745337 mapread 0x010331 thru 0x01044b (0x11b bytes) 000487 1782218667.750292 read 0x006a48 thru 0x0103f0 (0x99a9 bytes) 000489 1782218667.751876 mapread 0x00c077 thru 0x0193a9 (0xd333 bytes) 000490 1782218667.755592 mapwrite 0x00adb1 thru 0x0188a4 (0xdaf4 bytes) 000491 1782218667.809542 trunc from 0x03f10c to 0x035ec6 (0x9245 bytes) 000492 1782218667.831207 write 0x03f1be thru 0x03ffff (0xe42 bytes) 000493 1782218667.832447 trunc from 0x040000 to 0x02f69e (0x10961 bytes) 000494 1782218667.851927 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f69f): Mirror 7: 0xdf2bc8bb (stale) Mirror 11: 0x11190570 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f69f): Mirror 7: 0xdf2bc8bb (stale) Mirror 11: 0x11190570 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f69f: skipped 000495 1782218668.230866 trunc from 0x02f69f to 0x00868a (0x27014 bytes) 000496 1782218668.265613 mapwrite 0x03352f thru 0x03ffff (0xcad1 bytes) 000497 1782218668.516117 MIRROR_RESYNC 3 mirrors 000498 1782218670.015521 write 0x017928 thru 0x01c136 (0x480f bytes) 000499 1782218670.139250 mapwrite 0x034ef9 thru 0x03fc90 (0xad98 bytes) 000500 1782218670.267116 mapread 0x003660 thru 0x006473 (0x2e14 bytes) 000502 1782218670.313831 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd96c10d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd96c10d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000503 1782218670.837977 write 0x0323cd thru 0x03a02d (0x7c61 bytes) 000504 1782218670.865548 trunc from 0x040000 to 0x012108 (0x2def7 bytes) 000506 1782218670.909387 trunc from 0x012109 to 0x029e01 (0x17cf9 bytes) 000510 1782218670.954572 write 0x008aeb thru 0x0111d3 (0x86e9 bytes) 000511 1782218670.986401 mapwrite 0x030d80 thru 0x0331dc (0x245d bytes) 000513 1782218671.202105 read 0x01ade3 thru 0x01cb99 (0x1db7 bytes) 000514 1782218671.228417 read 0x00a9db thru 0x011a6a (0x7090 bytes) 000515 1782218671.231989 mapread 0x00435c thru 0x00ff68 (0xbc0d bytes) 000516 1782218671.264158 mapread 0x0080f4 thru 0x009f02 (0x1e0f bytes) 000517 1782218671.265601 mapwrite 0x03f976 thru 0x03ffff (0x68a bytes) 000519 1782218671.458846 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd96c10d3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa3024799 Mirror 8: 0xa3024799 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000522 1782218672.667261 mapwrite 0x02adb2 thru 0x034041 (0x9290 bytes) 000523 1782218672.977857 read 0x01ada3 thru 0x02038d (0x55eb bytes) 000524 1782218672.990977 read 0x00763c thru 0x01315d (0xbb22 bytes) 000525 1782218673.001882 write 0x01f1ae thru 0x0237b2 (0x4605 bytes) 000526 1782218673.007533 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd96c10d3 (stale) Mirror 8: 0xa3024799 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa4d15d9 Mirror 9: 0xa4d15d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000527 1782218674.248058 write 0x0278a6 thru 0x032ac0 (0xb21b bytes) 000530 1782218674.421753 mapwrite 0x011e72 thru 0x016df9 (0x4f88 bytes) 000533 1782218674.533027 trunc from 0x040000 to 0x02cf44 (0x130bb bytes) 000535 1782218674.582847 write 0x02cc81 thru 0x035edf (0x925f bytes) 000541 1782218674.589173 mapread 0x015221 thru 0x01b081 (0x5e61 bytes) 000542 1782218674.590434 write 0x03c1c4 thru 0x03ffff (0x3e3c bytes) 000543 1782218674.595854 read 0x02c2b2 thru 0x03b9b4 (0xf703 bytes) 000545 1782218674.611147 trunc from 0x040000 to 0x035b4c (0xa4b3 bytes) 000547 1782218674.666001 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35b4d): Mirror 8: 0x56ec07a1 (stale) Mirror 9: 0x1de486bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b4d): Mirror 8: 0x56ec07a1 (stale) Mirror 9: 0x1de486bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b4d: skipped 000549 1782218675.040120 MIRROR_RESYNC 3 mirrors 000550 1782218675.897578 mapwrite 0x00f356 thru 0x019953 (0xa5fe bytes) 000552 1782218676.189936 read 0x023c7c thru 0x03039c (0xc721 bytes) 000553 1782218676.205048 MIRROR_RESYNC 3 mirrors 000555 1782218676.895325 mapread 0x00489d thru 0x00e1ba (0x991e bytes) 000556 1782218676.923327 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) 000557 1782218677.346980 mapread 0x01da48 thru 0x0269eb (0x8fa4 bytes) 000558 1782218677.401142 MIRROR_RESYNC 2 mirrors 000559 1782218677.934318 mapwrite 0x010bb6 thru 0x018a4b (0x7e96 bytes) 000562 1782218678.104210 write 0x02e78f thru 0x03c8b2 (0xe124 bytes) 000563 1782218678.133406 trunc from 0x03c8b3 to 0x00912e (0x33784 bytes) 000564 1782218678.217801 mapread 0x00163a thru 0x00460a (0x2fd1 bytes) 000565 1782218678.243257 mapread 0x0026d0 thru 0x007f87 (0x58b8 bytes) 000566 1782218678.244672 mapwrite 0x01cdd3 thru 0x02b8e6 (0xeb14 bytes) 000567 1782218678.393388 mapwrite 0x01e7b5 thru 0x028026 (0x9872 bytes) 000568 1782218678.461006 mapwrite 0x03278e thru 0x03fbad (0xd420 bytes) 000569 1782218678.562913 read 0x033bb8 thru 0x039dfa (0x6243 bytes) 000571 1782218678.565148 trunc from 0x03fbae to 0x0105a2 (0x2f60b bytes) 000572 1782218678.588107 mapwrite 0x026de8 thru 0x02ec60 (0x7e79 bytes) 000575 1782218678.691912 read 0x02671f thru 0x02ec60 (0x8542 bytes) 000576 1782218678.693256 trunc from 0x02ec61 to 0x00b407 (0x23859 bytes) 000577 1782218678.718189 read 0x00957b thru 0x00b407 (0x1e8d bytes) 000580 1782218678.736198 trunc from 0x00b408 to 0x020f03 (0x15afc bytes) 000581 1782218678.774270 MIRROR_RESYNC 2 mirrors 000583 1782218679.704051 write 0x03a759 thru 0x03d376 (0x2c1e bytes) 000585 1782218679.807780 mapread 0x031b35 thru 0x038647 (0x6b13 bytes) 000587 1782218679.833093 write 0x020cd3 thru 0x0290ef (0x841d bytes) 000588 1782218679.844619 write 0x0008c3 thru 0x00d08f (0xc7cd bytes) 000589 1782218679.866615 trunc from 0x03d377 to 0x034897 (0x8adf bytes) 000590 1782218679.921309 mapwrite 0x0088b4 thru 0x00f016 (0x6763 bytes) 000594 1782218680.020295 read 0x0136c6 thru 0x013986 (0x2c1 bytes) 000595 1782218680.022051 read 0x024af8 thru 0x030646 (0xbb4f bytes) 000596 1782218680.037642 mapwrite 0x004c23 thru 0x011ced (0xd0cb bytes) 000598 1782218680.141759 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34898): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34898): Mirror 9: 0x5faf4b89 (stale) CRC-32 checksum value for chunk [0, 0x34898): Mirror 1: 0x7c180a4b Mirror 10: 0x7c180a4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34898: skipped 000599 1782218681.276269 mapread 0x00b125 thru 0x019598 (0xe474 bytes) 000600 1782218681.342153 write 0x01b3be thru 0x0270f0 (0xbd33 bytes) 000602 1782218681.481151 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34898): Mirror 10: 0x7c180a4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34898): Mirror 10: 0x7c180a4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34898: skipped 000603 1782218681.762857 mapwrite 0x00f097 thru 0x014817 (0x5781 bytes) 000605 1782218681.883989 write 0x00627f thru 0x00d232 (0x6fb4 bytes) 000606 1782218681.895177 MIRROR_RESYNC 2 mirrors 000607 1782218683.050747 MIRROR_RESYNC 2 mirrors 000608 1782218683.544687 mapread 0x021794 thru 0x026a03 (0x5270 bytes) 000611 1782218683.566327 mapread 0x02475d thru 0x026a00 (0x22a4 bytes) 000612 1782218683.568744 mapread 0x011708 thru 0x018d28 (0x7621 bytes) 000619 1782218683.600101 mapread 0x0043b9 thru 0x00b7f4 (0x743c bytes) 000620 1782218683.636076 MIRROR_EXTEND to 3 mirrors 000621 1782218684.645216 MIRROR_RESYNC 3 mirrors 000623 1782218685.124401 trunc from 0x034898 to 0x03cc49 (0x83b2 bytes) 000624 1782218685.250203 mapread 0x026433 thru 0x02c039 (0x5c07 bytes) 000626 1782218685.281210 read 0x0216bd thru 0x02b43a (0x9d7e bytes) 000627 1782218685.287909 MIRROR_RESYNC 3 mirrors 000635 1782218686.126334 mapread 0x03010c thru 0x03b9f1 (0xb8e6 bytes) 000636 1782218686.155929 trunc from 0x03cc4a to 0x01f388 (0x1d8c1 bytes) 000637 1782218686.276800 write 0x020604 thru 0x023de0 (0x37dd bytes) 000640 1782218686.281795 mapwrite 0x03cb42 thru 0x03e133 (0x15f2 bytes) 000641 1782218686.375920 write 0x01138e thru 0x014aba (0x372d bytes) 000642 1782218686.386969 trunc from 0x03e134 to 0x03703b (0x70f8 bytes) 000643 1782218686.416169 mapread 0x00512c thru 0x013330 (0xe205 bytes) 000646 1782218686.446102 trunc from 0x03703c to 0x00b2a5 (0x2bd96 bytes) 000647 1782218686.496645 mapread 0x002c93 thru 0x0074d4 (0x4842 bytes) 000648 1782218686.503235 mapread 0x0091f6 thru 0x00b2a5 (0x20b0 bytes) 000649 1782218686.510915 trunc from 0x00b2a6 to 0x019897 (0xe5f2 bytes) 000652 1782218686.546219 mapread 0x00f206 thru 0x018a82 (0x987d bytes) 000659 1782218686.579383 trunc from 0x019898 to 0x028cc7 (0xf430 bytes) 000661 1782218686.595560 mapwrite 0x022610 thru 0x025a25 (0x3416 bytes) 000662 1782218686.681074 mapwrite 0x005941 thru 0x010b76 (0xb236 bytes) 000663 1782218686.746077 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28cc8): Mirror 11: 0x9219ddf5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28cc8): Mirror 11: 0x9219ddf5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28cc8: skipped 000665 1782218687.169216 trunc from 0x028cc8 to 0x0054d6 (0x237f1 bytes) 000666 1782218687.192410 mapread 0x00033a thru 0x0054d6 (0x519d bytes) 000667 1782218687.215983 mapread 0x0028b4 thru 0x0054d6 (0x2c23 bytes) 000668 1782218687.217534 mapwrite 0x01101d thru 0x01b260 (0xa244 bytes) 000669 1782218687.331289 read 0x00c10b thru 0x01591c (0x9812 bytes) 000670 1782218687.334349 read 0x00bf83 thru 0x01afd3 (0xf051 bytes) 000671 1782218687.335921 trunc from 0x01b261 to 0x02a243 (0xefe3 bytes) 000673 1782218687.359176 write 0x0209a0 thru 0x0246d4 (0x3d35 bytes) 000674 1782218687.370867 mapread 0x013048 thru 0x019d4e (0x6d07 bytes) 000675 1782218687.372124 mapwrite 0x02d815 thru 0x036c12 (0x93fe bytes) 000676 1782218687.450197 mapwrite 0x01e7b0 thru 0x02aad0 (0xc321 bytes) 000677 1782218687.503447 write 0x0224eb thru 0x02e56d (0xc083 bytes) 000680 1782218687.509023 mapwrite 0x036c44 thru 0x03ffff (0x93bc bytes) 000681 1782218687.597079 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x24d65be5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x57251e96 Mirror 12: 0x57251e96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000684 1782218688.509095 write 0x02862b thru 0x03102d (0x8a03 bytes) 000685 1782218688.668204 mapwrite 0x001d24 thru 0x004926 (0x2c03 bytes) 000686 1782218688.778633 mapread 0x02dc4a thru 0x03b08a (0xd441 bytes) 000687 1782218688.804750 write 0x00ce24 thru 0x01703d (0xa21a bytes) 000689 1782218688.826135 write 0x03e2d5 thru 0x03ffff (0x1d2b bytes) 000690 1782218688.832374 mapread 0x03119d thru 0x03f624 (0xe488 bytes) 000693 1782218688.839830 write 0x03a3e3 thru 0x03ffff (0x5c1d bytes) 000695 1782218688.851448 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x24d65be5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x24d65be5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000696 1782218689.441365 mapread 0x01528e thru 0x01681e (0x1591 bytes) 000697 1782218689.452364 trunc from 0x040000 to 0x025157 (0x1aea8 bytes) 000698 1782218689.507069 read 0x011654 thru 0x01dee1 (0xc88e bytes) 000700 1782218689.527871 write 0x006530 thru 0x009f4f (0x3a20 bytes) 000702 1782218689.561623 write 0x0173f6 thru 0x01c21f (0x4e2a bytes) 000703 1782218689.574327 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25158): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25158): Mirror 11: 0xda2e1280 (stale) CRC-32 checksum value for chunk [0, 0x25158): Mirror 1: 0x4e966cd1 Mirror 12: 0x4e966cd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25158: skipped 000704 1782218690.747314 mapread 0x0141d0 thru 0x020b65 (0xc996 bytes) 000705 1782218690.879773 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25158): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25158): Mirror 11: 0xda2e1280 (stale) CRC-32 checksum value for chunk [0, 0x25158): Mirror 1: 0x4e966cd1 Mirror 12: 0x4e966cd1 Mirror 13: 0x4e966cd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25158: skipped 000706 1782218691.958537 mapread 0x01171d thru 0x01cb5c (0xb440 bytes) 000712 1782218692.050042 read 0x000e40 thru 0x0077f6 (0x69b7 bytes) 000713 1782218692.063137 read 0x00da75 thru 0x01b90f (0xde9b bytes) 000715 1782218692.064193 read 0x01f5d5 thru 0x025157 (0x5b83 bytes) 000717 1782218692.075097 trunc from 0x025158 to 0x025c9e (0xb47 bytes) 000720 1782218692.193995 write 0x0006c9 thru 0x0062c5 (0x5bfd bytes) 000721 1782218692.211897 mapread 0x011d45 thru 0x01d1b6 (0xb472 bytes) 000723 1782218692.295804 write 0x014651 thru 0x022279 (0xdc29 bytes) 000725 1782218692.306865 read 0x023417 thru 0x025c9e (0x2888 bytes) 000728 1782218692.326717 MIRROR_RESYNC 4 mirrors 000729 1782218693.093385 read 0x022c93 thru 0x023633 (0x9a1 bytes) 000733 1782218693.102358 trunc from 0x025c9f to 0x02232c (0x3972 bytes) 000734 1782218693.253193 trunc from 0x02232d to 0x019335 (0x8ff7 bytes) 000735 1782218693.284947 mapwrite 0x0297a2 thru 0x03495c (0xb1bb bytes) 000738 1782218693.371594 mapwrite 0x00557d thru 0x00f6ae (0xa132 bytes) 000739 1782218693.457043 mapwrite 0x039aee thru 0x03ffff (0x6512 bytes) 000740 1782218693.536299 mapwrite 0x0193d2 thru 0x02537f (0xbfae bytes) 000742 1782218693.725783 mapwrite 0x023d75 thru 0x03326c (0xf4f8 bytes) 000743 1782218693.827247 mapread 0x0164ed thru 0x01b608 (0x511c bytes) 000746 1782218693.830129 read 0x01a91f thru 0x01ca59 (0x213b bytes) 000749 1782218693.832436 mapwrite 0x03f73a thru 0x03ffff (0x8c6 bytes) 000752 1782218693.871293 write 0x018b47 thru 0x024da2 (0xc25c bytes) 000753 1782218693.873436 read 0x0170da thru 0x01a6f9 (0x3620 bytes) 000754 1782218693.874105 write 0x008110 thru 0x011cf6 (0x9be7 bytes) 000757 1782218693.889398 read 0x01e95c thru 0x02d771 (0xee16 bytes) 000759 1782218693.894084 read 0x027abd thru 0x036d0a (0xf24e bytes) 000760 1782218693.896050 mapread 0x011550 thru 0x0206d5 (0xf186 bytes) 000761 1782218693.905267 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xae5c6ca4 (stale) Mirror 12: 0xae5c6ca4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xae5c6ca4 (stale) Mirror 12: 0xae5c6ca4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000762 1782218694.286618 write 0x01e5d7 thru 0x01f5ae (0xfd8 bytes) 000764 1782218694.296819 write 0x011a8a thru 0x01d2f0 (0xb867 bytes) 000765 1782218694.309661 mapread 0x0036a6 thru 0x00c470 (0x8dcb bytes) 000767 1782218694.356145 read 0x008daa thru 0x00f4d7 (0x672e bytes) 000769 1782218694.358074 write 0x00f22b thru 0x01c3cb (0xd1a1 bytes) 000771 1782218694.361323 trunc from 0x040000 to 0x020dd2 (0x1f22d bytes) 000772 1782218694.394647 trunc from 0x020dd3 to 0x02eded (0xe01b bytes) 000774 1782218694.431721 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2edee): Mirror 12: 0xc2c4b2d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edee): Mirror 12: 0xc2c4b2d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edee: skipped 000776 1782218694.767113 mapwrite 0x00b99c thru 0x013e00 (0x8465 bytes) 000777 1782218694.910183 trunc from 0x02edee to 0x00e129 (0x20cc4 bytes) 000778 1782218694.939818 trunc from 0x00e12a to 0x0036bc (0xaa6d bytes) 000780 1782218694.961173 mapread 0x00026f thru 0x0036bc (0x344e bytes) 000781 1782218694.968149 trunc from 0x0036bd to 0x03ee41 (0x3b785 bytes) 000783 1782218694.990061 read 0x036517 thru 0x03ebbb (0x86a5 bytes) 000784 1782218695.001535 mapread 0x01fc13 thru 0x02f116 (0xf504 bytes) 000785 1782218695.020908 mapread 0x00262f thru 0x012264 (0xfc36 bytes) 000786 1782218695.039232 write 0x039bf7 thru 0x03ffff (0x6409 bytes) 000788 1782218695.041907 trunc from 0x040000 to 0x02cffb (0x13004 bytes) 000791 1782218695.064630 write 0x032166 thru 0x03d690 (0xb52b bytes) 000793 1782218695.070040 read 0x013e61 thru 0x01422d (0x3cd bytes) 000795 1782218695.073341 mapread 0x0152fd thru 0x01cc8a (0x798e bytes) 000796 1782218695.074884 mapread 0x008f37 thru 0x00fee9 (0x6fb3 bytes) 000798 1782218695.082155 mapread 0x013f7b thru 0x023400 (0xf486 bytes) 000800 1782218695.100782 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) 000801 1782218695.250596 mapread 0x01d892 thru 0x02cef4 (0xf663 bytes) 000802 1782218695.305194 trunc from 0x03d691 to 0x01af0b (0x22785 bytes) 000805 1782218695.339026 mapwrite 0x023548 thru 0x028f19 (0x59d2 bytes) 000808 1782218695.419808 mapread 0x02174a thru 0x028f19 (0x77d0 bytes) 000810 1782218695.422212 mapwrite 0x00f73e thru 0x014d06 (0x55c9 bytes) 000812 1782218695.497189 mapread 0x0183b5 thru 0x026849 (0xe495 bytes) 000813 1782218695.512795 trunc from 0x028f1a to 0x03a930 (0x11a17 bytes) 000814 1782218695.552812 MIRROR_EXTEND to 2 mirrors 000816 1782218696.249773 trunc from 0x03a931 to 0x00e3de (0x2c552 bytes) 000817 1782218696.337739 mapread 0x006452 thru 0x00e3de (0x7f8d bytes) 000818 1782218696.389787 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) 000819 1782218696.625051 write 0x03eabe thru 0x03ffff (0x1542 bytes) 000820 1782218696.626246 mapwrite 0x02b35d thru 0x03a853 (0xf4f7 bytes) 000823 1782218696.752071 mapwrite 0x026cc6 thru 0x0286ad (0x19e8 bytes) 000825 1782218696.798631 trunc from 0x040000 to 0x017c5e (0x283a1 bytes) 000827 1782218696.832335 mapread 0x0112ac thru 0x01213e (0xe93 bytes) 000830 1782218696.848574 mapwrite 0x0316be thru 0x036fe4 (0x5927 bytes) 000831 1782218696.940781 trunc from 0x036fe5 to 0x02ec5f (0x8385 bytes) 000834 1782218696.961685 mapread 0x00b249 thru 0x00e742 (0x34fa bytes) 000836 1782218696.964326 mapread 0x00dbcd thru 0x01664a (0x8a7e bytes) 000837 1782218696.966677 read 0x00bd89 thru 0x0107a0 (0x4a18 bytes) 000839 1782218696.968716 write 0x01a54f thru 0x025606 (0xb0b8 bytes) 000840 1782218696.986768 mapwrite 0x034fc6 thru 0x03ffff (0xb03a bytes) 000842 1782218697.146733 mapread 0x0124d7 thru 0x01e2d3 (0xbdfd bytes) 000845 1782218697.160342 read 0x020ae0 thru 0x02d000 (0xc521 bytes) 000847 1782218697.183077 mapwrite 0x01ae25 thru 0x0200b5 (0x5291 bytes) 000850 1782218697.259138 write 0x02cd35 thru 0x03748e (0xa75a bytes) 000851 1782218697.262353 trunc from 0x040000 to 0x030b1b (0xf4e4 bytes) 000852 1782218697.288628 trunc from 0x030b1c to 0x02ced6 (0x3c45 bytes) 000857 1782218697.328129 read 0x020531 thru 0x029ca5 (0x9775 bytes) 000858 1782218697.329356 mapwrite 0x0154dd thru 0x01f92e (0xa452 bytes) 000860 1782218697.432400 write 0x01c0c1 thru 0x023ddd (0x7d1d bytes) 000861 1782218697.434560 trunc from 0x02ced7 to 0x007491 (0x25a45 bytes) 000865 1782218697.486082 mapread 0x006d74 thru 0x007491 (0x71e bytes) 000866 1782218697.491128 write 0x02021d thru 0x0240d3 (0x3eb7 bytes) 000867 1782218697.495242 read 0x0185f4 thru 0x01c38d (0x3d9a bytes) 000870 1782218697.506568 mapread 0x00a391 thru 0x01181b (0x748b bytes) 000871 1782218697.509380 mapwrite 0x03345b thru 0x034b70 (0x1716 bytes) 000872 1782218697.607068 mapwrite 0x02245a thru 0x02a15e (0x7d05 bytes) 000873 1782218697.707273 MIRROR_EXTEND to 2 mirrors 000874 1782218698.457756 mapread 0x024b09 thru 0x02d709 (0x8c01 bytes) 000877 1782218698.480476 trunc from 0x034b71 to 0x03292e (0x2242 bytes) 000878 1782218698.597896 MIRROR_RESYNC 2 mirrors 000879 1782218699.472889 MIRROR_EXTEND to 3 mirrors 000881 1782218700.382200 mapwrite 0x03ae26 thru 0x03ffff (0x51da bytes) 000882 1782218700.636906 mapwrite 0x01b2cf thru 0x0274d8 (0xc20a bytes) 000883 1782218700.697846 mapread 0x02d314 thru 0x030292 (0x2f7f bytes) 000884 1782218700.707810 mapwrite 0x017c46 thru 0x01dc28 (0x5fe3 bytes) 000885 1782218700.750889 mapread 0x032c88 thru 0x038c8b (0x6004 bytes) 000886 1782218700.755940 MIRROR_RESYNC 3 mirrors 000887 1782218701.718007 mapwrite 0x00db54 thru 0x01cc6e (0xf11b bytes) 000888 1782218702.019233 mapwrite 0x03e606 thru 0x03ffff (0x19fa bytes) 000889 1782218702.070251 mapwrite 0x01cd63 thru 0x028216 (0xb4b4 bytes) 000890 1782218702.193526 trunc from 0x040000 to 0x00ec60 (0x3139f bytes) 000891 1782218702.226451 mapread 0x000fd3 thru 0x0014be (0x4ec bytes) 000892 1782218702.247908 mapread 0x004412 thru 0x004c45 (0x834 bytes) 000894 1782218702.252021 mapwrite 0x016dcd thru 0x021819 (0xaa4d bytes) 000895 1782218702.445883 write 0x0344c7 thru 0x03ffff (0xbb39 bytes) 000896 1782218702.448746 trunc from 0x040000 to 0x00a8e8 (0x35717 bytes) 000899 1782218702.473908 mapwrite 0x0214d6 thru 0x023783 (0x22ae bytes) 000900 1782218702.692759 MIRROR_RESYNC 3 mirrors 000901 1782218703.755116 mapread 0x015f60 thru 0x0222bb (0xc35c bytes) 000902 1782218703.794406 read 0x020b2e thru 0x023783 (0x2c56 bytes) 000904 1782218703.802188 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) 000905 1782218704.139980 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) 000907 1782218704.272377 trunc from 0x023784 to 0x00dd7a (0x15a09 bytes) 000908 1782218704.303749 mapwrite 0x0222ca thru 0x02ae5f (0x8b96 bytes) 000914 1782218704.443999 mapread 0x0118d0 thru 0x01dd58 (0xc489 bytes) 000922 1782218704.486336 write 0x009b13 thru 0x010966 (0x6e54 bytes) 000923 1782218704.497239 trunc from 0x02ae60 to 0x01b7c6 (0xf699 bytes) 000924 1782218704.532058 trunc from 0x01b7c7 to 0x00155b (0x1a26b bytes) 000932 1782218704.564194 read 0x00006d thru 0x00155b (0x14ef bytes) 000933 1782218704.571024 read 0x000de5 thru 0x00155b (0x777 bytes) 000934 1782218704.572006 mapread 0x000e5e thru 0x00155b (0x6fe bytes) 000935 1782218704.573858 read 0x0002b9 thru 0x00155b (0x12a3 bytes) 000938 1782218704.576940 mapwrite 0x03487d thru 0x03f7e9 (0xaf6d bytes) 000940 1782218704.672092 mapwrite 0x01e21a thru 0x025ee5 (0x7ccc bytes) 000943 1782218704.726066 trunc from 0x03f7ea to 0x01c1b9 (0x23630 bytes) 000944 1782218704.745292 read 0x011327 thru 0x01c1b9 (0xae93 bytes) 000945 1782218704.746999 mapwrite 0x03530a thru 0x03a7ef (0x54e6 bytes) 000946 1782218704.837454 trunc from 0x03a7f0 to 0x00c8f3 (0x2defc bytes) 000947 1782218704.854996 read 0x0053c8 thru 0x00c8f3 (0x752c bytes) 000948 1782218704.855832 write 0x000998 thru 0x00c38f (0xb9f8 bytes) 000951 1782218704.860694 mapwrite 0x00b2af thru 0x0120f3 (0x6e45 bytes) 000956 1782218704.951315 MIRROR_EXTEND to 2 mirrors 000958 1782218705.637569 MIRROR_EXTEND to 3 mirrors 000959 1782218706.487312 MIRROR_EXTEND to 4 mirrors 000961 1782218707.401542 mapwrite 0x02a78c thru 0x0358a3 (0xb118 bytes) 000963 1782218707.683077 mapread 0x018726 thru 0x026d09 (0xe5e4 bytes) 000967 1782218707.706702 mapread 0x00e239 thru 0x015e1a (0x7be2 bytes) 000968 1782218707.719983 mapwrite 0x015c3e thru 0x01b254 (0x5617 bytes) 000969 1782218707.782138 write 0x0172f3 thru 0x01cec9 (0x5bd7 bytes) 000971 1782218707.792148 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x358a4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358a4): Mirror 2: 0x6a61a59 (stale) Mirror 3: 0x6a61a59 (stale) Mirror 4: 0x6a61a59 (stale) CRC-32 checksum value for chunk [0, 0x358a4): Mirror 1: 0x56d6c614 Mirror 5: 0x56d6c614 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358a4: skipped 000972 1782218708.620550 read 0x0155d6 thru 0x01bd51 (0x677c bytes) 000973 1782218708.628357 trunc from 0x0358a4 to 0x02062f (0x15274 bytes) 000975 1782218708.719660 mapread 0x014a31 thru 0x01e93e (0x9f0e bytes) 000977 1782218708.740894 mapread 0x0067c2 thru 0x00aa52 (0x4291 bytes) 000978 1782218708.761878 write 0x00c1a9 thru 0x018257 (0xc0af bytes) 000979 1782218708.763412 mapread 0x0082b1 thru 0x00fe10 (0x7b60 bytes) 000981 1782218708.766705 write 0x03bf1a thru 0x03d5dd (0x16c4 bytes) 000984 1782218708.773836 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d5de): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5de): Mirror 2: 0xd4d6daf8 (stale) Mirror 3: 0xd4d6daf8 (stale) Mirror 4: 0xd4d6daf8 (stale) Mirror 5: 0x78e50e17 (stale) CRC-32 checksum value for chunk [0, 0x3d5de): Mirror 1: 0xd52ee7d2 Mirror 6: 0xd52ee7d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5de: skipped 000985 1782218710.095663 read 0x02dc2d thru 0x02efcd (0x13a1 bytes) 000986 1782218710.109885 trunc from 0x03d5de to 0x000322 (0x3d2bb bytes) 000988 1782218710.204235 mapread 0x00029a thru 0x000322 (0x89 bytes) 000989 1782218710.211688 read 0x000022 thru 0x000322 (0x301 bytes) 000990 1782218710.216375 MIRROR_RESYNC 6 mirrors 000991 1782218711.536543 mapread 0x00021a thru 0x000322 (0x109 bytes) 000992 1782218711.549683 mapwrite 0x00f82c thru 0x0160e1 (0x68b6 bytes) 000993 1782218711.826341 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x162:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x160e2): Mirror 3: 0x75e6c05b (stale) Mirror 4: 0x75e6c05b (stale) Mirror 5: 0x75e6c05b (stale) Mirror 6: 0x75e6c05b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160e2): Mirror 3: 0x75e6c05b (stale) Mirror 4: 0x75e6c05b (stale) Mirror 5: 0x75e6c05b (stale) Mirror 6: 0x75e6c05b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160e2: skipped 000994 1782218712.417319 read 0x01167d thru 0x013e4b (0x27cf bytes) 000995 1782218712.440375 read 0x007ab3 thru 0x008407 (0x955 bytes) 000996 1782218712.456482 read 0x00568d thru 0x00a43b (0x4daf bytes) 000998 1782218712.465546 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x160e2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160e2): Mirror 3: 0x75e6c05b (stale) Mirror 4: 0x75e6c05b (stale) Mirror 5: 0x75e6c05b (stale) Mirror 6: 0x75e6c05b (stale) CRC-32 checksum value for chunk [0, 0x160e2): Mirror 1: 0xa7ed02e7 Mirror 7: 0xa7ed02e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160e2: skipped 000999 1782218713.524873 read 0x015e15 thru 0x0160e1 (0x2cd bytes) 001000 1782218713.546029 trunc from 0x0160e2 to 0x02a873 (0x14792 bytes) All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 08:45:19 (1782218719) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 08:45:23 (1782218723) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275616 s, 7.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262912 s, 4.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 08:45:32 (1782218732) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7897952 .. Extending file size to 5131264 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5612800 .. Extending file size to 8642080 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8269792 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8936256 .. Extending file size to 8984480 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56763 56764 56765 56768 56770 done 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) 192.168.203.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg310-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:46:52 (1782218812) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8387616 .. Extending file size to 6509568 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8475840 .. Extending file size to 6582784 .. Extending file size to 7902016 .. Extending file size to 8559008 .. Extending file size to 8245344 .. Extending file size to 9243648 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9343360 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58277 58278 58280 58281 58285 done 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) 192.168.203.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg310-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (89s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:48:22 (1782218902) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199095 s, 10.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.592125 s, 3.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6bae7c6a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) PASS 200c (24s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:48:47 (1782218927) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189539 s, 11.1 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 08:48:56 (1782218936) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:49:04 (1782218944) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222955 s, 9.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:49:12 (1782218952) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239992 s, 8.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:49:19 (1782218959) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.32859 s, 6.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:49:53 (1782218993) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg310-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126952 s, 8.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:50:50 (1782219050) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg310-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250865 s, 8.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:51:51 (1782219111) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg310-server mds-ost sync done. Creating new pool test_204f oleg310-server: Pool lustre.test_204f created Adding targets to pool oleg310-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg310-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.305908 s, 6.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg310-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg310-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg310-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:53:11 (1782219191) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:53:21 (1782219201) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:53:30 (1782219210) oleg310-server: Pool lustre.test_206 created Adding targets to pool oleg310-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg310-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg310-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg310-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg310-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:53:56 (1782219236) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796796 s, 13.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040794 s, 25.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfdf2073f Mirror 3: 0xfdf2073f PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:54:06 (1782219246) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:54:10 (1782219250) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:54:13 (1782219253) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.101147 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1900000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x7c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x7c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x7c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x16a0000 Read this (4 bytes): BBBB 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:54:53 (1782219293) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0160996 s, 509 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:55:00 (1782219300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460308 s, 22.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d5 lmm_fid: [0x200000404:0x1d5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d5 lmm_fid: [0x200000404:0x1d5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834389 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-23 08:55:01.000000000 -0400 Modify: 2026-06-23 08:55:01.000000000 -0400 Change: 2026-06-23 08:55:03.000000000 -0400 Birth: 2026-06-23 08:55:01.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:55:08 (1782219308) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:55:12 (1782219312) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.671493 s, 61.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0480215 s, 85.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 23 08:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 23 08:55 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:55:21 (1782219321) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:55:29 (1782219329) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1769 sec =========== 08:55:39 (1782219339) === sanity-flr: start cleanup 08:55:41 (1782219341) === === sanity-flr: finish cleanup 08:55:49 (1782219349) ===