-----============= acceptance-small: sanity-flr ============----- Thu Sep 10 19:25:12 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg248-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:25:34 (1789082734) === oleg248-client.virtnet: executing check_config_client /mnt/lustre oleg248-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg248-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8967932c6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8967932c6000.idle_timeout=debug disable quota as required oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:26:13 (1789082773) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7603] == sanity-flr test 0a: lfs mirror create with -N option == 19:26:16 (1789082776) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:26:37 (1789082797) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:26:42 (1789082802) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:26:46 (1789082806) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:27:14 (1789082834) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:27:18 (1789082838) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:27:23 (1789082843) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8967932c6000.stats=clear osc.lustre-OST0001-osc-ffff8967932c6000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:27:32 (1789082852) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:27:46 (1789082866) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:27:57 (1789082877) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:28:02 (1789082882) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:28:12 (1789082892) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 19:28:22 (1789082902) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:28:30 (1789082910) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:28:33 (1789082913) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:28:42 (1789082922) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:28:53 (1789082933) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:29:02 (1789082942) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 1.14047 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150991 s, 6.9 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:29:15 (1789082955) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.412323 s, 38.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:29:24 (1789082964) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:29:38 (1789082978) Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (96s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:31:14 (1789083074) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.230436 s, 13.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:32:03 (1789083123) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:33:31 (1789083211) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.99728 s, 10.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 10 19:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.113s user 0m0.001s sys 0m0.199s Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8967932c6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.241s user 0m0.001s sys 0m0.219s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.330s user 0m0.001s sys 0m0.190s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8967932c6000.ost_server_uuid in FULL state after 0 sec PASS 33b (106s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:35:17 (1789083317) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:35:21 (1789083321) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:35:25 (1789083325) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 19:35:29 (1789083329) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 19:35:38 (1789083338) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.662083 s, 19.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:35:54 (1789083354) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647049 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457457 s, 22.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:36:02 (1789083362) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:36:10 (1789083370) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121204 s, 845 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147526 s, 694 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00183333 s, 559 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00157243 s, 651 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 19:36:23 (1789083383) create test files with size 15755297 .. 1d4b34349aac755082e900ad0370ed79 - 73d98056c965e3c033600c84926c9dca - 0379c7340a9807903c9333cc99456ac4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre PASS 37 (32s) == sanity-flr test 38: resync ============================ 19:36:55 (1789083415) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:37:11 (1789083431) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:37:19 (1789083439) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188567 s, 5.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458645 s, 22.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 19:37:30 (1789083450) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207982 s, 10.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.424588 s, 9.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137382 s, 15.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 19:37:47 (1789083467) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 19:37:51 (1789083471) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:37:55 (1789083475) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789865 s, 13.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 19:38:05 (1789083485) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:38:09 (1789083489) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:38:22 (1789083502) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.365911 s, 28.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:38:35 (1789083515) llite.lustre-ffff8967869e7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304396 s, 34.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff89679262d000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:38:54 (1789083534) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:39:43 (1789083583) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:40:29 (1789083629) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:40:33 (1789083633) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.246043 s, 12.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.131122 s, 24.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfe4157f9 (nosync,stale) Mirror 2: 0xfe4157f9 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 91ff0e9a56eeac1e6bb8e9a1acccace3 - mirror 1 checksum: eeaf969186d863cb6d22673220c965e5 - mirror 2 checksum: 91ff0e9a56eeac1e6bb8e9a1acccace3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T19:40:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T19:40:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 91ff0e9a56eeac1e6bb8e9a1acccace3 - mirror 1 checksum: eeaf969186d863cb6d22673220c965e5 - mirror 2 checksum: eeaf969186d863cb6d22673220c965e5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T19:40:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T19:40:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:40:46 (1789083646) Failing mds1 on oleg248-server Stopping /mnt/lustre-mds1 (opts:) on oleg248-server 19:40:54 (1789083654) shut down facet: mds1 facet_host: oleg248-server facet_failover_host: oleg248-server Failover mds1 to oleg248-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 19:41:17 (1789083677) targets are mounted 19:41:17 (1789083677) facet_failover done oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365008 s, 28.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xeb:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:41:35 (1789083695) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:41:39 (1789083699) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:41:43 (1789083703) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00801344 s, 2.0 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:41:51 (1789083711) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:41:55 (1789083715) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0131133 s, 312 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.103681 s, 39.5 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:42:06 (1789083726) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:42:34 (1789083754) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:43:06 (1789083786) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:43:33 (1789083813) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:43:59 (1789083839) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 19:44:07 (1789083847) mirror create pid 54242 mirror split pid 54243 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 54242 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 54243 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 19:45:15 (1789083915) Seed set to 3917 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1789083918.153574 write 0x026b53 thru 0x030ae9 (0x9f97 bytes) truncating to largest ever: 0x28cea 000003 1789083918.161119 trunc from 0x030aea to 0x028ce9 (0x7e00 bytes) 000004 1789083918.187967 mapread 0x0067f8 thru 0x010e04 (0xa60d bytes) 000006 1789083918.229342 mapread 0x008758 thru 0x016562 (0xde0b bytes) truncating to largest ever: 0x2d416 000008 1789083918.243856 trunc from 0x028cea to 0x02d415 (0x472c bytes) 000009 1789083918.289017 read 0x00ae34 thru 0x0139d1 (0x8b9e bytes) 000012 1789083918.295412 write 0x03291d thru 0x03e636 (0xbd1a bytes) 000013 1789083918.313088 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x31342 000014 1789083919.645498 trunc from 0x03e637 to 0x031341 (0xd2f5 bytes) 000017 1789083919.748865 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31342): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31342): Mirror 2: 0xc7c91290 (stale) CRC-32 checksum value for chunk [0, 0x31342): Mirror 1: 0xc7c91290 Mirror 3: 0xc7c91290 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31342: skipped 000018 1789083921.117461 read 0x0027a9 thru 0x00b64d (0x8ea5 bytes) truncating to largest ever: 0x33a3d 000019 1789083921.144009 trunc from 0x031342 to 0x033a3c (0x26fb bytes) 000020 1789083921.347447 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a3d): Mirror 2: 0xdbc58454 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a3d): Mirror 2: 0xdbc58454 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a3d: skipped 000022 1789083921.736686 write 0x02279f thru 0x0265a5 (0x3e07 bytes) 000024 1789083921.752753 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33a3d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a3d): Mirror 2: 0xdbc58454 (stale) CRC-32 checksum value for chunk [0, 0x33a3d): Mirror 1: 0x11fb3ef4 Mirror 3: 0x11fb3ef4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a3d: skipped 000026 1789083922.950049 mapread 0x01b812 thru 0x02a626 (0xee15 bytes) 000028 1789083923.043120 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000029 1789083923.436036 mapread 0x027f11 thru 0x033a3c (0xbb2c bytes) 000030 1789083923.493539 trunc from 0x033a3d to 0x0002d5 (0x33767 bytes) 000031 1789083923.600801 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d6): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d6): Mirror 3: 0x6fcfd414 (stale) CRC-32 checksum value for chunk [0, 0x2d6): Mirror 1: 0x6fcfd414 Mirror 4: 0x6fcfd414 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d6: skipped 000032 1789083924.609438 mapread 0x000076 thru 0x0002d5 (0x260 bytes) 000033 1789083924.652282 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000034 1789083925.066183 mapwrite 0x03c276 thru 0x03ffff (0x3d8a bytes) 000036 1789083925.237388 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000044 1789083925.424594 write 0x031efd thru 0x03ffff (0xe103 bytes) 000046 1789083925.435742 mapread 0x0177e1 thru 0x023149 (0xb969 bytes) 000048 1789083925.474338 read 0x01b2fe thru 0x01cc5b (0x195e bytes) 000049 1789083925.475460 mapwrite 0x0068f6 thru 0x00e9f9 (0x8104 bytes) 000050 1789083925.566602 mapread 0x030e89 thru 0x037f55 (0x70cd bytes) 000054 1789083925.589935 mapwrite 0x035757 thru 0x03ffff (0xa8a9 bytes) 000055 1789083925.664156 write 0x033c22 thru 0x034791 (0xb70 bytes) 000057 1789083925.676657 mapread 0x01340a thru 0x013851 (0x448 bytes) 000060 1789083925.683379 mapread 0x031be9 thru 0x03632d (0x4745 bytes) 000061 1789083925.688671 write 0x023f0b thru 0x029313 (0x5409 bytes) 000062 1789083925.692919 mapread 0x00f69f thru 0x014510 (0x4e72 bytes) 000063 1789083925.701742 mapwrite 0x022132 thru 0x028860 (0x672f bytes) 000064 1789083925.752707 read 0x02e423 thru 0x03e1cd (0xfdab bytes) 000065 1789083925.759580 MIRROR_EXTEND to 2 mirrors 000067 1789083927.064460 mapread 0x01f65e thru 0x0285c3 (0x8f66 bytes) 000068 1789083927.121607 MIRROR_RESYNC 2 mirrors 000069 1789083927.733505 MIRROR_EXTEND to 3 mirrors 000070 1789083928.909221 mapread 0x03039c thru 0x030a01 (0x666 bytes) 000071 1789083928.943406 write 0x0257a7 thru 0x03150a (0xbd64 bytes) 000073 1789083929.098503 trunc from 0x040000 to 0x025457 (0x1aba8 bytes) 000074 1789083929.163544 read 0x009551 thru 0x01838e (0xee3e bytes) 000075 1789083929.200912 mapread 0x00dded thru 0x0181ec (0xa400 bytes) 000076 1789083929.225823 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25458): Mirror 3: 0x9b75484c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25458): Mirror 3: 0x9b75484c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25458: skipped 000078 1789083929.637153 read 0x0008b9 thru 0x002ab9 (0x2201 bytes) 000082 1789083929.650806 trunc from 0x025458 to 0x02853e (0x30e7 bytes) 000085 1789083929.678388 read 0x00b13e thru 0x0172c6 (0xc189 bytes) 000086 1789083929.695444 trunc from 0x02853f to 0x030db2 (0x8874 bytes) 000088 1789083929.774475 read 0x02797b thru 0x02a3f2 (0x2a78 bytes) 000089 1789083929.793888 write 0x01dcf9 thru 0x025f05 (0x820d bytes) 000090 1789083929.795422 mapread 0x006527 thru 0x0111b9 (0xac93 bytes) 000091 1789083929.803782 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000093 1789083930.059459 trunc from 0x030db3 to 0x030bd2 (0x1e0 bytes) 000094 1789083930.080954 trunc from 0x030bd3 to 0x011d46 (0x1ee8c bytes) 000095 1789083930.098842 mapread 0x001aa9 thru 0x0093f7 (0x794f bytes) 000096 1789083930.122086 write 0x0166de thru 0x01de26 (0x7749 bytes) 000098 1789083930.130337 MIRROR_EXTEND to 2 mirrors 000099 1789083931.091219 mapread 0x00bcfd thru 0x017dc8 (0xc0cc bytes) 000101 1789083931.135478 trunc from 0x01de27 to 0x00bfc5 (0x11e61 bytes) 000103 1789083931.268533 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xbfc6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbfc6): Mirror 2: 0x365ccef2 (stale) CRC-32 checksum value for chunk [0, 0xbfc6): Mirror 1: 0x365ccef2 Mirror 3: 0x365ccef2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbfc6: skipped 000105 1789083932.791826 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000107 1789083933.330709 mapwrite 0x014b11 thru 0x020b69 (0xc059 bytes) 000109 1789083933.593704 mapwrite 0x03be9d thru 0x03ffff (0x4163 bytes) 000110 1789083933.684383 MIRROR_RESYNC 2 mirrors 000111 1789083934.475205 read 0x0302ba thru 0x03f229 (0xef70 bytes) 000112 1789083934.487810 trunc from 0x040000 to 0x030dd4 (0xf22b bytes) 000113 1789083934.594093 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30dd5): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30dd5): Mirror 3: 0x3424724e (stale) CRC-32 checksum value for chunk [0, 0x30dd5): Mirror 1: 0x3424724e Mirror 4: 0x3424724e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30dd5: skipped 000115 1789083935.589709 MIRROR_RESYNC 3 mirrors 000116 1789083936.744830 read 0x0267b9 thru 0x030dd4 (0xa61c bytes) 000118 1789083936.765353 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000119 1789083937.243273 trunc from 0x030dd5 to 0x015c82 (0x1b152 bytes) 000120 1789083937.398487 mapwrite 0x0042b5 thru 0x00cf64 (0x8cb0 bytes) 000122 1789083937.539333 MIRROR_RESYNC 2 mirrors 000123 1789083938.482278 trunc from 0x015c83 to 0x007fb7 (0xdccb bytes) 000125 1789083938.554148 mapwrite 0x03cde8 thru 0x03ffff (0x3218 bytes) 000126 1789083938.643234 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3578ec84 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa33a467b Mirror 5: 0xa33a467b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000127 1789083939.850967 trunc from 0x040000 to 0x020ec3 (0x1f13c bytes) 000132 1789083939.941502 mapread 0x01adbc thru 0x020ec3 (0x6108 bytes) 000133 1789083939.964144 mapwrite 0x014958 thru 0x01a616 (0x5cbf bytes) 000137 1789083940.027008 read 0x01a089 thru 0x020ec3 (0x6e3b bytes) 000138 1789083940.027597 write 0x03dbba thru 0x03ffff (0x2446 bytes) 000139 1789083940.028958 mapwrite 0x02d8ea thru 0x039a08 (0xc11f bytes) 000140 1789083940.082999 write 0x03402a thru 0x03772d (0x3704 bytes) 000141 1789083940.084089 write 0x01d628 thru 0x01df8a (0x963 bytes) 000143 1789083940.086323 mapwrite 0x0190c6 thru 0x020447 (0x7382 bytes) 000145 1789083940.136053 mapread 0x0192eb thru 0x01a1d5 (0xeeb bytes) 000147 1789083940.138806 read 0x02caa5 thru 0x03a37c (0xd8d8 bytes) 000148 1789083940.145992 MIRROR_RESYNC 3 mirrors 000150 1789083941.112332 read 0x012763 thru 0x01e862 (0xc100 bytes) 000153 1789083941.123378 read 0x017003 thru 0x01bb40 (0x4b3e bytes) 000154 1789083941.123838 read 0x02a245 thru 0x03504d (0xae09 bytes) 000155 1789083941.138187 read 0x02c03b thru 0x030d73 (0x4d39 bytes) 000157 1789083941.139016 write 0x01664b thru 0x018f48 (0x28fe bytes) 000158 1789083941.200604 mapwrite 0x03f743 thru 0x03ffff (0x8bd bytes) 000161 1789083941.224769 trunc from 0x040000 to 0x029360 (0x16c9f bytes) 000163 1789083941.251223 read 0x01dbe9 thru 0x027169 (0x9581 bytes) 000164 1789083941.251941 mapwrite 0x038acf thru 0x03cdb2 (0x42e4 bytes) 000165 1789083941.303577 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cdb3): Mirror 5: 0xb878a0b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdb3): Mirror 5: 0xb878a0b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdb3: skipped 000166 1789083941.669613 trunc from 0x03cdb3 to 0x03079c (0xc616 bytes) 000168 1789083941.732816 mapread 0x026dae thru 0x03079c (0x99ef bytes) 000170 1789083941.780545 trunc from 0x03079d to 0x0275b2 (0x91ea bytes) 000174 1789083941.808366 MIRROR_RESYNC 2 mirrors 000179 1789083942.599283 write 0x009d3e thru 0x00b327 (0x15ea bytes) 000182 1789083942.707937 read 0x018ccf thru 0x022b23 (0x9e55 bytes) 000183 1789083942.719355 write 0x00c698 thru 0x01ae1b (0xe784 bytes) 000184 1789083942.730573 trunc from 0x0275b3 to 0x014942 (0x12c70 bytes) 000185 1789083942.797924 write 0x034b5b thru 0x03a5af (0x5a55 bytes) 000186 1789083942.801815 trunc from 0x03a5b0 to 0x01e12c (0x1c483 bytes) 000188 1789083942.838833 mapwrite 0x00d4b1 thru 0x013d91 (0x68e1 bytes) 000189 1789083942.939679 mapread 0x019e4b thru 0x01e12c (0x42e2 bytes) 000191 1789083942.985211 mapwrite 0x01ff86 thru 0x02a9ae (0xaa29 bytes) 000192 1789083943.165593 write 0x000a11 thru 0x00b957 (0xaf47 bytes) 000193 1789083943.179451 mapread 0x02316a thru 0x02a9ae (0x7845 bytes) 000194 1789083943.182762 mapread 0x026365 thru 0x02a9ae (0x464a bytes) 000196 1789083943.188032 mapread 0x01e3e5 thru 0x025f21 (0x7b3d bytes) 000198 1789083943.192840 trunc from 0x02a9af to 0x00c03a (0x1e974 bytes) 000200 1789083943.223340 mapread 0x004b6f thru 0x00c03a (0x74cc bytes) 000201 1789083943.226524 write 0x00ea01 thru 0x014b76 (0x6176 bytes) 000204 1789083943.233327 read 0x008779 thru 0x01051b (0x7da3 bytes) 000209 1789083943.256159 read 0x00a417 thru 0x012742 (0x832c bytes) 000210 1789083943.266225 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000211 1789083943.555001 write 0x01a75d thru 0x01da74 (0x3318 bytes) 000215 1789083943.564656 write 0x027211 thru 0x02cb65 (0x5955 bytes) 000217 1789083943.578246 write 0x02569d thru 0x03099d (0xb301 bytes) 000219 1789083943.598243 mapwrite 0x00e20c thru 0x0143ff (0x61f4 bytes) 000222 1789083943.803569 read 0x00e34c thru 0x011cd9 (0x398e bytes) 000224 1789083943.809451 trunc from 0x03099e to 0x021326 (0xf677 bytes) 000226 1789083943.838398 mapread 0x010fd3 thru 0x01c920 (0xb94e bytes) 000229 1789083943.854226 write 0x016027 thru 0x01a393 (0x436d bytes) 000230 1789083943.859701 read 0x01137c thru 0x020b8f (0xf814 bytes) 000232 1789083943.870389 mapread 0x01f8cc thru 0x021326 (0x1a5b bytes) 000233 1789083943.872706 trunc from 0x021327 to 0x01fb2f (0x17f7 bytes) 000234 1789083943.894958 mapread 0x00fa00 thru 0x01982e (0x9e2f bytes) 000235 1789083943.899603 read 0x01336f thru 0x01d830 (0xa4c2 bytes) 000237 1789083943.901057 read 0x009caf thru 0x010f86 (0x72d8 bytes) 000240 1789083943.904611 trunc from 0x01fb30 to 0x00fa92 (0x1009d bytes) 000242 1789083943.927261 mapread 0x008c30 thru 0x00e372 (0x5743 bytes) 000245 1789083943.946448 MIRROR_EXTEND to 2 mirrors 000246 1789083944.709423 mapwrite 0x0194d7 thru 0x01e1d9 (0x4d03 bytes) 000248 1789083944.844463 write 0x02b078 thru 0x02cdb7 (0x1d40 bytes) truncating to largest ever: 0x3a922 000250 1789083944.848234 trunc from 0x02cdb8 to 0x03a921 (0xdb6a bytes) 000253 1789083944.877825 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000254 1789083945.025746 read 0x00218c thru 0x00b0c5 (0x8f3a bytes) 000255 1789083945.043852 MIRROR_EXTEND to 2 mirrors 000258 1789083945.912910 write 0x0265d1 thru 0x02bef3 (0x5923 bytes) 000259 1789083946.005185 mapwrite 0x006ff1 thru 0x0124b5 (0xb4c5 bytes) 000260 1789083946.164583 write 0x0075ac thru 0x00a2af (0x2d04 bytes) 000261 1789083946.166560 mapread 0x037a54 thru 0x03a921 (0x2ece bytes) 000262 1789083946.179577 trunc from 0x03a922 to 0x01d61c (0x1d305 bytes) 000263 1789083946.209454 mapwrite 0x0143e3 thru 0x01dc1e (0x983c bytes) 000264 1789083946.302360 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1dc1f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dc1f): Mirror 2: 0xb41db19b (stale) CRC-32 checksum value for chunk [0, 0x1dc1f): Mirror 1: 0x61446319 Mirror 3: 0x61446319 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dc1f: skipped 000266 1789083947.147223 write 0x00f082 thru 0x015a73 (0x69f2 bytes) 000267 1789083947.227586 MIRROR_RESYNC 3 mirrors 000269 1789083948.183866 read 0x018560 thru 0x01b09b (0x2b3c bytes) 000270 1789083948.193150 read 0x01c00d thru 0x01c221 (0x215 bytes) 000271 1789083948.208442 mapwrite 0x005e38 thru 0x00c4a9 (0x6672 bytes) 000272 1789083948.412886 write 0x02cee2 thru 0x03b06b (0xe18a bytes) 000273 1789083948.417140 mapread 0x032747 thru 0x036cba (0x4574 bytes) 000274 1789083948.420127 trunc from 0x03b06c to 0x033357 (0x7d14 bytes) 000277 1789083948.454212 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33358): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33358): Mirror 2: 0x45f739a9 (stale) Mirror 3: 0x45f739a9 (stale) CRC-32 checksum value for chunk [0, 0x33358): Mirror 1: 0x1ad155fd Mirror 4: 0x1ad155fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33358: skipped 000278 1789083949.954885 write 0x0320b6 thru 0x03aaf9 (0x8a44 bytes) 000279 1789083950.023118 mapwrite 0x00730e thru 0x012e37 (0xbb2a bytes) 000280 1789083950.118765 trunc from 0x03aafa to 0x024e3b (0x15cbe bytes) 000281 1789083950.163100 mapwrite 0x02d9bb thru 0x03846e (0xaab4 bytes) 000283 1789083950.244449 read 0x003181 thru 0x0061a1 (0x3021 bytes) truncating to largest ever: 0x3db8f 000285 1789083950.247373 trunc from 0x03846f to 0x03db8e (0x5720 bytes) 000286 1789083950.282264 write 0x035db6 thru 0x03e26d (0x84b8 bytes) 000289 1789083950.293069 MIRROR_RESYNC 4 mirrors 000291 1789083951.465885 write 0x03ac5c thru 0x03bacf (0xe74 bytes) 000292 1789083951.606898 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 2: 0xbfcdc1f (stale) Mirror 3: 0xbfcdc1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 2: 0xbfcdc1f (stale) Mirror 3: 0xbfcdc1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e26e: skipped 000293 1789083952.167831 trunc from 0x03e26e to 0x03a5e3 (0x3c8a bytes) 000294 1789083952.193420 mapwrite 0x016798 thru 0x020db6 (0xa61f bytes) 000295 1789083952.282294 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a5e4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5e4): Mirror 2: 0x3a3c40cd (stale) Mirror 3: 0x3a3c40cd (stale) CRC-32 checksum value for chunk [0, 0x3a5e4): Mirror 1: 0x997b3334 Mirror 4: 0x997b3334 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5e4: skipped 000298 1789083953.661010 mapread 0x032822 thru 0x03a5e3 (0x7dc2 bytes) 000300 1789083953.705560 write 0x03adc9 thru 0x03f11d (0x4355 bytes) 000304 1789083953.854939 read 0x024194 thru 0x031d8b (0xdbf8 bytes) 000305 1789083953.870558 read 0x01f6be thru 0x026f8c (0x78cf bytes) 000306 1789083953.902306 trunc from 0x03f11e to 0x00fde7 (0x2f336 bytes) 000307 1789083954.007675 read 0x007765 thru 0x00ef49 (0x77e5 bytes) 000308 1789083954.009133 write 0x023058 thru 0x031c1c (0xebc5 bytes) 000310 1789083954.016202 mapread 0x019fb7 thru 0x026eb0 (0xcefa bytes) 000311 1789083954.042555 read 0x001d8a thru 0x00d83f (0xbab6 bytes) 000312 1789083954.046411 read 0x01161d thru 0x01306a (0x1a4e bytes) 000316 1789083954.057450 trunc from 0x031c1d to 0x0213f1 (0x1082b bytes) 000318 1789083954.108207 trunc from 0x0213f2 to 0x005205 (0x1c1ec bytes) 000319 1789083954.164906 write 0x0001c7 thru 0x00bea3 (0xbcdd bytes) 000320 1789083954.166409 mapwrite 0x0067f2 thru 0x010e71 (0xa680 bytes) 000321 1789083954.394826 mapread 0x001be8 thru 0x00eb7a (0xcf93 bytes) 000322 1789083954.400290 mapwrite 0x01f490 thru 0x02edee (0xf95f bytes) 000323 1789083954.695863 write 0x027219 thru 0x03167e (0xa466 bytes) 000324 1789083954.704699 mapwrite 0x028b11 thru 0x02ec3a (0x612a bytes) 000325 1789083954.831408 mapwrite 0x031a07 thru 0x03e2cd (0xc8c7 bytes) 000326 1789083954.979425 read 0x017bbd thru 0x019256 (0x169a bytes) 000328 1789083954.981645 mapwrite 0x03139b thru 0x032a6c (0x16d2 bytes) 000329 1789083955.060846 write 0x01a738 thru 0x021bb5 (0x747e bytes) 000330 1789083955.065716 mapread 0x03ae36 thru 0x03e2cd (0x3498 bytes) 000331 1789083955.069092 read 0x010af6 thru 0x01bd0c (0xb217 bytes) 000332 1789083955.069938 mapread 0x0219d4 thru 0x030041 (0xe66e bytes) 000335 1789083955.075257 mapread 0x036947 thru 0x03e2cd (0x7987 bytes) 000337 1789083955.078115 mapread 0x00ceca thru 0x00e6f0 (0x1827 bytes) 000339 1789083955.080670 read 0x0312db thru 0x03a8e8 (0x960e bytes) 000340 1789083955.089413 trunc from 0x03e2ce to 0x00b0fe (0x331cf bytes) 000341 1789083955.135745 read 0x004620 thru 0x004a51 (0x432 bytes) 000342 1789083955.138738 mapwrite 0x021082 thru 0x02bfd9 (0xaf58 bytes) 000343 1789083955.306230 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfda): Mirror 3: 0x4d62d02 (stale) Mirror 4: 0xabe41956 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfda): Mirror 3: 0x4d62d02 (stale) Mirror 4: 0xabe41956 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfda: skipped 000344 1789083955.734831 trunc from 0x02bfda to 0x0299c0 (0x2619 bytes) 000345 1789083955.755559 mapread 0x006166 thru 0x0082c2 (0x215d bytes) 000346 1789083955.813818 mapwrite 0x00cbc9 thru 0x019612 (0xca4a bytes) 000348 1789083956.001282 mapread 0x027c0c thru 0x0299c0 (0x1db5 bytes) 000350 1789083956.020772 MIRROR_RESYNC 3 mirrors 000351 1789083957.229056 read 0x0224c6 thru 0x0299c0 (0x74fb bytes) 000352 1789083957.242827 trunc from 0x0299c1 to 0x00c86d (0x1d153 bytes) 000355 1789083957.356850 read 0x00ba92 thru 0x00c86d (0xddc bytes) 000357 1789083957.392400 read 0x0021e5 thru 0x0028a2 (0x6be bytes) 000358 1789083957.396591 mapwrite 0x036e3e thru 0x03ffff (0x91c2 bytes) 000359 1789083957.540434 mapwrite 0x01312d thru 0x014dde (0x1cb2 bytes) 000360 1789083957.642759 write 0x02f1f5 thru 0x03ab13 (0xb91f bytes) 000362 1789083957.650680 mapread 0x0131d0 thru 0x0193a5 (0x61d6 bytes) 000365 1789083957.663218 mapread 0x00b1c2 thru 0x00f147 (0x3f86 bytes) 000366 1789083957.668683 mapwrite 0x001b06 thru 0x00d5b0 (0xbaab bytes) 000367 1789083957.802862 trunc from 0x040000 to 0x0175bf (0x28a40 bytes) 000370 1789083957.846797 read 0x00d983 thru 0x0152dd (0x795b bytes) 000371 1789083957.856137 read 0x01191a thru 0x0175bf (0x5ca6 bytes) 000374 1789083957.865321 read 0x007f71 thru 0x01270d (0xa79d bytes) 000375 1789083957.867980 mapread 0x000ad7 thru 0x009f12 (0x943c bytes) 000376 1789083957.874871 mapread 0x007661 thru 0x00a60d (0x2fad bytes) 000378 1789083957.891290 mapread 0x005d0f thru 0x0092e9 (0x35db bytes) 000382 1789083957.902948 read 0x00b425 thru 0x015985 (0xa561 bytes) 000383 1789083957.905059 mapwrite 0x02cf64 thru 0x0305e9 (0x3686 bytes) 000384 1789083958.075973 mapwrite 0x022d62 thru 0x031c3d (0xeedc bytes) 000385 1789083958.238354 mapwrite 0x004382 thru 0x006356 (0x1fd5 bytes) 000387 1789083958.385209 mapread 0x02e3f7 thru 0x031c3d (0x3847 bytes) 000389 1789083958.406712 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31c3e): Mirror 3: 0x888ea04d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c3e): Mirror 3: 0x888ea04d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c3e: skipped 000390 1789083958.939450 mapread 0x02e021 thru 0x031c2f (0x3c0f bytes) 000391 1789083958.978585 mapwrite 0x011c4b thru 0x020631 (0xe9e7 bytes) 000392 1789083959.176792 mapwrite 0x0201e3 thru 0x0281e0 (0x7ffe bytes) 000393 1789083959.328105 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000395 1789083959.643433 mapread 0x0237cb thru 0x0309e7 (0xd21d bytes) 000397 1789083959.682650 mapread 0x003c1e thru 0x00be0d (0x81f0 bytes) 000398 1789083959.694523 mapwrite 0x01c1d3 thru 0x026a2d (0xa85b bytes) 000400 1789083959.769678 read 0x0129eb thru 0x02038b (0xd9a1 bytes) 000401 1789083959.777255 trunc from 0x031c3e to 0x00cea7 (0x24d96 bytes) 000402 1789083959.798104 mapwrite 0x03e70f thru 0x03ffff (0x18f1 bytes) 000403 1789083960.002276 mapwrite 0x02dbac thru 0x03933d (0xb792 bytes) 000404 1789083960.103234 read 0x004fbd thru 0x0081a3 (0x31e7 bytes) 000405 1789083960.105488 write 0x03c7f9 thru 0x03ffff (0x3807 bytes) 000406 1789083960.109508 write 0x02b842 thru 0x02ffd0 (0x478f bytes) 000409 1789083960.114000 trunc from 0x040000 to 0x01b8f3 (0x2470c bytes) 000413 1789083960.147503 read 0x00649e thru 0x00cfcd (0x6b30 bytes) 000414 1789083960.149170 mapwrite 0x01cb26 thru 0x02360f (0x6aea bytes) 000415 1789083960.260451 mapwrite 0x015355 thru 0x016d9c (0x1a48 bytes) 000417 1789083960.340530 MIRROR_EXTEND to 2 mirrors 000421 1789083961.039945 mapread 0x019033 thru 0x0220d0 (0x909e bytes) 000423 1789083961.096825 trunc from 0x023610 to 0x01e1de (0x5431 bytes) 000424 1789083961.285037 trunc from 0x01e1df to 0x026190 (0x7fb2 bytes) 000425 1789083961.317052 read 0x0005e9 thru 0x00a0aa (0x9ac2 bytes) 000427 1789083961.326813 mapread 0x01ef72 thru 0x026190 (0x721f bytes) 000428 1789083961.336613 write 0x006c19 thru 0x00cb85 (0x5f6d bytes) 000431 1789083961.343700 trunc from 0x026191 to 0x026e3b (0xcab bytes) 000434 1789083961.368559 trunc from 0x026e3c to 0x011aad (0x1538e bytes) 000435 1789083961.386202 trunc from 0x011aae to 0x00d94b (0x4162 bytes) 000437 1789083961.409019 mapread 0x009176 thru 0x00d94b (0x47d6 bytes) 000439 1789083961.421163 read 0x006755 thru 0x00d94b (0x71f7 bytes) 000441 1789083961.423204 trunc from 0x00d94c to 0x01eebb (0x11570 bytes) 000442 1789083961.436514 trunc from 0x01eebc to 0x033710 (0x14855 bytes) 000443 1789083961.456344 trunc from 0x033711 to 0x0128b1 (0x20e5f bytes) 000446 1789083961.474294 trunc from 0x0128b2 to 0x035d89 (0x234d8 bytes) 000448 1789083961.503139 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35d8a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d8a): Mirror 2: 0x20b23d04 (stale) CRC-32 checksum value for chunk [0, 0x35d8a): Mirror 1: 0x9331c734 Mirror 3: 0x9331c734 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d8a: skipped 000450 1789083963.315482 write 0x030a4e thru 0x03f19a (0xe74d bytes) 000451 1789083963.465473 mapwrite 0x013a70 thru 0x01e297 (0xa828 bytes) 000452 1789083963.636606 mapread 0x012f7b thru 0x018531 (0x55b7 bytes) 000453 1789083963.648245 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f19b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f19b): Mirror 2: 0x5c591d06 (stale) Mirror 3: 0x85c119b1 (stale) CRC-32 checksum value for chunk [0, 0x3f19b): Mirror 1: 0x5d52dd0c Mirror 4: 0x5d52dd0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f19b: skipped 000454 1789083965.681502 trunc from 0x03f19b to 0x02c4de (0x12cbc bytes) 000455 1789083965.848407 mapwrite 0x014630 thru 0x020b53 (0xc524 bytes) 000457 1789083966.097559 mapwrite 0x034369 thru 0x035215 (0xead bytes) 000458 1789083966.268497 mapwrite 0x01f316 thru 0x027254 (0x7f3f bytes) 000462 1789083966.376714 trunc from 0x035216 to 0x004b49 (0x306cc bytes) 000463 1789083966.412815 read 0x00225b thru 0x004b49 (0x28ef bytes) 000465 1789083966.434984 MIRROR_RESYNC 4 mirrors 000468 1789083967.830115 write 0x012aeb thru 0x019861 (0x6d77 bytes) 000469 1789083967.902712 read 0x012aee thru 0x019861 (0x6d74 bytes) 000470 1789083967.903428 mapread 0x0178d9 thru 0x019861 (0x1f89 bytes) 000471 1789083967.904489 write 0x02b009 thru 0x0389fa (0xd9f2 bytes) 000472 1789083967.911589 mapread 0x0129ad thru 0x01efad (0xc601 bytes) 000473 1789083967.931981 write 0x03184d thru 0x03e559 (0xcd0d bytes) 000474 1789083967.939159 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e55a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e55a): Mirror 2: 0xf5343d5b (stale) Mirror 3: 0xf5343d5b (stale) Mirror 4: 0xf5343d5b (stale) CRC-32 checksum value for chunk [0, 0x3e55a): Mirror 1: 0xc1382861 Mirror 5: 0xc1382861 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e55a: skipped 000476 1789083968.852581 trunc from 0x03e55a to 0x00b1e3 (0x33376 bytes) 000480 1789083968.965868 mapread 0x0047dc thru 0x00b1e3 (0x6a08 bytes) 000482 1789083969.022385 write 0x00cdc0 thru 0x01bead (0xf0ee bytes) 000484 1789083969.028078 write 0x018c44 thru 0x026558 (0xd915 bytes) 000485 1789083969.034755 mapwrite 0x020c43 thru 0x02aef4 (0xa2b2 bytes) 000486 1789083969.131228 mapread 0x011267 thru 0x013e72 (0x2c0c bytes) 000488 1789083969.133055 trunc from 0x02aef5 to 0x014602 (0x168f2 bytes) 000490 1789083969.149377 mapwrite 0x00d370 thru 0x00ec93 (0x1924 bytes) 000491 1789083969.178394 read 0x00ffd3 thru 0x014602 (0x4630 bytes) 000494 1789083969.180152 mapread 0x011bcf thru 0x0121a5 (0x5d7 bytes) 000496 1789083969.182150 read 0x010d16 thru 0x014602 (0x38ed bytes) 000497 1789083969.186056 MIRROR_RESYNC 5 mirrors 000498 1789083970.540757 write 0x01a985 thru 0x01f3e8 (0x4a64 bytes) 000500 1789083970.717013 mapread 0x0044cc thru 0x004afb (0x630 bytes) 000501 1789083970.726501 write 0x016471 thru 0x01c072 (0x5c02 bytes) 000502 1789083970.735125 mapread 0x01819f thru 0x0184f3 (0x355 bytes) 000503 1789083970.736100 mapread 0x012f46 thru 0x019f04 (0x6fbf bytes) 000504 1789083970.765217 mapread 0x01dfbc thru 0x01f3e8 (0x142d bytes) 000506 1789083970.768190 mapread 0x011721 thru 0x01f3e8 (0xdcc8 bytes) 000509 1789083970.773127 mapread 0x00e186 thru 0x01cdc6 (0xec41 bytes) 000511 1789083970.777889 write 0x038452 thru 0x03e4af (0x605e bytes) 000512 1789083970.784500 mapwrite 0x039361 thru 0x03aaa2 (0x1742 bytes) 000514 1789083970.956735 mapread 0x01c45f thru 0x023a17 (0x75b9 bytes) 000518 1789083970.990433 trunc from 0x03e4b0 to 0x0226cd (0x1bde2 bytes) 000519 1789083971.110458 MIRROR_RESYNC 5 mirrors 000522 1789083973.445067 write 0x01cf08 thru 0x01f697 (0x2790 bytes) 000523 1789083973.554984 read 0x01c8a0 thru 0x0226cd (0x5e2e bytes) 000524 1789083973.569492 read 0x002e6b thru 0x00e58a (0xb720 bytes) 000525 1789083973.580286 read 0x013f91 thru 0x0226cd (0xe73d bytes) 000527 1789083973.582040 mapwrite 0x038ae9 thru 0x03a7ac (0x1cc4 bytes) 000528 1789083973.691058 write 0x02a221 thru 0x035597 (0xb377 bytes) 000529 1789083973.693356 mapwrite 0x035644 thru 0x0395d0 (0x3f8d bytes) 000531 1789083973.763475 trunc from 0x03a7ad to 0x023230 (0x1757c bytes) 000534 1789083973.814492 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23231): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23231): Mirror 2: 0xc0927d0d (stale) Mirror 3: 0xc0927d0d (stale) Mirror 4: 0xc0927d0d (stale) Mirror 5: 0xc0927d0d (stale) CRC-32 checksum value for chunk [0, 0x23231): Mirror 1: 0x8855889e Mirror 6: 0x8855889e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23231: skipped 000535 1789083975.495266 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23231): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23231): Mirror 2: 0xc0927d0d (stale) Mirror 3: 0xc0927d0d (stale) Mirror 4: 0xc0927d0d (stale) Mirror 5: 0xc0927d0d (stale) CRC-32 checksum value for chunk [0, 0x23231): Mirror 1: 0x8855889e Mirror 6: 0x8855889e Mirror 7: 0x8855889e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23231: skipped 000536 1789083977.071109 write 0x00ee2d thru 0x01953e (0xa712 bytes) 000537 1789083977.181257 MIRROR_RESYNC 7 mirrors 000538 1789083978.789900 write 0x0119b0 thru 0x018ea1 (0x74f2 bytes) 000541 1789083978.890131 write 0x02f07b thru 0x03e8a5 (0xf82b bytes) 000542 1789083978.902124 write 0x03a72b thru 0x03fd76 (0x564c bytes) 000543 1789083978.914075 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd77): Mirror 2: 0xb50e098c (stale) Mirror 3: 0xb50e098c (stale) Mirror 5: 0xb50e098c (stale) Mirror 6: 0xb50e098c (stale) Mirror 7: 0xb50e098c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd77): Mirror 2: 0xb50e098c (stale) Mirror 3: 0xb50e098c (stale) Mirror 5: 0xb50e098c (stale) Mirror 6: 0xb50e098c (stale) Mirror 7: 0xb50e098c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd77: skipped 000544 1789083979.843390 MIRROR_RESYNC 6 mirrors 000546 1789083981.523872 read 0x004119 thru 0x0075d1 (0x34b9 bytes) 000548 1789083981.535084 mapread 0x0113a4 thru 0x01ee13 (0xda70 bytes) 000549 1789083981.584393 mapwrite 0x00153f thru 0x00c81c (0xb2de bytes) 000550 1789083982.012499 write 0x0236a6 thru 0x028e64 (0x57bf bytes) 000551 1789083982.037689 write 0x003c19 thru 0x0117cc (0xdbb4 bytes) 000554 1789083982.067248 mapread 0x01135b thru 0x01a520 (0x91c6 bytes) 000555 1789083982.095577 mapread 0x0147c7 thru 0x01e86f (0xa0a9 bytes) 000556 1789083982.107200 mapwrite 0x03a329 thru 0x03b725 (0x13fd bytes) 000557 1789083982.285336 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fd77): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd77): Mirror 2: 0xb38230c8 (stale) Mirror 3: 0xb38230c8 (stale) Mirror 5: 0xb38230c8 (stale) Mirror 6: 0xb38230c8 (stale) Mirror 7: 0xb38230c8 (stale) CRC-32 checksum value for chunk [0, 0x3fd77): Mirror 1: 0x6bc2547 Mirror 8: 0x6bc2547 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd77: skipped 000558 1789083983.518096 write 0x02639b thru 0x02a43c (0x40a2 bytes) 000559 1789083983.660051 trunc from 0x03fd77 to 0x014245 (0x2bb31 bytes) 000560 1789083983.737560 write 0x016c5a thru 0x0186c7 (0x1a6e bytes) 000561 1789083983.739020 write 0x00f200 thru 0x0133d6 (0x41d7 bytes) 000562 1789083983.747998 trunc from 0x0186c8 to 0x029331 (0x10c6a bytes) 000566 1789083983.769494 mapwrite 0x02705f thru 0x036cac (0xfc4e bytes) 000567 1789083983.899770 mapwrite 0x028fd6 thru 0x02bd15 (0x2d40 bytes) 000568 1789083983.951827 read 0x036b68 thru 0x036cac (0x145 bytes) 000569 1789083983.953778 trunc from 0x036cad to 0x015fa9 (0x20d03 bytes) 000571 1789083983.984409 mapread 0x0034d2 thru 0x0123fa (0xef29 bytes) 000572 1789083983.999855 read 0x00d7e2 thru 0x011233 (0x3a52 bytes) 000576 1789083984.003609 write 0x00be73 thru 0x00eff2 (0x3180 bytes) 000577 1789083984.005841 mapwrite 0x031d40 thru 0x03c278 (0xa539 bytes) 000579 1789083984.092782 write 0x031224 thru 0x036784 (0x5561 bytes) 000581 1789083984.097034 trunc from 0x03c279 to 0x017b9d (0x246db bytes) 000582 1789083984.118929 mapread 0x0036a3 thru 0x00c7de (0x913c bytes) 000583 1789083984.122070 read 0x005133 thru 0x0086b2 (0x3580 bytes) 000584 1789083984.123070 mapwrite 0x009884 thru 0x016e54 (0xd5d1 bytes) 000586 1789083984.198126 mapread 0x006ae8 thru 0x00d0f1 (0x660a bytes) 000587 1789083984.200130 mapread 0x009490 thru 0x00f7bf (0x6330 bytes) 000588 1789083984.202571 read 0x00256e thru 0x011d28 (0xf7bb bytes) 000589 1789083984.204102 trunc from 0x017b9e to 0x0307d2 (0x18c35 bytes) 000590 1789083984.221723 read 0x019d77 thru 0x0239e1 (0x9c6b bytes) 000591 1789083984.234588 mapwrite 0x0148ed thru 0x0198a5 (0x4fb9 bytes) 000592 1789083984.278331 write 0x030d24 thru 0x039408 (0x86e5 bytes) 000596 1789083984.283035 mapwrite 0x02f9e2 thru 0x034960 (0x4f7f bytes) 000598 1789083984.334952 trunc from 0x039409 to 0x018ef4 (0x20514 bytes) 000600 1789083984.356748 mapwrite 0x015ed7 thru 0x016dce (0xef8 bytes) 000601 1789083984.405002 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18ef5): Mirror 2: 0xb9bf91b4 (stale) Mirror 3: 0xb9bf91b4 (stale) Mirror 5: 0xb9bf91b4 (stale) Mirror 6: 0xb9bf91b4 (stale) Mirror 8: 0x87171137 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ef5): Mirror 2: 0xb9bf91b4 (stale) Mirror 3: 0xb9bf91b4 (stale) Mirror 5: 0xb9bf91b4 (stale) Mirror 6: 0xb9bf91b4 (stale) Mirror 8: 0x87171137 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ef5: skipped 000602 1789083984.814897 mapread 0x015015 thru 0x018ef4 (0x3ee0 bytes) 000608 1789083984.856874 mapwrite 0x011ca8 thru 0x017333 (0x568c bytes) 000609 1789083984.928786 trunc from 0x018ef5 to 0x02256b (0x9677 bytes) 000611 1789083984.960211 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2256c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2256c): Mirror 2: 0x653754a7 (stale) Mirror 3: 0x653754a7 (stale) Mirror 5: 0x653754a7 (stale) Mirror 6: 0x653754a7 (stale) Mirror 8: 0x47dfd11e (stale) CRC-32 checksum value for chunk [0, 0x2256c): Mirror 1: 0x279b770f Mirror 9: 0x279b770f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2256c: skipped 000612 1789083986.695584 trunc from 0x02256c to 0x024cef (0x2784 bytes) 000614 1789083986.828967 mapread 0x01e630 thru 0x024cef (0x66c0 bytes) 000615 1789083986.859235 MIRROR_RESYNC 7 mirrors 000616 1789083988.497144 MIRROR_EXTEND to 8 mirrors 000618 1789083989.226778 mapread 0x01e029 thru 0x024cef (0x6cc7 bytes) 000619 1789083989.250109 read 0x0201d9 thru 0x023b1a (0x3942 bytes) 000620 1789083989.250560 mapread 0x01d72d thru 0x02218f (0x4a63 bytes) 000621 1789083989.251318 write 0x02039d thru 0x028a98 (0x86fc bytes) 000623 1789083989.352247 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28a99): Mirror 2: 0xecf5a1b0 (stale) Mirror 3: 0xecf5a1b0 (stale) Mirror 6: 0xecf5a1b0 (stale) Mirror 8: 0xecf5a1b0 (stale) Mirror 9: 0xecf5a1b0 (stale) Mirror 10: 0xecf5a1b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28a99): Mirror 2: 0xecf5a1b0 (stale) Mirror 3: 0xecf5a1b0 (stale) Mirror 6: 0xecf5a1b0 (stale) Mirror 8: 0xecf5a1b0 (stale) Mirror 9: 0xecf5a1b0 (stale) Mirror 10: 0xecf5a1b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28a99: skipped 000624 1789083989.792410 trunc from 0x028a99 to 0x037713 (0xec7b bytes) 000625 1789083989.817308 MIRROR_RESYNC 7 mirrors 000626 1789083991.407524 mapwrite 0x00469d thru 0x009b55 (0x54b9 bytes) 000627 1789083991.568233 read 0x035ecf thru 0x037713 (0x1845 bytes) 000629 1789083991.579403 mapwrite 0x03db74 thru 0x03ffff (0x248c bytes) 000634 1789083991.662041 mapwrite 0x01ac27 thru 0x0275c1 (0xc99b bytes) 000635 1789083991.732710 mapread 0x016361 thru 0x01dd8d (0x7a2d bytes) 000636 1789083991.734176 read 0x006dff thru 0x00af4e (0x4150 bytes) 000639 1789083991.738063 mapwrite 0x0063bd thru 0x013966 (0xd5aa bytes) 000640 1789083991.795336 mapread 0x0153a2 thru 0x02099d (0xb5fc bytes) 000644 1789083991.801988 read 0x0077e3 thru 0x010737 (0x8f55 bytes) 000646 1789083991.805279 mapwrite 0x00628e thru 0x0094e0 (0x3253 bytes) 000649 1789083991.868316 mapwrite 0x01587a thru 0x01bfd5 (0x675c bytes) 000650 1789083991.922378 mapread 0x0069bc thru 0x00838a (0x19cf bytes) 000654 1789083991.930805 mapwrite 0x01ccf5 thru 0x021d4c (0x5058 bytes) 000655 1789083991.993678 read 0x01991e thru 0x0290d9 (0xf7bc bytes) 000656 1789083991.994482 mapread 0x02e563 thru 0x033b83 (0x5621 bytes) 000658 1789083992.000302 trunc from 0x040000 to 0x038be6 (0x7419 bytes) 000659 1789083992.031417 write 0x0047cf thru 0x01428d (0xfabf bytes) 000661 1789083992.035945 read 0x035a58 thru 0x037b22 (0x20cb bytes) 000663 1789083992.038860 trunc from 0x038be7 to 0x039445 (0x85f bytes) 000664 1789083992.061087 mapwrite 0x0286fc thru 0x037f6c (0xf871 bytes) 000667 1789083992.146304 write 0x0264f2 thru 0x027a37 (0x1546 bytes) 000673 1789083992.155833 mapwrite 0x02e3e7 thru 0x038342 (0x9f5c bytes) 000674 1789083992.218289 write 0x02fe74 thru 0x03c92c (0xcab9 bytes) 000675 1789083992.220609 trunc from 0x03c92d to 0x025c07 (0x16d25 bytes) 000676 1789083992.243300 mapread 0x02063e thru 0x024e09 (0x47cc bytes) 000677 1789083992.250102 MIRROR_RESYNC 7 mirrors 000678 1789083993.374832 MIRROR_EXTEND to 8 mirrors 000679 1789083994.394302 read 0x023316 thru 0x025c07 (0x28f2 bytes) 000680 1789083994.402140 mapread 0x015cb3 thru 0x01694f (0xc9d bytes) 000681 1789083994.414147 mapwrite 0x028fd8 thru 0x0305d2 (0x75fb bytes) 000683 1789083994.580378 mapread 0x01642d thru 0x018226 (0x1dfa bytes) 000684 1789083994.592457 write 0x03b775 thru 0x03ffff (0x488b bytes) 000686 1789083994.595072 mapread 0x019c92 thru 0x022553 (0x88c2 bytes) 000687 1789083994.602469 trunc from 0x040000 to 0x015620 (0x2a9df bytes) 000688 1789083994.625206 read 0x007c0b thru 0x00feaa (0x82a0 bytes) 000689 1789083994.634464 write 0x02315e thru 0x024070 (0xf13 bytes) 000690 1789083994.635807 mapread 0x010b99 thru 0x01dc6b (0xd0d3 bytes) 000691 1789083994.648731 MIRROR_RESYNC 8 mirrors 000692 1789083996.708837 trunc from 0x024071 to 0x03bf7c (0x17f0c bytes) 000693 1789083996.815209 mapwrite 0x0126d5 thru 0x01c05e (0x998a bytes) 000695 1789083996.913150 trunc from 0x03bf7d to 0x021964 (0x1a618 bytes) 000696 1789083996.944639 trunc from 0x021965 to 0x01b865 (0x60ff bytes) 000697 1789083996.963279 mapwrite 0x030c7e thru 0x03456b (0x38ee bytes) 000699 1789083997.048033 read 0x029557 thru 0x0314cd (0x7f77 bytes) 000700 1789083997.049323 write 0x000e0a thru 0x003a56 (0x2c4d bytes) 000701 1789083997.069009 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3456c): Mirror 3: 0xa0b3766b (stale) Mirror 6: 0xa0b3766b (stale) Mirror 8: 0xa0b3766b (stale) Mirror 9: 0xa0b3766b (stale) Mirror 10: 0xa0b3766b (stale) Mirror 11: 0xa0b3766b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3456c): Mirror 3: 0xa0b3766b (stale) Mirror 6: 0xa0b3766b (stale) Mirror 8: 0xa0b3766b (stale) Mirror 9: 0xa0b3766b (stale) Mirror 10: 0xa0b3766b (stale) Mirror 11: 0xa0b3766b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3456c: skipped 000702 1789083997.635205 mapread 0x00532a thru 0x0145c8 (0xf29f bytes) 000703 1789083997.681484 write 0x03c171 thru 0x03ffff (0x3e8f bytes) 000704 1789083997.682897 write 0x027925 thru 0x036b6e (0xf24a bytes) 000705 1789083997.698283 trunc from 0x040000 to 0x00f860 (0x3079f bytes) 000706 1789083997.723333 mapwrite 0x03852a thru 0x03ffff (0x7ad6 bytes) 000708 1789083997.822822 mapread 0x0085ca thru 0x015c4e (0xd685 bytes) 000711 1789083997.837526 mapwrite 0x016c1b thru 0x017d62 (0x1148 bytes) 000713 1789083997.896108 MIRROR_RESYNC 7 mirrors 000719 1789084000.112964 mapread 0x03fed6 thru 0x03ffff (0x12a bytes) 000720 1789084000.121795 mapread 0x007191 thru 0x00fe44 (0x8cb4 bytes) 000721 1789084000.146031 MIRROR_EXTEND to 8 mirrors 000722 1789084001.480746 MIRROR_RESYNC 8 mirrors 000725 1789084002.083923 mapread 0x03ef01 thru 0x03ffff (0x10ff bytes) 000726 1789084002.096093 read 0x029b96 thru 0x031492 (0x78fd bytes) 000727 1789084002.106381 mapwrite 0x006bec thru 0x00b77d (0x4b92 bytes) 000729 1789084002.308826 mapwrite 0x02b914 thru 0x038e0c (0xd4f9 bytes) 000730 1789084002.377781 mapread 0x02dcec thru 0x03d2da (0xf5ef bytes) 000731 1789084002.390144 mapread 0x01c1c1 thru 0x01ffaf (0x3def bytes) 000734 1789084002.422584 read 0x00a7d6 thru 0x016979 (0xc1a4 bytes) 000735 1789084002.434751 write 0x00a44f thru 0x0180db (0xdc8d bytes) 000736 1789084002.443758 MIRROR_RESYNC 8 mirrors 000737 1789084003.888502 trunc from 0x040000 to 0x02878f (0x17870 bytes) 000738 1789084003.963608 mapread 0x006b0f thru 0x00a40c (0x38fe bytes) 000741 1789084003.991875 trunc from 0x028790 to 0x004ad1 (0x23cbe bytes) 000742 1789084004.040005 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4ad2): Mirror 3: 0xf50fd13b (stale) Mirror 6: 0xf50fd13b (stale) Mirror 8: 0xf50fd13b (stale) Mirror 9: 0xf50fd13b (stale) Mirror 10: 0xf50fd13b (stale) Mirror 11: 0xf50fd13b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4ad2): Mirror 3: 0xf50fd13b (stale) Mirror 6: 0xf50fd13b (stale) Mirror 8: 0xf50fd13b (stale) Mirror 9: 0xf50fd13b (stale) Mirror 10: 0xf50fd13b (stale) Mirror 11: 0xf50fd13b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4ad2: skipped 000745 1789084004.529230 write 0x010bef thru 0x013520 (0x2932 bytes) 000747 1789084004.536233 write 0x00a1cb thru 0x00a6ef (0x525 bytes) 000750 1789084004.547919 write 0x00a51f thru 0x019f59 (0xfa3b bytes) 000751 1789084004.551208 mapwrite 0x013e56 thru 0x016cfa (0x2ea5 bytes) 000752 1789084004.603526 mapread 0x001a68 thru 0x00fcbf (0xe258 bytes) 000753 1789084004.641199 trunc from 0x019f5a to 0x029601 (0xf6a8 bytes) 000756 1789084004.673545 read 0x003346 thru 0x004a05 (0x16c0 bytes) 000759 1789084004.688652 read 0x00a0e1 thru 0x00b085 (0xfa5 bytes) 000760 1789084004.690575 write 0x02315f thru 0x026df3 (0x3c95 bytes) 000763 1789084004.735573 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29602): Mirror 3: 0x4fc49bca (stale) Mirror 6: 0x4fc49bca (stale) Mirror 8: 0x4fc49bca (stale) Mirror 9: 0x4fc49bca (stale) Mirror 11: 0x4fc49bca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29602): Mirror 3: 0x4fc49bca (stale) Mirror 6: 0x4fc49bca (stale) Mirror 8: 0x4fc49bca (stale) Mirror 9: 0x4fc49bca (stale) Mirror 11: 0x4fc49bca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29602: skipped 000764 1789084005.443798 trunc from 0x029602 to 0x019476 (0x1018b bytes) 000767 1789084005.494817 trunc from 0x019477 to 0x029717 (0x102a1 bytes) 000772 1789084005.533726 read 0x00e4f2 thru 0x013c38 (0x5747 bytes) 000773 1789084005.554231 mapread 0x023eba thru 0x02424a (0x391 bytes) 000774 1789084005.569219 trunc from 0x029718 to 0x01f82d (0x9eea bytes) 000775 1789084005.594989 write 0x01505b thru 0x01dbbe (0x8b64 bytes) 000776 1789084005.614421 mapwrite 0x00ce91 thru 0x0147ac (0x791c bytes) 000779 1789084005.719883 read 0x0171b3 thru 0x01f5b2 (0x8400 bytes) 000780 1789084005.742331 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f82e): Mirror 6: 0x4e8b50cf (stale) Mirror 8: 0x4e8b50cf (stale) Mirror 9: 0x4e8b50cf (stale) Mirror 11: 0x4e8b50cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f82e): Mirror 6: 0x4e8b50cf (stale) Mirror 8: 0x4e8b50cf (stale) Mirror 9: 0x4e8b50cf (stale) Mirror 11: 0x4e8b50cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f82e: skipped 000781 1789084006.334608 mapwrite 0x0245ec thru 0x02f3ef (0xae04 bytes) 000782 1789084006.525551 mapwrite 0x02969e thru 0x02cd1b (0x367e bytes) 000788 1789084006.668879 MIRROR_RESYNC 5 mirrors 000789 1789084007.716843 mapread 0x0089a4 thru 0x016ff6 (0xe653 bytes) 000790 1789084007.754975 mapread 0x0248ca thru 0x02f3ef (0xab26 bytes) 000793 1789084007.772551 read 0x0250fd thru 0x02f3ef (0xa2f3 bytes) 000794 1789084007.773185 mapread 0x001434 thru 0x001b11 (0x6de bytes) 000795 1789084007.783138 MIRROR_RESYNC 5 mirrors 000796 1789084008.268932 mapwrite 0x03baa2 thru 0x03e1a9 (0x2708 bytes) 000798 1789084008.425166 mapread 0x026cbf thru 0x031292 (0xa5d4 bytes) 000799 1789084008.445941 mapwrite 0x034d11 thru 0x03a0eb (0x53db bytes) 000800 1789084008.490202 mapread 0x00d4ec thru 0x01ccc9 (0xf7de bytes) 000801 1789084008.524782 write 0x033a54 thru 0x033caa (0x257 bytes) 000804 1789084008.527900 read 0x0207e2 thru 0x02a4ef (0x9d0e bytes) 000805 1789084008.529378 mapread 0x034ea0 thru 0x03e1a9 (0x930a bytes) 000806 1789084008.536017 MIRROR_RESYNC 5 mirrors 000807 1789084009.498205 trunc from 0x03e1aa to 0x017490 (0x26d19 bytes) 000808 1789084009.570007 trunc from 0x017491 to 0x0036f1 (0x13d9f bytes) 000809 1789084009.600762 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36f2): Mirror 8: 0xca89b8ac (stale) Mirror 9: 0xca89b8ac (stale) Mirror 11: 0xca89b8ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f2): Mirror 8: 0xca89b8ac (stale) Mirror 9: 0xca89b8ac (stale) Mirror 11: 0xca89b8ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f2: skipped 000810 1789084009.918003 read 0x0010a7 thru 0x0036f1 (0x264b bytes) 000812 1789084009.931000 trunc from 0x0036f2 to 0x00bd4e (0x865d bytes) 000813 1789084009.948546 write 0x033f26 thru 0x037c5c (0x3d37 bytes) 000816 1789084009.953680 write 0x002254 thru 0x004869 (0x2616 bytes) 000817 1789084009.966942 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c5d): Mirror 8: 0x6c38bb57 (stale) Mirror 11: 0x6c38bb57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c5d): Mirror 8: 0x6c38bb57 (stale) Mirror 11: 0x6c38bb57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c5d: skipped 000821 1789084010.384737 write 0x033638 thru 0x03ffff (0xc9c8 bytes) 000823 1789084010.395391 write 0x01edfb thru 0x02b0d9 (0xc2df bytes) 000824 1789084010.410360 trunc from 0x040000 to 0x023d3b (0x1c2c4 bytes) 000825 1789084010.437866 trunc from 0x023d3c to 0x0176ea (0xc651 bytes) 000826 1789084010.465029 trunc from 0x0176eb to 0x026fb9 (0xf8cf bytes) 000828 1789084010.496876 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26fba): Mirror 8: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26fba): Mirror 8: 0x506f3e4d (stale) Mirror 11: 0x506f3e4d (stale) CRC-32 checksum value for chunk [0, 0x26fba): Mirror 1: 0x83a8572 Mirror 12: 0x83a8572 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26fba: skipped 000829 1789084011.930482 write 0x0233f7 thru 0x029af1 (0x66fb bytes) 000830 1789084012.137221 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29af2): Mirror 11: 0x261fb0a (stale) Mirror 12: 0xb192b195 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29af2): Mirror 11: 0x261fb0a (stale) Mirror 12: 0xb192b195 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29af2: skipped 000831 1789084012.505581 mapwrite 0x008bbb thru 0x0117c4 (0x8c0a bytes) 000832 1789084012.613817 mapread 0x022800 thru 0x029af1 (0x72f2 bytes) 000834 1789084012.641967 write 0x021f29 thru 0x024087 (0x215f bytes) 000835 1789084012.643775 mapwrite 0x02423c thru 0x0287c7 (0x458c bytes) 000836 1789084012.708814 write 0x02f1a2 thru 0x03b3e6 (0xc245 bytes) 000841 1789084012.721933 read 0x0350a6 thru 0x037bce (0x2b29 bytes) 000844 1789084012.725796 trunc from 0x03b3e7 to 0x026918 (0x14ace bytes) 000845 1789084012.748251 trunc from 0x026919 to 0x0284d2 (0x1bba bytes) 000847 1789084012.805920 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x284d3): Mirror 11: 0x8cc57e01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x284d3): Mirror 11: 0x8cc57e01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x284d3: skipped 000848 1789084013.171326 MIRROR_RESYNC 2 mirrors 000850 1789084013.958099 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000852 1789084014.110983 MIRROR_EXTEND to 2 mirrors 000853 1789084014.836166 trunc from 0x0284d3 to 0x01d530 (0xafa2 bytes) 000856 1789084014.935870 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d531): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d531): Mirror 2: 0xff066333 (stale) CRC-32 checksum value for chunk [0, 0x1d531): Mirror 1: 0xff066333 Mirror 3: 0xff066333 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d531: skipped 000857 1789084015.866538 write 0x029c90 thru 0x02a8c4 (0xc35 bytes) truncating to largest ever: 0x3eecc 000858 1789084016.027039 trunc from 0x02a8c5 to 0x03eecb (0x14607 bytes) 000859 1789084016.068301 trunc from 0x03eecc to 0x02067f (0x1e84c bytes) 000861 1789084016.091924 mapread 0x01bb47 thru 0x02067f (0x4b39 bytes) 000863 1789084016.126742 mapread 0x00f24f thru 0x0144a4 (0x5256 bytes) 000864 1789084016.150954 write 0x02a0b5 thru 0x02d2bc (0x3208 bytes) 000865 1789084016.152413 mapread 0x018303 thru 0x02570d (0xd40b bytes) 000866 1789084016.183124 mapread 0x00a58a thru 0x019e50 (0xf8c7 bytes) 000869 1789084016.189932 write 0x0372cd thru 0x03ffff (0x8d33 bytes) 000871 1789084016.200895 trunc from 0x040000 to 0x019242 (0x26dbd bytes) 000872 1789084016.246143 mapread 0x007bba thru 0x013dea (0xc231 bytes) 000875 1789084016.267073 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19243): Mirror 3: 0xb1873bb7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19243): Mirror 3: 0xb1873bb7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19243: skipped 000876 1789084016.801520 write 0x00045c thru 0x004e59 (0x49fe bytes) 000878 1789084016.837953 write 0x03ad4b thru 0x03ffff (0x52b5 bytes) 000879 1789084016.843846 mapread 0x015d9e thru 0x024602 (0xe865 bytes) 000880 1789084016.885537 read 0x00fb23 thru 0x016586 (0x6a64 bytes) 000881 1789084016.894682 trunc from 0x040000 to 0x03cb0a (0x34f5 bytes) 000884 1789084016.924940 write 0x0052cb thru 0x008781 (0x34b7 bytes) 000885 1789084016.938612 read 0x01f111 thru 0x02c071 (0xcf61 bytes) 000887 1789084016.957137 read 0x0260a2 thru 0x02e7df (0x873e bytes) 000888 1789084016.959342 read 0x0196df thru 0x01d36a (0x3c8c bytes) 000889 1789084016.961949 mapwrite 0x018f7e thru 0x0243ae (0xb431 bytes) 000890 1789084017.041673 mapread 0x02ca5b thru 0x0325ad (0x5b53 bytes) 000891 1789084017.043236 mapwrite 0x03462c thru 0x03ee42 (0xa817 bytes) 000894 1789084017.171263 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000895 1789084017.398258 read 0x02929e thru 0x02eac8 (0x582b bytes) 000896 1789084017.426764 MIRROR_EXTEND to 2 mirrors 000898 1789084018.107088 trunc from 0x03ee43 to 0x03994e (0x54f4 bytes) 000900 1789084018.300895 mapwrite 0x0395f5 thru 0x03ffff (0x6a0b bytes) 000902 1789084018.496332 read 0x03a661 thru 0x03ffff (0x599f bytes) 000904 1789084018.507199 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000905 1789084018.728006 MIRROR_EXTEND to 2 mirrors 000906 1789084019.701103 read 0x00e945 thru 0x01151c (0x2bd8 bytes) 000907 1789084019.739399 mapwrite 0x011811 thru 0x01dbdb (0xc3cb bytes) 000908 1789084020.013267 mapwrite 0x0084c7 thru 0x00d602 (0x513c bytes) 000911 1789084020.069817 mapwrite 0x0003e7 thru 0x005912 (0x552c bytes) 000913 1789084020.128031 mapwrite 0x012132 thru 0x01c88a (0xa759 bytes) 000917 1789084020.186068 mapread 0x03e219 thru 0x03ffff (0x1de7 bytes) 000918 1789084020.199722 mapread 0x0153e4 thru 0x0226d6 (0xd2f3 bytes) 000919 1789084020.201352 write 0x016a4d thru 0x023192 (0xc746 bytes) 000920 1789084020.207053 mapread 0x03c927 thru 0x03ffff (0x36d9 bytes) 000921 1789084020.208977 write 0x0230a2 thru 0x02c27e (0x91dd bytes) 000924 1789084020.239236 MIRROR_RESYNC 2 mirrors 000925 1789084021.015461 trunc from 0x040000 to 0x01d7cc (0x22833 bytes) 000927 1789084021.132421 trunc from 0x01d7cd to 0x02b907 (0xe13b bytes) 000928 1789084021.156434 write 0x01c144 thru 0x02bcde (0xfb9b bytes) 000929 1789084021.169036 read 0x007f24 thru 0x0112e5 (0x93c2 bytes) 000930 1789084021.179086 read 0x01f5a4 thru 0x020b82 (0x15df bytes) 000932 1789084021.180839 mapwrite 0x00a218 thru 0x013821 (0x960a bytes) 000933 1789084021.246248 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bcdf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcdf): Mirror 2: 0xb1ddcceb (stale) CRC-32 checksum value for chunk [0, 0x2bcdf): Mirror 1: 0x5e74ba7c Mirror 3: 0x5e74ba7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcdf: skipped 000936 1789084022.509451 mapread 0x02973d thru 0x02bcde (0x25a2 bytes) 000937 1789084022.564549 trunc from 0x02bcdf to 0x02097f (0xb35f bytes) 000938 1789084022.666858 MIRROR_RESYNC 3 mirrors 000940 1789084023.412866 mapwrite 0x020c6b thru 0x02afe4 (0xa37a bytes) 000941 1789084023.595809 mapwrite 0x026823 thru 0x02d729 (0x6f07 bytes) 000942 1789084023.669035 mapread 0x027182 thru 0x02b84c (0x46cb bytes) 000944 1789084023.671882 read 0x0226a4 thru 0x02d729 (0xb086 bytes) 000945 1789084023.673181 read 0x0277b3 thru 0x02903d (0x188b bytes) 000946 1789084023.674545 mapwrite 0x010e04 thru 0x01b2a1 (0xa49e bytes) 000947 1789084023.755057 trunc from 0x02d72a to 0x03739f (0x9c76 bytes) 000949 1789084023.783606 trunc from 0x0373a0 to 0x02a406 (0xcf99 bytes) 000953 1789084023.812939 mapwrite 0x031cf3 thru 0x036323 (0x4631 bytes) 000955 1789084023.909064 mapwrite 0x03e926 thru 0x03ffff (0x16da bytes) 000956 1789084023.992780 read 0x028093 thru 0x0330e7 (0xb055 bytes) 000958 1789084024.005153 mapwrite 0x034aa8 thru 0x03cd1f (0x8278 bytes) 000961 1789084024.089943 trunc from 0x040000 to 0x011d19 (0x2e2e6 bytes) 000962 1789084024.106442 mapwrite 0x0383b9 thru 0x03b130 (0x2d78 bytes) 000964 1789084024.186982 mapwrite 0x0164b8 thru 0x01c7b9 (0x6302 bytes) 000965 1789084024.304306 trunc from 0x03b131 to 0x03aa44 (0x6ec bytes) 000967 1789084024.332477 mapwrite 0x01329c thru 0x01ace7 (0x7a4c bytes) 000968 1789084024.396324 mapread 0x038bb6 thru 0x03aa44 (0x1e8f bytes) 000972 1789084024.399808 trunc from 0x03aa45 to 0x016bfc (0x23e48 bytes) 000977 1789084024.501062 mapread 0x00931d thru 0x010675 (0x7359 bytes) 000978 1789084024.521975 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16bfd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16bfd): Mirror 2: 0x6ff17676 (stale) Mirror 3: 0x6ff17676 (stale) CRC-32 checksum value for chunk [0, 0x16bfd): Mirror 1: 0xcbad8a11 Mirror 4: 0xcbad8a11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16bfd: skipped 000979 1789084026.417859 MIRROR_RESYNC 4 mirrors 000980 1789084027.515814 mapread 0x0098af thru 0x00dbda (0x432c bytes) 000981 1789084027.536829 write 0x011e65 thru 0x020a45 (0xebe1 bytes) 000986 1789084027.608591 write 0x03ddcb thru 0x03ef3c (0x1172 bytes) 000988 1789084027.610740 trunc from 0x03ef3d to 0x03043c (0xeb00 bytes) 000990 1789084027.655962 write 0x00378e thru 0x0038ca (0x13d bytes) 000991 1789084027.671980 trunc from 0x03043d to 0x00b326 (0x25116 bytes) 000994 1789084027.716730 mapread 0x0038dd thru 0x00b326 (0x7a4a bytes) 000995 1789084027.729211 mapwrite 0x028e81 thru 0x029b0d (0xc8d bytes) 000997 1789084027.811433 read 0x021fe4 thru 0x029b0d (0x7b2a bytes) 001000 1789084027.814838 mapread 0x00c455 thru 0x00fadc (0x3688 bytes) All operations completed A-OK! PASS 70a (119s) == sanity-flr test 71: check mirror extend parent fid ==== 19:47:14 (1789084034) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 19:47:18 (1789084038) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224983 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237259 s, 4.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 19:47:28 (1789084048) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4756640 .. Extending file size to 1836928 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5570720 .. Extending file size to 3226240 .. Extending file size to 4091712 .. Extending file size to 6431840 .. Extending file size to 5605888 .. Extending file size to 7535424 .. Extending file size to 7464416 .. Extending file size to 7717280 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7853888 .. Extending file size to 7999296 .. Extending file size to 8400448 .. Extending file size to 8005120 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8091104 .. Extending file size to 8650208 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57292 57293 57295 57296 57298 done 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) 192.168.202.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:48:54 (1789084134) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1124704 .. Extending file size to 3231008 .. done Extending file size to 5935712 .. Extending file size to 3094880 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8430976 .. Extending file size to 7364512 .. Extending file size to 9107136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8561088 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8771904 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58786 58787 58789 58790 58791 done 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) 192.168.202.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:50:22 (1789084222) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112129 s, 18.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.34194 s, 6.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe22b1466 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:50:44 (1789084244) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0959298 s, 21.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:50:52 (1789084252) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:51:02 (1789084262) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24188 s, 8.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:51:14 (1789084274) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.406128 s, 5.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:51:25 (1789084285) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.355444 s, 5.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:51:59 (1789084319) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251914 s, 4.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:53:03 (1789084383) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23153 s, 9.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:54:09 (1789084449) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. Creating new pool test_204f oleg248-server: Pool lustre.test_204f created Adding targets to pool oleg248-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg248-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.256809 s, 8.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg248-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg248-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg248-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:55:35 (1789084535) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:55:43 (1789084543) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:55:54 (1789084554) oleg248-server: Pool lustre.test_206 created Adding targets to pool oleg248-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg248-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg248-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg248-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg248-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:56:21 (1789084581) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785241 s, 13.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315464 s, 33.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1d12c5e0 Mirror 3: 0x1d12c5e0 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:56:33 (1789084593) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:56:38 (1789084598) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:56:42 (1789084602) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.117044 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x990000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x18c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x18c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x18c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x830000 Read this (4 bytes): BBBB 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:57:22 (1789084642) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0447565 s, 183 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:57:31 (1789084651) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483794 s, 21.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834453 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 19:57:33.000000000 -0400 Modify: 2026-09-10 19:57:33.000000000 -0400 Change: 2026-09-10 19:57:35.000000000 -0400 Birth: 2026-09-10 19:57:33.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:57:40 (1789084660) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:57:45 (1789084665) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.601946 s, 68.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.110142 s, 37.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 10 19:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x217:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 10 19:57 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:57:55 (1789084675) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:58:06 (1789084686) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1985 sec =========== 19:58:19 (1789084699) === sanity-flr: start cleanup 19:58:21 (1789084701) === === sanity-flr: finish cleanup 19:58:29 (1789084709) ===