-----============= acceptance-small: sanity-flr ============----- Tue Jun 9 03:21:35 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg627-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:21:57 (1780989717) === oleg627-client.virtnet: executing check_config_client /mnt/lustre oleg627-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg627-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff96b2614a2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff96b2614a2800.idle_timeout=debug disable quota as required oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:22:30 (1780989750) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7575] == sanity-flr test 0a: lfs mirror create with -N option == 03:22:32 (1780989752) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:22:51 (1780989771) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:22:55 (1780989775) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:23:00 (1780989780) PASS 0d (34s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:23:34 (1780989814) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:23:37 (1780989817) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 03:23:41 (1780989821) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff96b2614a2800.stats=clear osc.lustre-OST0001-osc-ffff96b2614a2800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:23:50 (1780989830) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:24:02 (1780989842) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:24:11 (1780989851) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:24:14 (1780989854) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:24:23 (1780989863) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 03:24:34 (1780989874) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:24:42 (1780989882) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:24:46 (1780989886) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:24:55 (1780989895) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:25:06 (1780989906) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:25:13 (1780989913) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.8904 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174062 s, 6.0 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:25:26 (1780989926) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.444542 s, 30.7 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:25:36 (1780989936) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:25:47 (1780989947) Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg627-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:26:56 (1780990016) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.364837 s, 17.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:27:40 (1780990060) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg627-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:29:02 (1780990142) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.67607 s, 9.2 MB/s -rw-r--r-- 1 root root 52428800 Jun 9 03:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.729s user 0m0.000s sys 0m0.218s Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff96b2614a2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.310s user 0m0.001s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.519s user 0m0.000s sys 0m0.183s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg627-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff96b2614a2800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:30:40 (1780990240) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:30:44 (1780990244) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:30:48 (1780990248) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 03:30:52 (1780990252) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 03:31:01 (1780990261) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.708585 s, 17.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:31:16 (1780990276) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326908 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370136 s, 28.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:31:24 (1780990284) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:31:33 (1780990293) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000828187 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118274 s, 866 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000821531 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000813053 s, 1.3 MB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 03:31:45 (1780990305) create test files with size 15747477 .. 18b1d09f7fb1e8f9d01d7f254e9f5687 - 7e37e8a7a4828aed11cc996696de00bd - a3fc6cac0a36a4884ed5f4036a7a4d92 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg627-client.virtnet /mnt/lustre (opts:) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 03:32:13 (1780990333) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:32:29 (1780990349) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:32:36 (1780990356) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164633 s, 6.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364684 s, 28.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 03:32:46 (1780990366) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139037 s, 15.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.469701 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.381266 s, 5.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (22s) == sanity-flr test 42: lfs mirror verify ================= 03:33:08 (1780990388) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:33:12 (1780990392) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:33:17 (1780990397) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394579 s, 26.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 03:33:26 (1780990406) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:33:29 (1780990409) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:33:41 (1780990421) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.342202 s, 30.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg627-client.virtnet /mnt/lustre (opts:) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:33:55 (1780990435) llite.lustre-ffff96b2782b6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.40126 s, 26.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg627-client.virtnet /mnt/lustre (opts:) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff96b2457b6000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:34:12 (1780990452) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (49s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:35:01 (1780990501) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:35:44 (1780990544) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 03:35:47 (1780990547) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.316504 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.190347 s, 16.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9997ef4b (stale) Mirror 2: 0x9997ef4b (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 68467fc2ee947158d682692d1dae7c64 - mirror 1 checksum: 1310781b136e01336d3cd92648551a9d - mirror 2 checksum: 68467fc2ee947158d682692d1dae7c64 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 9 03:35:51 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 9 03:35:51 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 68467fc2ee947158d682692d1dae7c64 - mirror 1 checksum: 1310781b136e01336d3cd92648551a9d - mirror 2 checksum: 1310781b136e01336d3cd92648551a9d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 9 03:35:51 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 9 03:35:51 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:36:01 (1780990561) Failing mds1 on oleg627-server Stopping /mnt/lustre-mds1 (opts:) on oleg627-server 03:36:08 (1780990568) shut down facet: mds1 facet_host: oleg627-server facet_failover_host: oleg627-server Failover mds1 to oleg627-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0000 03:36:41 (1780990601) targets are mounted 03:36:41 (1780990601) facet_failover done oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827365 s, 12.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:37:02 (1780990622) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:37:06 (1780990626) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:37:09 (1780990629) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.01605 s, 1.0 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:37:18 (1780990638) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:37:22 (1780990642) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:37:25 (1780990645) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:37:53 (1780990673) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:38:25 (1780990705) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:38:51 (1780990731) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:39:17 (1780990757) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 03:39:27 (1780990767) mirror create pid 53551 mirror split pid 53552 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7730: 53551 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53552 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 03:40:36 (1780990836) Seed set to 838 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1780990839.106589 mapwrite 0x037d21 thru 0x03ffff (0x82df bytes) 000004 1780990839.311948 mapwrite 0x030c8f thru 0x03cd02 (0xc074 bytes) truncating to largest ever: 0xeaf 000005 1780990839.459613 trunc from 0x040000 to 0x000eae (0x3f151 bytes) 000007 1780990839.502746 read 0x0003f8 thru 0x000eae (0xab7 bytes) truncating to largest ever: 0x12128 000009 1780990839.515726 trunc from 0x000eaf to 0x012127 (0x11279 bytes) 000010 1780990839.542030 write 0x01f80a thru 0x021a05 (0x21fc bytes) 000011 1780990839.544121 read 0x0094b1 thru 0x00df9a (0x4aea bytes) 000015 1780990839.563000 mapwrite 0x01af4d thru 0x01c176 (0x122a bytes) 000018 1780990839.664691 mapread 0x000645 thru 0x00c531 (0xbeed bytes) 000019 1780990839.687067 MIRROR_EXTEND to 2 mirrors 000020 1780990840.851856 read 0x000688 thru 0x00d609 (0xcf82 bytes) 000021 1780990840.885427 write 0x01264a thru 0x014354 (0x1d0b bytes) 000022 1780990841.037669 trunc from 0x021a06 to 0x00b9b4 (0x16051 bytes) 000023 1780990841.076541 mapread 0x000360 thru 0x004b18 (0x47b9 bytes) 000024 1780990841.103519 mapread 0x001239 thru 0x002c66 (0x1a2e bytes) 000027 1780990841.115557 mapread 0x00a5bd thru 0x00b9b4 (0x13f8 bytes) 000029 1780990841.152514 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) 000031 1780990841.515532 write 0x0293c5 thru 0x034d76 (0xb9b2 bytes) 000032 1780990841.519330 trunc from 0x034d77 to 0x00cb4a (0x2822c bytes) 000034 1780990841.543265 mapread 0x008b0e thru 0x00cb4a (0x403d bytes) 000036 1780990841.567460 read 0x000b1f thru 0x00cb4a (0xc02c bytes) truncating to largest ever: 0x1c281 000039 1780990841.575350 trunc from 0x00cb4b to 0x01c280 (0xf736 bytes) 000040 1780990841.605638 mapwrite 0x00797e thru 0x00ce40 (0x54c3 bytes) 000044 1780990841.671954 mapread 0x0192b8 thru 0x01c280 (0x2fc9 bytes) truncating to largest ever: 0x1cae7 000046 1780990841.690284 trunc from 0x01c281 to 0x01cae6 (0x866 bytes) 000047 1780990841.718325 read 0x000c81 thru 0x00fc4d (0xefcd bytes) 000048 1780990841.731189 write 0x03c6a0 thru 0x03ffff (0x3960 bytes) 000050 1780990841.734481 read 0x03ebcb thru 0x03eda0 (0x1d6 bytes) 000051 1780990841.735573 read 0x022447 thru 0x024e90 (0x2a4a bytes) 000054 1780990841.762187 MIRROR_EXTEND to 2 mirrors 000055 1780990842.796867 read 0x0240f4 thru 0x025c65 (0x1b72 bytes) 000056 1780990842.816999 mapread 0x0230cc thru 0x023818 (0x74d bytes) 000058 1780990842.872283 trunc from 0x040000 to 0x00a03f (0x35fc0 bytes) truncating to largest ever: 0x244a9 000061 1780990843.065790 trunc from 0x00a040 to 0x0244a8 (0x1a469 bytes) truncating to largest ever: 0x33c09 000062 1780990843.118745 trunc from 0x0244a9 to 0x033c08 (0xf760 bytes) 000063 1780990843.142497 mapwrite 0x037393 thru 0x03863a (0x12a8 bytes) 000064 1780990843.252566 mapread 0x027ca7 thru 0x036708 (0xea62 bytes) 000065 1780990843.273481 read 0x02a593 thru 0x02b6ec (0x115a bytes) 000066 1780990843.274282 write 0x01c8ed thru 0x01d908 (0x101c bytes) 000069 1780990843.288160 mapread 0x001611 thru 0x00dd2f (0xc71f bytes) 000071 1780990843.291570 write 0x031acd thru 0x038c27 (0x715b bytes) 000073 1780990843.294134 mapwrite 0x02adca thru 0x03972d (0xe964 bytes) 000075 1780990843.374564 read 0x0236a4 thru 0x02be20 (0x877d bytes) 000076 1780990843.375487 trunc from 0x03972e to 0x022ebd (0x16870 bytes) 000080 1780990843.397726 trunc from 0x022ebe to 0x01e727 (0x4796 bytes) 000081 1780990843.418999 mapwrite 0x011441 thru 0x0135f3 (0x21b3 bytes) 000083 1780990843.474613 write 0x0194ec thru 0x0240c2 (0xabd7 bytes) 000085 1780990843.477704 trunc from 0x0240c3 to 0x00749c (0x1cc26 bytes) 000088 1780990843.521350 MIRROR_RESYNC 2 mirrors 000089 1780990844.477596 write 0x030a38 thru 0x03a647 (0x9c10 bytes) 000091 1780990844.606717 read 0x0053b4 thru 0x00af27 (0x5b74 bytes) 000093 1780990844.618534 mapread 0x007e84 thru 0x00ab00 (0x2c7d bytes) 000095 1780990844.627623 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) 000096 1780990844.859451 mapwrite 0x000cd5 thru 0x002e7f (0x21ab bytes) 000097 1780990844.982633 mapread 0x0052d0 thru 0x01045b (0xb18c bytes) 000098 1780990845.014894 MIRROR_EXTEND to 2 mirrors 000099 1780990846.147788 trunc from 0x03a648 to 0x03126a (0x93dd bytes) 000100 1780990846.334070 mapread 0x02ed54 thru 0x03126a (0x2517 bytes) 000101 1780990846.372341 trunc from 0x03126b to 0x00c94b (0x2491f bytes) truncating to largest ever: 0x38710 000102 1780990846.424745 trunc from 0x00c94c to 0x03870f (0x2bdc4 bytes) 000103 1780990846.457988 read 0x02d6c8 thru 0x03781d (0xa156 bytes) truncating to largest ever: 0x3fd76 000104 1780990846.473539 trunc from 0x038710 to 0x03fd75 (0x7666 bytes) 000106 1780990846.496716 trunc from 0x03fd76 to 0x031eb6 (0xdebf bytes) 000109 1780990846.532821 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) 000111 1780990846.742812 trunc from 0x031eb7 to 0x007215 (0x2aca1 bytes) 000112 1780990846.775146 mapread 0x005079 thru 0x007215 (0x219d bytes) 000114 1780990846.818419 mapwrite 0x032d88 thru 0x039ebb (0x7134 bytes) 000115 1780990846.960571 read 0x02021e thru 0x024b6c (0x494f bytes) 000116 1780990846.973856 read 0x0288b1 thru 0x031e44 (0x9594 bytes) 000117 1780990847.000532 read 0x02d872 thru 0x039ebb (0xc64a bytes) 000118 1780990847.001383 mapread 0x003c1d thru 0x012875 (0xec59 bytes) 000125 1780990847.011146 write 0x02d557 thru 0x03b7d8 (0xe282 bytes) 000126 1780990847.016469 trunc from 0x03b7d9 to 0x02f514 (0xc2c4 bytes) 000127 1780990847.069127 write 0x02e3b1 thru 0x03b24d (0xce9d bytes) 000128 1780990847.072635 write 0x00c0f2 thru 0x01412f (0x803e bytes) 000129 1780990847.077740 write 0x038e7b thru 0x03ffff (0x7185 bytes) 000131 1780990847.080886 read 0x03af9e thru 0x03ffff (0x5062 bytes) 000133 1780990847.084864 read 0x0059c5 thru 0x00b98d (0x5fc9 bytes) 000137 1780990847.091918 write 0x01b090 thru 0x025959 (0xa8ca bytes) 000138 1780990847.093639 read 0x002407 thru 0x0073d3 (0x4fcd bytes) 000139 1780990847.098508 trunc from 0x040000 to 0x018746 (0x278b9 bytes) 000140 1780990847.120588 mapwrite 0x028a6b thru 0x02f069 (0x65ff bytes) 000142 1780990847.246385 write 0x023e6a thru 0x030328 (0xc4bf bytes) 000143 1780990847.250113 write 0x024f91 thru 0x028f04 (0x3f74 bytes) 000144 1780990847.251550 write 0x027269 thru 0x031215 (0x9fad bytes) 000146 1780990847.261219 mapread 0x018863 thru 0x01961c (0xdba bytes) 000148 1780990847.265312 write 0x03a5da thru 0x03ffff (0x5a26 bytes) 000149 1780990847.268699 write 0x01a545 thru 0x02872a (0xe1e6 bytes) 000150 1780990847.275127 mapwrite 0x031228 thru 0x03764a (0x6423 bytes) 000152 1780990847.381417 mapwrite 0x018fec thru 0x01b43a (0x244f bytes) 000154 1780990847.497621 mapread 0x03520c thru 0x03f1c0 (0x9fb5 bytes) 000155 1780990847.500443 trunc from 0x040000 to 0x002191 (0x3de6e bytes) 000157 1780990847.540441 trunc from 0x002192 to 0x019f2d (0x17d9c bytes) 000160 1780990847.570830 read 0x0123e0 thru 0x019f2d (0x7b4e bytes) 000164 1780990847.591293 MIRROR_EXTEND to 2 mirrors 000165 1780990848.733640 read 0x00d6b6 thru 0x00e122 (0xa6d bytes) 000168 1780990848.796360 mapwrite 0x0313a3 thru 0x0314fa (0x158 bytes) 000169 1780990849.062655 read 0x029c33 thru 0x0314fa (0x78c8 bytes) 000170 1780990849.136683 mapread 0x0060c9 thru 0x00713f (0x1077 bytes) 000171 1780990849.137920 read 0x00554d thru 0x0065a3 (0x1057 bytes) 000172 1780990849.139008 read 0x02bd8f thru 0x030f3a (0x51ac bytes) 000174 1780990849.141689 read 0x0068b1 thru 0x008caa (0x23fa bytes) 000175 1780990849.142556 write 0x03d647 thru 0x03ffff (0x29b9 bytes) 000176 1780990849.145335 read 0x024157 thru 0x028d8c (0x4c36 bytes) 000178 1780990849.147100 mapread 0x005db4 thru 0x010cdb (0xaf28 bytes) 000180 1780990849.167434 mapwrite 0x01ff64 thru 0x021703 (0x17a0 bytes) 000182 1780990849.288735 read 0x02a57d thru 0x03986e (0xf2f2 bytes) 000183 1780990849.305792 trunc from 0x040000 to 0x021446 (0x1ebb9 bytes) 000184 1780990849.382511 mapwrite 0x00dd42 thru 0x01bc60 (0xdf1f bytes) 000186 1780990849.602960 mapread 0x013990 thru 0x01f6c7 (0xbd38 bytes) 000191 1780990849.615818 mapwrite 0x01f180 thru 0x02dfa9 (0xee2a bytes) 000195 1780990849.952661 MIRROR_RESYNC 2 mirrors 000198 1780990850.788251 read 0x019aa2 thru 0x024af7 (0xb056 bytes) 000199 1780990850.805521 MIRROR_EXTEND to 3 mirrors 000202 1780990851.658878 mapread 0x001a9a thru 0x0068bb (0x4e22 bytes) 000203 1780990851.715387 mapwrite 0x000b30 thru 0x009406 (0x88d7 bytes) 000204 1780990851.945849 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dfaa): Mirror 3: 0x363960b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dfaa): Mirror 3: 0x363960b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dfaa: skipped 000205 1780990852.470501 read 0x02a728 thru 0x02dfa9 (0x3882 bytes) 000208 1780990852.489891 write 0x0212da thru 0x030c73 (0xf99a bytes) 000210 1780990852.504420 read 0x01f3af thru 0x01ff83 (0xbd5 bytes) 000214 1780990852.534495 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) 000215 1780990852.774916 trunc from 0x030c74 to 0x00d634 (0x2363f bytes) 000216 1780990852.806947 trunc from 0x00d635 to 0x014073 (0x6a3f bytes) 000219 1780990852.836234 mapwrite 0x01ab62 thru 0x01afde (0x47d bytes) 000220 1780990852.950721 write 0x0334e9 thru 0x03dd92 (0xa8aa bytes) 000222 1780990852.953927 write 0x00ae2f thru 0x00c85f (0x1a31 bytes) 000223 1780990852.965509 mapwrite 0x00ff85 thru 0x016cc5 (0x6d41 bytes) 000224 1780990853.057474 mapwrite 0x039cba thru 0x03ffff (0x6346 bytes) 000226 1780990853.144477 mapread 0x01d680 thru 0x02d187 (0xfb08 bytes) 000227 1780990853.164535 read 0x0100d9 thru 0x019b01 (0x9a29 bytes) 000229 1780990853.166554 trunc from 0x040000 to 0x018cbf (0x27340 bytes) 000230 1780990853.185733 write 0x02acb1 thru 0x038033 (0xd383 bytes) 000232 1780990853.188483 read 0x00c388 thru 0x0153a1 (0x901a bytes) 000233 1780990853.202350 MIRROR_EXTEND to 2 mirrors 000234 1780990854.037651 trunc from 0x038034 to 0x03c3f0 (0x43bd bytes) 000235 1780990854.288917 read 0x0240ea thru 0x033a62 (0xf979 bytes) 000236 1780990854.306592 MIRROR_RESYNC 2 mirrors 000238 1780990855.045809 mapwrite 0x02f900 thru 0x0361ca (0x68cb bytes) 000239 1780990855.266528 write 0x017490 thru 0x021c30 (0xa7a1 bytes) 000240 1780990855.292790 mapread 0x026e34 thru 0x03458d (0xd75a bytes) 000241 1780990855.310530 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: 0x829640fc (stale) CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 1: 0xa3d95e52 Mirror 3: 0xa3d95e52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3f1: skipped 000242 1780990856.509364 mapwrite 0x005aa3 thru 0x00ad76 (0x52d4 bytes) 000243 1780990856.712784 read 0x0388ff thru 0x03c3f0 (0x3af2 bytes) 000244 1780990856.726391 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: 0x829640fc (stale) Mirror 3: 0xa3d95e52 (stale) CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 1: 0xe5769334 Mirror 4: 0xe5769334 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3f1: skipped 000245 1780990857.936901 read 0x01a22c thru 0x020641 (0x6416 bytes) 000246 1780990857.962035 read 0x0279f9 thru 0x029f8e (0x2596 bytes) 000247 1780990858.004619 mapwrite 0x02f8dd thru 0x03009e (0x7c2 bytes) 000251 1780990858.239639 read 0x007268 thru 0x014132 (0xcecb bytes) 000252 1780990858.251576 mapwrite 0x00b048 thru 0x01a354 (0xf30d bytes) 000253 1780990858.346443 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: 0x829640fc (stale) Mirror 3: 0xa3d95e52 (stale) Mirror 4: 0xe5769334 (stale) CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 1: 0x903a30f Mirror 5: 0x903a30f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3f1: skipped 000254 1780990859.610484 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 2: 0x829640fc (stale) Mirror 3: 0xa3d95e52 (stale) Mirror 4: 0xe5769334 (stale) CRC-32 checksum value for chunk [0, 0x3c3f1): Mirror 1: 0x903a30f Mirror 5: 0x903a30f Mirror 6: 0x903a30f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3f1: skipped 000255 1780990860.450767 write 0x03c0c2 thru 0x03ffff (0x3f3e bytes) 000256 1780990860.559982 mapwrite 0x01a8d0 thru 0x01d675 (0x2da6 bytes) 000261 1780990860.628419 mapread 0x02886c thru 0x02a33e (0x1ad3 bytes) 000262 1780990860.636771 write 0x01315c thru 0x020573 (0xd418 bytes) 000263 1780990860.644871 write 0x016251 thru 0x019476 (0x3226 bytes) 000265 1780990860.646542 write 0x03a390 thru 0x03a701 (0x372 bytes) 000268 1780990860.658581 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc549d201 (stale) Mirror 4: 0x3d44e683 (stale) Mirror 5: 0xb761822 (stale) Mirror 6: 0xb761822 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc549d201 (stale) Mirror 4: 0x3d44e683 (stale) Mirror 5: 0xb761822 (stale) Mirror 6: 0xb761822 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000269 1780990861.183434 read 0x038d8a thru 0x03ffff (0x7276 bytes) 000270 1780990861.205557 mapwrite 0x00621c thru 0x0107ab (0xa590 bytes) 000271 1780990861.437363 mapwrite 0x031ab8 thru 0x03afbd (0x9506 bytes) 000273 1780990861.644496 mapwrite 0x01229c thru 0x0154f7 (0x325c bytes) 000274 1780990861.794741 trunc from 0x040000 to 0x03072d (0xf8d2 bytes) 000275 1780990861.917408 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3072e): Mirror 4: 0x2d1b9bf (stale) Mirror 5: 0x557bbb99 (stale) Mirror 6: 0x557bbb99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3072e): Mirror 4: 0x2d1b9bf (stale) Mirror 5: 0x557bbb99 (stale) Mirror 6: 0x557bbb99 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3072e: skipped 000277 1780990862.484207 write 0x006009 thru 0x01550a (0xf502 bytes) 000280 1780990862.504016 read 0x026a9f thru 0x02d55c (0x6abe bytes) 000281 1780990862.525991 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3072e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3072e): Mirror 4: 0x2d1b9bf (stale) Mirror 5: 0x557bbb99 (stale) Mirror 6: 0x557bbb99 (stale) CRC-32 checksum value for chunk [0, 0x3072e): Mirror 1: 0xb9df69e8 Mirror 7: 0xb9df69e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3072e: skipped 000282 1780990864.456022 write 0x01410a thru 0x01c744 (0x863b bytes) 000284 1780990864.671179 mapwrite 0x022219 thru 0x02569a (0x3482 bytes) 000285 1780990864.836548 mapread 0x01bfe5 thru 0x027aaf (0xbacb bytes) 000286 1780990864.840997 read 0x012f63 thru 0x019daa (0x6e48 bytes) 000287 1780990864.858033 mapwrite 0x02797c thru 0x0318c4 (0x9f49 bytes) 000288 1780990865.178030 trunc from 0x0318c5 to 0x0091cf (0x286f5 bytes) 000290 1780990865.253985 trunc from 0x0091d0 to 0x0340bf (0x2aef0 bytes) 000294 1780990865.319002 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x340c0): Mirror 5: 0x3d673c6d (stale) Mirror 6: 0x3d673c6d (stale) Mirror 7: 0x9cca0feb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340c0): Mirror 5: 0x3d673c6d (stale) Mirror 6: 0x3d673c6d (stale) Mirror 7: 0x9cca0feb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340c0: skipped 000295 1780990865.831836 mapread 0x01cbf5 thru 0x01f2bd (0x26c9 bytes) 000297 1780990865.858708 trunc from 0x0340c0 to 0x03c2c7 (0x8208 bytes) 000298 1780990865.894986 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c2c8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2c8): Mirror 5: 0xb9c2fdd1 (stale) Mirror 6: 0xb9c2fdd1 (stale) Mirror 7: 0xdc21dd19 (stale) CRC-32 checksum value for chunk [0, 0x3c2c8): Mirror 1: 0x54a59e44 Mirror 8: 0x54a59e44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2c8: skipped 000299 1780990867.633182 write 0x002a16 thru 0x0123fc (0xf9e7 bytes) 000301 1780990867.816940 read 0x0230aa thru 0x02b1b1 (0x8108 bytes) 000305 1780990867.851006 write 0x00ece9 thru 0x011f04 (0x321c bytes) 000306 1780990867.856383 mapread 0x019edf thru 0x028f19 (0xf03b bytes) 000308 1780990867.911738 MIRROR_RESYNC 5 mirrors 000309 1780990870.222860 mapread 0x00dac0 thru 0x01b700 (0xdc41 bytes) 000311 1780990870.291765 mapread 0x031688 thru 0x0357b0 (0x4129 bytes) 000312 1780990870.318812 write 0x011245 thru 0x016fd6 (0x5d92 bytes) 000314 1780990870.491224 mapread 0x03c05a thru 0x03c2c7 (0x26e bytes) 000315 1780990870.515762 mapwrite 0x00cd65 thru 0x019182 (0xc41e bytes) 000317 1780990870.714567 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c2c8): Mirror 6: 0x235dd118 (stale) Mirror 7: 0x235dd118 (stale) Mirror 8: 0x235dd118 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2c8): Mirror 6: 0x235dd118 (stale) Mirror 7: 0x235dd118 (stale) Mirror 8: 0x235dd118 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2c8: skipped 000319 1780990871.349696 mapread 0x00e64a thru 0x01c01f (0xd9d6 bytes) 000320 1780990871.386814 mapread 0x01975d thru 0x01fe4a (0x66ee bytes) 000322 1780990871.389529 trunc from 0x03c2c8 to 0x019b56 (0x22771 bytes) 000332 1780990871.417736 write 0x03e82e thru 0x03ffff (0x17d2 bytes) 000335 1780990871.421840 write 0x002e7c thru 0x003368 (0x4ed bytes) 000337 1780990871.432389 read 0x003d32 thru 0x0136f4 (0xf9c3 bytes) 000338 1780990871.433407 mapwrite 0x02fd40 thru 0x0325b9 (0x287a bytes) 000342 1780990871.593853 mapread 0x0160d5 thru 0x025967 (0xf893 bytes) 000343 1780990871.625728 write 0x02f4d3 thru 0x031d38 (0x2866 bytes) 000344 1780990871.638664 MIRROR_RESYNC 4 mirrors 000348 1780990873.471338 mapread 0x00fe55 thru 0x011767 (0x1913 bytes) 000351 1780990873.521040 trunc from 0x040000 to 0x0351bd (0xae42 bytes) 000352 1780990873.720230 read 0x025273 thru 0x029686 (0x4414 bytes) 000353 1780990873.742282 trunc from 0x0351be to 0x021b10 (0x136ad bytes) 000354 1780990873.819747 MIRROR_RESYNC 4 mirrors 000355 1780990875.415233 write 0x02d5df thru 0x03c8bd (0xf2df bytes) 000356 1780990875.528517 trunc from 0x03c8be to 0x025ade (0x16ddf bytes) 000357 1780990875.580845 trunc from 0x025adf to 0x020202 (0x58dc bytes) 000358 1780990875.621957 write 0x03c772 thru 0x03ffff (0x388e bytes) 000360 1780990875.624000 write 0x03b5bb thru 0x03ffff (0x4a45 bytes) 000361 1780990875.655458 trunc from 0x040000 to 0x029bce (0x16431 bytes) 000363 1780990875.709141 mapread 0x020b14 thru 0x029bce (0x90bb bytes) 000364 1780990875.740848 read 0x00bc75 thru 0x015c46 (0x9fd2 bytes) 000365 1780990875.761516 read 0x002f52 thru 0x0093c7 (0x6476 bytes) 000366 1780990875.775393 read 0x01f445 thru 0x01f54a (0x106 bytes) 000368 1780990875.776751 mapwrite 0x030851 thru 0x03fec9 (0xf679 bytes) 000369 1780990875.940354 mapwrite 0x025cd0 thru 0x02ad09 (0x503a bytes) 000371 1780990876.091345 write 0x02d1f5 thru 0x030204 (0x3010 bytes) 000372 1780990876.097567 write 0x01a695 thru 0x0250c6 (0xaa32 bytes) 000373 1780990876.102369 trunc from 0x03feca to 0x017068 (0x28e61 bytes) 000374 1780990876.132170 trunc from 0x017069 to 0x00d525 (0x9b43 bytes) 000375 1780990876.177419 read 0x00110b thru 0x00d525 (0xc41b bytes) 000377 1780990876.178851 mapwrite 0x03fb0a thru 0x03ffff (0x4f6 bytes) 000379 1780990876.380944 write 0x00bae4 thru 0x00d875 (0x1d92 bytes) 000381 1780990876.386230 read 0x039f97 thru 0x03cb04 (0x2b6e bytes) 000383 1780990876.392914 read 0x01e7b8 thru 0x01e880 (0xc9 bytes) 000385 1780990876.399238 mapwrite 0x03b728 thru 0x03ffff (0x48d8 bytes) 000386 1780990876.478525 write 0x029cbe thru 0x031c83 (0x7fc6 bytes) 000388 1780990876.482277 read 0x03bcc9 thru 0x03ffff (0x4337 bytes) 000390 1780990876.484556 read 0x0032bc thru 0x009433 (0x6178 bytes) 000391 1780990876.486213 mapread 0x027b7d thru 0x0369ea (0xee6e bytes) 000392 1780990876.489288 trunc from 0x040000 to 0x02e97f (0x11680 bytes) 000394 1780990876.513520 trunc from 0x02e980 to 0x009e95 (0x24aea bytes) 000395 1780990876.548479 write 0x02afed thru 0x033d28 (0x8d3c bytes) 000396 1780990876.553723 mapwrite 0x03baab thru 0x03f132 (0x3688 bytes) 000398 1780990876.704229 mapwrite 0x019325 thru 0x01eda4 (0x5a80 bytes) 000399 1780990876.847343 mapread 0x01c369 thru 0x0298c4 (0xd55c bytes) 000400 1780990876.848859 mapwrite 0x02d254 thru 0x03bdfa (0xeba7 bytes) 000405 1780990877.025905 read 0x020138 thru 0x029cc1 (0x9b8a bytes) 000408 1780990877.031484 mapwrite 0x0105ec thru 0x0167dc (0x61f1 bytes) 000410 1780990877.137958 trunc from 0x03f133 to 0x02140a (0x1dd28 bytes) 000411 1780990877.183367 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2140b): Mirror 7: 0x3df2b727 (stale) Mirror 8: 0x3df2b727 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2140b): Mirror 7: 0x3df2b727 (stale) Mirror 8: 0x3df2b727 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2140b: skipped 000412 1780990877.806191 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2140b): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2140b): Mirror 7: 0x3df2b727 (stale) Mirror 8: 0x3df2b727 (stale) CRC-32 checksum value for chunk [0, 0x2140b): Mirror 1: 0x61cfc49b Mirror 9: 0x61cfc49b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2140b: skipped 000413 1780990879.802711 trunc from 0x02140b to 0x020a62 (0x9a8 bytes) 000414 1780990879.957693 read 0x012468 thru 0x01c609 (0xa1a2 bytes) 000415 1780990879.979611 write 0x030e34 thru 0x039af0 (0x8cbd bytes) 000416 1780990879.991566 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39af1): Mirror 8: 0xc02a2763 (stale) Mirror 9: 0xef260325 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39af1): Mirror 8: 0xc02a2763 (stale) Mirror 9: 0xef260325 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39af1: skipped 000417 1780990880.527847 mapread 0x02573b thru 0x030db2 (0xb678 bytes) 000418 1780990880.581767 write 0x010c97 thru 0x012572 (0x18dc bytes) 000419 1780990880.620845 MIRROR_RESYNC 3 mirrors 000422 1780990882.275816 read 0x013fc6 thru 0x016eab (0x2ee6 bytes) 000423 1780990882.296019 mapwrite 0x02a989 thru 0x036490 (0xbb08 bytes) 000424 1780990882.612323 write 0x0304af thru 0x039eae (0x9a00 bytes) 000425 1780990882.613904 mapwrite 0x000dc1 thru 0x00f41f (0xe65f bytes) 000427 1780990882.775491 trunc from 0x039eaf to 0x0111fa (0x28cb4 bytes) 000428 1780990882.841293 write 0x029e15 thru 0x03796d (0xdb59 bytes) 000431 1780990882.845496 read 0x01fbef thru 0x027c3e (0x8050 bytes) 000432 1780990882.855861 trunc from 0x03796e to 0x009927 (0x2e046 bytes) 000433 1780990882.883152 trunc from 0x009928 to 0x0035e0 (0x6347 bytes) 000434 1780990882.908891 read 0x002c34 thru 0x0035e0 (0x9ad bytes) 000439 1780990882.914934 write 0x00e525 thru 0x015147 (0x6c23 bytes) 000441 1780990882.927294 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x15148): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15148): Mirror 8: 0xd44b80fc (stale) Mirror 9: 0xd44b80fc (stale) CRC-32 checksum value for chunk [0, 0x15148): Mirror 1: 0x45eb32a5 Mirror 10: 0x45eb32a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15148: skipped 000442 1780990884.378902 trunc from 0x015148 to 0x037f81 (0x22e3a bytes) 000445 1780990884.622308 trunc from 0x037f82 to 0x00b27f (0x2cd02 bytes) 000446 1780990884.692106 mapread 0x008fa8 thru 0x00b27f (0x22d8 bytes) 000447 1780990884.778505 trunc from 0x00b280 to 0x0324ba (0x2723b bytes) 000448 1780990884.820440 read 0x02a9a1 thru 0x0320d6 (0x7736 bytes) 000449 1780990884.840581 write 0x001c5f thru 0x011579 (0xf91b bytes) 000450 1780990884.888597 write 0x0048fa thru 0x01368f (0xed96 bytes) 000452 1780990884.903295 write 0x01a1f3 thru 0x02679d (0xc5ab bytes) 000455 1780990884.925135 read 0x00c636 thru 0x00f655 (0x3020 bytes) 000457 1780990884.928333 mapwrite 0x00a7f4 thru 0x00e293 (0x3aa0 bytes) 000459 1780990885.067006 write 0x020947 thru 0x020deb (0x4a5 bytes) 000462 1780990885.069129 trunc from 0x0324bb to 0x008703 (0x29db7 bytes) 000464 1780990885.123703 mapread 0x006b0e thru 0x0074f4 (0x9e7 bytes) 000467 1780990885.130534 mapwrite 0x016856 thru 0x022c2c (0xc3d7 bytes) 000469 1780990885.339364 mapwrite 0x02bfd2 thru 0x037729 (0xb758 bytes) 000471 1780990885.491748 mapread 0x030310 thru 0x037729 (0x741a bytes) 000473 1780990885.495305 read 0x03074e thru 0x035c1e (0x54d1 bytes) 000474 1780990885.496343 mapread 0x00497b thru 0x011767 (0xcded bytes) 000475 1780990885.499302 trunc from 0x03772a to 0x03bb5a (0x4431 bytes) 000476 1780990885.562978 mapwrite 0x03a3b7 thru 0x03ffff (0x5c49 bytes) 000477 1780990885.710979 trunc from 0x040000 to 0x001b31 (0x3e4ce bytes) 000478 1780990885.730988 mapread 0x00032f thru 0x001b31 (0x1803 bytes) 000479 1780990885.732024 write 0x02dbe4 thru 0x02fe25 (0x2242 bytes) 000480 1780990885.735154 write 0x0295c6 thru 0x0346b4 (0xb0ef bytes) 000483 1780990885.746261 mapwrite 0x00edcd thru 0x019d7e (0xafb2 bytes) 000484 1780990885.915028 mapread 0x005729 thru 0x0129e7 (0xd2bf bytes) 000486 1780990885.958681 mapread 0x00be9e thru 0x019eba (0xe01d bytes) 000487 1780990885.960940 write 0x01cd95 thru 0x01d9d4 (0xc40 bytes) 000488 1780990885.974683 read 0x02f6b0 thru 0x031b94 (0x24e5 bytes) 000491 1780990885.977791 mapread 0x008535 thru 0x010154 (0x7c20 bytes) 000492 1780990885.982419 mapread 0x007854 thru 0x008446 (0xbf3 bytes) 000493 1780990885.985213 mapwrite 0x0055d2 thru 0x00d9b6 (0x83e5 bytes) 000496 1780990886.128266 read 0x02a04e thru 0x0346b4 (0xa667 bytes) 000497 1780990886.128960 read 0x019ac2 thru 0x02249e (0x89dd bytes) 000498 1780990886.141818 write 0x036840 thru 0x03c67a (0x5e3b bytes) 000499 1780990886.144704 trunc from 0x03c67b to 0x034d19 (0x7961 bytes) 000501 1780990886.159523 read 0x00591e thru 0x010c3b (0xb31e bytes) 000502 1780990886.160315 write 0x00f851 thru 0x01105e (0x180e bytes) 000503 1780990886.161295 trunc from 0x034d1a to 0x02006a (0x14caf bytes) 000504 1780990886.192921 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2006b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2006b): Mirror 8: 0x56dd389 (stale) Mirror 9: 0x56dd389 (stale) Mirror 10: 0x823f8f88 (stale) CRC-32 checksum value for chunk [0, 0x2006b): Mirror 1: 0x1b0b2b22 Mirror 11: 0x1b0b2b22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2006b: skipped 000505 1780990887.425601 read 0x00092a thru 0x00ccaf (0xc386 bytes) 000506 1780990887.445919 read 0x00f34d thru 0x016b89 (0x783d bytes) 000507 1780990887.489136 write 0x033a32 thru 0x0362e4 (0x28b3 bytes) 000511 1780990887.597748 mapwrite 0x03061a thru 0x03a09c (0x9a83 bytes) 000512 1780990887.716132 mapwrite 0x02163e thru 0x027b18 (0x64db bytes) 000513 1780990887.781204 mapread 0x009452 thru 0x0103a4 (0x6f53 bytes) 000514 1780990887.864129 mapread 0x008197 thru 0x009ae4 (0x194e bytes) 000515 1780990887.875196 MIRROR_RESYNC 5 mirrors 000516 1780990889.284277 trunc from 0x03a09d to 0x029a73 (0x10629 bytes) 000518 1780990889.449344 write 0x03a6e2 thru 0x03b52c (0xe4b bytes) 000522 1780990889.456748 read 0x00d640 thru 0x01bf2a (0xe8eb bytes) 000523 1780990889.474228 read 0x00778e thru 0x00d154 (0x59c7 bytes) 000524 1780990889.485386 trunc from 0x03b52d to 0x0209b8 (0x1ab74 bytes) 000526 1780990889.509653 mapwrite 0x03411b thru 0x034dc6 (0xcac bytes) 000527 1780990889.668436 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34dc7): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34dc7): Mirror 8: 0xe17a9252 (stale) Mirror 9: 0xe17a9252 (stale) Mirror 10: 0xe17a9252 (stale) Mirror 11: 0xe17a9252 (stale) CRC-32 checksum value for chunk [0, 0x34dc7): Mirror 1: 0x4eec3c68 Mirror 12: 0x4eec3c68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34dc7: skipped 000530 1780990891.062946 MIRROR_RESYNC 6 mirrors 000531 1780990892.656869 mapread 0x029860 thru 0x031d3a (0x84db bytes) 000532 1780990892.690577 trunc from 0x034dc7 to 0x039c09 (0x4e43 bytes) 000533 1780990892.855895 trunc from 0x039c0a to 0x02706e (0x12b9b bytes) 000536 1780990892.902872 write 0x02a4da thru 0x038d07 (0xe82e bytes) 000538 1780990892.906227 mapwrite 0x03e74f thru 0x03ffff (0x18b1 bytes) 000541 1780990893.021505 read 0x02b60b thru 0x03a855 (0xf24b bytes) 000542 1780990893.022294 write 0x013f5d thru 0x01cb26 (0x8bca bytes) 000544 1780990893.049965 mapread 0x020d5c thru 0x02e7c4 (0xda69 bytes) 000545 1780990893.080774 mapread 0x039cf2 thru 0x03ffff (0x630e bytes) 000547 1780990893.086119 write 0x014efc thru 0x016896 (0x199b bytes) 000548 1780990893.088028 trunc from 0x040000 to 0x0069f6 (0x39609 bytes) 000554 1780990893.113834 write 0x019248 thru 0x023b02 (0xa8bb bytes) 000555 1780990893.115748 mapread 0x00877e thru 0x00e1e5 (0x5a68 bytes) 000560 1780990893.131563 mapwrite 0x01071b thru 0x017ceb (0x75d1 bytes) 000562 1780990893.223205 read 0x01c32b thru 0x023b02 (0x77d8 bytes) 000564 1780990893.225553 read 0x00415c thru 0x00feec (0xbd91 bytes) 000566 1780990893.227005 write 0x02fac3 thru 0x030d52 (0x1290 bytes) 000568 1780990893.229316 write 0x0291f8 thru 0x035ecb (0xccd4 bytes) 000571 1780990893.244874 read 0x001493 thru 0x00540e (0x3f7c bytes) 000574 1780990893.248774 mapwrite 0x00c219 thru 0x016364 (0xa14c bytes) 000576 1780990893.361343 write 0x00353e thru 0x00dbeb (0xa6ae bytes) 000577 1780990893.363738 read 0x027403 thru 0x034213 (0xce11 bytes) 000580 1780990893.366308 trunc from 0x035ecc to 0x030843 (0x5688 bytes) 000581 1780990893.407721 mapread 0x01abbe thru 0x01e96b (0x3dae bytes) 000582 1780990893.417458 mapread 0x004180 thru 0x009acb (0x594c bytes) 000584 1780990893.422421 mapread 0x0096ea thru 0x009d64 (0x67b bytes) 000586 1780990893.449124 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30844): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30844): Mirror 8: 0x588afeb9 (stale) Mirror 9: 0x588afeb9 (stale) Mirror 10: 0x588afeb9 (stale) Mirror 11: 0x588afeb9 (stale) Mirror 12: 0x588afeb9 (stale) CRC-32 checksum value for chunk [0, 0x30844): Mirror 1: 0x6ab73455 Mirror 13: 0x6ab73455 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30844: skipped 000588 1780990895.146036 mapread 0x02ae41 thru 0x030843 (0x5a03 bytes) 000590 1780990895.226601 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30844): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30844): Mirror 8: 0x588afeb9 (stale) Mirror 9: 0x588afeb9 (stale) Mirror 10: 0x588afeb9 (stale) Mirror 11: 0x588afeb9 (stale) Mirror 12: 0x588afeb9 (stale) CRC-32 checksum value for chunk [0, 0x30844): Mirror 1: 0x6ab73455 Mirror 13: 0x6ab73455 Mirror 14: 0x6ab73455 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30844: skipped 000592 1780990896.687338 trunc from 0x030844 to 0x02a355 (0x64ee bytes) 000593 1780990896.829825 read 0x011af2 thru 0x01bba8 (0xa0b7 bytes) 000594 1780990896.842552 mapwrite 0x016b3b thru 0x023607 (0xcacd bytes) 000596 1780990897.086204 read 0x023092 thru 0x028f18 (0x5e87 bytes) 000599 1780990897.108373 mapread 0x01a1b4 thru 0x01d508 (0x3355 bytes) 000600 1780990897.110196 mapread 0x0009cf thru 0x00987d (0x8eaf bytes) 000601 1780990897.112869 mapwrite 0x00c687 thru 0x01a8c4 (0xe23e bytes) 000603 1780990897.212380 mapwrite 0x0330bb thru 0x03b899 (0x87df bytes) 000604 1780990897.414255 write 0x018d85 thru 0x01f8c7 (0x6b43 bytes) 000605 1780990897.417095 trunc from 0x03b89a to 0x023c2a (0x17c6f bytes) 000606 1780990897.455853 write 0x01dbf3 thru 0x02755c (0x996a bytes) 000607 1780990897.457248 read 0x019765 thru 0x01e5d9 (0x4e75 bytes) 000609 1780990897.486216 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2755d): Mirror 8: 0xaa847b5b (stale) Mirror 9: 0xaa847b5b (stale) Mirror 11: 0xaa847b5b (stale) Mirror 12: 0xaa847b5b (stale) Mirror 13: 0xb0f97255 (stale) Mirror 14: 0xb0f97255 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2755d): Mirror 8: 0xaa847b5b (stale) Mirror 9: 0xaa847b5b (stale) Mirror 11: 0xaa847b5b (stale) Mirror 12: 0xaa847b5b (stale) Mirror 13: 0xb0f97255 (stale) Mirror 14: 0xb0f97255 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2755d: skipped 000612 1780990898.364150 read 0x026cc3 thru 0x02755c (0x89a bytes) 000613 1780990898.372962 mapread 0x000860 thru 0x0085c3 (0x7d64 bytes) 000614 1780990898.436689 read 0x01d924 thru 0x02755c (0x9c39 bytes) 000615 1780990898.455968 mapwrite 0x029818 thru 0x030cba (0x74a3 bytes) 000616 1780990898.633009 trunc from 0x030cbb to 0x01133e (0x1f97c bytes) 000617 1780990898.673742 write 0x010f2c thru 0x017646 (0x671b bytes) 000618 1780990898.676149 trunc from 0x017647 to 0x039870 (0x2222a bytes) 000619 1780990898.726860 MIRROR_RESYNC 7 mirrors 000620 1780990900.539817 MIRROR_EXTEND to 8 mirrors 000621 1780990901.802992 write 0x039b75 thru 0x03b627 (0x1ab3 bytes) 000623 1780990901.926596 read 0x012d35 thru 0x01bfe8 (0x92b4 bytes) 000626 1780990901.974488 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b628): Mirror 8: 0xf9686347 (stale) Mirror 9: 0xf9686347 (stale) Mirror 11: 0xf9686347 (stale) Mirror 12: 0xf9686347 (stale) Mirror 14: 0xf9686347 (stale) Mirror 15: 0xf9686347 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b628): Mirror 8: 0xf9686347 (stale) Mirror 9: 0xf9686347 (stale) Mirror 11: 0xf9686347 (stale) Mirror 12: 0xf9686347 (stale) Mirror 14: 0xf9686347 (stale) Mirror 15: 0xf9686347 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b628: skipped 000627 1780990902.710587 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b628): Mirror 8: 0xf9686347 (stale) Mirror 11: 0xf9686347 (stale) Mirror 12: 0xf9686347 (stale) Mirror 14: 0xf9686347 (stale) Mirror 15: 0xf9686347 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b628): Mirror 8: 0xf9686347 (stale) Mirror 11: 0xf9686347 (stale) Mirror 12: 0xf9686347 (stale) Mirror 14: 0xf9686347 (stale) Mirror 15: 0xf9686347 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b628: skipped 000630 1780990903.416736 mapwrite 0x00b4f3 thru 0x0103ec (0x4efa bytes) 000632 1780990903.556883 mapread 0x038116 thru 0x03b627 (0x3512 bytes) 000633 1780990903.574305 read 0x009ef5 thru 0x00f7af (0x58bb bytes) 000635 1780990903.576359 mapwrite 0x000e98 thru 0x00faf1 (0xec5a bytes) 000637 1780990903.663834 write 0x018efe thru 0x023728 (0xa82b bytes) 000638 1780990903.679261 mapread 0x0271ae thru 0x02ee9c (0x7cef bytes) 000640 1780990903.695357 mapwrite 0x024a80 thru 0x031022 (0xc5a3 bytes) 000641 1780990903.856873 mapread 0x02198e thru 0x02d4a7 (0xbb1a bytes) 000642 1780990903.860013 read 0x025b15 thru 0x0297a7 (0x3c93 bytes) 000643 1780990903.868239 read 0x01b111 thru 0x02612a (0xb01a bytes) 000644 1780990903.883226 trunc from 0x03b628 to 0x01bfec (0x1f63b bytes) 000648 1780990903.925702 write 0x0221e7 thru 0x02933b (0x7155 bytes) 000649 1780990903.928082 read 0x010b29 thru 0x01d5b7 (0xca8f bytes) 000650 1780990903.941499 trunc from 0x02933c to 0x021d99 (0x75a2 bytes) 000652 1780990903.966026 write 0x012049 thru 0x01a353 (0x830b bytes) 000653 1780990903.969253 write 0x020e45 thru 0x02e315 (0xd4d1 bytes) 000654 1780990903.990810 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e316): Mirror 8: 0x88ddd5a4 (stale) Mirror 11: 0x88ddd5a4 (stale) Mirror 14: 0x88ddd5a4 (stale) Mirror 15: 0x88ddd5a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e316): Mirror 8: 0x88ddd5a4 (stale) Mirror 11: 0x88ddd5a4 (stale) Mirror 14: 0x88ddd5a4 (stale) Mirror 15: 0x88ddd5a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e316: skipped 000655 1780990904.613254 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2e316): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e316): Mirror 8: 0x88ddd5a4 (stale) Mirror 11: 0x88ddd5a4 (stale) Mirror 14: 0x88ddd5a4 (stale) Mirror 15: 0x88ddd5a4 (stale) CRC-32 checksum value for chunk [0, 0x2e316): Mirror 1: 0x86d3904c Mirror 16: 0x86d3904c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e316: skipped 000656 1780990906.221950 trunc from 0x02e316 to 0x006c33 (0x276e2 bytes) 000657 1780990906.518271 mapwrite 0x01c413 thru 0x02b963 (0xf551 bytes) 000660 1780990906.838472 trunc from 0x02b964 to 0x02614e (0x5815 bytes) 000665 1780990906.892203 write 0x003496 thru 0x0052ee (0x1e59 bytes) 000666 1780990906.909170 trunc from 0x02614f to 0x019c28 (0xc526 bytes) 000668 1780990906.962690 MIRROR_RESYNC 6 mirrors 000669 1780990908.627434 MIRROR_EXTEND to 7 mirrors 000672 1780990909.920294 write 0x00f154 thru 0x01ad5a (0xbc07 bytes) 000673 1780990910.052448 mapwrite 0x025881 thru 0x02a6a9 (0x4e29 bytes) 000674 1780990910.233972 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2a6aa): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a6aa): Mirror 8: 0x7ca468fe (stale) Mirror 11: 0x7ca468fe (stale) Mirror 14: 0x7ca468fe (stale) Mirror 15: 0x7ca468fe (stale) Mirror 16: 0x7ca468fe (stale) Mirror 17: 0x7ca468fe (stale) CRC-32 checksum value for chunk [0, 0x2a6aa): Mirror 1: 0x27542e67 Mirror 18: 0x27542e67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a6aa: skipped 000676 1780990911.760900 mapwrite 0x00c21c thru 0x012604 (0x63e9 bytes) 000677 1780990912.164336 trunc from 0x02a6aa to 0x03dc54 (0x135ab bytes) 000678 1780990912.260165 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc55): Mirror 8: 0xa5ae6090 (stale) Mirror 11: 0xa5ae6090 (stale) Mirror 14: 0xa5ae6090 (stale) Mirror 15: 0xa5ae6090 (stale) Mirror 17: 0xa5ae6090 (stale) Mirror 18: 0xf3f1d98f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc55): Mirror 8: 0xa5ae6090 (stale) Mirror 11: 0xa5ae6090 (stale) Mirror 14: 0xa5ae6090 (stale) Mirror 15: 0xa5ae6090 (stale) Mirror 17: 0xa5ae6090 (stale) Mirror 18: 0xf3f1d98f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc55: skipped 000679 1780990913.059026 trunc from 0x03dc55 to 0x031379 (0xc8db bytes) 000681 1780990913.098007 MIRROR_RESYNC 7 mirrors 000682 1780990914.862582 mapread 0x02b626 thru 0x031379 (0x5d54 bytes) 000684 1780990914.886318 MIRROR_EXTEND to 8 mirrors 000687 1780990915.840608 mapwrite 0x03c43c thru 0x03ffff (0x3bc4 bytes) 000689 1780990916.247551 MIRROR_RESYNC 8 mirrors 000692 1780990918.162436 write 0x02d612 thru 0x0318ea (0x42d9 bytes) 000693 1780990918.253979 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe3ee1eb9 (stale) Mirror 11: 0xe3ee1eb9 (stale) Mirror 14: 0xe3ee1eb9 (stale) Mirror 15: 0xe3ee1eb9 (stale) Mirror 18: 0xe3ee1eb9 (stale) Mirror 19: 0xe3ee1eb9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe3ee1eb9 (stale) Mirror 11: 0xe3ee1eb9 (stale) Mirror 14: 0xe3ee1eb9 (stale) Mirror 15: 0xe3ee1eb9 (stale) Mirror 18: 0xe3ee1eb9 (stale) Mirror 19: 0xe3ee1eb9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1780990918.935981 read 0x0269f0 thru 0x0304da (0x9aeb bytes) 000703 1780990918.960923 write 0x03d6d8 thru 0x03ffff (0x2928 bytes) 000704 1780990918.973150 mapread 0x0067b0 thru 0x013178 (0xc9c9 bytes) 000705 1780990919.014227 mapwrite 0x033b2c thru 0x03886d (0x4d42 bytes) 000707 1780990919.102210 read 0x03d9c4 thru 0x03ffff (0x263c bytes) 000708 1780990919.103284 mapwrite 0x01a3e9 thru 0x01f4fd (0x5115 bytes) 000709 1780990919.207524 read 0x007605 thru 0x012dc8 (0xb7c4 bytes) 000711 1780990919.227098 mapread 0x0328e6 thru 0x03b4fe (0x8c19 bytes) 000713 1780990919.245532 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe3ee1eb9 (stale) Mirror 11: 0xe3ee1eb9 (stale) Mirror 14: 0xe3ee1eb9 (stale) Mirror 15: 0xe3ee1eb9 (stale) Mirror 18: 0xe3ee1eb9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe3ee1eb9 (stale) Mirror 11: 0xe3ee1eb9 (stale) Mirror 14: 0xe3ee1eb9 (stale) Mirror 15: 0xe3ee1eb9 (stale) Mirror 18: 0xe3ee1eb9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000715 1780990919.942225 read 0x009537 thru 0x011fa4 (0x8a6e bytes) 000717 1780990919.953233 mapwrite 0x031aec thru 0x03a05a (0x856f bytes) 000718 1780990920.030658 read 0x010e66 thru 0x01eca2 (0xde3d bytes) 000719 1780990920.048158 read 0x03f062 thru 0x03ffff (0xf9e bytes) 000723 1780990920.053979 read 0x03a459 thru 0x03bbe7 (0x178f bytes) 000724 1780990920.059534 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe3ee1eb9 (stale) Mirror 14: 0xe3ee1eb9 (stale) Mirror 15: 0xe3ee1eb9 (stale) Mirror 18: 0xe3ee1eb9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe3ee1eb9 (stale) Mirror 14: 0xe3ee1eb9 (stale) Mirror 15: 0xe3ee1eb9 (stale) Mirror 18: 0xe3ee1eb9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000725 1780990920.561721 mapread 0x010236 thru 0x01d7eb (0xd5b6 bytes) 000726 1780990920.599481 mapread 0x032a35 thru 0x0334fa (0xac6 bytes) 000729 1780990920.610706 mapread 0x02db83 thru 0x031d7b (0x41f9 bytes) 000732 1780990920.618594 MIRROR_RESYNC 5 mirrors 000734 1780990922.410823 write 0x00c976 thru 0x011941 (0x4fcc bytes) 000736 1780990922.508285 mapread 0x00d09c thru 0x01aa6e (0xd9d3 bytes) 000738 1780990922.521076 read 0x017eca thru 0x01870e (0x845 bytes) 000739 1780990922.522687 trunc from 0x040000 to 0x00427b (0x3bd84 bytes) 000741 1780990922.557111 MIRROR_RESYNC 5 mirrors 000742 1780990924.016311 write 0x01a389 thru 0x01a78a (0x402 bytes) 000744 1780990924.134941 mapread 0x007de9 thru 0x00a6ff (0x2917 bytes) 000749 1780990924.191729 mapread 0x00ad64 thru 0x01385b (0x8af8 bytes) 000750 1780990924.208977 read 0x0176a8 thru 0x01a78a (0x30e3 bytes) 000754 1780990924.217958 mapread 0x001ce2 thru 0x002ab1 (0xdd0 bytes) 000755 1780990924.229893 mapwrite 0x015cd9 thru 0x0255fc (0xf924 bytes) 000756 1780990924.403079 mapread 0x014582 thru 0x01dd5d (0x97dc bytes) 000757 1780990924.404317 mapwrite 0x0248f8 thru 0x033f54 (0xf65d bytes) 000758 1780990924.531437 trunc from 0x033f55 to 0x0215d2 (0x12982 bytes) 000761 1780990924.554708 mapread 0x00f084 thru 0x016d59 (0x7cd6 bytes) 000765 1780990924.562127 mapread 0x01d900 thru 0x020ea5 (0x35a6 bytes) 000766 1780990924.564108 write 0x00a861 thru 0x012bda (0x837a bytes) 000767 1780990924.578197 read 0x00fc3b thru 0x01c71f (0xcae5 bytes) 000768 1780990924.578975 read 0x000c6b thru 0x00e60e (0xd9a4 bytes) 000775 1780990924.603706 mapread 0x020b3b thru 0x0215d2 (0xa98 bytes) 000776 1780990924.605280 mapwrite 0x03f6d7 thru 0x03ffff (0x929 bytes) 000777 1780990924.721117 read 0x014a98 thru 0x02140c (0xc975 bytes) 000780 1780990924.722987 mapwrite 0x02e862 thru 0x034dd8 (0x6577 bytes) 000782 1780990924.807481 MIRROR_RESYNC 5 mirrors 000784 1780990926.512023 write 0x01ae01 thru 0x02481b (0x9a1b bytes) 000785 1780990926.663432 mapwrite 0x005b5b thru 0x007c29 (0x20cf bytes) 000788 1780990926.732825 mapwrite 0x00aad9 thru 0x015802 (0xad2a bytes) 000791 1780990926.841915 read 0x035439 thru 0x03ffff (0xabc7 bytes) 000792 1780990926.871902 write 0x03ce3f thru 0x03ffff (0x31c1 bytes) 000795 1780990926.889148 write 0x00e4e2 thru 0x01c061 (0xdb80 bytes) 000796 1780990926.915012 MIRROR_RESYNC 5 mirrors 000800 1780990928.475997 trunc from 0x040000 to 0x005130 (0x3aecf bytes) 000801 1780990928.615748 MIRROR_RESYNC 5 mirrors 000803 1780990930.395947 MIRROR_RESYNC 5 mirrors 000805 1780990931.263955 write 0x022258 thru 0x025d10 (0x3ab9 bytes) 000807 1780990931.374436 mapwrite 0x000573 thru 0x00c00f (0xba9d bytes) 000808 1780990931.557602 mapread 0x010878 thru 0x014538 (0x3cc1 bytes) 000809 1780990931.575063 write 0x026cbd thru 0x035420 (0xe764 bytes) 000811 1780990931.580212 write 0x0270f7 thru 0x029e9a (0x2da4 bytes) 000814 1780990931.584437 read 0x0010e8 thru 0x00e4a4 (0xd3bd bytes) 000816 1780990931.586972 trunc from 0x035421 to 0x0250b0 (0x10370 bytes) 000818 1780990931.630702 read 0x004fb5 thru 0x012481 (0xd4cd bytes) 000819 1780990931.646863 MIRROR_RESYNC 5 mirrors 000820 1780990933.206828 trunc from 0x0250b1 to 0x00997e (0x1b732 bytes) 000821 1780990933.314593 mapread 0x00658d thru 0x00997e (0x33f2 bytes) 000823 1780990933.342622 mapwrite 0x00a635 thru 0x012cf5 (0x86c1 bytes) 000824 1780990933.456982 mapread 0x0113e3 thru 0x012cf5 (0x1913 bytes) 000828 1780990933.461561 trunc from 0x012cf6 to 0x012d35 (0x40 bytes) 000829 1780990933.488157 read 0x008de4 thru 0x00cf2d (0x414a bytes) 000830 1780990933.492946 read 0x011a12 thru 0x012d35 (0x1324 bytes) 000831 1780990933.499035 trunc from 0x012d36 to 0x01595c (0x2c27 bytes) 000832 1780990933.526873 write 0x0277da thru 0x02d2e9 (0x5b10 bytes) 000834 1780990933.529036 read 0x0297f4 thru 0x02af49 (0x1756 bytes) 000835 1780990933.532513 write 0x00d57d thru 0x01b5c5 (0xe049 bytes) 000836 1780990933.542353 write 0x011840 thru 0x0196a0 (0x7e61 bytes) 000837 1780990933.543865 trunc from 0x02d2ea to 0x02468c (0x8c5d bytes) 000839 1780990933.564770 MIRROR_RESYNC 5 mirrors 000840 1780990935.214578 mapread 0x00c59a thru 0x0173c2 (0xae29 bytes) 000842 1780990935.257877 mapwrite 0x02337a thru 0x02e4f3 (0xb17a bytes) 000846 1780990935.621786 mapread 0x0261e6 thru 0x02e4f3 (0x830e bytes) 000847 1780990935.623098 trunc from 0x02e4f4 to 0x039170 (0xac7d bytes) 000848 1780990935.683900 read 0x02704f thru 0x034db9 (0xdd6b bytes) 000849 1780990935.699182 mapwrite 0x032ca6 thru 0x03e769 (0xbac4 bytes) 000850 1780990935.864874 read 0x009989 thru 0x0167d6 (0xce4e bytes) 000851 1780990935.882984 write 0x031b5b thru 0x0327f3 (0xc99 bytes) 000852 1780990935.884021 read 0x009542 thru 0x0139b6 (0xa475 bytes) 000855 1780990935.907931 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e76a): Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e76a): Mirror 8: 0xdb9d3836 (stale) Mirror 14: 0xdb9d3836 (stale) Mirror 15: 0xdb9d3836 (stale) Mirror 18: 0xdb9d3836 (stale) CRC-32 checksum value for chunk [0, 0x3e76a): Mirror 1: 0xa931749 Mirror 19: 0xa931749 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e76a: skipped 000856 1780990937.141917 mapwrite 0x01231c thru 0x019819 (0x74fe bytes) 000857 1780990937.467887 trunc from 0x03e76a to 0x00a0e2 (0x34687 bytes) 000859 1780990937.531812 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa0e3): Mirror 8: 0x1a8ea458 (stale) Mirror 15: 0x1a8ea458 (stale) Mirror 18: 0x1a8ea458 (stale) Mirror 19: 0x1a8ea458 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa0e3): Mirror 8: 0x1a8ea458 (stale) Mirror 15: 0x1a8ea458 (stale) Mirror 18: 0x1a8ea458 (stale) Mirror 19: 0x1a8ea458 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa0e3: skipped 000860 1780990938.085547 write 0x03e1d6 thru 0x03ffff (0x1e2a bytes) 000861 1780990938.089179 mapwrite 0x0208f2 thru 0x02f5bf (0xecce bytes) 000863 1780990938.213867 read 0x03a86e thru 0x03ffff (0x5792 bytes) 000866 1780990938.224419 mapread 0x024f66 thru 0x02d4f5 (0x8590 bytes) 000867 1780990938.228955 read 0x018dcf thru 0x0275c9 (0xe7fb bytes) 000869 1780990938.263753 MIRROR_RESYNC 5 mirrors 000870 1780990940.058306 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) 000872 1780990940.647567 write 0x0159a9 thru 0x018681 (0x2cd9 bytes) 000873 1780990940.779974 trunc from 0x040000 to 0x02f98e (0x10671 bytes) 000875 1780990940.830014 mapread 0x00096e thru 0x00f4fc (0xeb8f bytes) 000879 1780990940.874463 mapwrite 0x00f405 thru 0x01424c (0x4e48 bytes) 000881 1780990941.023521 read 0x00d1a4 thru 0x013630 (0x648d bytes) 000882 1780990941.025866 trunc from 0x02f98f to 0x02ff7a (0x5ec bytes) 000890 1780990941.089696 trunc from 0x02ff7b to 0x02f3b0 (0xbca bytes) 000891 1780990941.141887 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f3b1): Mirror 8: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f3b1): Mirror 8: 0x1c003f37 (stale) Mirror 18: 0x1c003f37 (stale) Mirror 19: 0x1c003f37 (stale) CRC-32 checksum value for chunk [0, 0x2f3b1): Mirror 1: 0x61893f1d Mirror 20: 0x61893f1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f3b1: skipped 000892 1780990942.454734 mapread 0x01b34d thru 0x01ef13 (0x3bc7 bytes) 000895 1780990942.511600 mapwrite 0x03dcf0 thru 0x03ffff (0x2310 bytes) 000897 1780990942.705547 mapread 0x026954 thru 0x02b83d (0x4eea bytes) 000898 1780990942.759570 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x7bffb3be (stale) Mirror 19: 0x7bffb3be (stale) Mirror 20: 0x8818d1e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x7bffb3be (stale) Mirror 19: 0x7bffb3be (stale) Mirror 20: 0x8818d1e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000903 1780990943.443290 trunc from 0x040000 to 0x01a0bb (0x25f44 bytes) 000904 1780990943.486470 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a0bc): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0bc): Mirror 18: 0x5381dfd6 (stale) Mirror 19: 0x5381dfd6 (stale) Mirror 20: 0x3b18887e (stale) CRC-32 checksum value for chunk [0, 0x1a0bc): Mirror 1: 0x3b18887e Mirror 21: 0x3b18887e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0bc: skipped 000905 1780990945.020967 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0bc): Mirror 18: 0x5381dfd6 (stale) Mirror 20: 0x3b18887e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0bc): Mirror 18: 0x5381dfd6 (stale) Mirror 20: 0x3b18887e (stale) CRC-32 checksum value for chunk [0, 0x1a0bc): Mirror 1: 0x3b18887e Mirror 21: 0x3b18887e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0bc: skipped 000908 1780990946.243029 trunc from 0x01a0bc to 0x01105e (0x905d bytes) 000913 1780990946.422007 mapwrite 0x039412 thru 0x03ffff (0x6bee bytes) 000915 1780990946.697829 write 0x02e42a thru 0x03323e (0x4e15 bytes) 000917 1780990946.717444 read 0x0138e9 thru 0x02196d (0xe085 bytes) 000918 1780990946.747141 MIRROR_RESYNC 4 mirrors 000919 1780990948.147402 trunc from 0x040000 to 0x030fa1 (0xf05e bytes) 000921 1780990948.299365 mapread 0x008458 thru 0x0111be (0x8d67 bytes) 000922 1780990948.370305 trunc from 0x030fa2 to 0x000855 (0x3074c bytes) 000924 1780990948.436775 read 0x00040c thru 0x000855 (0x44a bytes) 000926 1780990948.448663 trunc from 0x000856 to 0x01944a (0x18bf5 bytes) 000927 1780990948.500325 trunc from 0x01944b to 0x007c82 (0x117c8 bytes) 000928 1780990948.542515 trunc from 0x007c83 to 0x03dfa9 (0x36327 bytes) 000929 1780990948.592774 read 0x0004c8 thru 0x00df20 (0xda59 bytes) 000931 1780990948.640744 MIRROR_RESYNC 4 mirrors 000933 1780990950.252830 MIRROR_RESYNC 4 mirrors 000935 1780990950.725582 MIRROR_EXTEND to 5 mirrors 000939 1780990951.835151 trunc from 0x03dfaa to 0x036689 (0x7920 bytes) 000940 1780990952.117865 mapread 0x0149ac thru 0x014d63 (0x3b8 bytes) 000941 1780990952.137858 mapwrite 0x032fc2 thru 0x034c46 (0x1c85 bytes) 000943 1780990952.325227 MIRROR_RESYNC 5 mirrors 000944 1780990954.166845 write 0x00652f thru 0x0112f6 (0xadc8 bytes) 000945 1780990954.366775 mapwrite 0x008037 thru 0x011802 (0x97cc bytes) 000946 1780990954.511963 write 0x036ae7 thru 0x03ffff (0x9519 bytes) 000948 1780990954.535804 write 0x02db53 thru 0x033839 (0x5ce7 bytes) 000951 1780990954.585095 trunc from 0x040000 to 0x00a734 (0x358cb bytes) 000952 1780990954.649090 trunc from 0x00a735 to 0x007044 (0x36f0 bytes) 000954 1780990954.708439 read 0x000745 thru 0x0064c8 (0x5d84 bytes) 000956 1780990954.730409 read 0x003b7f thru 0x0061a5 (0x2627 bytes) 000957 1780990954.732900 read 0x005408 thru 0x007044 (0x1c3d bytes) 000958 1780990954.745417 trunc from 0x007045 to 0x0073db (0x397 bytes) 000962 1780990954.775317 read 0x0041f3 thru 0x005f72 (0x1d80 bytes) 000964 1780990954.779375 mapwrite 0x006480 thru 0x00b1fa (0x4d7b bytes) 000965 1780990955.002894 trunc from 0x00b1fb to 0x018aac (0xd8b2 bytes) 000966 1780990955.050404 trunc from 0x018aad to 0x015687 (0x3425 bytes) 000967 1780990955.120958 write 0x008960 thru 0x00e79c (0x5e3d bytes) 000970 1780990955.164927 mapwrite 0x0129ef thru 0x015c5c (0x326e bytes) 000973 1780990955.288425 trunc from 0x015c5d to 0x03cbe4 (0x26f88 bytes) 000974 1780990955.356338 write 0x0221fa thru 0x0304c6 (0xe2cd bytes) 000975 1780990955.385844 trunc from 0x03cbe5 to 0x004d64 (0x37e80 bytes) 000976 1780990955.438798 mapread 0x002698 thru 0x004d64 (0x26cd bytes) 000978 1780990955.448780 mapwrite 0x01df1b thru 0x026d79 (0x8e5f bytes) 000979 1780990955.728832 mapwrite 0x03363a thru 0x03458a (0xf51 bytes) 000980 1780990955.813719 read 0x00daef thru 0x01b7a1 (0xdcb3 bytes) 000981 1780990955.829996 trunc from 0x03458b to 0x022f4b (0x1163f bytes) 000983 1780990955.854660 mapwrite 0x033040 thru 0x037469 (0x442a bytes) 000985 1780990956.038191 read 0x02cd5a thru 0x031730 (0x49d7 bytes) 000986 1780990956.070027 mapwrite 0x009985 thru 0x012715 (0x8d91 bytes) 000988 1780990956.218086 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3746a): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3746a): Mirror 18: 0x16800943 (stale) Mirror 20: 0x16800943 (stale) Mirror 21: 0x16800943 (stale) Mirror 22: 0x16800943 (stale) CRC-32 checksum value for chunk [0, 0x3746a): Mirror 1: 0x379faef Mirror 23: 0x379faef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3746a: skipped 000991 1780990957.799184 mapwrite 0x028dd0 thru 0x02a092 (0x12c3 bytes) 000992 1780990958.066836 write 0x00ec05 thru 0x01d799 (0xeb95 bytes) 000993 1780990958.086171 write 0x0266c2 thru 0x031e67 (0xb7a6 bytes) 000994 1780990958.087731 read 0x03630a thru 0x037469 (0x1160 bytes) 000997 1780990958.111372 read 0x000a4e thru 0x00c7b9 (0xbd6c bytes) All operations completed A-OK! PASS 70a (128s) == sanity-flr test 71: check mirror extend parent fid ==== 03:42:45 (1780990965) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 03:42:49 (1780990969) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.29774 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15278 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 03:42:59 (1780990979) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5110400 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3320768 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5992576 .. Extending file size to 7454432 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6179360 .. Extending file size to 7749728 .. Extending file size to 7883552 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8982048 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7907680 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8770016 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56514 56515 56517 56518 56519 done 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) 192.168.206.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:44:26 (1780991066) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8058144 .. Extending file size to 7325824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7677504 .. Extending file size to 8882176 .. Extending file size to 7734336 .. Extending file size to 8686496 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8990784 .. Extending file size to 9039968 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9079936 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57805 57806 57807 57809 57810 done 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) 192.168.206.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (89s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:45:56 (1780991156) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.057739 s, 36.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.472307 s, 4.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x2549963 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) PASS 200c (24s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:46:22 (1780991182) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220908 s, 9.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 03:46:31 (1780991191) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x191:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:46:40 (1780991200) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.40918 s, 5.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:46:50 (1780991210) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.34656 s, 6.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x286:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:47:00 (1780991220) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.371319 s, 5.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:47:37 (1780991257) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg627-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182533 s, 5.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:48:42 (1780991322) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg627-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.526103 s, 4.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:49:46 (1780991386) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg627-server mds-ost sync done. Creating new pool test_204f oleg627-server: Pool lustre.test_204f created Adding targets to pool oleg627-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg627-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242895 s, 8.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg627-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg627-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg627-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:51:11 (1780991471) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:51:20 (1780991480) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:51:29 (1780991489) oleg627-server: Pool lustre.test_206 created Adding targets to pool oleg627-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg627-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg627-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg627-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg627-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:51:56 (1780991516) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115447 s, 9.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560293 s, 18.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x617cc04 Mirror 3: 0x617cc04 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:52:10 (1780991530) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:52:15 (1780991535) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:52:19 (1780991539) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.169182 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1960000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ac0000 Read this (4 bytes): BBBB 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:53:03 (1780991583) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0414162 s, 198 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:53:12 (1780991592) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570825 s, 18.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1b6 lmm_fid: [0x200000404:0x1b6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1b6 lmm_fid: [0x200000404:0x1b6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834358 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 03:53:14.000000000 -0400 Modify: 2026-06-09 03:53:14.000000000 -0400 Change: 2026-06-09 03:53:16.000000000 -0400 Birth: 2026-06-09 03:53:15.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:53:23 (1780991603) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:53:28 (1780991608) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.283 s, 31.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0475311 s, 86.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 9 03:53 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 9 03:53 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:53:40 (1780991620) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:53:49 (1780991629) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (15s) == sanity-flr test complete, duration 1947 sec =========== 03:54:04 (1780991644) === sanity-flr: start cleanup 03:54:06 (1780991646) === === sanity-flr: finish cleanup 03:54:16 (1780991656) ===