-----============= acceptance-small: sanity-flr ============----- Sun Mar 15 16:54:30 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg129-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:54:58 (1773608098) === oleg129-client.virtnet: executing check_config_client /mnt/lustre oleg129-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg129-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f46c8022000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f46c8022000.idle_timeout=debug disable quota as required oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:55:45 (1773608145) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7593] == sanity-flr test 0a: lfs mirror create with -N option == 16:55:49 (1773608149) PASS 0a (24s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:56:13 (1773608173) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:56:18 (1773608178) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:56:23 (1773608183) PASS 0d (33s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:56:56 (1773608216) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:57:01 (1773608221) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 16:57:06 (1773608226) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f46c8022000.stats=clear osc.lustre-OST0001-osc-ffff8f46c8022000.stats=clear PASS 0g (12s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:57:17 (1773608237) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (16s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:57:33 (1773608253) PASS 0j (15s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:57:48 (1773608268) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:57:52 (1773608272) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (16s) == sanity-flr test 2: create components from existing files ========================================================== 16:58:08 (1773608288) PASS 2 (12s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:58:20 (1773608300) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:58:25 (1773608305) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:58:35 (1773608315) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:58:50 (1773608330) PASS 7 (11s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:59:01 (1773608341) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 3.52409 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195771 s, 5.4 MB/s PASS 21 (17s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:59:18 (1773608358) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.786399 s, 20.0 MB/s PASS 22 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:59:30 (1773608370) Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (94s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:01:04 (1773608464) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.959295 s, 10.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (57s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:02:01 (1773608521) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (101s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:03:42 (1773608622) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 8.92133 s, 5.9 MB/s -rw-r--r-- 1 root root 52428800 Mar 15 17:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.556s user 0m0.002s sys 0m0.235s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f46c8022000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.688s user 0m0.002s sys 0m0.207s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.736s user 0m0.000s sys 0m0.213s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg129-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f46c8022000.ost_server_uuid in FULL state after 0 sec PASS 33b (122s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:05:45 (1773608745) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:05:50 (1773608750) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:05:53 (1773608753) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 17:05:57 (1773608757) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 17:06:07 (1773608767) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.938846 s, 13.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (20s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:06:27 (1773608787) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513634 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062864 s, 16.7 MB/s PASS 36b (12s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:06:39 (1773608799) PASS 36c (13s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:06:52 (1773608812) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012777 s, 801 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000919191 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128416 s, 797 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00272725 s, 375 kB/s flv=10 olv=10 PASS 36d (18s) == sanity-flr test 37: mirror I/O API verification ======= 17:07:10 (1773608830) create test files with size 15732964 .. 5528cb0ce0648a0e885cc8104394f320 - e6fa44d7424e0ecd7584a1ff26e68e48 - 6132c5bb0d13fb647c6724e68d1cffb4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 17:07:40 (1773608860) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:07:57 (1773608877) PASS 39 (12s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:08:10 (1773608890) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195556 s, 5.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441034 s, 23.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 17:08:22 (1773608902) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191288 s, 11.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.578939 s, 7.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.403966 s, 5.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (24s) == sanity-flr test 42: lfs mirror verify ================= 17:08:46 (1773608926) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 17:08:51 (1773608931) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:08:56 (1773608936) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362578 s, 28.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (12s) == sanity-flr test 44a: lfs mirror split check =========== 17:09:08 (1773608948) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:09:13 (1773608953) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (14s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:09:30 (1773608970) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.24253 s, 43.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:09:48 (1773608988) llite.lustre-ffff8f46d631e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.294049 s, 35.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8f46c8bff800.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:10:08 (1773609008) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (60s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:11:08 (1773609068) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (60s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:12:08 (1773609128) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 17:12:13 (1773609133) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.544261 s, 5.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223397 s, 14.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf2673ffa (stale) Mirror 2: 0xf2673ffa (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 2f0a6df49cd7b294ea282f22f8f5c259 - mirror 1 checksum: 017e9d97c89b5d23db84855e3b697b35 - mirror 2 checksum: 2f0a6df49cd7b294ea282f22f8f5c259 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Mar 15 17:12:16 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Mar 15 17:12:16 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2f0a6df49cd7b294ea282f22f8f5c259 - mirror 1 checksum: 017e9d97c89b5d23db84855e3b697b35 - mirror 2 checksum: 017e9d97c89b5d23db84855e3b697b35 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Mar 15 17:12:16 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Mar 15 17:12:16 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:12:31 (1773609151) Failing mds1 on oleg129-server Stopping /mnt/lustre-mds1 (opts:) on oleg129-server 17:12:41 (1773609161) shut down facet: mds1 facet_host: oleg129-server facet_failover_host: oleg129-server Failover mds1 to oleg129-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 17:13:17 (1773609197) targets are mounted 17:13:17 (1773609197) facet_failover done oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607389 s, 17.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (68s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:13:40 (1773609220) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:13:45 (1773609225) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:13:51 (1773609231) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0187055 s, 876 kB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:14:04 (1773609244) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:14:10 (1773609250) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:14:16 (1773609256) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (39s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:14:55 (1773609295) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (35s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:15:30 (1773609330) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (29s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:16:00 (1773609360) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (29s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:16:28 (1773609388) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (11s) == sanity-flr test 70: mirror create and split race ====== 17:16:40 (1773609400) mirror create pid 52387 mirror split pid 52389 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52387 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52389 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (72s) == sanity-flr test 70a: flr mode fsx test ================ 17:17:52 (1773609472) Seed set to 9475 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x31943 000002 1773609476.299100 trunc from 00000000 to 0x031942 (0x31943 bytes) 000003 1773609476.339858 trunc from 0x031943 to 0x0064e8 (0x2b45a bytes) 000005 1773609476.369367 read 0x000b4f thru 0x0064e8 (0x599a bytes) 000007 1773609476.382708 read 0x00126c thru 0x0064e8 (0x527d bytes) 000008 1773609476.384829 trunc from 0x0064e9 to 0x012893 (0xc3ab bytes) 000010 1773609476.419311 write 0x02dccd thru 0x02ee54 (0x1188 bytes) 000011 1773609476.421285 mapwrite 0x02c5ae thru 0x03a157 (0xdbaa bytes) 000012 1773609476.693035 write 0x01f0a6 thru 0x02dc5e (0xebb9 bytes) 000014 1773609476.698089 mapwrite 0x01b002 thru 0x01d7e7 (0x27e6 bytes) 000019 1773609476.886489 read 0x02e1a6 thru 0x03a157 (0xbfb2 bytes) 000024 1773609476.904953 trunc from 0x03a158 to 0x02dd2b (0xc42c bytes) 000025 1773609476.957738 mapwrite 0x00c155 thru 0x01612d (0x9fd9 bytes) 000030 1773609477.092697 write 0x01542a thru 0x024dce (0xf9a5 bytes) 000031 1773609477.095894 mapread 0x015287 thru 0x023b3e (0xe8b8 bytes) 000032 1773609477.107315 read 0x00f35f thru 0x01c057 (0xccf9 bytes) 000034 1773609477.108609 mapread 0x0188b1 thru 0x01f5fa (0x6d4a bytes) 000035 1773609477.114928 write 0x010f53 thru 0x01f316 (0xe3c4 bytes) 000036 1773609477.144268 MIRROR_EXTEND to 2 mirrors 000037 1773609478.442235 write 0x00d987 thru 0x00f3e7 (0x1a61 bytes) 000038 1773609478.812274 read 0x018b23 thru 0x027852 (0xed30 bytes) 000041 1773609478.855715 trunc from 0x02dd2c to 0x0099c6 (0x24365 bytes) 000045 1773609479.075292 MIRROR_RESYNC 2 mirrors 000047 1773609480.553416 read 0x0055b5 thru 0x006a70 (0x14bc bytes) 000048 1773609480.563158 read 0x00433a thru 0x0099c6 (0x568d bytes) 000049 1773609480.577749 mapwrite 0x017b63 thru 0x02333d (0xb7db bytes) 000053 1773609481.048126 write 0x02267d thru 0x022e2e (0x7b2 bytes) 000054 1773609481.049619 trunc from 0x02333e to 0x0143b4 (0xef89 bytes) 000055 1773609481.085323 mapread 0x000ba5 thru 0x0039bf (0x2e1b bytes) truncating to largest ever: 0x37308 000057 1773609481.117132 trunc from 0x0143b5 to 0x037307 (0x22f53 bytes) 000059 1773609481.148043 mapwrite 0x0087e3 thru 0x01200e (0x982c bytes) 000063 1773609481.254156 write 0x035386 thru 0x03ffff (0xac7a bytes) 000064 1773609481.257829 write 0x03aba6 thru 0x03ffff (0x545a bytes) 000065 1773609481.259792 mapwrite 0x03a629 thru 0x03f0c5 (0x4a9d bytes) 000067 1773609481.323693 trunc from 0x040000 to 0x01a0a6 (0x25f59 bytes) 000068 1773609481.345992 mapwrite 0x01ee2b thru 0x0238c2 (0x4a98 bytes) 000070 1773609481.601096 read 0x0120d8 thru 0x01a567 (0x8490 bytes) 000071 1773609481.633862 read 0x0071b2 thru 0x016a19 (0xf868 bytes) 000073 1773609481.635555 mapwrite 0x020be5 thru 0x02c3aa (0xb7c6 bytes) 000078 1773609481.924406 write 0x00649d thru 0x010359 (0x9ebd bytes) 000079 1773609481.930111 mapwrite 0x011fa2 thru 0x020e66 (0xeec5 bytes) 000080 1773609482.059640 mapwrite 0x0170fc thru 0x020fb3 (0x9eb8 bytes) 000081 1773609482.196525 trunc from 0x02c3ab to 0x0266da (0x5cd0 bytes) 000084 1773609482.263723 write 0x024f92 thru 0x02d155 (0x81c4 bytes) 000087 1773609482.283217 write 0x013c2b thru 0x0239e3 (0xfdb9 bytes) 000088 1773609482.294419 mapwrite 0x03638a thru 0x03ffff (0x9c76 bytes) 000089 1773609482.454053 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd8d606c8 Mirror 3: 0xd8d606c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000090 1773609483.808013 trunc from 0x040000 to 0x02863a (0x179c5 bytes) 000092 1773609483.979570 write 0x039be7 thru 0x03da3d (0x3e57 bytes) 000093 1773609483.983705 write 0x02ac34 thru 0x034842 (0x9c0f bytes) 000094 1773609484.007094 trunc from 0x03da3e to 0x0134c6 (0x2a577 bytes) 000096 1773609484.082054 read 0x005753 thru 0x01221b (0xcac9 bytes) 000097 1773609484.108122 trunc from 0x0134c7 to 0x0040a5 (0xf421 bytes) 000098 1773609484.153120 mapread 0x002571 thru 0x0040a5 (0x1b35 bytes) 000100 1773609484.163311 mapwrite 0x03c779 thru 0x03ffff (0x3887 bytes) 000101 1773609484.289236 MIRROR_RESYNC 3 mirrors 000102 1773609485.194091 mapread 0x02cbaf thru 0x03c1b1 (0xf603 bytes) 000103 1773609485.250221 write 0x000c3c thru 0x004f7b (0x4340 bytes) 000104 1773609485.372073 write 0x033cca thru 0x03ffff (0xc336 bytes) 000106 1773609485.385240 mapwrite 0x019446 thru 0x021c84 (0x883f bytes) 000107 1773609485.528367 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x12106f23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x12106f23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000108 1773609486.335265 mapwrite 0x01b83f thru 0x021f23 (0x66e5 bytes) 000112 1773609486.658119 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000113 1773609486.997130 write 0x03f61d thru 0x03ffff (0x9e3 bytes) 000114 1773609487.007866 mapwrite 0x02115c thru 0x025f03 (0x4da8 bytes) 000115 1773609487.110344 trunc from 0x040000 to 0x03068c (0xf973 bytes) 000116 1773609487.165424 mapwrite 0x01cd0a thru 0x029a8c (0xcd83 bytes) 000118 1773609487.257336 mapread 0x02b045 thru 0x03068c (0x5648 bytes) 000120 1773609487.282688 MIRROR_EXTEND to 2 mirrors 000121 1773609488.221409 read 0x02ff0f thru 0x03068c (0x77e bytes) 000122 1773609488.235214 mapwrite 0x022b13 thru 0x024498 (0x1986 bytes) 000124 1773609488.622865 trunc from 0x03068d to 0x02158a (0xf102 bytes) 000127 1773609488.737295 mapwrite 0x00b246 thru 0x00e14c (0x2f07 bytes) 000128 1773609488.878560 mapread 0x016847 thru 0x02158a (0xad44 bytes) 000130 1773609488.909259 trunc from 0x02158b to 0x022e7e (0x18f4 bytes) 000131 1773609488.984063 MIRROR_RESYNC 2 mirrors 000132 1773609490.204342 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000133 1773609490.637001 mapread 0x000556 thru 0x00f2dc (0xed87 bytes) 000134 1773609490.706256 mapread 0x00e112 thru 0x01dc71 (0xfb60 bytes) 000136 1773609490.755777 mapread 0x0008e4 thru 0x00d792 (0xceaf bytes) 000137 1773609490.757232 mapwrite 0x014807 thru 0x02324a (0xea44 bytes) 000138 1773609490.970228 trunc from 0x02324b to 0x0185dc (0xac6e bytes) 000142 1773609490.995926 write 0x016035 thru 0x016b2e (0xafa bytes) 000143 1773609490.997382 mapread 0x0150dc thru 0x0185dc (0x3501 bytes) 000148 1773609491.005062 mapwrite 0x037219 thru 0x03bbd5 (0x49bd bytes) 000149 1773609491.108333 read 0x0299c0 thru 0x031564 (0x7ba5 bytes) 000150 1773609491.122152 trunc from 0x03bbd6 to 0x00e4f7 (0x2d6de bytes) 000151 1773609491.145181 read 0x008fc0 thru 0x00d4d2 (0x4513 bytes) 000157 1773609491.168361 read 0x00a5c5 thru 0x00a7e3 (0x21f bytes) 000158 1773609491.169400 write 0x01978d thru 0x0282a6 (0xeb1a bytes) 000159 1773609491.174265 mapwrite 0x012ca6 thru 0x017ba9 (0x4f04 bytes) 000160 1773609491.273333 trunc from 0x0282a7 to 0x013be3 (0x146c3 bytes) 000161 1773609491.301308 write 0x028851 thru 0x02ec6c (0x641c bytes) 000164 1773609491.307081 write 0x0083fb thru 0x00a6d7 (0x22dd bytes) 000166 1773609491.313405 mapwrite 0x013436 thru 0x01de4a (0xaa15 bytes) 000167 1773609491.461910 mapread 0x019eac thru 0x022c01 (0x8d56 bytes) 000169 1773609491.511318 write 0x030725 thru 0x039583 (0x8e5f bytes) 000171 1773609491.519333 trunc from 0x039584 to 0x032667 (0x6f1c bytes) 000173 1773609491.560322 trunc from 0x032668 to 0x034ad6 (0x246f bytes) 000174 1773609491.594108 read 0x03136e thru 0x033326 (0x1fb9 bytes) 000176 1773609491.618816 read 0x02e7b0 thru 0x034ad6 (0x6327 bytes) 000179 1773609491.626111 mapread 0x0157dc thru 0x01af18 (0x573d bytes) 000180 1773609491.627913 write 0x01ad59 thru 0x01b9d6 (0xc7e bytes) 000186 1773609491.642421 read 0x019057 thru 0x023c3a (0xabe4 bytes) 000187 1773609491.643447 mapread 0x022170 thru 0x02d675 (0xb506 bytes) 000188 1773609491.659723 MIRROR_EXTEND to 2 mirrors 000189 1773609493.052409 mapwrite 0x0269cc thru 0x0324de (0xbb13 bytes) 000190 1773609493.571399 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34ad7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ad7): Mirror 2: 0x96cc0a1a (stale) CRC-32 checksum value for chunk [0, 0x34ad7): Mirror 1: 0xe2c0a4a2 Mirror 3: 0xe2c0a4a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ad7: skipped 000191 1773609495.119311 mapwrite 0x01eba3 thru 0x0225fe (0x3a5c bytes) 000193 1773609495.494257 mapwrite 0x030250 thru 0x033755 (0x3506 bytes) 000194 1773609495.592281 write 0x01998b thru 0x026fad (0xd623 bytes) 000195 1773609495.621411 write 0x014d88 thru 0x024a13 (0xfc8c bytes) 000197 1773609495.644286 read 0x021320 thru 0x02d326 (0xc007 bytes) 000199 1773609495.654376 write 0x019500 thru 0x0279a7 (0xe4a8 bytes) 000200 1773609495.659203 write 0x0367eb thru 0x03ffff (0x9815 bytes) 000202 1773609495.667315 mapread 0x02ca39 thru 0x0353bc (0x8984 bytes) 000203 1773609495.675389 mapwrite 0x03c84f thru 0x03e869 (0x201b bytes) 000206 1773609495.894963 read 0x01882a thru 0x01c333 (0x3b0a bytes) 000207 1773609495.901182 mapwrite 0x007dc1 thru 0x00c503 (0x4743 bytes) 000208 1773609496.027849 mapread 0x038335 thru 0x03ffff (0x7ccb bytes) 000209 1773609496.030131 trunc from 0x040000 to 0x02b966 (0x14699 bytes) 000210 1773609496.062070 mapread 0x0082fd thru 0x012bd5 (0xa8d9 bytes) 000211 1773609496.063750 write 0x03959d thru 0x03df6a (0x49ce bytes) 000212 1773609496.068089 write 0x00acaf thru 0x00e6ab (0x39fd bytes) 000213 1773609496.069555 trunc from 0x03df6b to 0x02dc73 (0x102f7 bytes) 000214 1773609496.108050 mapwrite 0x014a62 thru 0x022149 (0xd6e8 bytes) 000215 1773609496.187431 read 0x0011dd thru 0x002b62 (0x1986 bytes) 000216 1773609496.189235 write 0x004615 thru 0x007e27 (0x3813 bytes) 000218 1773609496.196377 mapread 0x00c7d7 thru 0x01486a (0x8094 bytes) 000219 1773609496.198531 write 0x0178ff thru 0x018606 (0xd08 bytes) 000220 1773609496.201303 mapwrite 0x0211ad thru 0x0238e3 (0x2737 bytes) 000221 1773609496.250446 mapwrite 0x0251d8 thru 0x029fe8 (0x4e11 bytes) 000223 1773609496.360008 MIRROR_RESYNC 3 mirrors 000224 1773609497.913796 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000225 1773609498.709852 mapread 0x017841 thru 0x021351 (0x9b11 bytes) 000226 1773609498.751885 trunc from 0x02dc74 to 0x025080 (0x8bf3 bytes) 000227 1773609498.984973 mapwrite 0x0068ac thru 0x0113d9 (0xab2e bytes) 000228 1773609499.112035 mapwrite 0x0168de thru 0x01dc9d (0x73c0 bytes) 000230 1773609499.247478 trunc from 0x025081 to 0x00aab0 (0x1a5d0 bytes) 000233 1773609499.307128 write 0x0169c6 thru 0x01aeb7 (0x44f2 bytes) 000234 1773609499.308819 trunc from 0x01aeb8 to 0x0057ab (0x1570c bytes) 000235 1773609499.355426 write 0x008dd2 thru 0x0138fa (0xab29 bytes) 000237 1773609499.358961 write 0x016942 thru 0x02315f (0xc81e bytes) 000239 1773609499.365507 trunc from 0x023160 to 0x0159c9 (0xd796 bytes) 000240 1773609499.402251 mapwrite 0x005f28 thru 0x00b333 (0x540c bytes) 000241 1773609499.580341 write 0x012683 thru 0x01f9cf (0xd34d bytes) 000243 1773609499.609432 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000250 1773609499.977984 trunc from 0x01f9d0 to 0x02dc62 (0xe293 bytes) 000254 1773609500.032808 trunc from 0x02dc63 to 0x02b750 (0x2512 bytes) 000255 1773609500.079091 write 0x028fe8 thru 0x032f71 (0x9f8a bytes) 000256 1773609500.089207 mapwrite 0x014ce4 thru 0x01ddf8 (0x9115 bytes) 000258 1773609500.268960 trunc from 0x032f72 to 0x005513 (0x2da5e bytes) 000259 1773609500.325860 write 0x015a82 thru 0x01b8aa (0x5e29 bytes) 000260 1773609500.331250 mapwrite 0x017955 thru 0x01a78a (0x2e36 bytes) truncating to largest ever: 0x3fc08 000261 1773609500.526715 trunc from 0x01b8ab to 0x03fc07 (0x2435d bytes) 000263 1773609500.563144 mapwrite 0x0234c2 thru 0x02b702 (0x8241 bytes) 000264 1773609500.772250 mapwrite 0x03a15e thru 0x03b377 (0x121a bytes) 000265 1773609500.946823 trunc from 0x03fc08 to 0x037df7 (0x7e10 bytes) 000266 1773609500.991143 trunc from 0x037df8 to 0x023148 (0x14caf bytes) 000267 1773609501.037118 trunc from 0x023149 to 0x00c553 (0x16bf5 bytes) 000274 1773609501.108116 mapwrite 0x0306ce thru 0x0364ba (0x5ded bytes) 000276 1773609501.293894 read 0x03414d thru 0x0364ba (0x236e bytes) 000277 1773609501.305381 write 0x00bcd3 thru 0x016051 (0xa37f bytes) 000280 1773609501.364974 MIRROR_EXTEND to 2 mirrors 000281 1773609502.823540 mapread 0x015051 thru 0x02293b (0xd8eb bytes) 000283 1773609502.973091 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000284 1773609503.407021 mapread 0x0085c5 thru 0x00d96b (0x53a7 bytes) 000285 1773609503.462354 mapread 0x01579c thru 0x02578f (0xfff4 bytes) 000286 1773609503.596082 mapread 0x02c30e thru 0x02f263 (0x2f56 bytes) 000287 1773609503.635778 mapread 0x0340d6 thru 0x0364ba (0x23e5 bytes) 000289 1773609503.668739 trunc from 0x0364bb to 0x03a3a2 (0x3ee8 bytes) 000290 1773609503.803455 read 0x015b84 thru 0x018601 (0x2a7e bytes) 000291 1773609503.846960 trunc from 0x03a3a3 to 0x00fd56 (0x2a64c bytes) 000292 1773609503.979287 write 0x004e1e thru 0x011ddd (0xcfc0 bytes) 000297 1773609504.041962 write 0x0280d3 thru 0x034cd4 (0xcc02 bytes) 000298 1773609504.045353 write 0x00cc87 thru 0x017d76 (0xb0f0 bytes) 000300 1773609504.069892 mapread 0x0304a3 thru 0x034cd4 (0x4832 bytes) 000301 1773609504.071071 read 0x03233c thru 0x034cd4 (0x2999 bytes) 000302 1773609504.077180 write 0x03a7e6 thru 0x03ffff (0x581a bytes) 000303 1773609504.078929 write 0x01ed11 thru 0x02479a (0x5a8a bytes) 000304 1773609504.099233 trunc from 0x040000 to 0x02a109 (0x15ef6 bytes) 000305 1773609504.136067 trunc from 0x02a10a to 0x03ca4d (0x12944 bytes) 000307 1773609504.176152 mapwrite 0x034766 thru 0x03b20a (0x6aa5 bytes) 000308 1773609504.397033 mapwrite 0x001570 thru 0x010700 (0xf191 bytes) 000309 1773609504.536349 write 0x03bb30 thru 0x03ffff (0x44d0 bytes) 000310 1773609504.539056 trunc from 0x040000 to 0x024eb5 (0x1b14a bytes) 000311 1773609504.641971 trunc from 0x024eb6 to 0x031ef0 (0xd03b bytes) 000312 1773609504.687088 write 0x02bacb thru 0x038f82 (0xd4b8 bytes) 000313 1773609504.699391 mapwrite 0x03413f thru 0x03d048 (0x8f0a bytes) 000314 1773609504.970852 mapread 0x02086a thru 0x02bd09 (0xb4a0 bytes) 000317 1773609505.073833 mapwrite 0x03a807 thru 0x03ffff (0x57f9 bytes) 000318 1773609505.205359 write 0x0386f2 thru 0x03b00d (0x291c bytes) 000320 1773609505.208024 read 0x010a8b thru 0x019abb (0x9031 bytes) 000321 1773609505.209230 trunc from 0x040000 to 0x03e06a (0x1f95 bytes) 000324 1773609505.236396 trunc from 0x03e06b to 0x0255f7 (0x18a73 bytes) 000327 1773609505.262116 write 0x02966f thru 0x02e257 (0x4be9 bytes) 000328 1773609505.265377 write 0x02b5d4 thru 0x0316ad (0x60da bytes) 000330 1773609505.269253 mapread 0x00ff17 thru 0x012796 (0x2880 bytes) 000331 1773609505.271115 write 0x034437 thru 0x03a130 (0x5cfa bytes) 000334 1773609505.275336 mapwrite 0x026d5c thru 0x034471 (0xd716 bytes) 000335 1773609505.341057 mapread 0x021f32 thru 0x0262b4 (0x4383 bytes) 000336 1773609505.353727 MIRROR_EXTEND to 2 mirrors 000338 1773609506.423438 MIRROR_EXTEND to 3 mirrors 000339 1773609507.182391 mapwrite 0x0324c2 thru 0x034fc6 (0x2b05 bytes) 000340 1773609507.431330 trunc from 0x03a131 to 0x029354 (0x10ddc bytes) 000341 1773609507.514194 trunc from 0x029355 to 0x03c759 (0x13405 bytes) 000343 1773609507.550968 mapwrite 0x035bdb thru 0x03c4e4 (0x690a bytes) 000345 1773609507.633723 mapread 0x0201ff thru 0x02171d (0x151f bytes) 000346 1773609507.674268 MIRROR_RESYNC 3 mirrors 000347 1773609508.835420 read 0x00e600 thru 0x0134ca (0x4ecb bytes) 000348 1773609508.858245 write 0x003de7 thru 0x0114f3 (0xd70d bytes) 000350 1773609509.159256 write 0x03bef7 thru 0x03ffff (0x4109 bytes) 000352 1773609509.226136 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd5bc8628 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd5bc8628 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000353 1773609509.973962 write 0x039e75 thru 0x03a0ae (0x23a bytes) 000355 1773609510.030130 MIRROR_RESYNC 2 mirrors 000357 1773609510.991057 read 0x020365 thru 0x024f6a (0x4c06 bytes) 000358 1773609511.006021 read 0x017f6e thru 0x01c32f (0x43c2 bytes) 000363 1773609511.047765 mapread 0x013ad8 thru 0x021f97 (0xe4c0 bytes) 000365 1773609511.063206 MIRROR_RESYNC 2 mirrors 000367 1773609511.726747 MIRROR_EXTEND to 3 mirrors 000369 1773609513.120868 write 0x01c4b9 thru 0x01fcef (0x3837 bytes) 000370 1773609513.222244 read 0x01ed34 thru 0x026487 (0x7754 bytes) 000371 1773609513.233210 trunc from 0x040000 to 0x004163 (0x3be9c bytes) 000372 1773609513.264009 MIRROR_RESYNC 3 mirrors 000374 1773609514.494155 read 0x0018d9 thru 0x004163 (0x288b bytes) 000375 1773609514.505515 read 0x000982 thru 0x0017b5 (0xe34 bytes) 000376 1773609514.512327 mapwrite 0x029f7b thru 0x033c61 (0x9ce7 bytes) 000377 1773609514.979956 mapread 0x028693 thru 0x030314 (0x7c82 bytes) 000378 1773609514.986776 read 0x01c3d0 thru 0x029bc7 (0xd7f8 bytes) 000379 1773609515.009360 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c62): Mirror 4: 0xda48add1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c62): Mirror 4: 0xda48add1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c62: skipped 000380 1773609515.839129 trunc from 0x033c62 to 0x01abef (0x19072 bytes) 000382 1773609515.939088 MIRROR_RESYNC 2 mirrors 000383 1773609517.559954 write 0x03d6ea thru 0x03ffff (0x2916 bytes) 000385 1773609517.772088 read 0x007797 thru 0x016011 (0xe87b bytes) 000386 1773609517.816884 trunc from 0x040000 to 0x01b54b (0x24ab4 bytes) 000391 1773609517.908138 mapread 0x00ff44 thru 0x014f96 (0x5053 bytes) 000394 1773609517.934934 read 0x00dda0 thru 0x0138b3 (0x5b14 bytes) 000395 1773609517.941321 mapwrite 0x00ac54 thru 0x01a311 (0xf6be bytes) 000397 1773609518.237858 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b54c): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b54c): Mirror 4: 0x65af081c (stale) CRC-32 checksum value for chunk [0, 0x1b54c): Mirror 1: 0x92d073db Mirror 5: 0x92d073db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b54c: skipped 000398 1773609520.138234 read 0x00a992 thru 0x012ceb (0x835a bytes) 000399 1773609520.211518 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000400 1773609521.106529 MIRROR_EXTEND to 3 mirrors 000401 1773609522.890383 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000403 1773609523.506782 mapwrite 0x01e04b thru 0x027221 (0x91d7 bytes) 000404 1773609523.839997 MIRROR_RESYNC 2 mirrors 000405 1773609524.936377 mapwrite 0x0155ef thru 0x01cb86 (0x7598 bytes) 000406 1773609525.208262 mapwrite 0x0355bb thru 0x03919d (0x3be3 bytes) 000408 1773609525.397538 trunc from 0x03919e to 0x0009c7 (0x387d6 bytes) 000409 1773609525.429738 mapread 0x000730 thru 0x0009c7 (0x298 bytes) 000410 1773609525.448108 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000411 1773609525.724430 read 0x000312 thru 0x0009c7 (0x6b6 bytes) 000414 1773609525.740352 mapread 0x000899 thru 0x0009c7 (0x12f bytes) 000415 1773609525.741619 write 0x02a75f thru 0x039cff (0xf5a1 bytes) 000416 1773609525.755553 mapwrite 0x000a1f thru 0x00889f (0x7e81 bytes) 000417 1773609526.045047 read 0x02e73e thru 0x039cff (0xb5c2 bytes) 000419 1773609526.063210 mapwrite 0x03be4f thru 0x03c28c (0x43e bytes) 000421 1773609526.261919 read 0x008d3c thru 0x00ca77 (0x3d3c bytes) 000422 1773609526.264951 mapread 0x02ed04 thru 0x03849c (0x9799 bytes) 000423 1773609526.267980 write 0x014a4d thru 0x015e32 (0x13e6 bytes) 000424 1773609526.280682 mapwrite 0x03f355 thru 0x03ffff (0xcab bytes) 000425 1773609526.505874 read 0x02cd84 thru 0x031c38 (0x4eb5 bytes) 000426 1773609526.506673 write 0x03d0c8 thru 0x03dbfa (0xb33 bytes) 000428 1773609526.526963 mapwrite 0x01a442 thru 0x01c9a7 (0x2566 bytes) 000429 1773609526.728879 mapread 0x0202fe thru 0x020620 (0x323 bytes) 000433 1773609526.736654 mapwrite 0x03e8a0 thru 0x03ffff (0x1760 bytes) 000435 1773609526.873443 mapwrite 0x00414d thru 0x011e2d (0xdce1 bytes) 000437 1773609526.994248 MIRROR_EXTEND to 2 mirrors 000440 1773609528.624920 read 0x009559 thru 0x0108a2 (0x734a bytes) 000442 1773609528.687383 MIRROR_RESYNC 2 mirrors 000443 1773609529.452836 write 0x025565 thru 0x02f057 (0x9af3 bytes) 000444 1773609529.732268 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x299d05a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x58622ec Mirror 3: 0x58622ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000445 1773609531.583839 MIRROR_RESYNC 3 mirrors 000446 1773609532.856190 read 0x0223c6 thru 0x02a0bd (0x7cf8 bytes) 000449 1773609532.890301 read 0x0070e9 thru 0x012305 (0xb21d bytes) 000450 1773609532.921727 mapwrite 0x03db67 thru 0x03f7e9 (0x1c83 bytes) 000451 1773609533.253311 mapwrite 0x005058 thru 0x00f44c (0xa3f5 bytes) 000452 1773609533.426325 trunc from 0x040000 to 0x02d0bc (0x12f43 bytes) 000455 1773609533.484346 read 0x02b034 thru 0x02d0bc (0x2089 bytes) 000458 1773609533.515762 mapread 0x012ecd thru 0x014354 (0x1488 bytes) 000461 1773609533.524837 trunc from 0x02d0bd to 0x01ccf1 (0x103cb bytes) 000462 1773609533.552830 mapwrite 0x022626 thru 0x02e473 (0xbe4e bytes) 000463 1773609533.716798 mapwrite 0x00c5df thru 0x01167e (0x50a0 bytes) 000464 1773609533.767738 write 0x033643 thru 0x039625 (0x5fe3 bytes) 000466 1773609533.771918 mapread 0x027b4e thru 0x02d1de (0x5691 bytes) 000467 1773609533.774899 mapwrite 0x0105a0 thru 0x01481c (0x427d bytes) 000468 1773609533.869760 mapread 0x028d08 thru 0x02a6d6 (0x19cf bytes) 000470 1773609533.871450 read 0x00ff8e thru 0x01688c (0x68ff bytes) 000471 1773609533.872221 mapwrite 0x01b24b thru 0x028e28 (0xdbde bytes) 000472 1773609533.985589 mapwrite 0x01aab9 thru 0x0261c9 (0xb711 bytes) 000474 1773609534.049385 trunc from 0x039626 to 0x01f961 (0x19cc4 bytes) 000475 1773609534.070284 mapwrite 0x02c7fd thru 0x02cd4f (0x553 bytes) 000479 1773609534.190745 mapread 0x000a60 thru 0x0101d6 (0xf777 bytes) 000480 1773609534.195929 mapwrite 0x0204c6 thru 0x02418e (0x3cc9 bytes) 000482 1773609534.419705 read 0x02ba36 thru 0x02cd4f (0x131a bytes) 000483 1773609534.458516 mapwrite 0x031241 thru 0x03ffff (0xedbf bytes) 000484 1773609534.668676 read 0x02c54f thru 0x02ea77 (0x2529 bytes) 000490 1773609534.690917 write 0x00140b thru 0x010e0a (0xfa00 bytes) 000491 1773609534.700336 trunc from 0x040000 to 0x0196b9 (0x26946 bytes) 000492 1773609534.735422 mapwrite 0x025512 thru 0x02da1a (0x8509 bytes) 000493 1773609535.035381 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2da1b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da1b): Mirror 2: 0x57f71b42 (stale) Mirror 3: 0x57f71b42 (stale) CRC-32 checksum value for chunk [0, 0x2da1b): Mirror 1: 0x5ee4ad9d Mirror 4: 0x5ee4ad9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da1b: skipped 000494 1773609536.793979 mapread 0x00c62b thru 0x00e996 (0x236c bytes) 000496 1773609536.876832 mapread 0x020677 thru 0x02cd1b (0xc6a5 bytes) 000497 1773609536.941815 write 0x005373 thru 0x00fa73 (0xa701 bytes) 000498 1773609537.071210 mapwrite 0x038242 thru 0x03c4b5 (0x4274 bytes) 000504 1773609537.287147 read 0x0367d4 thru 0x03681c (0x49 bytes) 000505 1773609537.294129 write 0x03d40f thru 0x03ffff (0x2bf1 bytes) 000506 1773609537.317402 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x58622ec (stale) Mirror 4: 0xe7f4564 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x58622ec (stale) Mirror 4: 0xe7f4564 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000507 1773609538.046261 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x58622ec (stale) Mirror 4: 0xe7f4564 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ced3ea4 Mirror 5: 0x2ced3ea4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1773609540.097931 mapwrite 0x00a3e3 thru 0x0154fb (0xb119 bytes) 000511 1773609540.761584 trunc from 0x040000 to 0x01d1d3 (0x22e2c bytes) 000514 1773609540.820660 mapwrite 0x01b285 thru 0x023591 (0x830d bytes) 000515 1773609541.130570 trunc from 0x023592 to 0x006cd2 (0x1c8bf bytes) 000516 1773609541.166373 write 0x032d44 thru 0x035c20 (0x2edd bytes) 000518 1773609541.168216 mapread 0x0143d8 thru 0x01a113 (0x5d3c bytes) 000519 1773609541.184334 write 0x03fcea thru 0x03ffff (0x316 bytes) 000520 1773609541.185973 mapread 0x01a25b thru 0x028a03 (0xe7a9 bytes) 000521 1773609541.210345 mapwrite 0x03367e thru 0x03ffff (0xc982 bytes) 000522 1773609541.318027 mapwrite 0x0256ec thru 0x033ced (0xe602 bytes) 000523 1773609541.413456 write 0x01a4b3 thru 0x023021 (0x8b6f bytes) 000525 1773609541.424714 MIRROR_RESYNC 4 mirrors 000529 1773609543.434695 write 0x0345c3 thru 0x034905 (0x343 bytes) 000530 1773609543.651520 mapwrite 0x025563 thru 0x02c4e9 (0x6f87 bytes) 000531 1773609543.844033 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa2075f26 (stale) Mirror 4: 0xa2075f26 (stale) Mirror 5: 0xa2075f26 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xacf525d3 Mirror 6: 0xacf525d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000532 1773609545.434915 mapwrite 0x02bde8 thru 0x03bdcb (0xffe4 bytes) 000533 1773609545.914982 read 0x0112a7 thru 0x0135a3 (0x22fd bytes) 000534 1773609545.929216 write 0x0258c3 thru 0x027547 (0x1c85 bytes) 000535 1773609545.930882 mapread 0x01539a thru 0x02387b (0xe4e2 bytes) 000536 1773609545.999835 read 0x030063 thru 0x03c11c (0xc0ba bytes) 000539 1773609546.004985 write 0x01d537 thru 0x0229b2 (0x547c bytes) 000541 1773609546.014569 mapread 0x0052ad thru 0x00b6e4 (0x6438 bytes) 000542 1773609546.045113 write 0x0225ed thru 0x029825 (0x7239 bytes) 000544 1773609546.063050 write 0x02ad02 thru 0x03324f (0x854e bytes) 000545 1773609546.084153 write 0x033870 thru 0x03ffff (0xc790 bytes) 000546 1773609546.097624 mapread 0x014031 thru 0x01c81b (0x87eb bytes) 000548 1773609546.101179 read 0x01c834 thru 0x01f6f8 (0x2ec5 bytes) 000552 1773609546.117260 mapwrite 0x025525 thru 0x02b7c8 (0x62a4 bytes) 000554 1773609546.245341 mapread 0x032787 thru 0x033114 (0x98e bytes) 000556 1773609546.252630 trunc from 0x040000 to 0x03d215 (0x2dea bytes) 000560 1773609546.326348 write 0x005574 thru 0x00dc3d (0x86ca bytes) 000561 1773609546.331373 mapwrite 0x022592 thru 0x02ebce (0xc63d bytes) 000564 1773609546.462558 read 0x02e924 thru 0x035e30 (0x750d bytes) 000565 1773609546.481565 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d216): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d216): Mirror 3: 0xa6eb79ae (stale) Mirror 4: 0xa6eb79ae (stale) Mirror 5: 0xa6eb79ae (stale) Mirror 6: 0x6666e5e9 (stale) CRC-32 checksum value for chunk [0, 0x3d216): Mirror 1: 0xeb5330bc Mirror 7: 0xeb5330bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d216: skipped 000567 1773609548.587036 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d216): Mirror 4: 0xa6eb79ae (stale) Mirror 5: 0xa6eb79ae (stale) Mirror 6: 0x6666e5e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d216): Mirror 4: 0xa6eb79ae (stale) Mirror 5: 0xa6eb79ae (stale) Mirror 6: 0x6666e5e9 (stale) CRC-32 checksum value for chunk [0, 0x3d216): Mirror 1: 0xeb5330bc Mirror 7: 0xeb5330bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d216: skipped 000568 1773609549.859244 read 0x02cd80 thru 0x03433a (0x75bb bytes) 000569 1773609549.897383 trunc from 0x03d216 to 0x02ee87 (0xe38e bytes) 000570 1773609550.068084 mapwrite 0x030362 thru 0x03c471 (0xc110 bytes) 000574 1773609550.214215 read 0x015379 thru 0x016e5e (0x1ae6 bytes) 000575 1773609550.272633 mapread 0x01eddc thru 0x0299f8 (0xac1d bytes) 000578 1773609550.280219 mapwrite 0x01a689 thru 0x01fb84 (0x54fc bytes) 000579 1773609550.497597 mapwrite 0x018aa7 thru 0x0252e5 (0xc83f bytes) 000581 1773609550.570355 trunc from 0x03c472 to 0x003a62 (0x38a0f bytes) 000582 1773609550.607208 trunc from 0x003a63 to 0x030e8c (0x2d42a bytes) 000583 1773609550.630055 write 0x01ec91 thru 0x01ffa9 (0x1319 bytes) 000585 1773609550.688286 mapwrite 0x039578 thru 0x03e814 (0x529d bytes) 000586 1773609550.889224 read 0x01f53e thru 0x026536 (0x6ff9 bytes) 000589 1773609550.916728 mapread 0x01fcd3 thru 0x0227b6 (0x2ae4 bytes) 000590 1773609550.921011 trunc from 0x03e815 to 0x031747 (0xd0cd bytes) 000592 1773609550.966345 trunc from 0x031748 to 0x02359a (0xe1ad bytes) 000595 1773609551.009214 write 0x03a4a7 thru 0x03c14d (0x1ca7 bytes) 000596 1773609551.015186 mapwrite 0x03c9e3 thru 0x03f77a (0x2d98 bytes) 000598 1773609551.209709 trunc from 0x03f77b to 0x01dfa3 (0x217d7 bytes) 000601 1773609551.272743 trunc from 0x01dfa4 to 0x03d7e2 (0x1f83f bytes) 000602 1773609551.329321 write 0x031d02 thru 0x03fb94 (0xde93 bytes) 000603 1773609551.347292 mapwrite 0x009dd6 thru 0x0189d5 (0xec00 bytes) 000605 1773609551.472098 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb95): Mirror 4: 0xf9184094 (stale) Mirror 6: 0x1a6d520f (stale) Mirror 7: 0x651f8029 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb95): Mirror 4: 0xf9184094 (stale) Mirror 6: 0x1a6d520f (stale) Mirror 7: 0x651f8029 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb95: skipped 000607 1773609552.507779 write 0x03f6e3 thru 0x03ffff (0x91d bytes) 000608 1773609552.565850 MIRROR_RESYNC 4 mirrors 000609 1773609554.386236 mapwrite 0x00d015 thru 0x0198b2 (0xc89e bytes) 000610 1773609554.861567 trunc from 0x040000 to 0x00d52a (0x32ad5 bytes) 000611 1773609554.953416 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd52b): Mirror 6: 0xfee98fca (stale) Mirror 7: 0xfee98fca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd52b): Mirror 6: 0xfee98fca (stale) Mirror 7: 0xfee98fca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd52b: skipped 000612 1773609555.608930 trunc from 0x00d52b to 0x004ab9 (0x8a71 bytes) 000613 1773609555.667388 trunc from 0x004aba to 0x028151 (0x23698 bytes) 000616 1773609555.758691 read 0x024ee8 thru 0x028151 (0x326a bytes) 000617 1773609555.786317 MIRROR_RESYNC 3 mirrors 000620 1773609557.616356 mapwrite 0x008acd thru 0x0131df (0xa713 bytes) 000626 1773609558.297530 mapwrite 0x02ef38 thru 0x034a06 (0x5acf bytes) 000627 1773609558.590635 MIRROR_RESYNC 3 mirrors 000629 1773609559.731275 mapwrite 0x019d56 thru 0x01f964 (0x5c0f bytes) 000633 1773609560.139093 read 0x00cfe5 thru 0x017925 (0xa941 bytes) 000635 1773609560.176778 mapread 0x033d29 thru 0x034a06 (0xcde bytes) 000637 1773609560.240130 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a07): Mirror 7: 0x7d8a6309 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a07): Mirror 7: 0x7d8a6309 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a07: skipped 000638 1773609560.906698 trunc from 0x034a07 to 0x00fa87 (0x24f7f bytes) 000641 1773609560.954665 mapwrite 0x00bdf6 thru 0x012e44 (0x704f bytes) 000642 1773609561.135848 mapread 0x0098a6 thru 0x012e44 (0x959f bytes) 000643 1773609561.138164 mapread 0x012981 thru 0x012e44 (0x4c4 bytes) 000644 1773609561.153332 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000645 1773609561.426931 write 0x0008a9 thru 0x005cfc (0x5454 bytes) 000646 1773609561.450399 mapwrite 0x00521c thru 0x00db11 (0x88f6 bytes) 000648 1773609561.585728 read 0x004657 thru 0x00b0ec (0x6a96 bytes) 000649 1773609561.588311 mapwrite 0x01cda0 thru 0x021e7c (0x50dd bytes) 000650 1773609561.738262 read 0x003f6a thru 0x00e4f0 (0xa587 bytes) 000652 1773609561.740122 write 0x000c49 thru 0x007675 (0x6a2d bytes) 000655 1773609561.749698 mapwrite 0x016cf5 thru 0x0171f5 (0x501 bytes) 000656 1773609561.868957 read 0x0213c8 thru 0x021e7c (0xab5 bytes) 000658 1773609561.871157 read 0x01498c thru 0x01f1b2 (0xa827 bytes) 000659 1773609561.898687 MIRROR_EXTEND to 2 mirrors 000662 1773609563.302215 write 0x01d78f thru 0x02a949 (0xd1bb bytes) 000663 1773609563.412779 mapwrite 0x031c6d thru 0x03a0c4 (0x8458 bytes) 000664 1773609563.701334 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000667 1773609564.121924 write 0x0007c6 thru 0x00c0fa (0xb935 bytes) 000671 1773609564.148185 trunc from 0x03a0c5 to 0x03b899 (0x17d5 bytes) 000673 1773609564.208343 trunc from 0x03b89a to 0x012ade (0x28dbb bytes) 000679 1773609564.243987 mapwrite 0x004bc8 thru 0x007384 (0x27bd bytes) 000680 1773609564.476453 trunc from 0x012adf to 0x00155c (0x11582 bytes) 000682 1773609564.513333 write 0x01100c thru 0x01bcb9 (0xacae bytes) 000683 1773609564.518123 mapwrite 0x02d94c thru 0x02fe43 (0x24f8 bytes) 000685 1773609564.716438 read 0x0211a2 thru 0x02c498 (0xb2f7 bytes) 000686 1773609564.744162 read 0x020ac8 thru 0x0284ef (0x7a28 bytes) 000687 1773609564.783179 write 0x012117 thru 0x01bb6a (0x9a54 bytes) 000689 1773609564.798396 read 0x015353 thru 0x01acff (0x59ad bytes) 000691 1773609564.815873 mapread 0x017532 thru 0x0243a1 (0xce70 bytes) 000692 1773609564.820902 mapwrite 0x03639c thru 0x03ffff (0x9c64 bytes) 000693 1773609565.082446 read 0x0364ee thru 0x03cc4b (0x675e bytes) 000695 1773609565.083997 read 0x0378bd thru 0x03d787 (0x5ecb bytes) 000696 1773609565.094345 write 0x02380c thru 0x029be2 (0x63d7 bytes) 000698 1773609565.108284 write 0x027931 thru 0x034374 (0xca44 bytes) 000699 1773609565.114188 read 0x003044 thru 0x00bce2 (0x8c9f bytes) 000700 1773609565.126259 write 0x02ed85 thru 0x033503 (0x477f bytes) 000701 1773609565.128532 mapwrite 0x026355 thru 0x02cef9 (0x6ba5 bytes) 000703 1773609565.266951 read 0x013f30 thru 0x0150b0 (0x1181 bytes) 000704 1773609565.270268 write 0x00ff3c thru 0x012052 (0x2117 bytes) 000706 1773609565.272644 read 0x00a486 thru 0x013f2d (0x9aa8 bytes) 000708 1773609565.276453 mapwrite 0x01f2ad thru 0x02be9b (0xcbef bytes) 000715 1773609565.371784 trunc from 0x040000 to 0x028b72 (0x1748d bytes) 000717 1773609565.400053 write 0x006a4f thru 0x012112 (0xb6c4 bytes) 000718 1773609565.403354 read 0x00141d thru 0x006614 (0x51f8 bytes) 000721 1773609565.405286 mapwrite 0x026e58 thru 0x02ff49 (0x90f2 bytes) 000722 1773609565.513308 write 0x01ba64 thru 0x02430f (0x88ac bytes) 000723 1773609565.519168 trunc from 0x02ff4a to 0x0086a6 (0x278a3 bytes) 000724 1773609565.551145 trunc from 0x0086a7 to 0x0107b4 (0x810e bytes) 000725 1773609565.570960 mapread 0x010713 thru 0x0107b4 (0xa2 bytes) 000726 1773609565.578161 read 0x00966b thru 0x00df17 (0x48ad bytes) 000731 1773609565.600063 trunc from 0x0107b5 to 0x002b73 (0xdc41 bytes) 000735 1773609565.627258 mapread 0x0008c5 thru 0x002b73 (0x22af bytes) 000737 1773609565.630325 read 0x0023a9 thru 0x002b73 (0x7cb bytes) 000739 1773609565.631999 mapwrite 0x03cebf thru 0x03ffff (0x3141 bytes) 000740 1773609565.722277 write 0x0177c4 thru 0x023e4c (0xc689 bytes) 000741 1773609565.732124 mapwrite 0x03bcce thru 0x03ffff (0x4332 bytes) 000743 1773609565.821761 mapwrite 0x01af02 thru 0x0222fc (0x73fb bytes) 000744 1773609565.890421 trunc from 0x040000 to 0x0192fd (0x26d02 bytes) 000745 1773609565.921111 mapread 0x010bb4 thru 0x013947 (0x2d94 bytes) 000747 1773609565.924386 write 0x00a3ec thru 0x00fe51 (0x5a66 bytes) 000749 1773609565.927789 mapread 0x00cc5d thru 0x0192fd (0xc6a1 bytes) 000756 1773609565.940783 write 0x0074f9 thru 0x007687 (0x18f bytes) 000757 1773609565.942318 read 0x0095aa thru 0x00a413 (0xe6a bytes) 000759 1773609565.945029 mapread 0x006567 thru 0x00aef7 (0x4991 bytes) 000760 1773609565.947860 mapread 0x01740c thru 0x0192fd (0x1ef2 bytes) 000761 1773609565.949319 trunc from 0x0192fe to 0x03828a (0x1ef8d bytes) 000766 1773609565.976362 mapread 0x008d83 thru 0x00e0f2 (0x5370 bytes) 000767 1773609565.978189 write 0x021b62 thru 0x02b2d9 (0x9778 bytes) 000769 1773609565.997040 read 0x0298bd thru 0x02eb41 (0x5285 bytes) 000771 1773609566.000347 mapwrite 0x0233ad thru 0x024dfc (0x1a50 bytes) 000772 1773609566.076230 mapread 0x00ce72 thru 0x00ecd8 (0x1e67 bytes) 000774 1773609566.077976 write 0x033c4a thru 0x0362a3 (0x265a bytes) 000776 1773609566.082864 trunc from 0x03828b to 0x028204 (0x10086 bytes) 000778 1773609566.113867 mapwrite 0x01d28e thru 0x021928 (0x469b bytes) 000780 1773609566.165802 write 0x004eb9 thru 0x0087ed (0x3935 bytes) 000781 1773609566.168223 mapwrite 0x003bd1 thru 0x013819 (0xfc49 bytes) 000782 1773609566.304693 read 0x004bcc thru 0x0138ed (0xed22 bytes) 000785 1773609566.307739 read 0x007ea9 thru 0x014bfa (0xcd52 bytes) 000792 1773609566.336949 MIRROR_EXTEND to 2 mirrors 000794 1773609567.119700 mapread 0x006b85 thru 0x00713f (0x5bb bytes) 000798 1773609567.181762 write 0x03ccac thru 0x03eebd (0x2212 bytes) 000799 1773609567.296362 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000800 1773609567.510199 MIRROR_EXTEND to 2 mirrors 000801 1773609568.723101 trunc from 0x03eebe to 0x02bf86 (0x12f37 bytes) 000803 1773609568.977835 MIRROR_RESYNC 2 mirrors 000804 1773609570.546254 write 0x017bf9 thru 0x0227d4 (0xabdc bytes) 000805 1773609570.867571 write 0x004ee6 thru 0x012eff (0xe01a bytes) 000807 1773609570.934350 trunc from 0x02bf87 to 0x000886 (0x2b700 bytes) 000808 1773609571.000965 trunc from 0x000887 to 0x039553 (0x38ccd bytes) 000811 1773609571.104211 trunc from 0x039554 to 0x00f594 (0x29fbf bytes) 000816 1773609571.185314 trunc from 0x00f595 to 0x01d415 (0xde81 bytes) 000818 1773609571.235570 mapread 0x014074 thru 0x01d415 (0x93a2 bytes) 000819 1773609571.299270 mapwrite 0x02a3f1 thru 0x02e8d7 (0x44e7 bytes) 000820 1773609571.488185 MIRROR_RESYNC 2 mirrors 000821 1773609572.413645 read 0x00b188 thru 0x016852 (0xb6cb bytes) 000824 1773609572.429834 read 0x014f2c thru 0x02446d (0xf542 bytes) 000825 1773609572.441889 read 0x02bde7 thru 0x02e8d7 (0x2af1 bytes) 000826 1773609572.442431 trunc from 0x02e8d8 to 0x02aa6c (0x3e6b bytes) 000827 1773609572.537349 trunc from 0x02aa6d to 0x027b11 (0x2f5b bytes) 000828 1773609572.567938 read 0x01016f thru 0x010e66 (0xcf8 bytes) 000829 1773609572.586795 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000830 1773609572.771079 mapwrite 0x01c7c8 thru 0x021a17 (0x5250 bytes) 000831 1773609572.871969 trunc from 0x027b12 to 0x000f63 (0x26bae bytes) 000832 1773609572.893382 read 0x0005b6 thru 0x000f63 (0x9ae bytes) 000833 1773609572.906155 write 0x03a015 thru 0x03dc21 (0x3c0d bytes) 000835 1773609572.909290 mapwrite 0x007a3b thru 0x0148ea (0xceb0 bytes) 000836 1773609572.988026 write 0x0200d3 thru 0x0232b1 (0x31df bytes) 000837 1773609573.007256 MIRROR_EXTEND to 2 mirrors 000838 1773609573.866286 mapwrite 0x01650a thru 0x01e30d (0x7e04 bytes) 000840 1773609574.252895 mapwrite 0x029cbe thru 0x030aff (0x6e42 bytes) 000841 1773609574.364277 mapwrite 0x032e51 thru 0x03d7a5 (0xa955 bytes) 000842 1773609574.575160 write 0x03ac8b thru 0x03da4a (0x2dc0 bytes) 000845 1773609574.605348 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000846 1773609574.903796 trunc from 0x03dc22 to 0x004ea3 (0x38d7e bytes) 000847 1773609574.928545 trunc from 0x004ea4 to 0x014048 (0xf1a5 bytes) 000849 1773609574.957410 write 0x0387ec thru 0x038d83 (0x598 bytes) 000852 1773609574.964083 write 0x032723 thru 0x039000 (0x68de bytes) 000854 1773609574.977430 trunc from 0x039001 to 0x01823d (0x20dc3 bytes) 000863 1773609575.016347 trunc from 0x01823e to 0x0395bb (0x2137e bytes) 000864 1773609575.041740 mapwrite 0x02bde4 thru 0x030fd5 (0x51f2 bytes) 000865 1773609575.191447 trunc from 0x0395bc to 0x031174 (0x8447 bytes) 000866 1773609575.242781 trunc from 0x031175 to 0x016fc0 (0x1a1b4 bytes) 000867 1773609575.286807 read 0x01045d thru 0x016fc0 (0x6b64 bytes) 000868 1773609575.310925 mapwrite 0x00b134 thru 0x00f3c8 (0x4295 bytes) 000872 1773609575.580822 trunc from 0x016fc1 to 0x0335a8 (0x1c5e8 bytes) 000875 1773609575.648795 mapwrite 0x02dddf thru 0x032d74 (0x4f96 bytes) 000876 1773609575.796783 read 0x0118aa thru 0x01d713 (0xbe6a bytes) 000877 1773609575.833448 write 0x03ac2f thru 0x03d139 (0x250b bytes) 000880 1773609575.869321 mapwrite 0x03024d thru 0x033345 (0x30f9 bytes) 000881 1773609576.016856 mapwrite 0x03717e thru 0x03b186 (0x4009 bytes) 000882 1773609576.113848 mapread 0x025a90 thru 0x03412b (0xe69c bytes) 000883 1773609576.119695 write 0x004248 thru 0x00ee99 (0xac52 bytes) 000886 1773609576.126757 mapwrite 0x023283 thru 0x02de56 (0xabd4 bytes) 000887 1773609576.274576 mapread 0x0174d4 thru 0x01c5ed (0x511a bytes) 000888 1773609576.285869 mapread 0x01d113 thru 0x02101e (0x3f0c bytes) 000889 1773609576.287367 trunc from 0x03d13a to 0x029295 (0x13ea4 bytes) 000892 1773609576.353877 write 0x02811b thru 0x0354c9 (0xd3af bytes) 000893 1773609576.358617 mapwrite 0x0389d1 thru 0x03ffff (0x762f bytes) 000895 1773609576.513784 trunc from 0x040000 to 0x010a81 (0x2f57e bytes) 000896 1773609576.549403 trunc from 0x010a82 to 0x034448 (0x239c7 bytes) 000898 1773609576.586798 MIRROR_EXTEND to 2 mirrors 000899 1773609577.861680 mapwrite 0x002b6d thru 0x00d186 (0xa61a bytes) 000900 1773609578.414164 write 0x02337e thru 0x02ef69 (0xbbec bytes) 000902 1773609578.460810 mapread 0x00ffcc thru 0x019419 (0x944e bytes) 000903 1773609578.485289 mapwrite 0x008537 thru 0x00a984 (0x244e bytes) 000904 1773609578.621714 write 0x014955 thru 0x01fa40 (0xb0ec bytes) 000906 1773609578.635983 trunc from 0x034449 to 0x022c55 (0x117f3 bytes) 000907 1773609578.711655 mapread 0x0003c7 thru 0x00a0b9 (0x9cf3 bytes) 000908 1773609578.726135 mapread 0x00bad4 thru 0x013978 (0x7ea5 bytes) 000909 1773609578.727411 mapwrite 0x023207 thru 0x023e3e (0xc38 bytes) 000910 1773609578.944424 write 0x035b15 thru 0x03ffff (0xa4eb bytes) 000911 1773609578.948157 mapread 0x0082a1 thru 0x0112b7 (0x9017 bytes) 000912 1773609578.959396 mapread 0x00e95b thru 0x013f08 (0x55ae bytes) 000913 1773609578.964378 write 0x002c13 thru 0x012095 (0xf483 bytes) 000914 1773609578.971161 mapwrite 0x01750b thru 0x022afc (0xb5f2 bytes) 000915 1773609579.046741 read 0x036808 thru 0x03ffff (0x97f8 bytes) 000916 1773609579.048060 read 0x03d375 thru 0x03ffff (0x2c8b bytes) 000917 1773609579.049881 mapwrite 0x007290 thru 0x00cbf4 (0x5965 bytes) 000919 1773609579.115112 write 0x009363 thru 0x00daa2 (0x4740 bytes) 000920 1773609579.117220 trunc from 0x040000 to 0x0376f8 (0x8907 bytes) 000922 1773609579.152556 mapread 0x00ddfa thru 0x01046e (0x2675 bytes) 000924 1773609579.156056 mapread 0x017e01 thru 0x019555 (0x1755 bytes) 000925 1773609579.158453 mapwrite 0x02ff1b thru 0x039930 (0x9a16 bytes) 000926 1773609579.265453 read 0x01b11f thru 0x02b024 (0xff06 bytes) 000927 1773609579.266843 mapwrite 0x015d92 thru 0x022dd5 (0xd044 bytes) 000928 1773609579.353951 write 0x02c3d6 thru 0x035514 (0x913f bytes) 000932 1773609579.360824 write 0x023608 thru 0x02a796 (0x718f bytes) 000933 1773609579.362629 mapread 0x029825 thru 0x02aba0 (0x137c bytes) 000934 1773609579.367611 trunc from 0x039931 to 0x002932 (0x36ffe bytes) 000936 1773609579.411838 read 0x0001fe thru 0x002014 (0x1e17 bytes) 000937 1773609579.415807 mapwrite 0x018555 thru 0x02084c (0x82f8 bytes) 000938 1773609579.529160 mapread 0x008530 thru 0x00e77f (0x6250 bytes) 000939 1773609579.530671 trunc from 0x02084d to 0x00b5eb (0x15261 bytes) 000940 1773609579.560209 trunc from 0x00b5ec to 0x00cbdc (0x15f1 bytes) 000942 1773609579.585949 mapread 0x00aaec thru 0x00cbdc (0x20f1 bytes) 000943 1773609579.595912 mapwrite 0x014331 thru 0x01b8cb (0x759b bytes) 000944 1773609579.779210 read 0x013629 thru 0x01893d (0x5315 bytes) 000945 1773609579.802287 read 0x015c0b thru 0x01b042 (0x5438 bytes) 000946 1773609579.806425 mapread 0x01311b thru 0x014208 (0x10ee bytes) 000948 1773609579.808406 mapread 0x0183f7 thru 0x01b8cb (0x34d5 bytes) 000949 1773609579.814726 read 0x00fd41 thru 0x01b8cb (0xbb8b bytes) 000950 1773609579.816125 read 0x010511 thru 0x019080 (0x8b70 bytes) 000951 1773609579.817755 trunc from 0x01b8cc to 0x005087 (0x16844 bytes) 000952 1773609579.875953 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000954 1773609580.246848 MIRROR_EXTEND to 2 mirrors 000956 1773609581.682717 MIRROR_RESYNC 2 mirrors 000957 1773609582.403838 mapwrite 0x01d2a8 thru 0x02393e (0x6697 bytes) 000958 1773609582.922785 read 0x0049cf thru 0x00cd81 (0x83b3 bytes) 000965 1773609582.987716 MIRROR_RESYNC 2 mirrors 000967 1773609584.460227 mapread 0x0174c6 thru 0x02393e (0xc479 bytes) 000968 1773609584.533717 read 0x01473f thru 0x02393e (0xf200 bytes) 000969 1773609584.557685 MIRROR_RESYNC 2 mirrors 000970 1773609585.175287 read 0x022425 thru 0x02393e (0x151a bytes) 000973 1773609585.180390 mapread 0x007b54 thru 0x008dd8 (0x1285 bytes) 000974 1773609585.205367 write 0x018a5d thru 0x0226df (0x9c83 bytes) 000975 1773609585.426619 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x112:0x0]: Invalid argument (22) 000978 1773609585.732563 trunc from 0x02393f to 0x00818d (0x1b7b1 bytes) 000979 1773609585.772329 write 0x03ac2f thru 0x03b79a (0xb6c bytes) 000980 1773609585.782922 mapwrite 0x02e37f thru 0x03a537 (0xc1b9 bytes) 000981 1773609586.081388 write 0x007225 thru 0x00f235 (0x8011 bytes) 000982 1773609586.126658 read 0x001a97 thru 0x0082cb (0x6835 bytes) 000983 1773609586.127667 mapread 0x013f3c thru 0x0185ea (0x46af bytes) 000987 1773609586.186417 mapwrite 0x00eec8 thru 0x00ff74 (0x10ad bytes) 000995 1773609586.332901 write 0x02b071 thru 0x02cdab (0x1d3b bytes) 000997 1773609586.335943 mapread 0x008c1b thru 0x0113e6 (0x87cc bytes) 000999 1773609586.340910 trunc from 0x03b79b to 0x03d1a5 (0x1a0b bytes) 001000 1773609586.383974 mapwrite 0x0191db thru 0x01f168 (0x5f8e bytes) All operations completed A-OK! PASS 70a (122s) == sanity-flr test 71: check mirror extend parent fid ==== 17:19:54 (1773609594) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 17:19:58 (1773609598) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x192:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x196:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.487336 s, 4.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x192:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x193:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x196:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42157 s, 2.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x192:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x193:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x197:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x196:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x192:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x193:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x197:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x196:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x194:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (15s) == sanity-flr test 200a: stress test ===================== 17:20:13 (1773609613) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1110240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6976672 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5375200 .. failed Extending file size to 7936736 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9087648 .. Extending file size to 8366752 .. Extending file size to 8849568 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 55131 55132 55134 55137 55138 done 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (97s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:21:51 (1773609711) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7085280 .. Extending file size to 5836128 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7390816 .. Extending file size to 7312352 .. Extending file size to 7846944 .. Extending file size to 8871008 .. Extending file size to 7754912 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8240640 .. Extending file size to 9003648 .. Extending file size to 8666080 .. Extending file size to 8879712 .. Waiting 56578 56580 56581 56583 56584 done 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (90s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:23:20 (1773609800) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3840 2204672 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175795 s, 11.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.74346 s, 2.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcffffaed (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 200c (28s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:23:51 (1773609831) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16634 s, 12.6 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 17:24:01 (1773609841) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x151:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1ad:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1ad:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1ae:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x1a9:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:24:11 (1773609851) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308543 s, 6.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1ab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:24:22 (1773609862) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216185 s, 9.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1ad:0x0] } PASS 204b (13s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:24:35 (1773609875) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.757258 s, 2.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1af:0x0] } PASS 204c (43s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:25:19 (1773609919) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342568 s, 3.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b4:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:26:25 (1773609985) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.370883 s, 5.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:27:31 (1773610051) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. Creating new pool test_204f oleg129-server: Pool lustre.test_204f created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.360438 s, 5.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x1b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg129-server: Pool lustre.test_204f destroyed PASS 204f (92s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:29:03 (1773610143) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:29:15 (1773610155) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:29:27 (1773610167) oleg129-server: Pool lustre.test_206 created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg129-server: Pool lustre.test_206 destroyed PASS 206 (33s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:29:59 (1773610199) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138467 s, 7.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536083 s, 19.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe9ca04ad Mirror 3: 0xe9ca04ad PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:30:15 (1773610215) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:30:20 (1773610220) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:30:25 (1773610225) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.184761 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x15e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2390000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2390000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2390000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xfc0000 Read this (4 bytes): BBBB 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 209a (48s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:31:13 (1773610273) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0507895 s, 161 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:31:23 (1773610283) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105617 s, 9.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x176 lmm_fid: [0x200000404:0x176:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x176 lmm_fid: [0x200000404:0x176:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1c0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834294 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-15 17:31:25.000000000 -0400 Modify: 2026-03-15 17:31:25.000000000 -0400 Change: 2026-03-15 17:31:27.000000000 -0400 Birth: 2026-03-15 17:31:25.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:31:35 (1773610295) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:31:39 (1773610299) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.825495 s, 49.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0779526 s, 52.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 15 17:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x178:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 15 17:31 /mnt/lustre/f211.sanity-flr PASS 211 (14s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:31:53 (1773610313) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:32:05 (1773610325) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2264 sec =========== 17:32:17 (1773610337) === sanity-flr: start cleanup 17:32:20 (1773610340) === === sanity-flr: finish cleanup 17:32:29 (1773610349) ===