-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 11:21:44 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg457-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:21:55 (1780327315) === oleg457-client.virtnet: executing check_config_client /mnt/lustre oleg457-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg457-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c2591124000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c2591124000.idle_timeout=debug disable quota as required oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:22:12 (1780327332) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7560] == sanity-flr test 0a: lfs mirror create with -N option == 11:22:13 (1780327333) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:22:21 (1780327341) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:22:23 (1780327343) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:22:24 (1780327344) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:22:36 (1780327356) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:22:38 (1780327358) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 11:22:39 (1780327359) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c2591124000.stats=clear osc.lustre-OST0001-osc-ffff9c2591124000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:22:43 (1780327363) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:22:49 (1780327369) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:22:53 (1780327373) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:22:55 (1780327375) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:22:59 (1780327379) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 11:23:03 (1780327383) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:23:07 (1780327387) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:23:09 (1780327389) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:23:12 (1780327392) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:23:17 (1780327397) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:23:21 (1780327401) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.26003 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679139 s, 15.4 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:23:27 (1780327407) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.264343 s, 79.3 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:23:31 (1780327411) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:23:36 (1780327416) Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (36s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:24:13 (1780327453) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.195185 s, 37.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:24:31 (1780327471) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (40s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:25:11 (1780327511) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.81701 s, 18.6 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 11:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.606s user 0m0.002s sys 0m0.161s Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2591124000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.612s user 0m0.000s sys 0m0.172s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.568s user 0m0.003s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2591124000.ost_server_uuid in FULL state after 0 sec PASS 33b (47s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:25:58 (1780327558) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:26:00 (1780327560) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:26:02 (1780327562) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 11:26:04 (1780327564) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 11:26:08 (1780327568) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.372106 s, 33.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:26:15 (1780327575) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164948 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175957 s, 59.6 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:26:18 (1780327578) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:26:23 (1780327583) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000909173 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000750074 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00081888 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000874018 s, 1.2 MB/s flv=10 olv=10 PASS 36d (6s) == sanity-flr test 37: mirror I/O API verification ======= 11:26:29 (1780327589) create test files with size 15737006 .. 04aa660627bf97e9512e189361352d48 - 984b334bf9fd7f15047a796af485250e - bd549bb68c09a67a68f92aba8554962a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre PASS 37 (12s) == sanity-flr test 38: resync ============================ 11:26:41 (1780327601) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:26:50 (1780327610) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:26:53 (1780327613) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578485 s, 18.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176455 s, 59.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 11:26:58 (1780327618) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0628166 s, 33.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.168525 s, 24.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126458 s, 16.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 11:27:06 (1780327626) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 11:27:08 (1780327628) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:27:09 (1780327629) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174202 s, 60.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 11:27:13 (1780327633) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:27:15 (1780327635) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:27:20 (1780327640) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.119491 s, 87.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:27:34 (1780327654) llite.lustre-ffff9c25908ea800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.126079 s, 83.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c25852db800.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:27:41 (1780327661) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (19s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:28:01 (1780327681) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (18s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:28:20 (1780327700) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 11:28:21 (1780327701) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.12281 s, 25.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0733877 s, 42.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x71367d63 (nosync,stale) Mirror 2: 0x71367d63 (nosync,stale) ** verify mirror 2 doesn't change original checksum: a0aa9902369e644099c390e7c150ad8d - mirror 1 checksum: cdb157eaad05396a1b2d21acacd8e0bf - mirror 2 checksum: a0aa9902369e644099c390e7c150ad8d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 11:28:22 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 11:28:22 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a0aa9902369e644099c390e7c150ad8d - mirror 1 checksum: cdb157eaad05396a1b2d21acacd8e0bf - mirror 2 checksum: cdb157eaad05396a1b2d21acacd8e0bf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 11:28:22 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 11:28:22 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (5s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:28:27 (1780327707) Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 11:28:30 (1780327710) shut down facet: mds1 facet_host: oleg457-server facet_failover_host: oleg457-server Failover mds1 to oleg457-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 11:28:46 (1780327726) targets are mounted 11:28:46 (1780327726) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177836 s, 59.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:28:54 (1780327734) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:28:55 (1780327735) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:28:57 (1780327737) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0026533 s, 6.2 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:29:01 (1780327741) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:29:02 (1780327742) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:29:04 (1780327744) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:29:15 (1780327755) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:29:36 (1780327776) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:29:56 (1780327796) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:30:15 (1780327815) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 11:30:19 (1780327819) mirror create pid 53442 mirror split pid 53443 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53442 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53443 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 11:31:26 (1780327886) Seed set to 7888 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000006 1780327889.434832 MIRROR_EXTEND to 2 mirrors 000008 1780327890.213516 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000010 1780327890.495682 mapwrite 0x02c628 thru 0x032a88 (0x6461 bytes) 000011 1780327890.711058 read 0x03251f thru 0x032a88 (0x56a bytes) truncating to largest ever: 0x2b1e8 000012 1780327890.711851 trunc from 0x032a89 to 0x02b1e7 (0x78a1 bytes) 000013 1780327890.739688 mapwrite 0x025d2a thru 0x02d2cb (0x75a2 bytes) 000014 1780327890.852721 trunc from 0x02d2cc to 0x009db0 (0x2351b bytes) 000015 1780327890.890018 mapwrite 0x008aaf thru 0x00e860 (0x5db2 bytes) 000016 1780327891.061723 trunc from 0x00e861 to 0x002b13 (0xbd4d bytes) 000017 1780327891.101298 mapwrite 0x02244d thru 0x032108 (0xfcbc bytes) truncating to largest ever: 0x3fd20 000018 1780327891.399749 trunc from 0x032109 to 0x03fd1f (0xdc17 bytes) 000021 1780327891.446778 mapwrite 0x036e4c thru 0x03b7b9 (0x496e bytes) 000024 1780327891.583979 read 0x00a131 thru 0x0132de (0x91ae bytes) 000025 1780327891.610781 mapwrite 0x016f82 thru 0x01907b (0x20fa bytes) 000026 1780327891.721917 read 0x00f12c thru 0x01c4c3 (0xd398 bytes) 000028 1780327891.730256 mapread 0x00de4a thru 0x01714a (0x9301 bytes) 000030 1780327891.735894 mapwrite 0x03c7d0 thru 0x03f3b6 (0x2be7 bytes) 000032 1780327891.862356 MIRROR_EXTEND to 2 mirrors 000033 1780327893.392974 write 0x00d926 thru 0x016928 (0x9003 bytes) 000034 1780327893.552257 MIRROR_RESYNC 2 mirrors 000036 1780327894.649006 mapread 0x03783f thru 0x03f6b5 (0x7e77 bytes) 000038 1780327894.673493 write 0x0393c9 thru 0x03ffff (0x6c37 bytes) 000040 1780327894.783058 mapread 0x026b5a thru 0x032aac (0xbf53 bytes) 000043 1780327894.818021 trunc from 0x040000 to 0x01aa54 (0x255ab bytes) 000044 1780327894.848909 read 0x018e3e thru 0x01aa2d (0x1bf0 bytes) 000048 1780327894.859602 read 0x001507 thru 0x002417 (0xf11 bytes) 000049 1780327894.885328 mapwrite 0x009ea0 thru 0x0107b7 (0x6918 bytes) 000050 1780327894.975945 trunc from 0x01aa55 to 0x0342bc (0x19868 bytes) 000051 1780327895.018520 trunc from 0x0342bd to 0x00a43d (0x29e7f bytes) 000052 1780327895.056461 write 0x0388d4 thru 0x03d459 (0x4b86 bytes) 000053 1780327895.057912 mapread 0x003cc8 thru 0x00d658 (0x9991 bytes) 000054 1780327895.104727 mapwrite 0x011f0e thru 0x014d53 (0x2e46 bytes) 000055 1780327895.251839 write 0x02d099 thru 0x03754e (0xa4b6 bytes) 000059 1780327895.278916 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d45a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d45a): Mirror 2: 0x75cc427a (stale) CRC-32 checksum value for chunk [0, 0x3d45a): Mirror 1: 0x8538addc Mirror 3: 0x8538addc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d45a: skipped 000060 1780327896.513362 trunc from 0x03d45a to 0x0130a3 (0x2a3b6 bytes) 000061 1780327896.734138 mapwrite 0x0312f6 thru 0x033c86 (0x2991 bytes) 000062 1780327896.911243 read 0x011141 thru 0x012de8 (0x1ca8 bytes) 000063 1780327896.932565 read 0x0056a5 thru 0x0077b0 (0x210c bytes) 000064 1780327896.983983 write 0x00ae64 thru 0x00b97b (0xb18 bytes) 000066 1780327896.994979 mapwrite 0x00aef3 thru 0x01acd7 (0xfde5 bytes) 000069 1780327897.223513 mapwrite 0x020858 thru 0x022be1 (0x238a bytes) 000071 1780327897.401899 write 0x039e09 thru 0x03ffff (0x61f7 bytes) 000073 1780327897.412646 read 0x03db26 thru 0x03ffff (0x24da bytes) 000076 1780327897.430784 mapwrite 0x02d5b0 thru 0x039710 (0xc161 bytes) 000081 1780327897.547043 read 0x00c5e7 thru 0x01787c (0xb296 bytes) 000082 1780327897.548036 mapwrite 0x033135 thru 0x03b85f (0x872b bytes) 000083 1780327897.598101 mapwrite 0x032117 thru 0x035eb2 (0x3d9c bytes) 000084 1780327897.655429 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc973dada (stale) Mirror 3: 0x48495878 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe6a1be28 Mirror 4: 0xe6a1be28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000087 1780327899.376753 trunc from 0x040000 to 0x01c473 (0x23b8c bytes) 000088 1780327899.561268 mapread 0x0056a2 thru 0x0106ee (0xb04d bytes) 000090 1780327899.637966 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1c474): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c474): Mirror 2: 0xf8bd867a (stale) Mirror 3: 0x3ad90d3b (stale) Mirror 4: 0x4bad97d7 (stale) CRC-32 checksum value for chunk [0, 0x1c474): Mirror 1: 0x4bad97d7 Mirror 5: 0x4bad97d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c474: skipped 000093 1780327901.420525 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c474): Mirror 2: 0xf8bd867a (stale) Mirror 3: 0x3ad90d3b (stale) Mirror 4: 0x4bad97d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c474): Mirror 2: 0xf8bd867a (stale) Mirror 3: 0x3ad90d3b (stale) Mirror 4: 0x4bad97d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c474: skipped 000094 1780327902.133638 read 0x01475a thru 0x01c473 (0x7d1a bytes) 000095 1780327902.156501 read 0x01047a thru 0x011faa (0x1b31 bytes) 000096 1780327902.175981 write 0x006e40 thru 0x0154d8 (0xe699 bytes) truncating to largest ever: 0x3ffbf 000098 1780327902.230660 trunc from 0x01c474 to 0x03ffbe (0x23b4b bytes) 000099 1780327902.257093 trunc from 0x03ffbf to 0x030b56 (0xf468 bytes) 000101 1780327902.298775 mapread 0x020ef7 thru 0x021a17 (0xb21 bytes) 000102 1780327902.319837 write 0x019697 thru 0x0246c0 (0xb02a bytes) 000103 1780327902.338207 mapwrite 0x03e017 thru 0x03ffff (0x1fe9 bytes) 000104 1780327902.563689 mapwrite 0x001c38 thru 0x00944c (0x7815 bytes) 000107 1780327902.907658 mapread 0x00aadc thru 0x019867 (0xed8c bytes) 000108 1780327902.943698 read 0x01f3d8 thru 0x029038 (0x9c61 bytes) 000109 1780327902.944452 mapread 0x029a0d thru 0x037728 (0xdd1c bytes) 000112 1780327902.966557 mapread 0x024d73 thru 0x025582 (0x810 bytes) 000115 1780327902.973087 mapread 0x019010 thru 0x028b3e (0xfb2f bytes) 000119 1780327902.983934 mapread 0x01a87f thru 0x022cc6 (0x8448 bytes) 000121 1780327902.991012 mapwrite 0x031c48 thru 0x03ffff (0xe3b8 bytes) 000122 1780327903.190790 read 0x03d496 thru 0x03ffff (0x2b6a bytes) 000124 1780327903.202602 read 0x031433 thru 0x03b134 (0x9d02 bytes) 000127 1780327903.220313 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc973dada (stale) Mirror 4: 0xe6a1be28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc973dada (stale) Mirror 4: 0xe6a1be28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000128 1780327903.763261 mapwrite 0x01cacd thru 0x027513 (0xaa47 bytes) 000129 1780327903.938583 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe6a1be28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe6a1be28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000132 1780327904.344850 write 0x00c3e7 thru 0x00ff29 (0x3b43 bytes) 000133 1780327904.355072 mapread 0x01711f thru 0x01c4e3 (0x53c5 bytes) 000135 1780327904.366822 mapread 0x021c02 thru 0x024201 (0x2600 bytes) 000136 1780327904.375945 mapwrite 0x010a87 thru 0x02093a (0xfeb4 bytes) 000139 1780327904.460233 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000141 1780327904.579109 MIRROR_EXTEND to 2 mirrors 000143 1780327905.568776 read 0x01bf55 thru 0x023a10 (0x7abc bytes) 000144 1780327905.582309 trunc from 0x040000 to 0x021618 (0x1e9e7 bytes) 000148 1780327905.658777 write 0x00d2a1 thru 0x012054 (0x4db4 bytes) 000153 1780327905.677133 mapread 0x001d31 thru 0x00cfc4 (0xb294 bytes) 000154 1780327905.692585 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000155 1780327905.857904 mapread 0x0136f8 thru 0x016bfd (0x3506 bytes) 000156 1780327905.902926 trunc from 0x021619 to 0x001aa9 (0x1fb6f bytes) 000158 1780327905.920772 write 0x008c6b thru 0x011138 (0x84ce bytes) 000159 1780327905.923270 mapwrite 0x024812 thru 0x028026 (0x3815 bytes) 000160 1780327905.992611 write 0x01a2fe thru 0x021f3f (0x7c42 bytes) 000164 1780327906.002097 mapread 0x026046 thru 0x028026 (0x1fe1 bytes) 000166 1780327906.004239 trunc from 0x028027 to 0x0213ec (0x6c3a bytes) 000167 1780327906.022374 mapwrite 0x02f677 thru 0x035344 (0x5cce bytes) 000168 1780327906.099413 mapread 0x013925 thru 0x01dc28 (0xa304 bytes) 000170 1780327906.101089 mapread 0x017865 thru 0x018fb8 (0x1754 bytes) 000171 1780327906.103095 trunc from 0x035345 to 0x01a95a (0x1a9ea bytes) 000172 1780327906.119296 mapread 0x0085ec thru 0x00b5dc (0x2ff1 bytes) 000173 1780327906.120615 trunc from 0x01a95b to 0x02d899 (0x12f3f bytes) 000175 1780327906.135843 mapwrite 0x008b28 thru 0x00d828 (0x4d01 bytes) 000176 1780327906.209345 read 0x0199e1 thru 0x0209d7 (0x6ff7 bytes) 000177 1780327906.223012 mapwrite 0x03e451 thru 0x03ffff (0x1baf bytes) 000179 1780327906.278762 read 0x031166 thru 0x034c39 (0x3ad4 bytes) 000181 1780327906.296760 trunc from 0x040000 to 0x0081f0 (0x37e0f bytes) 000182 1780327906.331032 write 0x01193e thru 0x020d37 (0xf3fa bytes) 000184 1780327906.340674 trunc from 0x020d38 to 0x02fc26 (0xeeef bytes) 000185 1780327906.367965 MIRROR_EXTEND to 2 mirrors 000186 1780327907.156785 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000187 1780327907.341420 MIRROR_EXTEND to 2 mirrors 000188 1780327908.720808 trunc from 0x02fc27 to 0x01d52b (0x126fb bytes) 000190 1780327908.944088 mapwrite 0x00d2af thru 0x00d67b (0x3cd bytes) 000191 1780327909.099080 mapread 0x01b102 thru 0x01d52b (0x242a bytes) 000193 1780327909.139681 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000195 1780327909.385290 write 0x021862 thru 0x0245c2 (0x2d61 bytes) 000198 1780327909.388437 mapwrite 0x03a6f9 thru 0x03ffa5 (0x58ad bytes) 000199 1780327909.465112 write 0x011ecc thru 0x01211c (0x251 bytes) 000202 1780327909.482795 read 0x012114 thru 0x01c719 (0xa606 bytes) 000203 1780327909.494574 read 0x002eb5 thru 0x00a28c (0x73d8 bytes) 000205 1780327909.512000 mapwrite 0x01b8a3 thru 0x026c47 (0xb3a5 bytes) 000206 1780327909.750530 read 0x0106d1 thru 0x014c5f (0x458f bytes) 000207 1780327909.752231 mapread 0x02ed4b thru 0x038994 (0x9c4a bytes) 000208 1780327909.753579 mapread 0x0093a7 thru 0x0098c4 (0x51e bytes) 000209 1780327909.757077 write 0x034d72 thru 0x03ffff (0xb28e bytes) 000211 1780327909.761323 read 0x018a9f thru 0x01ec24 (0x6186 bytes) 000212 1780327909.762032 mapread 0x03fd88 thru 0x03ffff (0x278 bytes) 000213 1780327909.763911 mapwrite 0x0303eb thru 0x03ab45 (0xa75b bytes) 000214 1780327909.917631 read 0x032e04 thru 0x032e85 (0x82 bytes) 000215 1780327909.918787 write 0x007a59 thru 0x0130e6 (0xb68e bytes) 000216 1780327909.921082 read 0x012486 thru 0x01fc0a (0xd785 bytes) 000217 1780327909.922954 mapread 0x01b7b2 thru 0x028be8 (0xd437 bytes) 000218 1780327909.925660 write 0x01c4e0 thru 0x02259b (0x60bc bytes) 000219 1780327909.929409 mapwrite 0x034fc7 thru 0x036c0b (0x1c45 bytes) 000221 1780327910.055891 read 0x030c57 thru 0x03bc74 (0xb01e bytes) 000223 1780327910.057297 mapread 0x0219ff thru 0x02f689 (0xdc8b bytes) 000224 1780327910.070105 mapread 0x01d4bb thru 0x02b42b (0xdf71 bytes) 000225 1780327910.073404 mapread 0x005171 thru 0x0100d2 (0xaf62 bytes) 000227 1780327910.095982 trunc from 0x040000 to 0x000c43 (0x3f3bc bytes) 000230 1780327910.159513 mapread 0x000972 thru 0x000c43 (0x2d2 bytes) 000232 1780327910.171569 trunc from 0x000c44 to 0x03a9ec (0x39da9 bytes) 000236 1780327910.249036 mapwrite 0x02f771 thru 0x03b334 (0xbbc4 bytes) 000237 1780327910.508551 write 0x03b0a1 thru 0x03ffff (0x4f5f bytes) 000238 1780327910.514105 mapwrite 0x038deb thru 0x03a221 (0x1437 bytes) 000242 1780327910.707758 write 0x02261b thru 0x02519b (0x2b81 bytes) 000243 1780327910.718571 mapread 0x03a5df thru 0x03ffff (0x5a21 bytes) 000244 1780327910.721684 trunc from 0x040000 to 0x027033 (0x18fcc bytes) 000245 1780327910.790383 read 0x021d4d thru 0x027033 (0x52e7 bytes) 000247 1780327910.791684 trunc from 0x027034 to 0x024082 (0x2fb1 bytes) 000249 1780327910.842330 read 0x01aac1 thru 0x02240f (0x794f bytes) 000250 1780327910.844416 mapread 0x0219a0 thru 0x024082 (0x26e3 bytes) 000251 1780327910.845959 read 0x021d88 thru 0x024082 (0x22fb bytes) 000252 1780327910.854143 mapread 0x011072 thru 0x01b87f (0xa80e bytes) 000253 1780327910.856249 read 0x013ca4 thru 0x01dbd0 (0x9f2d bytes) 000254 1780327910.863236 mapread 0x017ad9 thru 0x01a7a4 (0x2ccc bytes) 000255 1780327910.867939 mapwrite 0x025453 thru 0x02648c (0x103a bytes) 000256 1780327911.078127 write 0x034e3c thru 0x037ad7 (0x2c9c bytes) 000257 1780327911.081794 write 0x010e24 thru 0x01b4db (0xa6b8 bytes) 000259 1780327911.092533 mapwrite 0x02311f thru 0x0296b1 (0x6593 bytes) 000260 1780327911.291467 read 0x02ad3b thru 0x02e791 (0x3a57 bytes) 000261 1780327911.293221 mapread 0x03766d thru 0x037ad7 (0x46b bytes) 000262 1780327911.308693 read 0x002e81 thru 0x003cac (0xe2c bytes) 000265 1780327911.310540 mapread 0x025870 thru 0x035228 (0xf9b9 bytes) 000267 1780327911.316694 read 0x00e493 thru 0x01b469 (0xcfd7 bytes) 000268 1780327911.317633 trunc from 0x037ad8 to 0x02900c (0xeacb bytes) 000271 1780327911.387948 read 0x01c963 thru 0x02900c (0xc6aa bytes) 000273 1780327911.392798 mapread 0x011ab6 thru 0x01e1d2 (0xc71d bytes) 000275 1780327911.413052 mapwrite 0x02c7f1 thru 0x037ce6 (0xb4f6 bytes) 000277 1780327911.651285 mapread 0x00697a thru 0x00dc05 (0x728c bytes) 000278 1780327911.662596 write 0x03be57 thru 0x03e8b6 (0x2a60 bytes) 000279 1780327911.671959 trunc from 0x03e8b7 to 0x016ff0 (0x278c6 bytes) 000280 1780327911.715932 mapread 0x00cc95 thru 0x016ff0 (0xa35c bytes) 000281 1780327911.725261 read 0x00ed39 thru 0x012c98 (0x3f60 bytes) 000282 1780327911.730958 write 0x02612e thru 0x02bacf (0x59a2 bytes) 000283 1780327911.736241 mapwrite 0x032140 thru 0x03f90a (0xd7cb bytes) 000285 1780327911.928857 write 0x000738 thru 0x00bd2e (0xb5f7 bytes) 000286 1780327911.939273 trunc from 0x03f90b to 0x009e1d (0x35aed bytes) 000287 1780327911.965925 mapread 0x006c35 thru 0x00756e (0x93a bytes) 000288 1780327911.975029 MIRROR_EXTEND to 2 mirrors 000290 1780327913.060366 trunc from 0x009e1e to 0x019112 (0xf2f5 bytes) 000294 1780327913.210417 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000299 1780327913.597022 trunc from 0x019113 to 0x013989 (0x5789 bytes) 000300 1780327913.708829 trunc from 0x01398a to 0x0290d9 (0x15750 bytes) 000301 1780327913.938039 mapread 0x016cd9 thru 0x025ef8 (0xf220 bytes) 000302 1780327914.064752 read 0x01105e thru 0x01bc20 (0xabc3 bytes) 000303 1780327914.067930 mapread 0x028572 thru 0x0290d9 (0xb68 bytes) 000306 1780327914.078028 mapwrite 0x002a10 thru 0x007ea5 (0x5496 bytes) 000307 1780327914.291927 mapread 0x025039 thru 0x0290d9 (0x40a1 bytes) 000309 1780327914.313826 mapread 0x01b1a2 thru 0x027fbd (0xce1c bytes) 000311 1780327914.325556 write 0x0247f2 thru 0x024c0c (0x41b bytes) 000314 1780327914.351593 trunc from 0x0290da to 0x0049b0 (0x24729 bytes) 000315 1780327914.398067 write 0x02033d thru 0x02cbb8 (0xc87c bytes) 000316 1780327914.409968 read 0x00ea3e thru 0x01acc8 (0xc28b bytes) 000319 1780327914.422549 mapwrite 0x01f240 thru 0x021f12 (0x2cd3 bytes) 000320 1780327914.519613 write 0x016782 thru 0x0209bb (0xa23a bytes) 000321 1780327914.522583 write 0x038a55 thru 0x03e6a5 (0x5c51 bytes) 000322 1780327914.524859 trunc from 0x03e6a6 to 0x01e5a3 (0x20102 bytes) 000323 1780327914.548092 mapread 0x01981c thru 0x01e5a3 (0x4d88 bytes) 000326 1780327914.556555 trunc from 0x01e5a4 to 0x02ed7c (0x107d9 bytes) 000327 1780327914.609557 mapread 0x02dd82 thru 0x02e51b (0x79a bytes) 000328 1780327914.623427 trunc from 0x02ed7d to 0x030d39 (0x1fbd bytes) 000330 1780327914.660819 mapwrite 0x00ae2d thru 0x013700 (0x88d4 bytes) 000331 1780327914.768369 read 0x0090d4 thru 0x016fc6 (0xdef3 bytes) 000332 1780327914.770628 trunc from 0x030d3a to 0x0344fe (0x37c5 bytes) 000333 1780327914.788426 trunc from 0x0344ff to 0x022399 (0x12165 bytes) 000334 1780327914.806983 mapread 0x0023c3 thru 0x0088a6 (0x64e4 bytes) 000335 1780327914.808133 trunc from 0x02239a to 0x00264e (0x1fd4b bytes) 000340 1780327914.830327 mapread 0x0003de thru 0x00264e (0x2271 bytes) 000342 1780327914.833754 write 0x0165d5 thru 0x0233ec (0xce18 bytes) 000346 1780327914.847941 mapwrite 0x023e42 thru 0x032aec (0xecab bytes) 000351 1780327914.960638 mapread 0x02c54e thru 0x032aec (0x659f bytes) 000352 1780327914.963925 trunc from 0x032aed to 0x0127c2 (0x2032a bytes) 000354 1780327914.982352 read 0x009170 thru 0x010b95 (0x7a26 bytes) 000355 1780327914.983160 read 0x010218 thru 0x0127c2 (0x25ab bytes) 000356 1780327914.989386 read 0x0107b7 thru 0x0127c2 (0x200c bytes) 000357 1780327914.990662 trunc from 0x0127c3 to 0x02e4a5 (0x1bce3 bytes) 000358 1780327915.008369 mapwrite 0x0194e9 thru 0x028eb3 (0xf9cb bytes) 000359 1780327915.124846 write 0x03eae5 thru 0x03ffff (0x151b bytes) 000360 1780327915.126006 trunc from 0x040000 to 0x0197ba (0x26845 bytes) 000361 1780327915.149407 trunc from 0x0197bb to 0x019474 (0x346 bytes) 000362 1780327915.169879 mapread 0x011f54 thru 0x012b18 (0xbc5 bytes) 000363 1780327915.170860 mapread 0x006440 thru 0x012e27 (0xc9e8 bytes) 000365 1780327915.178619 write 0x01bd58 thru 0x01c387 (0x630 bytes) 000368 1780327915.186935 trunc from 0x01c388 to 0x01b9e7 (0x9a0 bytes) 000369 1780327915.222353 read 0x00a7f6 thru 0x01847f (0xdc8a bytes) 000371 1780327915.223543 read 0x006b69 thru 0x01586e (0xed06 bytes) 000373 1780327915.224695 mapwrite 0x01545d thru 0x01f17e (0x9d22 bytes) 000375 1780327915.442677 read 0x0026c1 thru 0x00f51e (0xce5e bytes) 000376 1780327915.443633 mapwrite 0x0377a1 thru 0x03d2f2 (0x5b52 bytes) 000377 1780327915.502965 mapread 0x023dad thru 0x02c152 (0x83a6 bytes) 000379 1780327915.505794 trunc from 0x03d2f3 to 0x018974 (0x2497e bytes) 000381 1780327915.523703 mapwrite 0x03dc65 thru 0x03ffff (0x239b bytes) 000382 1780327915.657568 mapwrite 0x013b4d thru 0x0160c1 (0x2575 bytes) 000383 1780327915.740993 read 0x028c2f thru 0x029942 (0xd14 bytes) 000384 1780327915.784006 read 0x023bc1 thru 0x02ca0d (0x8e4d bytes) 000385 1780327915.789472 mapwrite 0x025509 thru 0x029ac7 (0x45bf bytes) 000386 1780327915.898719 mapwrite 0x005b68 thru 0x00f514 (0x99ad bytes) 000388 1780327916.026652 write 0x0049eb thru 0x007c79 (0x328f bytes) 000391 1780327916.028512 mapwrite 0x024cff thru 0x02f318 (0xa61a bytes) 000392 1780327916.198852 write 0x02d9e0 thru 0x03d239 (0xf85a bytes) 000395 1780327916.211240 read 0x03e549 thru 0x03e800 (0x2b8 bytes) 000403 1780327916.237873 read 0x01ef23 thru 0x023db4 (0x4e92 bytes) 000404 1780327916.238596 mapread 0x014523 thru 0x0232a6 (0xed84 bytes) 000408 1780327916.260009 read 0x03bbd3 thru 0x03ffff (0x442d bytes) 000410 1780327916.262387 read 0x03a160 thru 0x03ffff (0x5ea0 bytes) 000411 1780327916.283160 mapwrite 0x003a11 thru 0x013633 (0xfc23 bytes) 000414 1780327916.505596 write 0x003473 thru 0x004244 (0xdd2 bytes) 000418 1780327916.516980 read 0x03b3a5 thru 0x03ffff (0x4c5b bytes) 000419 1780327916.517881 read 0x0296be thru 0x02e8e7 (0x522a bytes) 000420 1780327916.534655 mapread 0x02fc8b thru 0x03c1c8 (0xc53e bytes) 000422 1780327916.553829 write 0x014c44 thru 0x023148 (0xe505 bytes) 000424 1780327916.567768 mapread 0x0364f0 thru 0x03e4eb (0x7ffc bytes) 000428 1780327916.579656 write 0x03272d thru 0x03dd07 (0xb5db bytes) 000431 1780327916.585431 read 0x03813f thru 0x03d05c (0x4f1e bytes) 000433 1780327916.590226 read 0x0334a3 thru 0x034cc3 (0x1821 bytes) 000435 1780327916.592141 trunc from 0x040000 to 0x032c9e (0xd361 bytes) 000441 1780327916.620843 mapread 0x00f43c thru 0x01de90 (0xea55 bytes) 000443 1780327916.627109 read 0x026518 thru 0x02fd49 (0x9832 bytes) 000444 1780327916.633253 write 0x01fbc9 thru 0x02ee41 (0xf279 bytes) 000445 1780327916.647894 MIRROR_EXTEND to 2 mirrors 000446 1780327917.839344 MIRROR_RESYNC 2 mirrors 000449 1780327918.463047 trunc from 0x032c9f to 0x01c4df (0x167bf bytes) 000450 1780327918.689790 read 0x00e0ab thru 0x018fb9 (0xaf0f bytes) 000451 1780327918.701795 trunc from 0x01c4e0 to 0x00f13c (0xd3a3 bytes) 000452 1780327918.732369 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf13d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf13d): Mirror 2: 0xd54a29d6 (stale) CRC-32 checksum value for chunk [0, 0xf13d): Mirror 1: 0xd54a29d6 Mirror 3: 0xd54a29d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf13d: skipped 000456 1780327920.520703 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf13d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf13d): Mirror 2: 0xd54a29d6 (stale) CRC-32 checksum value for chunk [0, 0xf13d): Mirror 1: 0xd54a29d6 Mirror 3: 0xd54a29d6 Mirror 4: 0xd54a29d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf13d: skipped 000457 1780327922.024593 read 0x004e9b thru 0x00af11 (0x6077 bytes) 000458 1780327922.057035 trunc from 0x00f13d to 0x02310d (0x13fd1 bytes) 000460 1780327922.314365 write 0x02a533 thru 0x02c661 (0x212f bytes) 000462 1780327922.324644 mapread 0x004b8d thru 0x006836 (0x1caa bytes) 000463 1780327922.364727 write 0x039231 thru 0x03e579 (0x5349 bytes) 000464 1780327922.371651 trunc from 0x03e57a to 0x004dff (0x3977a bytes) 000465 1780327922.421500 read 0x0040a5 thru 0x004dff (0xd5b bytes) 000467 1780327922.428764 mapread 0x003c84 thru 0x004dff (0x117c bytes) 000472 1780327922.440293 mapwrite 0x002fea thru 0x00ae7b (0x7e92 bytes) 000474 1780327922.667153 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xae7c): Mirror 3: 0x7748598e (stale) Mirror 4: 0x7748598e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae7c): Mirror 3: 0x7748598e (stale) Mirror 4: 0x7748598e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae7c: skipped 000475 1780327923.490585 trunc from 0x00ae7c to 0x008c9f (0x21dc bytes) 000476 1780327923.519332 mapwrite 0x01b649 thru 0x0235f4 (0x7fac bytes) 000483 1780327923.712023 mapwrite 0x026f7b thru 0x0369ae (0xfa34 bytes) 000486 1780327923.888327 trunc from 0x0369af to 0x039e95 (0x34e7 bytes) 000489 1780327923.911127 read 0x01cc34 thru 0x023a55 (0x6e22 bytes) 000490 1780327923.913118 mapread 0x012966 thru 0x018d27 (0x63c2 bytes) 000491 1780327923.937861 mapread 0x004fc0 thru 0x0051c5 (0x206 bytes) 000492 1780327923.977688 write 0x034204 thru 0x03cde1 (0x8bde bytes) 000494 1780327923.987608 write 0x01a70c thru 0x0281bb (0xdab0 bytes) 000495 1780327923.996449 trunc from 0x03cde2 to 0x02bfc7 (0x10e1a bytes) 000496 1780327924.083983 MIRROR_RESYNC 3 mirrors 000497 1780327925.633485 MIRROR_RESYNC 3 mirrors 000498 1780327926.391740 write 0x00bcfb thru 0x01a730 (0xea36 bytes) 000499 1780327926.727021 MIRROR_RESYNC 3 mirrors 000501 1780327928.411465 trunc from 0x02bfc8 to 0x03314b (0x7184 bytes) 000502 1780327928.711643 write 0x027345 thru 0x02c8da (0x5596 bytes) 000505 1780327928.746090 write 0x034159 thru 0x038984 (0x482c bytes) 000506 1780327928.753303 trunc from 0x038985 to 0x026bd2 (0x11db2 bytes) 000507 1780327928.847934 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26bd3): Mirror 3: 0x215abf15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26bd3): Mirror 3: 0x215abf15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26bd3: skipped 000509 1780327929.513337 mapread 0x0069fe thru 0x00bd1d (0x5320 bytes) 000510 1780327929.581731 MIRROR_RESYNC 2 mirrors 000512 1780327931.219072 mapwrite 0x01817b thru 0x0201d4 (0x805a bytes) 000514 1780327931.552662 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000517 1780327931.802093 mapread 0x01d82d thru 0x026bd2 (0x93a6 bytes) 000518 1780327931.827698 trunc from 0x026bd3 to 0x03851b (0x11949 bytes) 000519 1780327931.854425 mapread 0x018c06 thru 0x019a29 (0xe24 bytes) 000520 1780327931.855453 read 0x00dd97 thru 0x012ebb (0x5125 bytes) 000522 1780327931.879499 MIRROR_EXTEND to 2 mirrors 000525 1780327933.077810 MIRROR_EXTEND to 3 mirrors 000526 1780327934.764021 mapwrite 0x02c916 thru 0x03740a (0xaaf5 bytes) 000529 1780327935.247094 mapwrite 0x023c14 thru 0x02bc0c (0x7ff9 bytes) 000532 1780327935.324851 mapwrite 0x018290 thru 0x020429 (0x819a bytes) 000536 1780327935.429101 mapread 0x021901 thru 0x028133 (0x6833 bytes) 000538 1780327935.437493 MIRROR_RESYNC 3 mirrors 000539 1780327936.534163 read 0x016d24 thru 0x0260ed (0xf3ca bytes) 000543 1780327936.544499 read 0x00cf51 thru 0x015edd (0x8f8d bytes) 000544 1780327936.561752 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000545 1780327937.093838 mapread 0x030350 thru 0x03851b (0x81cc bytes) 000547 1780327937.160563 mapwrite 0x02e08a thru 0x03a5c7 (0xc53e bytes) 000548 1780327937.432772 write 0x0242b8 thru 0x02942c (0x5175 bytes) 000549 1780327937.454872 write 0x0300e9 thru 0x033b83 (0x3a9b bytes) 000550 1780327937.455905 trunc from 0x03a5c8 to 0x010acb (0x29afc bytes) 000554 1780327937.490725 mapread 0x00db11 thru 0x010acb (0x2fbb bytes) 000556 1780327937.516707 read 0x001e37 thru 0x006bcf (0x4d99 bytes) 000557 1780327937.532439 read 0x006059 thru 0x008630 (0x25d8 bytes) 000558 1780327937.538820 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10acc): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10acc): Mirror 3: 0x70fbb092 (stale) CRC-32 checksum value for chunk [0, 0x10acc): Mirror 1: 0x70fbb092 Mirror 4: 0x70fbb092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10acc: skipped 000561 1780327939.258436 MIRROR_RESYNC 3 mirrors 000566 1780327940.479279 trunc from 0x010acc to 0x0183cf (0x7904 bytes) 000567 1780327940.669327 read 0x0047d9 thru 0x00d733 (0x8f5b bytes) 000568 1780327940.691450 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x183d0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x183d0): Mirror 3: 0x1b31901 (stale) Mirror 4: 0x1b31901 (stale) CRC-32 checksum value for chunk [0, 0x183d0): Mirror 1: 0x1b31901 Mirror 5: 0x1b31901 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x183d0: skipped 000569 1780327941.770605 mapwrite 0x024bd1 thru 0x02cc51 (0x8081 bytes) 000570 1780327941.923097 write 0x007f79 thru 0x010017 (0x809f bytes) 000571 1780327941.939626 read 0x00f345 thru 0x01b505 (0xc1c1 bytes) 000572 1780327941.949934 read 0x029c01 thru 0x02cc51 (0x3051 bytes) 000575 1780327941.951971 read 0x01ec8e thru 0x0296ba (0xaa2d bytes) 000577 1780327941.958591 MIRROR_RESYNC 4 mirrors 000578 1780327943.143059 write 0x03644e thru 0x03ffff (0x9bb2 bytes) 000580 1780327943.269665 write 0x035d69 thru 0x03c7c7 (0x6a5f bytes) 000582 1780327943.277560 read 0x02837b thru 0x0365d6 (0xe25c bytes) 000583 1780327943.288088 trunc from 0x040000 to 0x032d9c (0xd263 bytes) 000584 1780327943.314685 mapwrite 0x026204 thru 0x030bf6 (0xa9f3 bytes) 000585 1780327943.398896 mapread 0x027c8a thru 0x02f081 (0x73f8 bytes) 000586 1780327943.401058 mapread 0x011b42 thru 0x015a43 (0x3f02 bytes) 000589 1780327943.424468 mapread 0x02d98f thru 0x032d9c (0x540e bytes) 000591 1780327943.431948 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32d9d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d9d): Mirror 3: 0xc21ca3da (stale) Mirror 4: 0xc21ca3da (stale) Mirror 5: 0xc21ca3da (stale) CRC-32 checksum value for chunk [0, 0x32d9d): Mirror 1: 0xe5bbc41 Mirror 6: 0xe5bbc41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d9d: skipped 000592 1780327944.201797 mapwrite 0x03cb26 thru 0x03ffff (0x34da bytes) 000593 1780327944.494912 mapwrite 0x021e10 thru 0x02e330 (0xc521 bytes) 000594 1780327944.596489 mapwrite 0x00d161 thru 0x0147ee (0x768e bytes) 000595 1780327944.650327 read 0x03b31a thru 0x03ffff (0x4ce6 bytes) 000596 1780327944.651383 mapwrite 0x020480 thru 0x029d47 (0x98c8 bytes) 000600 1780327944.687589 mapwrite 0x033eb6 thru 0x03a016 (0x6161 bytes) 000601 1780327944.728422 mapwrite 0x028508 thru 0x033f84 (0xba7d bytes) 000603 1780327944.792497 mapwrite 0x02937f thru 0x02d3c4 (0x4046 bytes) 000604 1780327944.858145 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2bb27ac2 (stale) Mirror 4: 0x2bb27ac2 (stale) Mirror 5: 0x2bb27ac2 (stale) Mirror 6: 0x31bb1d2c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5fec5f71 Mirror 7: 0x5fec5f71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000608 1780327945.939768 MIRROR_RESYNC 6 mirrors 000609 1780327947.005590 write 0x005d5c thru 0x0148bd (0xeb62 bytes) 000612 1780327947.156657 read 0x03f582 thru 0x03ffff (0xa7e bytes) 000614 1780327947.163772 mapread 0x03cbd6 thru 0x03ffff (0x342a bytes) 000617 1780327947.189914 mapwrite 0x00bf8e thru 0x0105ad (0x4620 bytes) 000618 1780327947.250526 mapread 0x03ed32 thru 0x03ffff (0x12ce bytes) 000621 1780327947.259538 trunc from 0x040000 to 0x03940b (0x6bf4 bytes) 000622 1780327947.298953 mapwrite 0x01db00 thru 0x020481 (0x2982 bytes) 000623 1780327947.389069 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3940c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3940c): Mirror 3: 0x2711eb28 (stale) Mirror 4: 0x2711eb28 (stale) Mirror 5: 0x2711eb28 (stale) Mirror 6: 0x2711eb28 (stale) Mirror 7: 0x2711eb28 (stale) CRC-32 checksum value for chunk [0, 0x3940c): Mirror 1: 0xeb4f5abd Mirror 8: 0xeb4f5abd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3940c: skipped 000625 1780327949.167021 mapread 0x02aaf0 thru 0x02d241 (0x2752 bytes) 000626 1780327949.210475 mapread 0x002bc9 thru 0x011133 (0xe56b bytes) 000628 1780327949.277553 write 0x000c32 thru 0x0026b6 (0x1a85 bytes) 000629 1780327949.397695 mapwrite 0x023e78 thru 0x02acc6 (0x6e4f bytes) 000631 1780327949.457378 mapread 0x01f43a thru 0x024d60 (0x5927 bytes) 000632 1780327949.464341 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3940c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3940c): Mirror 3: 0x2711eb28 (stale) Mirror 4: 0x2711eb28 (stale) Mirror 5: 0x2711eb28 (stale) Mirror 6: 0x2711eb28 (stale) Mirror 7: 0x2711eb28 (stale) Mirror 8: 0xeb4f5abd (stale) CRC-32 checksum value for chunk [0, 0x3940c): Mirror 1: 0xf3a39e0c Mirror 9: 0xf3a39e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3940c: skipped 000633 1780327950.604732 mapread 0x0014b6 thru 0x0052ac (0x3df7 bytes) 000634 1780327950.666859 mapread 0x00a726 thru 0x00cd04 (0x25df bytes) 000636 1780327950.712281 trunc from 0x03940c to 0x03b81b (0x2410 bytes) 000637 1780327950.879452 read 0x028120 thru 0x0356c4 (0xd5a5 bytes) 000638 1780327950.897043 mapread 0x0229b1 thru 0x031877 (0xeec7 bytes) 000646 1780327950.914979 mapwrite 0x00ad97 thru 0x012de8 (0x8052 bytes) 000647 1780327951.004666 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b81c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b81c): Mirror 3: 0x22d840ca (stale) Mirror 4: 0x22d840ca (stale) Mirror 5: 0x22d840ca (stale) Mirror 6: 0x22d840ca (stale) Mirror 7: 0x22d840ca (stale) Mirror 8: 0xcddb3d0d (stale) Mirror 9: 0x72caf8f1 (stale) CRC-32 checksum value for chunk [0, 0x3b81c): Mirror 1: 0xd250dc5a Mirror 10: 0xd250dc5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b81c: skipped 000648 1780327952.248088 mapread 0x01f59f thru 0x02146d (0x1ecf bytes) 000651 1780327952.345200 read 0x02a69a thru 0x030792 (0x60f9 bytes) 000654 1780327952.419470 read 0x0113ae thru 0x015d83 (0x49d6 bytes) 000656 1780327952.479113 read 0x034db1 thru 0x03b81b (0x6a6b bytes) 000658 1780327952.507723 trunc from 0x03b81c to 0x032a87 (0x8d94 bytes) 000659 1780327952.653339 mapread 0x01dc93 thru 0x02828d (0xa5fb bytes) 000660 1780327952.681108 write 0x003e94 thru 0x01117f (0xd2ec bytes) 000661 1780327952.685905 mapread 0x02f339 thru 0x032a87 (0x374f bytes) 000662 1780327952.688578 write 0x007f7b thru 0x00e974 (0x69fa bytes) 000663 1780327952.698055 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32a88): Mirror 3: 0x957c325a (stale) Mirror 4: 0x957c325a (stale) Mirror 5: 0x957c325a (stale) Mirror 6: 0x957c325a (stale) Mirror 7: 0x957c325a (stale) Mirror 9: 0x58cf7e4b (stale) Mirror 10: 0x2bc5c051 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a88): Mirror 3: 0x957c325a (stale) Mirror 4: 0x957c325a (stale) Mirror 5: 0x957c325a (stale) Mirror 6: 0x957c325a (stale) Mirror 7: 0x957c325a (stale) Mirror 9: 0x58cf7e4b (stale) Mirror 10: 0x2bc5c051 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a88: skipped 000664 1780327953.437562 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32a88): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a88): Mirror 3: 0x957c325a (stale) Mirror 4: 0x957c325a (stale) Mirror 5: 0x957c325a (stale) Mirror 6: 0x957c325a (stale) Mirror 7: 0x957c325a (stale) Mirror 9: 0x58cf7e4b (stale) Mirror 10: 0x2bc5c051 (stale) CRC-32 checksum value for chunk [0, 0x32a88): Mirror 1: 0x67cbe49b Mirror 11: 0x67cbe49b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a88: skipped 000665 1780327954.819768 trunc from 0x032a88 to 0x0013c0 (0x316c7 bytes) 000667 1780327954.946676 mapread 0x000a69 thru 0x0013c0 (0x958 bytes) 000669 1780327954.958500 mapread 0x00040d thru 0x0013c0 (0xfb4 bytes) 000673 1780327954.962716 write 0x01e2ed thru 0x027eae (0x9bc2 bytes) 000674 1780327954.964994 mapread 0x00849e thru 0x016638 (0xe19b bytes) 000676 1780327954.986634 write 0x029205 thru 0x030983 (0x777f bytes) 000677 1780327954.992735 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x30984): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30984): Mirror 3: 0x73072026 (stale) Mirror 4: 0x73072026 (stale) Mirror 5: 0x73072026 (stale) Mirror 6: 0x73072026 (stale) Mirror 7: 0x73072026 (stale) Mirror 9: 0xaf8d191c (stale) Mirror 10: 0xdcc6dfe4 (stale) Mirror 11: 0x23bea0d (stale) CRC-32 checksum value for chunk [0, 0x30984): Mirror 1: 0x8c2cb95c Mirror 12: 0x8c2cb95c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30984: skipped 000678 1780327956.061805 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30984): Mirror 4: 0x73072026 (stale) Mirror 5: 0x73072026 (stale) Mirror 6: 0x73072026 (stale) Mirror 7: 0x73072026 (stale) Mirror 9: 0xaf8d191c (stale) Mirror 10: 0xdcc6dfe4 (stale) Mirror 11: 0x23bea0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30984): Mirror 4: 0x73072026 (stale) Mirror 5: 0x73072026 (stale) Mirror 6: 0x73072026 (stale) Mirror 7: 0x73072026 (stale) Mirror 9: 0xaf8d191c (stale) Mirror 10: 0xdcc6dfe4 (stale) Mirror 11: 0x23bea0d (stale) CRC-32 checksum value for chunk [0, 0x30984): Mirror 1: 0x8c2cb95c Mirror 12: 0x8c2cb95c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30984: skipped 000679 1780327957.333055 read 0x01d001 thru 0x021f74 (0x4f74 bytes) 000680 1780327957.395565 read 0x02a946 thru 0x030983 (0x603e bytes) 000681 1780327957.472135 mapread 0x015cf3 thru 0x018fdc (0x32ea bytes) 000682 1780327957.534059 mapread 0x0292b0 thru 0x030983 (0x76d4 bytes) 000683 1780327957.578695 trunc from 0x030984 to 0x03eb7a (0xe1f7 bytes) 000687 1780327957.745741 mapread 0x0253a0 thru 0x02c70d (0x736e bytes) 000688 1780327957.794307 mapread 0x0136e6 thru 0x023323 (0xfc3e bytes) 000689 1780327957.888799 MIRROR_RESYNC 9 mirrors 000690 1780327959.299299 mapwrite 0x00474c thru 0x00da9d (0x9352 bytes) 000692 1780327959.641153 mapread 0x0301cf thru 0x03eb7a (0xe9ac bytes) 000695 1780327959.663792 mapread 0x02c6da thru 0x02fa60 (0x3387 bytes) 000696 1780327959.665343 write 0x011cde thru 0x019b29 (0x7e4c bytes) 000698 1780327959.688760 trunc from 0x03eb7b to 0x014f5c (0x29c1e bytes) 000701 1780327959.732643 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x14f5d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14f5d): Mirror 4: 0x7c24a9db (stale) Mirror 5: 0x7c24a9db (stale) Mirror 6: 0x7c24a9db (stale) Mirror 7: 0x7c24a9db (stale) Mirror 9: 0x7c24a9db (stale) Mirror 10: 0x7c24a9db (stale) Mirror 11: 0x7c24a9db (stale) Mirror 12: 0x7c24a9db (stale) CRC-32 checksum value for chunk [0, 0x14f5d): Mirror 1: 0xd0e46015 Mirror 13: 0xd0e46015 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14f5d: skipped 000702 1780327960.855435 read 0x0044ce thru 0x010cc8 (0xc7fb bytes) 000703 1780327960.877540 read 0x00bfa3 thru 0x014f5c (0x8fba bytes) 000705 1780327960.916970 trunc from 0x014f5d to 0x03ed5d (0x29e01 bytes) 000707 1780327961.074983 trunc from 0x03ed5e to 0x013cf8 (0x2b065 bytes) 000709 1780327961.111222 write 0x03c515 thru 0x03c67c (0x168 bytes) 000710 1780327961.112283 mapread 0x008336 thru 0x009b9d (0x1868 bytes) 000711 1780327961.140728 write 0x00ead4 thru 0x0184dc (0x9a09 bytes) 000713 1780327961.160640 mapwrite 0x035a4b thru 0x03ffff (0xa5b5 bytes) 000715 1780327961.282698 mapread 0x00c774 thru 0x01754d (0xadda bytes) 000716 1780327961.306613 write 0x020327 thru 0x023e23 (0x3afd bytes) 000718 1780327961.310040 mapwrite 0x00d40c thru 0x01ba8f (0xe684 bytes) 000719 1780327961.361959 mapwrite 0x03b2ba thru 0x03ffff (0x4d46 bytes) 000721 1780327961.419965 write 0x03fa54 thru 0x03ffff (0x5ac bytes) 000722 1780327961.421076 trunc from 0x040000 to 0x038b07 (0x74f8 bytes) 000723 1780327961.458100 MIRROR_RESYNC 10 mirrors 000725 1780327963.407153 mapread 0x018826 thru 0x020a51 (0x822c bytes) 000727 1780327963.428415 mapread 0x020467 thru 0x026a46 (0x65e0 bytes) 000730 1780327963.437514 mapwrite 0x038033 thru 0x03f87c (0x784a bytes) 000733 1780327963.646820 write 0x00e60f thru 0x01bbc5 (0xd5b7 bytes) 000736 1780327963.681517 write 0x030e89 thru 0x03fe3c (0xefb4 bytes) 000737 1780327963.701786 mapread 0x03e689 thru 0x03fe3c (0x17b4 bytes) 000738 1780327963.707156 mapread 0x038685 thru 0x03fe3c (0x77b8 bytes) 000739 1780327963.710843 mapread 0x00e32a thru 0x01b3db (0xd0b2 bytes) 000740 1780327963.715972 mapwrite 0x0051a1 thru 0x009abf (0x491f bytes) 000743 1780327963.806641 read 0x035dd4 thru 0x0382ea (0x2517 bytes) 000744 1780327963.812474 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3fe3d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe3d): Mirror 4: 0xf287f8e8 (stale) Mirror 5: 0xf287f8e8 (stale) Mirror 6: 0xf287f8e8 (stale) Mirror 7: 0xf287f8e8 (stale) Mirror 9: 0xf287f8e8 (stale) Mirror 10: 0xf287f8e8 (stale) Mirror 11: 0xf287f8e8 (stale) Mirror 12: 0xf287f8e8 (stale) Mirror 13: 0xf287f8e8 (stale) CRC-32 checksum value for chunk [0, 0x3fe3d): Mirror 1: 0x8afeff0 Mirror 14: 0x8afeff0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe3d: skipped 000746 1780327964.977280 MIRROR_RESYNC 11 mirrors 000747 1780327966.899581 MIRROR_RESYNC 11 mirrors 000748 1780327967.810843 mapread 0x0297f2 thru 0x02fb50 (0x635f bytes) 000752 1780327967.863624 trunc from 0x03fe3d to 0x03985c (0x65e0 bytes) 000753 1780327967.974700 read 0x003bf5 thru 0x00e43b (0xa847 bytes) 000755 1780327967.987931 read 0x00f4f7 thru 0x016d86 (0x7890 bytes) 000756 1780327968.004155 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3985d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3985d): Mirror 4: 0xf7094231 (stale) Mirror 5: 0xf7094231 (stale) Mirror 6: 0xf7094231 (stale) Mirror 7: 0xf7094231 (stale) Mirror 9: 0xf7094231 (stale) Mirror 10: 0xf7094231 (stale) Mirror 11: 0xf7094231 (stale) Mirror 12: 0xf7094231 (stale) Mirror 13: 0xf7094231 (stale) Mirror 14: 0xf7094231 (stale) CRC-32 checksum value for chunk [0, 0x3985d): Mirror 1: 0xf7094231 Mirror 15: 0xf7094231 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3985d: skipped 000758 1780327969.647565 trunc from 0x03985d to 0x011475 (0x283e7 bytes) 000765 1780327969.766022 write 0x01a879 thru 0x021723 (0x6eab bytes) 000767 1780327969.771937 mapwrite 0x026d5d thru 0x031514 (0xa7b8 bytes) 000768 1780327970.009524 trunc from 0x031515 to 0x01db5d (0x139b7 bytes) 000769 1780327970.031625 write 0x02ca7b thru 0x035baf (0x9135 bytes) 000770 1780327970.033253 write 0x000a8e thru 0x00ce1b (0xc38e bytes) 000771 1780327970.079518 write 0x031d38 thru 0x03cca9 (0xaf72 bytes) 000773 1780327970.083729 write 0x03d137 thru 0x03ffff (0x2ec9 bytes) 000774 1780327970.085318 trunc from 0x040000 to 0x02c2fa (0x13d05 bytes) 000778 1780327970.153131 mapwrite 0x02ac18 thru 0x02e35d (0x3746 bytes) 000779 1780327970.313222 mapread 0x024356 thru 0x02e35d (0xa008 bytes) 000780 1780327970.323854 mapread 0x02162d thru 0x02310f (0x1ae3 bytes) 000782 1780327970.370956 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2e35e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e35e): Mirror 4: 0xc68e529b (stale) Mirror 5: 0xc68e529b (stale) Mirror 6: 0xc68e529b (stale) Mirror 7: 0xc68e529b (stale) Mirror 9: 0xc68e529b (stale) Mirror 10: 0xc68e529b (stale) Mirror 11: 0xc68e529b (stale) Mirror 12: 0xc68e529b (stale) Mirror 13: 0xc68e529b (stale) Mirror 14: 0xc68e529b (stale) Mirror 15: 0xc68e529b (stale) CRC-32 checksum value for chunk [0, 0x2e35e): Mirror 1: 0x2dd38014 Mirror 16: 0x2dd38014 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e35e: skipped 000785 1780327972.272284 mapwrite 0x00655e thru 0x01269e (0xc141 bytes) 000787 1780327972.578029 read 0x021307 thru 0x028325 (0x701f bytes) 000788 1780327972.591396 mapread 0x0221f0 thru 0x0238a7 (0x16b8 bytes) 000792 1780327972.609099 mapread 0x029607 thru 0x02e35d (0x4d57 bytes) 000795 1780327972.632630 read 0x017201 thru 0x02199a (0xa79a bytes) 000798 1780327972.668500 write 0x00787c thru 0x016bdf (0xf364 bytes) 000799 1780327972.670967 read 0x026b2f thru 0x02d514 (0x69e6 bytes) 000801 1780327972.672557 write 0x013dc5 thru 0x01c764 (0x89a0 bytes) 000802 1780327972.677786 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2e35e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e35e): Mirror 4: 0xc68e529b (stale) Mirror 5: 0xc68e529b (stale) Mirror 6: 0xc68e529b (stale) Mirror 7: 0xc68e529b (stale) Mirror 9: 0xc68e529b (stale) Mirror 10: 0xc68e529b (stale) Mirror 11: 0xc68e529b (stale) Mirror 12: 0xc68e529b (stale) Mirror 13: 0xc68e529b (stale) Mirror 14: 0xc68e529b (stale) Mirror 15: 0xc68e529b (stale) Mirror 16: 0x2dd38014 (stale) CRC-32 checksum value for chunk [0, 0x2e35e): Mirror 1: 0xdc82515b Mirror 17: 0xdc82515b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e35e: skipped 000804 1780327973.574626 write 0x007aa7 thru 0x00baa8 (0x4002 bytes) 000805 1780327973.815859 mapwrite 0x00bf3c thru 0x00e619 (0x26de bytes) 000807 1780327973.922076 write 0x03a594 thru 0x03ffff (0x5a6c bytes) 000810 1780327973.927622 mapwrite 0x027cdb thru 0x034716 (0xca3c bytes) 000812 1780327974.088675 write 0x00639b thru 0x013aa1 (0xd707 bytes) 000814 1780327974.100609 mapread 0x0342a6 thru 0x03d5af (0x930a bytes) 000815 1780327974.118967 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9c4abfbf (stale) Mirror 5: 0x9c4abfbf (stale) Mirror 6: 0x9c4abfbf (stale) Mirror 7: 0x9c4abfbf (stale) Mirror 9: 0x9c4abfbf (stale) Mirror 10: 0x9c4abfbf (stale) Mirror 11: 0x9c4abfbf (stale) Mirror 12: 0x9c4abfbf (stale) Mirror 13: 0x9c4abfbf (stale) Mirror 14: 0x9c4abfbf (stale) Mirror 15: 0xc16a7dc4 (stale) Mirror 16: 0xff4c2199 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9c4abfbf (stale) Mirror 5: 0x9c4abfbf (stale) Mirror 6: 0x9c4abfbf (stale) Mirror 7: 0x9c4abfbf (stale) Mirror 9: 0x9c4abfbf (stale) Mirror 10: 0x9c4abfbf (stale) Mirror 11: 0x9c4abfbf (stale) Mirror 12: 0x9c4abfbf (stale) Mirror 13: 0x9c4abfbf (stale) Mirror 14: 0x9c4abfbf (stale) Mirror 15: 0xc16a7dc4 (stale) Mirror 16: 0xff4c2199 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000816 1780327975.007017 mapwrite 0x029cdc thru 0x036398 (0xc6bd bytes) 000819 1780327975.132088 trunc from 0x040000 to 0x026e7c (0x19183 bytes) 000821 1780327975.186965 trunc from 0x026e7d to 0x01a83e (0xc63e bytes) 000822 1780327975.219100 mapwrite 0x01d759 thru 0x01fdd5 (0x267d bytes) 000823 1780327975.336962 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1fdd6): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fdd6): Mirror 4: 0x9d8e377 (stale) Mirror 5: 0x9d8e377 (stale) Mirror 6: 0x9d8e377 (stale) Mirror 7: 0x9d8e377 (stale) Mirror 9: 0x9d8e377 (stale) Mirror 10: 0x9d8e377 (stale) Mirror 11: 0x9d8e377 (stale) Mirror 12: 0x9d8e377 (stale) Mirror 13: 0x9d8e377 (stale) Mirror 14: 0x9d8e377 (stale) Mirror 15: 0x9d8e377 (stale) Mirror 16: 0xb4841ec7 (stale) CRC-32 checksum value for chunk [0, 0x1fdd6): Mirror 1: 0xe275fca4 Mirror 17: 0xe275fca4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fdd6: skipped 000826 1780327976.558541 mapwrite 0x01324a thru 0x01d837 (0xa5ee bytes) 000828 1780327976.871650 mapwrite 0x025809 thru 0x028278 (0x2a70 bytes) 000831 1780327977.095331 read 0x028182 thru 0x028278 (0xf7 bytes) 000832 1780327977.103979 write 0x006c54 thru 0x011b61 (0xaf0e bytes) 000834 1780327977.138052 read 0x003cd3 thru 0x00eacf (0xadfd bytes) 000835 1780327977.138770 write 0x01b8ac thru 0x028412 (0xcb67 bytes) 000836 1780327977.144861 mapwrite 0x0268c5 thru 0x0291ec (0x2928 bytes) 000837 1780327977.331799 read 0x002197 thru 0x008764 (0x65ce bytes) 000838 1780327977.362995 MIRROR_RESYNC 14 mirrors 000841 1780327981.221942 trunc from 0x0291ed to 0x00694d (0x2289f bytes) 000843 1780327981.399110 mapwrite 0x0107c4 thru 0x018651 (0x7e8e bytes) 000844 1780327981.739945 mapread 0x01151f thru 0x018651 (0x7133 bytes) 000845 1780327981.741266 write 0x034513 thru 0x036a8c (0x257a bytes) 000847 1780327981.762516 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x36a8d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a8d): Mirror 4: 0x60d3752f (stale) Mirror 5: 0x60d3752f (stale) Mirror 6: 0x60d3752f (stale) Mirror 7: 0x60d3752f (stale) Mirror 9: 0x60d3752f (stale) Mirror 10: 0x60d3752f (stale) Mirror 11: 0x60d3752f (stale) Mirror 12: 0x60d3752f (stale) Mirror 13: 0x60d3752f (stale) Mirror 14: 0x60d3752f (stale) Mirror 15: 0x60d3752f (stale) Mirror 16: 0x60d3752f (stale) Mirror 17: 0x60d3752f (stale) CRC-32 checksum value for chunk [0, 0x36a8d): Mirror 1: 0xe1a9c864 Mirror 18: 0xe1a9c864 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a8d: skipped 000848 1780327983.762428 write 0x010c5a thru 0x014b1f (0x3ec6 bytes) 000850 1780327983.973793 read 0x01baae thru 0x01e11b (0x266e bytes) 000851 1780327983.982790 trunc from 0x036a8d to 0x026c0c (0xfe80 bytes) 000852 1780327984.030217 mapwrite 0x0395af thru 0x03ffff (0x6a51 bytes) 000853 1780327984.249357 mapwrite 0x006a58 thru 0x00c3cf (0x5978 bytes) 000855 1780327984.380883 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3322dc8a (stale) Mirror 5: 0x3322dc8a (stale) Mirror 6: 0x3322dc8a (stale) Mirror 7: 0x3322dc8a (stale) Mirror 9: 0x3322dc8a (stale) Mirror 10: 0x3322dc8a (stale) Mirror 11: 0x3322dc8a (stale) Mirror 12: 0x3322dc8a (stale) Mirror 13: 0x3322dc8a (stale) Mirror 14: 0x3322dc8a (stale) Mirror 15: 0x3322dc8a (stale) Mirror 16: 0x3322dc8a (stale) Mirror 18: 0x6b80b93f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3322dc8a (stale) Mirror 5: 0x3322dc8a (stale) Mirror 6: 0x3322dc8a (stale) Mirror 7: 0x3322dc8a (stale) Mirror 9: 0x3322dc8a (stale) Mirror 10: 0x3322dc8a (stale) Mirror 11: 0x3322dc8a (stale) Mirror 12: 0x3322dc8a (stale) Mirror 13: 0x3322dc8a (stale) Mirror 14: 0x3322dc8a (stale) Mirror 15: 0x3322dc8a (stale) Mirror 16: 0x3322dc8a (stale) Mirror 18: 0x6b80b93f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000856 1780327985.550672 write 0x01895f thru 0x02489a (0xbf3c bytes) 000860 1780327985.577897 trunc from 0x040000 to 0x021240 (0x1edbf bytes) 000862 1780327985.637100 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x21241): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21241): Mirror 4: 0x635b5463 (stale) Mirror 5: 0x635b5463 (stale) Mirror 6: 0x635b5463 (stale) Mirror 7: 0x635b5463 (stale) Mirror 9: 0x635b5463 (stale) Mirror 10: 0x635b5463 (stale) Mirror 11: 0x635b5463 (stale) Mirror 12: 0x635b5463 (stale) Mirror 13: 0x635b5463 (stale) Mirror 14: 0x635b5463 (stale) Mirror 15: 0x635b5463 (stale) Mirror 16: 0x635b5463 (stale) Mirror 18: 0x4997a293 (stale) CRC-32 checksum value for chunk [0, 0x21241): Mirror 1: 0xa2908a5e Mirror 19: 0xa2908a5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21241: skipped 000863 1780327987.365561 mapwrite 0x035dce thru 0x03ffff (0xa232 bytes) 000865 1780327987.850500 write 0x01d845 thru 0x025c15 (0x83d1 bytes) 000870 1780327987.885957 mapwrite 0x019c66 thru 0x021cd2 (0x806d bytes) 000871 1780327988.053744 write 0x00713e thru 0x009611 (0x24d4 bytes) 000876 1780327988.098277 read 0x0278f9 thru 0x03459d (0xcca5 bytes) 000878 1780327988.128110 trunc from 0x040000 to 0x0321e7 (0xde18 bytes) 000881 1780327988.152194 write 0x018b11 thru 0x025edf (0xd3cf bytes) 000884 1780327988.156755 write 0x038680 thru 0x03ffff (0x7980 bytes) 000885 1780327988.159378 read 0x00dead thru 0x01a95e (0xcab2 bytes) 000886 1780327988.168697 trunc from 0x040000 to 0x003591 (0x3ca6e bytes) 000887 1780327988.192653 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3592): Mirror 4: 0xaf1eab1b (stale) Mirror 6: 0xaf1eab1b (stale) Mirror 7: 0xaf1eab1b (stale) Mirror 9: 0xaf1eab1b (stale) Mirror 10: 0xaf1eab1b (stale) Mirror 11: 0xaf1eab1b (stale) Mirror 12: 0xaf1eab1b (stale) Mirror 13: 0xaf1eab1b (stale) Mirror 14: 0xaf1eab1b (stale) Mirror 15: 0xaf1eab1b (stale) Mirror 16: 0xaf1eab1b (stale) Mirror 18: 0xaf1eab1b (stale) Mirror 19: 0xaf1eab1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3592): Mirror 4: 0xaf1eab1b (stale) Mirror 6: 0xaf1eab1b (stale) Mirror 7: 0xaf1eab1b (stale) Mirror 9: 0xaf1eab1b (stale) Mirror 10: 0xaf1eab1b (stale) Mirror 11: 0xaf1eab1b (stale) Mirror 12: 0xaf1eab1b (stale) Mirror 13: 0xaf1eab1b (stale) Mirror 14: 0xaf1eab1b (stale) Mirror 15: 0xaf1eab1b (stale) Mirror 16: 0xaf1eab1b (stale) Mirror 18: 0xaf1eab1b (stale) Mirror 19: 0xaf1eab1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3592: skipped 000889 1780327988.899728 trunc from 0x003592 to 0x01a6b8 (0x17127 bytes) 000891 1780327988.920021 write 0x010fc2 thru 0x0136bc (0x26fb bytes) 000893 1780327988.938563 trunc from 0x01a6b9 to 0x01bc70 (0x15b8 bytes) 000894 1780327988.963335 mapwrite 0x00afba thru 0x017004 (0xc04b bytes) 000895 1780327989.043738 trunc from 0x01bc71 to 0x011711 (0xa55f bytes) 000898 1780327989.071083 read 0x0102ca thru 0x011711 (0x1448 bytes) 000900 1780327989.075323 mapwrite 0x00d606 thru 0x015a11 (0x840c bytes) 000902 1780327989.237429 mapwrite 0x00f079 thru 0x01b106 (0xc08e bytes) 000903 1780327989.401263 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b107): Mirror 4: 0x358908f (stale) Mirror 6: 0x358908f (stale) Mirror 7: 0x358908f (stale) Mirror 9: 0x358908f (stale) Mirror 11: 0x358908f (stale) Mirror 12: 0x358908f (stale) Mirror 13: 0x358908f (stale) Mirror 14: 0x358908f (stale) Mirror 15: 0x358908f (stale) Mirror 16: 0x358908f (stale) Mirror 18: 0x87420af1 (stale) Mirror 19: 0x14896a60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b107): Mirror 4: 0x358908f (stale) Mirror 6: 0x358908f (stale) Mirror 7: 0x358908f (stale) Mirror 9: 0x358908f (stale) Mirror 11: 0x358908f (stale) Mirror 12: 0x358908f (stale) Mirror 13: 0x358908f (stale) Mirror 14: 0x358908f (stale) Mirror 15: 0x358908f (stale) Mirror 16: 0x358908f (stale) Mirror 18: 0x87420af1 (stale) Mirror 19: 0x14896a60 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b107: skipped 000904 1780327990.332542 read 0x01aafe thru 0x01b106 (0x609 bytes) 000905 1780327990.340831 mapwrite 0x02b62f thru 0x0334a6 (0x7e78 bytes) 000906 1780327990.458382 mapwrite 0x02527f thru 0x027084 (0x1e06 bytes) 000911 1780327990.500934 trunc from 0x0334a7 to 0x03bdaa (0x8904 bytes) 000912 1780327990.522905 write 0x019e7b thru 0x01ab78 (0xcfe bytes) 000914 1780327990.546632 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bdab): Mirror 4: 0xeb212aad (stale) Mirror 6: 0xeb212aad (stale) Mirror 7: 0xeb212aad (stale) Mirror 9: 0xeb212aad (stale) Mirror 11: 0xeb212aad (stale) Mirror 12: 0xeb212aad (stale) Mirror 13: 0xeb212aad (stale) Mirror 14: 0xeb212aad (stale) Mirror 15: 0xeb212aad (stale) Mirror 16: 0xeb212aad (stale) Mirror 18: 0xbba7808a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdab): Mirror 4: 0xeb212aad (stale) Mirror 6: 0xeb212aad (stale) Mirror 7: 0xeb212aad (stale) Mirror 9: 0xeb212aad (stale) Mirror 11: 0xeb212aad (stale) Mirror 12: 0xeb212aad (stale) Mirror 13: 0xeb212aad (stale) Mirror 14: 0xeb212aad (stale) Mirror 15: 0xeb212aad (stale) Mirror 16: 0xeb212aad (stale) Mirror 18: 0xbba7808a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdab: skipped 000915 1780327991.521383 trunc from 0x03bdab to 0x036b5e (0x524c bytes) 000917 1780327991.543914 read 0x02cb0c thru 0x0332b6 (0x67ab bytes) 000918 1780327991.561250 mapwrite 0x017f73 thru 0x02182c (0x98ba bytes) 000919 1780327991.731115 trunc from 0x036b5f to 0x010530 (0x2662e bytes) 000920 1780327991.752801 trunc from 0x010531 to 0x01ea8b (0xe55b bytes) 000921 1780327991.770988 mapread 0x0148fd thru 0x01ea8b (0xa18f bytes) 000922 1780327991.791602 read 0x019eba thru 0x01ea8b (0x4bd2 bytes) 000923 1780327991.792501 trunc from 0x01ea8c to 0x018ff0 (0x5a9b bytes) 000925 1780327991.846332 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18ff1): Mirror 4: 0xc4624f0 (stale) Mirror 6: 0xc4624f0 (stale) Mirror 7: 0xc4624f0 (stale) Mirror 9: 0xc4624f0 (stale) Mirror 11: 0xc4624f0 (stale) Mirror 12: 0xc4624f0 (stale) Mirror 14: 0xc4624f0 (stale) Mirror 15: 0xc4624f0 (stale) Mirror 16: 0xc4624f0 (stale) Mirror 18: 0x7bae5d5c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ff1): Mirror 4: 0xc4624f0 (stale) Mirror 6: 0xc4624f0 (stale) Mirror 7: 0xc4624f0 (stale) Mirror 9: 0xc4624f0 (stale) Mirror 11: 0xc4624f0 (stale) Mirror 12: 0xc4624f0 (stale) Mirror 14: 0xc4624f0 (stale) Mirror 15: 0xc4624f0 (stale) Mirror 16: 0xc4624f0 (stale) Mirror 18: 0x7bae5d5c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ff1: skipped 000927 1780327992.710862 read 0x0081bd thru 0x012b30 (0xa974 bytes) 000930 1780327992.726862 MIRROR_RESYNC 11 mirrors 000932 1780327994.607732 mapread 0x001fdb thru 0x00b65e (0x9684 bytes) 000934 1780327994.638887 MIRROR_RESYNC 11 mirrors 000936 1780327995.115006 mapread 0x013da9 thru 0x018ff0 (0x5248 bytes) 000938 1780327995.139114 read 0x009f28 thru 0x016320 (0xc3f9 bytes) 000939 1780327995.139540 write 0x0127e2 thru 0x01e7a5 (0xbfc4 bytes) 000942 1780327995.203091 mapread 0x00ef7e thru 0x012375 (0x33f8 bytes) 000943 1780327995.207856 MIRROR_RESYNC 11 mirrors 000945 1780327996.633258 read 0x01b88b thru 0x01e7a5 (0x2f1b bytes) 000946 1780327996.656398 mapwrite 0x02aee6 thru 0x036cad (0xbdc8 bytes) 000948 1780327996.809143 write 0x023bf3 thru 0x03094e (0xcd5c bytes) 000951 1780327996.827023 trunc from 0x036cae to 0x01f4a3 (0x1780a bytes) 000955 1780327996.850965 read 0x00639f thru 0x00c1ed (0x5e4f bytes) 000956 1780327996.881899 read 0x019440 thru 0x01f4a3 (0x6064 bytes) 000957 1780327996.883268 mapread 0x013a92 thru 0x014913 (0xe82 bytes) 000958 1780327996.885299 mapread 0x00bb7c thru 0x00cd78 (0x11fd bytes) 000961 1780327996.894179 mapwrite 0x036015 thru 0x038faa (0x2f96 bytes) 000962 1780327996.962979 read 0x01412a thru 0x01b95f (0x7836 bytes) 000963 1780327996.963714 mapwrite 0x02a0cd thru 0x031cec (0x7c20 bytes) 000964 1780327997.031115 MIRROR_RESYNC 11 mirrors 000968 1780327999.061466 mapwrite 0x03b9a4 thru 0x03ffff (0x465c bytes) 000969 1780327999.209872 read 0x034734 thru 0x03997a (0x5247 bytes) 000970 1780327999.215818 read 0x0172b7 thru 0x026b0d (0xf857 bytes) 000971 1780327999.226684 mapwrite 0x004996 thru 0x009178 (0x47e3 bytes) 000974 1780327999.277327 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8191ea29 (stale) Mirror 6: 0x8191ea29 (stale) Mirror 7: 0x8191ea29 (stale) Mirror 9: 0x8191ea29 (stale) Mirror 11: 0x8191ea29 (stale) Mirror 12: 0x8191ea29 (stale) Mirror 14: 0x8191ea29 (stale) Mirror 16: 0x8191ea29 (stale) Mirror 18: 0x8191ea29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8191ea29 (stale) Mirror 6: 0x8191ea29 (stale) Mirror 7: 0x8191ea29 (stale) Mirror 9: 0x8191ea29 (stale) Mirror 11: 0x8191ea29 (stale) Mirror 12: 0x8191ea29 (stale) Mirror 14: 0x8191ea29 (stale) Mirror 16: 0x8191ea29 (stale) Mirror 18: 0x8191ea29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000977 1780327999.930030 mapwrite 0x03cbe1 thru 0x03ffff (0x341f bytes) 000981 1780327999.996604 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8191ea29 (stale) Mirror 6: 0x8191ea29 (stale) Mirror 7: 0x8191ea29 (stale) Mirror 9: 0x8191ea29 (stale) Mirror 11: 0x8191ea29 (stale) Mirror 12: 0x8191ea29 (stale) Mirror 14: 0x8191ea29 (stale) Mirror 16: 0x8191ea29 (stale) Mirror 18: 0x8191ea29 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9dc40dd Mirror 19: 0xf9dc40dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000982 1780328000.960112 MIRROR_RESYNC 11 mirrors 000983 1780328002.634548 read 0x031d0e thru 0x03f0eb (0xd3de bytes) 000984 1780328002.648402 mapwrite 0x0284a8 thru 0x033da9 (0xb902 bytes) 000986 1780328002.869745 mapwrite 0x013b49 thru 0x018291 (0x4749 bytes) 000987 1780328002.935991 trunc from 0x040000 to 0x010fec (0x2f013 bytes) 000988 1780328002.963403 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x10fed): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10fed): Mirror 4: 0x92179ab1 (stale) Mirror 6: 0x92179ab1 (stale) Mirror 7: 0x92179ab1 (stale) Mirror 9: 0x92179ab1 (stale) Mirror 11: 0x92179ab1 (stale) Mirror 12: 0x92179ab1 (stale) Mirror 14: 0x92179ab1 (stale) Mirror 16: 0x92179ab1 (stale) Mirror 18: 0x92179ab1 (stale) Mirror 19: 0x92179ab1 (stale) CRC-32 checksum value for chunk [0, 0x10fed): Mirror 1: 0x92179ab1 Mirror 20: 0x92179ab1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10fed: skipped 000991 1780328004.616698 mapwrite 0x02320a thru 0x02a3da (0x71d1 bytes) 000993 1780328004.793712 write 0x01da40 thru 0x027225 (0x97e6 bytes) 000996 1780328004.805143 write 0x002449 thru 0x009ec2 (0x7a7a bytes) 000999 1780328004.820971 trunc from 0x02a3db to 0x01aaff (0xf8db bytes) All operations completed A-OK! PASS 70a (124s) == sanity-flr test 71: check mirror extend parent fid ==== 11:33:30 (1780328010) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 11:33:33 (1780328013) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.260666 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16698 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 11:33:41 (1780328021) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3506048 .. Extending file size to 3882208 .. done Extending file size to 7279648 .. Extending file size to 8413632 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8657184 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8528576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8682112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8999872 .. Extending file size to 8948640 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9099168 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56508 56510 56511 56512 56513 done 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:35:00 (1780328100) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6503584 .. Extending file size to 2376960 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5276576 .. Extending file size to 5544896 .. Extending file size to 6832224 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7949632 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8379680 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5863296 .. Extending file size to 7748608 .. Extending file size to 8625600 .. Extending file size to 8530496 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58150 58151 58153 58155 58156 failed 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:36:22 (1780328182) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201308 s, 10.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.298066 s, 7.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1ea253da (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:36:46 (1780328206) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223413 s, 9.4 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 11:36:56 (1780328216) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } PASS 203 (12s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:37:08 (1780328228) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254148 s, 8.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:37:20 (1780328240) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.304985 s, 6.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:37:31 (1780328251) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420208 s, 5.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:38:04 (1780328284) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104606 s, 10.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:39:04 (1780328344) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19154 s, 10.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:40:05 (1780328405) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. Creating new pool test_204f oleg457-server: Pool lustre.test_204f created Adding targets to pool oleg457-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg457-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.278166 s, 7.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg457-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg457-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg457-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:41:24 (1780328484) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:41:32 (1780328492) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:41:41 (1780328501) oleg457-server: Pool lustre.test_206 created Adding targets to pool oleg457-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg457-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg457-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg457-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg457-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:42:06 (1780328526) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094856 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504273 s, 20.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd2371dbe Mirror 3: 0xd2371dbe PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:42:17 (1780328537) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:42:21 (1780328541) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:42:25 (1780328545) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.106639 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xbb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x790000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x790000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x790000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf60000 Read this (4 bytes): BBBB 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:43:00 (1780328580) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0296839 s, 276 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:43:07 (1780328587) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586632 s, 17.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f0 lmm_fid: [0x200000404:0x1f0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f0 lmm_fid: [0x200000404:0x1f0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834416 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:43:09.000000000 -0400 Modify: 2026-06-01 11:43:09.000000000 -0400 Change: 2026-06-01 11:43:10.000000000 -0400 Birth: 2026-06-01 11:43:08.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:43:15 (1780328595) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:43:18 (1780328598) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.717287 s, 57.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0515705 s, 79.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 11:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 11:43 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:43:26 (1780328606) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:43:33 (1780328613) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1317 sec =========== 11:43:42 (1780328622) === sanity-flr: start cleanup 11:43:43 (1780328623) === === sanity-flr: finish cleanup 11:43:49 (1780328629) ===