-----============= acceptance-small: sanity-flr ============----- Tue Jun 2 02:54:16 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg446-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 02:54:42 (1780383282) === oleg446-client.virtnet: executing check_config_client /mnt/lustre oleg446-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg446-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9dc952651000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9dc952651000.idle_timeout=debug disable quota as required oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 02:55:25 (1780383325) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7583] == sanity-flr test 0a: lfs mirror create with -N option == 02:55:28 (1780383328) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:55:49 (1780383349) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:55:53 (1780383353) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:55:57 (1780383357) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:56:29 (1780383389) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:56:33 (1780383393) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 02:56:36 (1780383396) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9dc952651000.stats=clear osc.lustre-OST0001-osc-ffff9dc952651000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:56:45 (1780383405) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:57:00 (1780383420) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:57:11 (1780383431) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:57:15 (1780383435) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:57:25 (1780383445) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (14s) == sanity-flr test 2: create components from existing files ========================================================== 02:57:39 (1780383459) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:57:50 (1780383470) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:57:55 (1780383475) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:58:05 (1780383485) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:58:17 (1780383497) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:58:24 (1780383504) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.836358 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891756 s, 11.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:58:35 (1780383515) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.219297 s, 33.5 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:58:45 (1780383525) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:58:59 (1780383539) Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg446-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:00:10 (1780383610) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.502784 s, 20.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:00:54 (1780383654) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg446-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:02:25 (1780383745) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.86575 s, 8.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 2 03:02 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.661s user 0m0.000s sys 0m0.198s Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc952651000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m2.322s user 0m0.001s sys 0m0.207s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.190s user 0m0.000s sys 0m0.194s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg446-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc952651000.ost_server_uuid in FULL state after 0 sec PASS 33b (102s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:04:07 (1780383847) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:04:11 (1780383851) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:04:15 (1780383855) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (6s) == sanity-flr test 35: allow to write to mirrored files == 03:04:21 (1780383861) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 03:04:30 (1780383870) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.688606 s, 18.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:04:45 (1780383885) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281859 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604206 s, 17.4 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:04:55 (1780383895) PASS 36c (14s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:05:09 (1780383909) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109406 s, 936 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00472511 s, 217 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00312621 s, 328 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00303969 s, 337 kB/s flv=10 olv=10 PASS 36d (16s) == sanity-flr test 37: mirror I/O API verification ======= 03:05:25 (1780383925) create test files with size 15731316 .. 3c44f92b2a9bfa74c33fa056c179b5b2 - 15902cf02bf2105e411e0ef64e4ca740 - 5c371abc61c00b9b96a5a7d6676dbe19 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg446-client.virtnet /mnt/lustre (opts:) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 03:05:54 (1780383954) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:06:12 (1780383972) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:06:20 (1780383980) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237864 s, 4.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521241 s, 20.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 03:06:31 (1780383991) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111748 s, 18.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.530878 s, 7.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309874 s, 6.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (23s) == sanity-flr test 42: lfs mirror verify ================= 03:06:55 (1780384015) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 03:07:00 (1780384020) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:07:04 (1780384024) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292287 s, 35.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (12s) == sanity-flr test 44a: lfs mirror split check =========== 03:07:16 (1780384036) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:07:21 (1780384041) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:07:33 (1780384053) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31106 s, 33.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg446-client.virtnet /mnt/lustre (opts:) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:07:46 (1780384066) llite.lustre-ffff9dc94873f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.302863 s, 34.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg446-client.virtnet /mnt/lustre (opts:) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9dc951712000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:08:02 (1780384082) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:08:41 (1780384121) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:09:22 (1780384162) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 03:09:25 (1780384165) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.351983 s, 8.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.224396 s, 14.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xaf952325 (nosync,stale) Mirror 2: 0xaf952325 (nosync,stale) ** verify mirror 2 doesn't change original checksum: e4b7dc36dbaa00f7edaf72ce121fa6d4 - mirror 1 checksum: b19aec6e8cc15cb97076da454ab4c3b0 - mirror 2 checksum: e4b7dc36dbaa00f7edaf72ce121fa6d4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 2 03:09:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 2 03:09:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e4b7dc36dbaa00f7edaf72ce121fa6d4 - mirror 1 checksum: b19aec6e8cc15cb97076da454ab4c3b0 - mirror 2 checksum: b19aec6e8cc15cb97076da454ab4c3b0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 2 03:09:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 2 03:09:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:09:39 (1780384179) Failing mds1 on oleg446-server Stopping /mnt/lustre-mds1 (opts:) on oleg446-server 03:09:46 (1780384186) shut down facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server Failover mds1 to oleg446-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 03:10:09 (1780384209) targets are mounted 03:10:09 (1780384209) facet_failover done oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372999 s, 28.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:10:24 (1780384224) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:10:28 (1780384228) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:10:31 (1780384231) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0266806 s, 614 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:10:40 (1780384240) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:10:43 (1780384243) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:10:47 (1780384247) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:11:14 (1780384274) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:11:43 (1780384303) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:12:08 (1780384328) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:12:33 (1780384353) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 03:12:40 (1780384360) mirror create pid 53582 mirror split pid 53584 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53582 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53584 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 03:13:47 (1780384427) Seed set to 4429 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x47a 000001 1780384430.271514 trunc from 00000000 to 0x000479 (0x47a bytes) 000003 1780384430.309947 read 0x0002e2 thru 0x000479 (0x198 bytes) 000004 1780384430.334431 read 0x000093 thru 0x000479 (0x3e7 bytes) 000008 1780384430.342486 write 0x008722 thru 0x0088a5 (0x184 bytes) 000009 1780384430.346925 mapwrite 0x012c6c thru 0x01395f (0xcf4 bytes) truncating to largest ever: 0x6d60 000010 1780384430.512746 trunc from 0x013960 to 0x006d5f (0xcc00 bytes) truncating to largest ever: 0x174f0 000011 1780384430.547212 trunc from 0x006d60 to 0x0174ef (0x10790 bytes) 000012 1780384430.582744 mapread 0x000d77 thru 0x00230d (0x1597 bytes) 000015 1780384430.584924 mapwrite 0x003afe thru 0x010560 (0xca63 bytes) 000017 1780384430.882446 mapread 0x014bd8 thru 0x0174ef (0x2918 bytes) 000020 1780384430.885948 mapwrite 0x01ce03 thru 0x025680 (0x887e bytes) 000021 1780384431.029366 mapread 0x00f364 thru 0x011c2e (0x28cb bytes) 000024 1780384431.053301 read 0x020b0f thru 0x025680 (0x4b72 bytes) truncating to largest ever: 0x38d2c 000025 1780384431.054551 trunc from 0x025681 to 0x038d2b (0x136ab bytes) 000031 1780384431.093292 mapread 0x010292 thru 0x011821 (0x1590 bytes) 000032 1780384431.094512 mapwrite 0x008be7 thru 0x016635 (0xda4f bytes) 000033 1780384431.186270 mapread 0x033aa3 thru 0x038d2b (0x5289 bytes) 000034 1780384431.190368 mapread 0x004bc5 thru 0x0143f2 (0xf82e bytes) 000035 1780384431.199627 read 0x01306b thru 0x018d8c (0x5d22 bytes) 000037 1780384431.203535 mapread 0x00f180 thru 0x013019 (0x3e9a bytes) 000039 1780384431.205442 read 0x00e0ad thru 0x01b2ec (0xd240 bytes) 000041 1780384431.213229 write 0x02605e thru 0x02d0ca (0x706d bytes) 000042 1780384431.217180 read 0x034083 thru 0x038d2b (0x4ca9 bytes) 000047 1780384431.234103 MIRROR_EXTEND to 2 mirrors 000048 1780384431.998523 MIRROR_EXTEND to 3 mirrors 000052 1780384433.072797 trunc from 0x038d2c to 0x03266e (0x66bd bytes) 000057 1780384433.217426 read 0x027271 thru 0x03266e (0xb3fe bytes) 000061 1780384433.229534 trunc from 0x03266f to 0x01a68f (0x17fdf bytes) 000063 1780384433.255602 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1a690): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a690): Mirror 2: 0x40c795db (stale) Mirror 3: 0x40c795db (stale) CRC-32 checksum value for chunk [0, 0x1a690): Mirror 1: 0x40c795db Mirror 4: 0x40c795db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a690: skipped 000064 1780384434.177818 mapwrite 0x02e441 thru 0x0337db (0x539b bytes) 000065 1780384434.397335 read 0x022803 thru 0x025400 (0x2bfe bytes) 000066 1780384434.423072 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x337dc): Mirror 2: 0x885d27c6 (stale) Mirror 3: 0x885d27c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337dc): Mirror 2: 0x885d27c6 (stale) Mirror 3: 0x885d27c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337dc: skipped 000067 1780384434.769323 write 0x007f73 thru 0x00d69f (0x572d bytes) 000068 1780384434.789189 mapwrite 0x022289 thru 0x02cca2 (0xaa1a bytes) truncating to largest ever: 0x3f13f 000069 1780384434.925200 trunc from 0x0337dc to 0x03f13e (0xb963 bytes) 000070 1780384434.949656 read 0x02de09 thru 0x039102 (0xb2fa bytes) 000071 1780384434.961776 read 0x00714b thru 0x007fa3 (0xe59 bytes) 000072 1780384434.963041 read 0x01cb00 thru 0x02a0fa (0xd5fb bytes) 000073 1780384434.964616 read 0x00a323 thru 0x019510 (0xf1ee bytes) 000074 1780384434.979215 read 0x001990 thru 0x005bb6 (0x4227 bytes) 000076 1780384434.984316 write 0x002835 thru 0x0053b2 (0x2b7e bytes) 000077 1780384434.992065 trunc from 0x03f13f to 0x0294f0 (0x15c4e bytes) 000078 1780384435.024386 trunc from 0x0294f1 to 0x01cd70 (0xc780 bytes) 000079 1780384435.051957 write 0x00e27b thru 0x01b69d (0xd423 bytes) 000081 1780384435.056286 write 0x02ae25 thru 0x035e1a (0xaff6 bytes) 000082 1780384435.063861 trunc from 0x035e1b to 0x0356aa (0x770 bytes) 000083 1780384435.092993 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x356ab): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x356ab): Mirror 2: 0xce2d611c (stale) Mirror 3: 0xce2d611c (stale) CRC-32 checksum value for chunk [0, 0x356ab): Mirror 1: 0x26e365d9 Mirror 4: 0x26e365d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x356ab: skipped 000084 1780384436.194999 write 0x004d1b thru 0x007137 (0x241d bytes) 000086 1780384436.310328 trunc from 0x0356ab to 0x03cf4b (0x78a1 bytes) 000087 1780384436.362921 write 0x030f50 thru 0x03769e (0x674f bytes) 000088 1780384436.377856 trunc from 0x03cf4c to 0x0379b3 (0x5598 bytes) 000091 1780384436.455882 write 0x03be2c thru 0x03ffff (0x41d4 bytes) 000092 1780384436.461747 mapread 0x017d0c thru 0x01ce48 (0x513d bytes) 000093 1780384436.494831 trunc from 0x040000 to 0x01d1cb (0x22e34 bytes) 000096 1780384436.554041 read 0x0173fd thru 0x01d1cb (0x5dcf bytes) 000097 1780384436.555492 trunc from 0x01d1cc to 0x022c13 (0x5a48 bytes) 000098 1780384436.589785 write 0x035adb thru 0x036088 (0x5ae bytes) 000099 1780384436.600576 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36089): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36089): Mirror 2: 0x3af92ca7 (stale) Mirror 3: 0x3af92ca7 (stale) Mirror 4: 0x714825ce (stale) CRC-32 checksum value for chunk [0, 0x36089): Mirror 1: 0x56ac056b Mirror 5: 0x56ac056b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36089: skipped 000100 1780384437.680903 trunc from 0x036089 to 0x020bd5 (0x154b3 bytes) 000101 1780384437.857150 trunc from 0x020bd6 to 0x008e63 (0x17d72 bytes) 000104 1780384437.885864 mapread 0x008373 thru 0x008e63 (0xaf1 bytes) 000105 1780384437.894401 mapwrite 0x018d1e thru 0x01af1a (0x21fd bytes) 000106 1780384437.975494 mapread 0x0140c1 thru 0x01af1a (0x6e5a bytes) 000107 1780384437.976711 mapwrite 0x0399e5 thru 0x03ffff (0x661b bytes) 000108 1780384438.059286 write 0x028dc8 thru 0x02ea87 (0x5cc0 bytes) 000110 1780384438.075751 mapwrite 0x0083e3 thru 0x014e6e (0xca8c bytes) 000111 1780384438.192775 read 0x02d6dc thru 0x03d4f9 (0xfe1e bytes) 000112 1780384438.207183 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x55d097bb (stale) Mirror 4: 0x8353756a (stale) Mirror 5: 0x2874b590 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x55d097bb (stale) Mirror 4: 0x8353756a (stale) Mirror 5: 0x2874b590 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000113 1780384438.627076 mapread 0x00ee09 thru 0x00ff96 (0x118e bytes) 000117 1780384438.645352 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8353756a (stale) Mirror 5: 0x2874b590 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8353756a (stale) Mirror 5: 0x2874b590 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000119 1780384439.019838 trunc from 0x040000 to 0x002175 (0x3de8a bytes) 000120 1780384439.042223 mapread 0x000c66 thru 0x002175 (0x1510 bytes) 000121 1780384439.068036 mapwrite 0x0215cd thru 0x02fb9b (0xe5cf bytes) 000123 1780384439.198432 write 0x032429 thru 0x033996 (0x156e bytes) 000124 1780384439.199541 read 0x022a4e thru 0x0324ce (0xfa81 bytes) 000125 1780384439.219723 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33997): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33997): Mirror 4: 0xa92d8406 (stale) Mirror 5: 0xd4abdb0 (stale) CRC-32 checksum value for chunk [0, 0x33997): Mirror 1: 0x4a9ebf90 Mirror 6: 0x4a9ebf90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33997: skipped 000126 1780384440.112886 trunc from 0x033997 to 0x02ac99 (0x8cfd bytes) 000127 1780384440.369402 mapwrite 0x03bb65 thru 0x03ffff (0x449b bytes) 000128 1780384440.512148 mapread 0x0379e2 thru 0x03e9e1 (0x7000 bytes) 000129 1780384440.517220 trunc from 0x040000 to 0x0005fb (0x3fa04 bytes) 000132 1780384440.543320 write 0x005072 thru 0x00a011 (0x4fa0 bytes) 000133 1780384440.546544 trunc from 0x00a012 to 0x010e90 (0x6e7f bytes) 000134 1780384440.580577 trunc from 0x010e91 to 0x026f5b (0x160cb bytes) 000135 1780384440.602037 write 0x01b92b thru 0x025b4c (0xa222 bytes) 000139 1780384440.633209 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26f5c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26f5c): Mirror 4: 0xaaaa85a (stale) Mirror 5: 0x4206eebb (stale) Mirror 6: 0xc8cc4e5b (stale) CRC-32 checksum value for chunk [0, 0x26f5c): Mirror 1: 0x1e19940e Mirror 7: 0x1e19940e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26f5c: skipped 000140 1780384441.726482 MIRROR_RESYNC 5 mirrors 000141 1780384443.069401 read 0x0245b0 thru 0x026f5b (0x29ac bytes) 000142 1780384443.091059 MIRROR_RESYNC 5 mirrors 000144 1780384443.639885 mapread 0x024c7d thru 0x026f5b (0x22df bytes) 000145 1780384443.640564 trunc from 0x026f5c to 0x035e3b (0xeee0 bytes) 000146 1780384443.778482 mapwrite 0x0222ec thru 0x02d6db (0xb3f0 bytes) 000147 1780384443.965228 mapread 0x010a2b thru 0x0162a3 (0x5879 bytes) 000148 1780384444.004268 write 0x0347a3 thru 0x03ffff (0xb85d bytes) 000150 1780384444.025036 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x48b54e1d (stale) Mirror 5: 0x48b54e1d (stale) Mirror 6: 0x48b54e1d (stale) Mirror 7: 0x48b54e1d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x305d7e73 Mirror 8: 0x305d7e73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000152 1780384445.178908 trunc from 0x040000 to 0x0184f3 (0x27b0c bytes) 000153 1780384445.311491 write 0x02882f thru 0x032dbe (0xa590 bytes) 000154 1780384445.317395 mapwrite 0x01aef0 thru 0x023233 (0x8344 bytes) 000156 1780384445.429529 mapwrite 0x003af5 thru 0x00d2cf (0x97db bytes) 000158 1780384445.556723 mapread 0x0266e6 thru 0x032dbe (0xc6d9 bytes) 000159 1780384445.570535 read 0x002093 thru 0x00f134 (0xd0a2 bytes) 000163 1780384445.579338 write 0x00febe thru 0x013d2b (0x3e6e bytes) 000166 1780384445.593177 mapwrite 0x014437 thru 0x0195ee (0x51b8 bytes) 000168 1780384445.734216 read 0x022eda thru 0x023989 (0xab0 bytes) 000169 1780384445.757183 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32dbf): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32dbf): Mirror 4: 0x2e5c8953 (stale) Mirror 5: 0x2e5c8953 (stale) Mirror 6: 0x2e5c8953 (stale) Mirror 7: 0x2e5c8953 (stale) Mirror 8: 0x7ccc9a2e (stale) CRC-32 checksum value for chunk [0, 0x32dbf): Mirror 1: 0x6795dea8 Mirror 9: 0x6795dea8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32dbf: skipped 000171 1780384446.952314 mapwrite 0x00ca8a thru 0x018b3d (0xc0b4 bytes) 000173 1780384447.289388 read 0x01b6f9 thru 0x01dca2 (0x25aa bytes) 000174 1780384447.291377 trunc from 0x032dbf to 0x0268bf (0xc4ff bytes) 000175 1780384447.321236 write 0x031a41 thru 0x03748d (0x5a4d bytes) 000176 1780384447.322841 mapwrite 0x038a5e thru 0x03ffff (0x75a2 bytes) 000177 1780384447.441765 write 0x002e89 thru 0x00a041 (0x71b9 bytes) 000178 1780384447.452966 mapwrite 0x01cd71 thru 0x02b4da (0xe76a bytes) 000179 1780384447.601018 MIRROR_RESYNC 7 mirrors 000180 1780384448.991699 trunc from 0x040000 to 0x0105f8 (0x2fa07 bytes) 000183 1780384449.112840 write 0x03e25f thru 0x03ffff (0x1da1 bytes) 000186 1780384449.116932 mapwrite 0x02a8cc thru 0x033220 (0x8955 bytes) 000187 1780384449.200729 mapread 0x03be52 thru 0x03ffff (0x41ae bytes) 000188 1780384449.219681 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x73ce9c3f (stale) Mirror 5: 0x73ce9c3f (stale) Mirror 6: 0x73ce9c3f (stale) Mirror 7: 0x73ce9c3f (stale) Mirror 9: 0x73ce9c3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x73ce9c3f (stale) Mirror 5: 0x73ce9c3f (stale) Mirror 6: 0x73ce9c3f (stale) Mirror 7: 0x73ce9c3f (stale) Mirror 9: 0x73ce9c3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1780384449.742726 mapwrite 0x028d55 thru 0x033cd0 (0xaf7c bytes) 000190 1780384449.869395 read 0x0223a7 thru 0x030db6 (0xea10 bytes) 000191 1780384449.875796 write 0x036702 thru 0x03e7e3 (0x80e2 bytes) 000192 1780384449.883705 read 0x01cb7e thru 0x027dd3 (0xb256 bytes) 000193 1780384449.910926 mapwrite 0x03d78d thru 0x03ffff (0x2873 bytes) 000194 1780384449.965592 trunc from 0x040000 to 0x01dc96 (0x22369 bytes) 000196 1780384449.994842 mapwrite 0x03b6bc thru 0x03ffff (0x4944 bytes) 000198 1780384450.141385 write 0x026567 thru 0x02eb56 (0x85f0 bytes) 000199 1780384450.158823 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x73ce9c3f (stale) Mirror 5: 0x73ce9c3f (stale) Mirror 6: 0x73ce9c3f (stale) Mirror 7: 0x73ce9c3f (stale) Mirror 9: 0x73ce9c3f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe1463fe5 Mirror 10: 0xe1463fe5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000200 1780384451.149190 write 0x0117e4 thru 0x019d40 (0x855d bytes) 000202 1780384451.274423 read 0x016977 thru 0x01d61a (0x6ca4 bytes) 000203 1780384451.282539 mapread 0x00d0d0 thru 0x01b6d1 (0xe602 bytes) 000204 1780384451.293286 write 0x01bdcf thru 0x022002 (0x6234 bytes) 000206 1780384451.301563 read 0x0361ff thru 0x03dc57 (0x7a59 bytes) 000207 1780384451.313881 write 0x006eec thru 0x010000 (0x9115 bytes) 000209 1780384451.322377 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x73ce9c3f (stale) Mirror 5: 0x73ce9c3f (stale) Mirror 6: 0x73ce9c3f (stale) Mirror 7: 0x73ce9c3f (stale) Mirror 9: 0x73ce9c3f (stale) Mirror 10: 0xe1463fe5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x223862ab Mirror 11: 0x223862ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000211 1780384452.206846 read 0x0221ad thru 0x02c701 (0xa555 bytes) 000212 1780384452.219066 write 0x02d03d thru 0x0310b4 (0x4078 bytes) 000213 1780384452.321122 mapread 0x017999 thru 0x0215c8 (0x9c30 bytes) 000214 1780384452.358769 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x73ce9c3f (stale) Mirror 5: 0x73ce9c3f (stale) Mirror 6: 0x73ce9c3f (stale) Mirror 7: 0x73ce9c3f (stale) Mirror 9: 0x73ce9c3f (stale) Mirror 10: 0xe1463fe5 (stale) Mirror 11: 0x223862ab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf153de89 Mirror 12: 0xf153de89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000215 1780384453.361337 mapread 0x03dffc thru 0x03ffff (0x2004 bytes) 000216 1780384453.390016 MIRROR_RESYNC 9 mirrors 000218 1780384454.989591 mapread 0x006db9 thru 0x00a246 (0x348e bytes) 000219 1780384455.012578 mapwrite 0x03588f thru 0x03ffff (0xa771 bytes) 000220 1780384455.264295 MIRROR_RESYNC 9 mirrors 000221 1780384457.025533 mapread 0x01264f thru 0x01e6ee (0xc0a0 bytes) 000222 1780384457.103756 MIRROR_EXTEND to 10 mirrors 000229 1780384458.258073 trunc from 0x040000 to 0x006ea9 (0x39156 bytes) 000231 1780384458.383237 read 0x001429 thru 0x006ea9 (0x5a81 bytes) 000232 1780384458.401241 write 0x02bbaf thru 0x03a49f (0xe8f1 bytes) 000233 1780384458.403033 read 0x01cc53 thru 0x026c73 (0xa021 bytes) 000235 1780384458.422645 read 0x02e6ce thru 0x03a49f (0xbdd2 bytes) 000236 1780384458.423556 mapwrite 0x0013bf thru 0x006328 (0x4f6a bytes) 000237 1780384458.508696 read 0x01fadd thru 0x024bf9 (0x511d bytes) 000238 1780384458.509412 read 0x011cc3 thru 0x01b0cf (0x940d bytes) 000239 1780384458.510103 write 0x0389ae thru 0x03ffff (0x7652 bytes) 000241 1780384458.513706 write 0x03688f thru 0x03d1f0 (0x6962 bytes) 000243 1780384458.516567 mapread 0x03a238 thru 0x03ffff (0x5dc8 bytes) 000246 1780384458.519722 trunc from 0x040000 to 0x03e200 (0x1dff bytes) 000248 1780384458.567022 write 0x017e1c thru 0x01883e (0xa23 bytes) 000249 1780384458.568949 mapwrite 0x01f729 thru 0x02d30a (0xdbe2 bytes) 000251 1780384458.701595 read 0x020dc2 thru 0x028956 (0x7b95 bytes) 000252 1780384458.705643 trunc from 0x03e201 to 0x02cc6f (0x11591 bytes) 000253 1780384458.733778 read 0x028e92 thru 0x02cc6f (0x3dde bytes) 000254 1780384458.734659 write 0x02a808 thru 0x02ccba (0x24b3 bytes) 000256 1780384458.736846 mapwrite 0x024a19 thru 0x029daa (0x5392 bytes) 000258 1780384458.863320 mapwrite 0x02dabb thru 0x03433e (0x6884 bytes) 000259 1780384459.023050 read 0x02156a thru 0x025c07 (0x469e bytes) 000261 1780384459.024595 mapread 0x01d93b thru 0x026cbb (0x9381 bytes) 000262 1780384459.032467 mapwrite 0x012d79 thru 0x022948 (0xfbd0 bytes) 000263 1780384459.194047 mapread 0x00de99 thru 0x00e273 (0x3db bytes) 000264 1780384459.199119 mapread 0x0226ab thru 0x028bf9 (0x654f bytes) 000268 1780384459.202343 trunc from 0x03433f to 0x03c5df (0x82a1 bytes) 000271 1780384459.223760 write 0x028e5f thru 0x02f305 (0x64a7 bytes) 000273 1780384459.230657 mapread 0x0033fa thru 0x00a08b (0x6c92 bytes) 000275 1780384459.253565 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c5e0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5e0): Mirror 4: 0x1e811ac8 (stale) Mirror 5: 0x1e811ac8 (stale) Mirror 6: 0x1e811ac8 (stale) Mirror 7: 0x1e811ac8 (stale) Mirror 9: 0x1e811ac8 (stale) Mirror 10: 0x1e811ac8 (stale) Mirror 11: 0x1e811ac8 (stale) Mirror 12: 0x1e811ac8 (stale) Mirror 13: 0x1e811ac8 (stale) CRC-32 checksum value for chunk [0, 0x3c5e0): Mirror 1: 0x3e10f9c2 Mirror 14: 0x3e10f9c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5e0: skipped 000276 1780384460.792487 MIRROR_RESYNC 11 mirrors 000279 1780384462.691725 trunc from 0x03c5e0 to 0x026a55 (0x15b8a bytes) 000280 1780384462.885447 mapwrite 0x032e1a thru 0x038afd (0x5ce4 bytes) 000281 1780384463.073119 write 0x02f69c thru 0x032a43 (0x33a8 bytes) 000283 1780384463.077459 write 0x00b68f thru 0x011f76 (0x68e8 bytes) 000284 1780384463.091255 write 0x02e43b thru 0x030c72 (0x2838 bytes) 000286 1780384463.096178 mapwrite 0x03cb23 thru 0x03ffff (0x34dd bytes) 000287 1780384463.179181 mapread 0x017710 thru 0x01d948 (0x6239 bytes) 000288 1780384463.195174 write 0x03dd73 thru 0x03ffff (0x228d bytes) 000289 1780384463.204011 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8575d493 (stale) Mirror 5: 0x8575d493 (stale) Mirror 6: 0x8575d493 (stale) Mirror 7: 0x8575d493 (stale) Mirror 9: 0x8575d493 (stale) Mirror 10: 0x8575d493 (stale) Mirror 11: 0x8575d493 (stale) Mirror 12: 0x8575d493 (stale) Mirror 13: 0x8575d493 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8575d493 (stale) Mirror 5: 0x8575d493 (stale) Mirror 6: 0x8575d493 (stale) Mirror 7: 0x8575d493 (stale) Mirror 9: 0x8575d493 (stale) Mirror 10: 0x8575d493 (stale) Mirror 11: 0x8575d493 (stale) Mirror 12: 0x8575d493 (stale) Mirror 13: 0x8575d493 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000290 1780384463.847321 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8575d493 (stale) Mirror 5: 0x8575d493 (stale) Mirror 6: 0x8575d493 (stale) Mirror 7: 0x8575d493 (stale) Mirror 9: 0x8575d493 (stale) Mirror 10: 0x8575d493 (stale) Mirror 11: 0x8575d493 (stale) Mirror 13: 0x8575d493 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8575d493 (stale) Mirror 5: 0x8575d493 (stale) Mirror 6: 0x8575d493 (stale) Mirror 7: 0x8575d493 (stale) Mirror 9: 0x8575d493 (stale) Mirror 10: 0x8575d493 (stale) Mirror 11: 0x8575d493 (stale) Mirror 13: 0x8575d493 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000291 1780384464.469476 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8575d493 (stale) Mirror 5: 0x8575d493 (stale) Mirror 6: 0x8575d493 (stale) Mirror 7: 0x8575d493 (stale) Mirror 9: 0x8575d493 (stale) Mirror 11: 0x8575d493 (stale) Mirror 13: 0x8575d493 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8575d493 (stale) Mirror 5: 0x8575d493 (stale) Mirror 6: 0x8575d493 (stale) Mirror 7: 0x8575d493 (stale) Mirror 9: 0x8575d493 (stale) Mirror 11: 0x8575d493 (stale) Mirror 13: 0x8575d493 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1780384465.063670 mapread 0x02bb0f thru 0x030ab1 (0x4fa3 bytes) 000294 1780384465.090375 mapwrite 0x0150f2 thru 0x016387 (0x1296 bytes) 000295 1780384465.141016 trunc from 0x040000 to 0x00e7a2 (0x3185d bytes) 000296 1780384465.157674 mapwrite 0x01f579 thru 0x022dd0 (0x3858 bytes) 000297 1780384465.244629 trunc from 0x022dd1 to 0x01b941 (0x748f bytes) 000298 1780384465.267797 read 0x000ab5 thru 0x00dd51 (0xd29d bytes) 000300 1780384465.278622 write 0x00cd35 thru 0x013b4b (0x6e17 bytes) 000301 1780384465.284201 mapwrite 0x0036aa thru 0x0117eb (0xe142 bytes) 000303 1780384465.344680 mapread 0x00f2a5 thru 0x01a853 (0xb5af bytes) 000305 1780384465.354111 mapwrite 0x02c8f5 thru 0x02e2f6 (0x1a02 bytes) 000306 1780384465.441902 mapwrite 0x02b216 thru 0x0326c1 (0x74ac bytes) 000308 1780384465.531228 write 0x016d7d thru 0x020264 (0x94e8 bytes) 000311 1780384465.543167 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326c2): Mirror 4: 0x1014ea8d (stale) Mirror 5: 0x1014ea8d (stale) Mirror 6: 0x1014ea8d (stale) Mirror 7: 0x1014ea8d (stale) Mirror 11: 0x1014ea8d (stale) Mirror 13: 0x1014ea8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326c2): Mirror 4: 0x1014ea8d (stale) Mirror 5: 0x1014ea8d (stale) Mirror 6: 0x1014ea8d (stale) Mirror 7: 0x1014ea8d (stale) Mirror 11: 0x1014ea8d (stale) Mirror 13: 0x1014ea8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326c2: skipped 000312 1780384466.057405 MIRROR_RESYNC 7 mirrors 000314 1780384467.401148 MIRROR_RESYNC 7 mirrors 000317 1780384467.977388 mapread 0x01d687 thru 0x022b64 (0x54de bytes) 000318 1780384468.006843 mapwrite 0x02c6b1 thru 0x02fdd6 (0x3726 bytes) 000319 1780384468.198349 MIRROR_RESYNC 7 mirrors 000321 1780384469.476455 mapread 0x02efde thru 0x0326c1 (0x36e4 bytes) 000322 1780384469.507318 write 0x0233ab thru 0x025e89 (0x2adf bytes) 000323 1780384469.611369 mapwrite 0x002e5a thru 0x00388f (0xa36 bytes) 000324 1780384469.682323 read 0x020657 thru 0x0302df (0xfc89 bytes) 000325 1780384469.693024 trunc from 0x0326c2 to 0x02a265 (0x845c bytes) 000326 1780384469.719168 mapwrite 0x02d4b4 thru 0x02f1e0 (0x1d2d bytes) 000327 1780384469.825689 write 0x03b566 thru 0x03ffff (0x4a9a bytes) 000328 1780384469.836207 MIRROR_RESYNC 7 mirrors 000329 1780384471.076414 trunc from 0x040000 to 0x03196e (0xe691 bytes) 000330 1780384471.167277 trunc from 0x03196f to 0x01113b (0x20833 bytes) 000331 1780384471.213881 write 0x0163b9 thru 0x019e3a (0x3a82 bytes) 000332 1780384471.216447 trunc from 0x019e3b to 0x021783 (0x7949 bytes) 000333 1780384471.242846 write 0x02a6cc thru 0x02df3f (0x3874 bytes) 000334 1780384471.246263 write 0x01f11a thru 0x028c88 (0x9b6f bytes) 000337 1780384471.267609 mapread 0x0168ed thru 0x019922 (0x3036 bytes) 000340 1780384471.285260 MIRROR_RESYNC 7 mirrors 000341 1780384472.563035 mapwrite 0x008040 thru 0x00c6a3 (0x4664 bytes) 000342 1780384472.762270 mapread 0x02a4ab thru 0x02df3f (0x3a95 bytes) 000343 1780384472.776426 write 0x001a64 thru 0x00eab1 (0xd04e bytes) 000344 1780384472.787304 mapwrite 0x01c3e1 thru 0x02738c (0xafac bytes) 000345 1780384472.846724 MIRROR_RESYNC 7 mirrors 000346 1780384474.278613 trunc from 0x02df40 to 0x00019a (0x2dda5 bytes) 000347 1780384474.364847 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19b): Mirror 5: 0xd796c9d4 (stale) Mirror 6: 0xd796c9d4 (stale) Mirror 7: 0xd796c9d4 (stale) Mirror 11: 0xd796c9d4 (stale) Mirror 13: 0xd796c9d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b): Mirror 5: 0xd796c9d4 (stale) Mirror 6: 0xd796c9d4 (stale) Mirror 7: 0xd796c9d4 (stale) Mirror 11: 0xd796c9d4 (stale) Mirror 13: 0xd796c9d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b: skipped 000348 1780384474.751377 trunc from 0x00019b to 0x003967 (0x37cd bytes) 000350 1780384474.787408 trunc from 0x003968 to 0x030854 (0x2ceed bytes) 000354 1780384474.821343 mapread 0x007532 thru 0x007cfd (0x7cc bytes) 000355 1780384474.831727 write 0x018d89 thru 0x022a9d (0x9d15 bytes) 000357 1780384474.849959 trunc from 0x030855 to 0x0325a2 (0x1d4e bytes) 000359 1780384474.876043 mapread 0x00e471 thru 0x01c15a (0xdcea bytes) 000360 1780384474.899756 read 0x027617 thru 0x0325a2 (0xaf8c bytes) 000361 1780384474.909625 write 0x01c25a thru 0x021235 (0x4fdc bytes) 000362 1780384474.911963 read 0x014758 thru 0x02221c (0xdac5 bytes) 000363 1780384474.913043 read 0x01359a thru 0x02140b (0xde72 bytes) 000364 1780384474.914431 trunc from 0x0325a3 to 0x039e20 (0x787e bytes) 000365 1780384474.932652 mapread 0x02730f thru 0x032cc4 (0xb9b6 bytes) 000368 1780384474.938061 write 0x007ccf thru 0x00d74e (0x5a80 bytes) 000369 1780384474.941036 mapwrite 0x02fdd7 thru 0x03a080 (0xa2aa bytes) 000370 1780384475.094881 mapwrite 0x02c390 thru 0x037b32 (0xb7a3 bytes) 000371 1780384475.182205 mapwrite 0x00a920 thru 0x01268c (0x7d6d bytes) 000373 1780384475.241665 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a081): Mirror 5: 0x77b66652 (stale) Mirror 6: 0x77b66652 (stale) Mirror 7: 0x77b66652 (stale) Mirror 13: 0x77b66652 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a081): Mirror 5: 0x77b66652 (stale) Mirror 6: 0x77b66652 (stale) Mirror 7: 0x77b66652 (stale) Mirror 13: 0x77b66652 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a081: skipped 000375 1780384475.724146 mapread 0x015884 thru 0x016213 (0x990 bytes) 000376 1780384475.741690 MIRROR_RESYNC 5 mirrors 000377 1780384476.919481 read 0x036785 thru 0x03a080 (0x38fc bytes) 000380 1780384476.934629 write 0x03b5b3 thru 0x03ffff (0x4a4d bytes) 000381 1780384477.023921 MIRROR_RESYNC 5 mirrors 000382 1780384478.154929 read 0x032ae6 thru 0x03eceb (0xc206 bytes) 000383 1780384478.169341 write 0x03f437 thru 0x03ffff (0xbc9 bytes) 000384 1780384478.257502 read 0x0061bd thru 0x00a470 (0x42b4 bytes) 000385 1780384478.277215 write 0x00cb0d thru 0x00d144 (0x638 bytes) 000389 1780384478.285710 write 0x027c69 thru 0x030e66 (0x91fe bytes) 000391 1780384478.302779 MIRROR_RESYNC 5 mirrors 000392 1780384479.348314 read 0x0114a1 thru 0x013895 (0x23f5 bytes) 000395 1780384479.368904 mapread 0x0348ae thru 0x03a308 (0x5a5b bytes) 000399 1780384479.396977 mapread 0x004105 thru 0x0072b3 (0x31af bytes) 000400 1780384479.417331 MIRROR_RESYNC 5 mirrors 000403 1780384479.952287 read 0x00ddb9 thru 0x0164f4 (0x873c bytes) 000406 1780384479.967342 mapread 0x0174fb thru 0x01f875 (0x837b bytes) 000407 1780384479.994065 mapwrite 0x020d99 thru 0x026787 (0x59ef bytes) 000408 1780384480.186178 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x35bffdfb (stale) Mirror 6: 0x35bffdfb (stale) Mirror 7: 0x35bffdfb (stale) Mirror 13: 0x35bffdfb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdc63f108 Mirror 14: 0xdc63f108 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000409 1780384481.184855 read 0x0127e7 thru 0x01c5a3 (0x9dbd bytes) 000410 1780384481.202976 write 0x008a3a thru 0x00c606 (0x3bcd bytes) 000411 1780384481.305197 mapread 0x00c9ce thru 0x01b362 (0xe995 bytes) 000412 1780384481.341910 trunc from 0x040000 to 0x03c30b (0x3cf4 bytes) 000413 1780384481.360658 write 0x00a9f3 thru 0x00ab9e (0x1ac bytes) 000416 1780384481.363430 write 0x03e3b4 thru 0x03ffff (0x1c4c bytes) 000417 1780384481.365367 mapread 0x00546f thru 0x00d4e6 (0x8078 bytes) 000418 1780384481.372512 trunc from 0x040000 to 0x025106 (0x1aef9 bytes) 000421 1780384481.395378 write 0x03c89c thru 0x03ffff (0x3764 bytes) 000422 1780384481.400750 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x35bffdfb (stale) Mirror 6: 0x35bffdfb (stale) Mirror 7: 0x35bffdfb (stale) Mirror 13: 0x35bffdfb (stale) Mirror 14: 0xdc63f108 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3c56634 Mirror 15: 0xc3c56634 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000423 1780384482.572526 read 0x025abe thru 0x02edd9 (0x931c bytes) 000424 1780384482.591244 mapread 0x02debd thru 0x033bbf (0x5d03 bytes) 000425 1780384482.638494 trunc from 0x040000 to 0x00450b (0x3baf4 bytes) 000427 1780384482.750254 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x450c): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x450c): Mirror 5: 0xf677f527 (stale) Mirror 6: 0xf677f527 (stale) Mirror 7: 0xf677f527 (stale) Mirror 13: 0xf677f527 (stale) Mirror 14: 0xf677f527 (stale) Mirror 15: 0xf677f527 (stale) CRC-32 checksum value for chunk [0, 0x450c): Mirror 1: 0xf677f527 Mirror 16: 0xf677f527 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x450c: skipped 000428 1780384483.784686 mapread 0x00041a thru 0x00450b (0x40f2 bytes) 000431 1780384483.869171 mapread 0x003b2e thru 0x00450b (0x9de bytes) 000432 1780384483.882711 write 0x0158bf thru 0x01be96 (0x65d8 bytes) 000435 1780384483.966450 mapwrite 0x00b004 thru 0x00b286 (0x283 bytes) 000438 1780384484.047031 mapwrite 0x012cbe thru 0x016990 (0x3cd3 bytes) 000441 1780384484.116159 mapread 0x0175f5 thru 0x01be96 (0x48a2 bytes) 000444 1780384484.122105 mapwrite 0x036564 thru 0x03ffff (0x9a9c bytes) 000445 1780384484.220017 trunc from 0x040000 to 0x015347 (0x2acb8 bytes) 000446 1780384484.252563 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x15348): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15348): Mirror 5: 0xc73a8697 (stale) Mirror 6: 0xc73a8697 (stale) Mirror 7: 0xc73a8697 (stale) Mirror 13: 0xc73a8697 (stale) Mirror 14: 0xc73a8697 (stale) Mirror 15: 0x91c5272f (stale) Mirror 16: 0x71aa57af (stale) CRC-32 checksum value for chunk [0, 0x15348): Mirror 1: 0xc0a0f64d Mirror 17: 0xc0a0f64d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15348: skipped 000447 1780384485.444257 trunc from 0x015348 to 0x02ed43 (0x199fc bytes) 000451 1780384485.589864 trunc from 0x02ed44 to 0x030272 (0x152f bytes) 000453 1780384485.616411 trunc from 0x030273 to 0x01554f (0x1ad23 bytes) 000454 1780384485.649642 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15550): Mirror 5: 0x8bc9b7c5 (stale) Mirror 6: 0x8bc9b7c5 (stale) Mirror 13: 0x8bc9b7c5 (stale) Mirror 14: 0x8bc9b7c5 (stale) Mirror 15: 0x6bec53bb (stale) Mirror 16: 0x3e0ab152 (stale) Mirror 17: 0x4d4cb2cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15550): Mirror 5: 0x8bc9b7c5 (stale) Mirror 6: 0x8bc9b7c5 (stale) Mirror 13: 0x8bc9b7c5 (stale) Mirror 14: 0x8bc9b7c5 (stale) Mirror 15: 0x6bec53bb (stale) Mirror 16: 0x3e0ab152 (stale) Mirror 17: 0x4d4cb2cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15550: skipped 000455 1780384486.122829 read 0x001116 thru 0x004353 (0x323e bytes) 000456 1780384486.140789 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15550): Mirror 6: 0x8bc9b7c5 (stale) Mirror 13: 0x8bc9b7c5 (stale) Mirror 14: 0x8bc9b7c5 (stale) Mirror 15: 0x6bec53bb (stale) Mirror 16: 0x3e0ab152 (stale) Mirror 17: 0x4d4cb2cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15550): Mirror 6: 0x8bc9b7c5 (stale) Mirror 13: 0x8bc9b7c5 (stale) Mirror 14: 0x8bc9b7c5 (stale) Mirror 15: 0x6bec53bb (stale) Mirror 16: 0x3e0ab152 (stale) Mirror 17: 0x4d4cb2cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15550: skipped 000459 1780384486.711343 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15550): Mirror 6: 0x8bc9b7c5 (stale) Mirror 13: 0x8bc9b7c5 (stale) Mirror 14: 0x8bc9b7c5 (stale) Mirror 16: 0x3e0ab152 (stale) Mirror 17: 0x4d4cb2cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15550): Mirror 6: 0x8bc9b7c5 (stale) Mirror 13: 0x8bc9b7c5 (stale) Mirror 14: 0x8bc9b7c5 (stale) Mirror 16: 0x3e0ab152 (stale) Mirror 17: 0x4d4cb2cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15550: skipped 000460 1780384487.100295 read 0x00365f thru 0x00ced0 (0x9872 bytes) 000461 1780384487.112645 mapwrite 0x026d38 thru 0x0328ca (0xbb93 bytes) 000463 1780384487.244331 write 0x01eec6 thru 0x01ff05 (0x1040 bytes) 000467 1780384487.256041 write 0x009b54 thru 0x00a78c (0xc39 bytes) 000468 1780384487.261618 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x328cb): Mirror 13: 0x48cf81b6 (stale) Mirror 14: 0x71030174 (stale) Mirror 16: 0xaab06cd8 (stale) Mirror 17: 0xa4100b00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x328cb): Mirror 13: 0x48cf81b6 (stale) Mirror 14: 0x71030174 (stale) Mirror 16: 0xaab06cd8 (stale) Mirror 17: 0xa4100b00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x328cb: skipped 000470 1780384487.634285 write 0x0382f7 thru 0x03ffff (0x7d09 bytes) 000474 1780384487.641745 write 0x003b0e thru 0x00f3a5 (0xb898 bytes) 000480 1780384487.664066 trunc from 0x040000 to 0x00552f (0x3aad0 bytes) 000482 1780384487.697394 read 0x00252d thru 0x00552f (0x3003 bytes) 000483 1780384487.725875 MIRROR_RESYNC 5 mirrors 000485 1780384489.016999 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000488 1780384489.398492 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000489 1780384489.766898 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000491 1780384490.103793 write 0x01e0a7 thru 0x027935 (0x988f bytes) 000492 1780384490.193716 mapread 0x015d6e thru 0x0172a8 (0x153b bytes) 000493 1780384490.220057 mapwrite 0x01e44c thru 0x023ace (0x5683 bytes) 000495 1780384490.281534 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27936): Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27936): Mirror 17: 0xaed34cd1 (stale) CRC-32 checksum value for chunk [0, 0x27936): Mirror 1: 0xdda8238c Mirror 18: 0xdda8238c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27936: skipped 000496 1780384490.963732 read 0x014ca4 thru 0x023f14 (0xf271 bytes) 000497 1780384490.976811 read 0x01427d thru 0x01cdfb (0x8b7f bytes) 000499 1780384490.989802 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27936): Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27936): Mirror 17: 0xaed34cd1 (stale) CRC-32 checksum value for chunk [0, 0x27936): Mirror 1: 0xdda8238c Mirror 18: 0xdda8238c Mirror 19: 0xdda8238c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27936: skipped 000502 1780384491.810535 mapread 0x021656 thru 0x022edb (0x1886 bytes) 000504 1780384491.826180 mapwrite 0x00a1aa thru 0x00fd44 (0x5b9b bytes) 000505 1780384492.059358 write 0x00de05 thru 0x01983c (0xba38 bytes) 000506 1780384492.066563 read 0x009e32 thru 0x012863 (0x8a32 bytes) 000512 1780384492.077740 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27936): Mirror 17: 0xaed34cd1 (stale) Mirror 18: 0xdda8238c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27936): Mirror 17: 0xaed34cd1 (stale) Mirror 18: 0xdda8238c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27936: skipped 000519 1780384492.462599 trunc from 0x027936 to 0x023c51 (0x3ce4 bytes) 000520 1780384492.502539 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x23c52): Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23c52): Mirror 17: 0x23c498fb (stale) Mirror 18: 0xf4ce2133 (stale) CRC-32 checksum value for chunk [0, 0x23c52): Mirror 1: 0xea09f5b4 Mirror 19: 0xea09f5b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23c52: skipped 000521 1780384493.356329 write 0x008bef thru 0x0160d7 (0xd4e9 bytes) 000523 1780384493.486780 mapwrite 0x0132ca thru 0x0192d0 (0x6007 bytes) 000527 1780384493.577031 trunc from 0x023c52 to 0x03c5f9 (0x189a8 bytes) 000529 1780384493.608590 read 0x00d5f7 thru 0x00dfd9 (0x9e3 bytes) 000532 1780384493.611728 mapread 0x01158b thru 0x01f9ef (0xe465 bytes) 000533 1780384493.613648 trunc from 0x03c5fa to 0x02503c (0x175bd bytes) 000534 1780384493.637586 trunc from 0x02503d to 0x03b770 (0x16734 bytes) 000539 1780384493.663425 mapread 0x013de6 thru 0x020412 (0xc62d bytes) 000541 1780384493.674606 trunc from 0x03b771 to 0x02a9f8 (0x10d78 bytes) 000542 1780384493.694627 trunc from 0x02a9f9 to 0x039eff (0xf507 bytes) 000543 1780384493.725700 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39f00): Mirror 17: 0x44029866 (stale) Mirror 18: 0x5668a922 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f00): Mirror 17: 0x44029866 (stale) Mirror 18: 0x5668a922 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f00: skipped 000547 1780384494.115826 MIRROR_RESYNC 3 mirrors 000549 1780384494.910833 trunc from 0x039f00 to 0x02353b (0x169c4 bytes) 000550 1780384494.998405 read 0x01b732 thru 0x01ba9b (0x36a bytes) 000551 1780384495.005311 trunc from 0x02353c to 0x0263e2 (0x2ea7 bytes) 000552 1780384495.039550 read 0x021fa7 thru 0x02252a (0x584 bytes) 000554 1780384495.050680 trunc from 0x0263e3 to 0x00da0b (0x189d7 bytes) 000555 1780384495.064500 write 0x00f08b thru 0x016890 (0x7806 bytes) 000556 1780384495.065854 read 0x002f19 thru 0x00fa30 (0xcb18 bytes) 000558 1780384495.073460 read 0x0097b6 thru 0x012a9a (0x92e5 bytes) 000562 1780384495.078686 trunc from 0x016891 to 0x029239 (0x129a9 bytes) 000563 1780384495.102684 mapread 0x0016fa thru 0x007ec8 (0x67cf bytes) 000564 1780384495.107817 read 0x022cd8 thru 0x029239 (0x6562 bytes) 000565 1780384495.131497 mapread 0x024c8b thru 0x028175 (0x34eb bytes) 000566 1780384495.135535 read 0x027a25 thru 0x029239 (0x1815 bytes) 000569 1780384495.145271 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2923a): Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2923a): Mirror 17: 0x808aee3e (stale) Mirror 18: 0x808aee3e (stale) CRC-32 checksum value for chunk [0, 0x2923a): Mirror 1: 0xfb07dd6f Mirror 19: 0xfb07dd6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2923a: skipped 000570 1780384496.062593 write 0x031a0f thru 0x038ba9 (0x719b bytes) 000571 1780384496.157037 trunc from 0x038baa to 0x03b700 (0x2b57 bytes) 000572 1780384496.211981 mapwrite 0x034eb7 thru 0x038a9e (0x3be8 bytes) 000573 1780384496.292126 mapwrite 0x028f30 thru 0x033c4f (0xad20 bytes) 000575 1780384496.385565 mapwrite 0x0264e2 thru 0x02f4d8 (0x8ff7 bytes) 000576 1780384496.462873 trunc from 0x03b701 to 0x02d36b (0xe395 bytes) 000577 1780384496.487891 write 0x0156dc thru 0x022728 (0xd04d bytes) 000580 1780384496.525545 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d36c): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d36c): Mirror 17: 0x7db7a850 (stale) Mirror 18: 0x7db7a850 (stale) Mirror 19: 0x446f08d4 (stale) CRC-32 checksum value for chunk [0, 0x2d36c): Mirror 1: 0xc0c35472 Mirror 20: 0xc0c35472 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d36c: skipped 000582 1780384497.528075 write 0x002d65 thru 0x00ef12 (0xc1ae bytes) 000584 1780384497.664649 write 0x036904 thru 0x03ffff (0x96fc bytes) 000586 1780384497.670744 read 0x016257 thru 0x0213de (0xb188 bytes) 000590 1780384497.698151 mapwrite 0x0215aa thru 0x02b67c (0xa0d3 bytes) 000595 1780384497.815767 write 0x035378 thru 0x03cccd (0x7956 bytes) 000598 1780384497.827934 trunc from 0x040000 to 0x005dc1 (0x3a23e bytes) 000599 1780384497.863456 trunc from 0x005dc2 to 0x00945a (0x3699 bytes) 000600 1780384497.882605 mapread 0x0077f1 thru 0x00945a (0x1c6a bytes) 000602 1780384497.896850 mapwrite 0x006bdc thru 0x009374 (0x2799 bytes) 000603 1780384497.979657 MIRROR_RESYNC 5 mirrors 000605 1780384499.129701 trunc from 0x00945b to 0x013c3b (0xa7e1 bytes) 000606 1780384499.220155 write 0x02b027 thru 0x02e70e (0x36e8 bytes) 000608 1780384499.229267 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e70f): Mirror 18: 0x4610911 (stale) Mirror 19: 0x4610911 (stale) Mirror 20: 0x4610911 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e70f): Mirror 18: 0x4610911 (stale) Mirror 19: 0x4610911 (stale) Mirror 20: 0x4610911 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e70f: skipped 000609 1780384499.633219 trunc from 0x02e70f to 0x000c6b (0x2daa3 bytes) 000610 1780384499.648943 mapwrite 0x02ae33 thru 0x0382f8 (0xd4c6 bytes) 000611 1780384499.818268 trunc from 0x0382f9 to 0x024873 (0x13a85 bytes) 000612 1780384499.843968 mapread 0x001ff4 thru 0x00879a (0x67a7 bytes) 000613 1780384499.866594 mapread 0x0000af thru 0x004837 (0x4789 bytes) 000614 1780384499.883655 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24874): Mirror 18: 0x87798b22 (stale) Mirror 19: 0x87798b22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24874): Mirror 18: 0x87798b22 (stale) Mirror 19: 0x87798b22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24874: skipped 000615 1780384500.241197 read 0x01ac03 thru 0x024873 (0x9c71 bytes) 000616 1780384500.251898 read 0x0247e8 thru 0x024873 (0x8c bytes) 000617 1780384500.252596 mapwrite 0x015950 thru 0x017a37 (0x20e8 bytes) 000619 1780384500.323540 mapread 0x013814 thru 0x02193c (0xe129 bytes) 000625 1780384500.329375 mapwrite 0x02521e thru 0x02d435 (0x8218 bytes) 000626 1780384500.421509 mapwrite 0x02bac3 thru 0x02bdd5 (0x313 bytes) 000629 1780384500.464857 mapwrite 0x00a7d4 thru 0x010467 (0x5c94 bytes) 000630 1780384500.524922 write 0x013e52 thru 0x023aa0 (0xfc4f bytes) 000631 1780384500.528593 mapwrite 0x0194f7 thru 0x025ea0 (0xc9aa bytes) 000632 1780384500.685145 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d436): Mirror 19: 0x9c0ab944 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d436): Mirror 19: 0x9c0ab944 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d436: skipped 000633 1780384501.126770 write 0x00f327 thru 0x01182a (0x2504 bytes) 000634 1780384501.149500 trunc from 0x02d436 to 0x026dc6 (0x666f bytes) 000635 1780384501.174870 mapread 0x024052 thru 0x026dc6 (0x2d75 bytes) 000636 1780384501.188401 mapwrite 0x00a91d thru 0x0155a2 (0xac86 bytes) 000641 1780384501.293987 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26dc7): Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26dc7): Mirror 19: 0x8cf95ca5 (stale) CRC-32 checksum value for chunk [0, 0x26dc7): Mirror 1: 0xf0e44c2b Mirror 20: 0xf0e44c2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26dc7: skipped 000642 1780384502.157510 mapwrite 0x03e981 thru 0x03ffff (0x167f bytes) 000643 1780384502.390233 mapwrite 0x03a39d thru 0x03e07e (0x3ce2 bytes) 000646 1780384502.517538 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x1a2d4876 (stale) Mirror 20: 0x4123547b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd98ac973 Mirror 21: 0xd98ac973 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000647 1780384503.375338 mapwrite 0x03da8f thru 0x03ffff (0x2571 bytes) 000650 1780384503.540727 read 0x039e49 thru 0x03ffff (0x61b7 bytes) 000652 1780384503.542294 mapread 0x03f43d thru 0x03ffff (0xbc3 bytes) 000653 1780384503.549074 MIRROR_RESYNC 4 mirrors 000654 1780384504.625647 write 0x021d0c thru 0x023326 (0x161b bytes) 000655 1780384504.708241 mapwrite 0x00b96e thru 0x01b917 (0xffaa bytes) 000656 1780384504.842057 trunc from 0x040000 to 0x0076b8 (0x38947 bytes) 000658 1780384504.888383 read 0x001f03 thru 0x0076b8 (0x57b6 bytes) 000660 1780384504.899464 trunc from 0x0076b9 to 0x03611a (0x2ea62 bytes) 000661 1780384504.924942 mapread 0x001aed thru 0x00b08f (0x95a3 bytes) 000662 1780384504.937407 mapwrite 0x024a6d thru 0x031580 (0xcb14 bytes) 000663 1780384505.068338 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3611b): Mirror 19: 0x72f4149d (stale) Mirror 21: 0x72f4149d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3611b): Mirror 19: 0x72f4149d (stale) Mirror 21: 0x72f4149d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3611b: skipped 000664 1780384505.482053 mapwrite 0x006536 thru 0x00e744 (0x820f bytes) 000665 1780384505.559350 mapwrite 0x026e1a thru 0x033557 (0xc73e bytes) 000666 1780384505.654461 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3611b): Mirror 21: 0x72f4149d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3611b): Mirror 21: 0x72f4149d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3611b: skipped 000667 1780384506.037104 mapwrite 0x02fd60 thru 0x037c0e (0x7eaf bytes) 000668 1780384506.169328 trunc from 0x037c0f to 0x00c47c (0x2b792 bytes) 000669 1780384506.203421 write 0x005a28 thru 0x0156c4 (0xfc9d bytes) 000670 1780384506.214500 write 0x0084b9 thru 0x00d492 (0x4fda bytes) 000671 1780384506.224140 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x156c5): Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x156c5): Mirror 21: 0x3050432c (stale) CRC-32 checksum value for chunk [0, 0x156c5): Mirror 1: 0x7f951bac Mirror 22: 0x7f951bac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x156c5: skipped 000673 1780384507.100666 mapwrite 0x03e870 thru 0x03ffff (0x1790 bytes) 000676 1780384507.275298 trunc from 0x040000 to 0x01bed4 (0x2412b bytes) 000677 1780384507.318408 mapread 0x00c9a6 thru 0x013c05 (0x7260 bytes) 000678 1780384507.339850 mapwrite 0x03e64f thru 0x03ffff (0x19b1 bytes) 000680 1780384507.425310 mapread 0x03f54e thru 0x03ffff (0xab2 bytes) 000681 1780384507.426553 mapwrite 0x013169 thru 0x018c55 (0x5aed bytes) 000682 1780384507.487981 MIRROR_RESYNC 3 mirrors 000684 1780384508.213444 read 0x02f0ca thru 0x039c06 (0xab3d bytes) 000685 1780384508.225758 mapwrite 0x02f813 thru 0x0379ce (0x81bc bytes) 000686 1780384508.432392 mapread 0x038807 thru 0x03a3a9 (0x1ba3 bytes) 000687 1780384508.441313 read 0x013467 thru 0x017ffe (0x4b98 bytes) 000688 1780384508.456847 mapread 0x00ab80 thru 0x00f3af (0x4830 bytes) 000689 1780384508.458261 trunc from 0x040000 to 0x0101d6 (0x2fe29 bytes) 000690 1780384508.520458 trunc from 0x0101d7 to 0x02ef4b (0x1ed75 bytes) 000691 1780384508.550320 mapwrite 0x01ce43 thru 0x0285f1 (0xb7af bytes) 000693 1780384508.744557 write 0x00dc38 thru 0x0141d2 (0x659b bytes) 000698 1780384508.758989 read 0x0190ad thru 0x024405 (0xb359 bytes) 000699 1780384508.771229 mapread 0x01491b thru 0x022dc9 (0xe4af bytes) 000700 1780384508.773261 trunc from 0x02ef4c to 0x0023ca (0x2cb81 bytes) 000701 1780384508.795386 read 0x001ce6 thru 0x0023ca (0x6e5 bytes) 000702 1780384508.800433 trunc from 0x0023cb to 0x0229d5 (0x2060b bytes) 000704 1780384508.840230 write 0x01d0c1 thru 0x02ae49 (0xdd89 bytes) 000706 1780384508.882037 trunc from 0x02ae4a to 0x033af8 (0x8caf bytes) 000707 1780384508.919477 MIRROR_RESYNC 3 mirrors 000708 1780384509.794045 mapwrite 0x028a42 thru 0x02b920 (0x2edf bytes) 000709 1780384510.035549 read 0x02ed8a thru 0x033af8 (0x4d6f bytes) 000710 1780384510.045802 mapread 0x01389c thru 0x01d07d (0x97e2 bytes) 000711 1780384510.079261 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33af9): Mirror 22: 0x4d0a654a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33af9): Mirror 22: 0x4d0a654a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33af9: skipped 000712 1780384510.454132 mapwrite 0x03b4c3 thru 0x03ffff (0x4b3d bytes) 000715 1780384510.589411 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x34fbc7a3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x873483c8 Mirror 23: 0x873483c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000718 1780384511.443586 read 0x026de8 thru 0x02930b (0x2524 bytes) 000720 1780384511.460570 write 0x031328 thru 0x038c06 (0x78df bytes) 000722 1780384511.544511 read 0x00c963 thru 0x0189d1 (0xc06f bytes) 000723 1780384511.566665 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x34fbc7a3 (stale) Mirror 23: 0x873483c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6d8c802e Mirror 24: 0x6d8c802e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1780384512.503821 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x873483c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x873483c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6d8c802e Mirror 24: 0x6d8c802e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000726 1780384513.204122 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x873483c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6d8c802e Mirror 24: 0x6d8c802e Mirror 25: 0x6d8c802e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000727 1780384514.109787 trunc from 0x040000 to 0x00f114 (0x30eeb bytes) 000728 1780384514.211904 write 0x026103 thru 0x02e408 (0x8306 bytes) 000731 1780384514.219707 mapread 0x010c16 thru 0x013453 (0x283e bytes) 000732 1780384514.242191 trunc from 0x02e409 to 0x03e19c (0xfd94 bytes) 000734 1780384514.267611 read 0x03b5ba thru 0x03e19c (0x2be3 bytes) 000736 1780384514.283719 MIRROR_RESYNC 4 mirrors 000737 1780384515.277438 mapwrite 0x015a96 thru 0x01ed7f (0x92ea bytes) 000738 1780384515.490284 mapread 0x025a7c thru 0x02c160 (0x66e5 bytes) 000739 1780384515.508708 write 0x02dca2 thru 0x033136 (0x5495 bytes) 000740 1780384515.519680 trunc from 0x03e19d to 0x01cae8 (0x216b4 bytes) 000741 1780384515.583814 mapwrite 0x000656 thru 0x0072a8 (0x6c53 bytes) 000743 1780384515.649386 mapread 0x006681 thru 0x0158dc (0xf25c bytes) 000749 1780384515.655131 trunc from 0x01cae9 to 0x020b35 (0x404d bytes) 000750 1780384515.686380 MIRROR_RESYNC 4 mirrors 000753 1780384516.541040 MIRROR_EXTEND to 5 mirrors 000754 1780384517.301818 write 0x01abaf thru 0x01f3be (0x4810 bytes) 000756 1780384517.445669 MIRROR_RESYNC 5 mirrors 000757 1780384518.535339 write 0x0259cb thru 0x0274de (0x1b14 bytes) 000758 1780384518.589034 trunc from 0x0274df to 0x004291 (0x2324d bytes) 000759 1780384518.609184 mapread 0x003e34 thru 0x004291 (0x45e bytes) 000762 1780384518.629656 trunc from 0x004292 to 0x012884 (0xe5f3 bytes) 000765 1780384518.678340 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12885): Mirror 23: 0xae00c252 (stale) Mirror 25: 0xae00c252 (stale) Mirror 26: 0xae00c252 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12885): Mirror 23: 0xae00c252 (stale) Mirror 25: 0xae00c252 (stale) Mirror 26: 0xae00c252 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12885: skipped 000769 1780384519.077379 read 0x011c60 thru 0x012884 (0xc25 bytes) 000770 1780384519.088117 mapread 0x004e9f thru 0x012884 (0xd9e6 bytes) 000771 1780384519.112920 mapread 0x00a591 thru 0x010329 (0x5d99 bytes) 000772 1780384519.115425 mapread 0x0027ab thru 0x0068d5 (0x412b bytes) 000773 1780384519.117317 mapread 0x01039c thru 0x012884 (0x24e9 bytes) 000774 1780384519.119313 mapread 0x006ac2 thru 0x012884 (0xbdc3 bytes) 000779 1780384519.127953 read 0x00e4cb thru 0x012884 (0x43ba bytes) 000780 1780384519.129936 mapwrite 0x01a8a1 thru 0x028481 (0xdbe1 bytes) 000781 1780384519.240854 mapread 0x023953 thru 0x028481 (0x4b2f bytes) 000782 1780384519.244182 write 0x03e68c thru 0x03ffff (0x1974 bytes) 000783 1780384519.246546 read 0x005b71 thru 0x00c92f (0x6dbf bytes) 000784 1780384519.249107 mapread 0x0323d8 thru 0x03e570 (0xc199 bytes) 000785 1780384519.268556 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x2f9cbb3d (stale) Mirror 25: 0x2f9cbb3d (stale) Mirror 26: 0x2f9cbb3d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2207e85d Mirror 27: 0x2207e85d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1780384520.232332 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x2f9cbb3d (stale) Mirror 25: 0x2f9cbb3d (stale) Mirror 26: 0x2f9cbb3d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2207e85d Mirror 27: 0x2207e85d Mirror 28: 0x2207e85d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000787 1780384521.220685 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x2f9cbb3d (stale) Mirror 25: 0x2f9cbb3d (stale) Mirror 26: 0x2f9cbb3d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2207e85d Mirror 27: 0x2207e85d Mirror 28: 0x2207e85d Mirror 29: 0x2207e85d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000788 1780384522.187351 write 0x02b55a thru 0x039a5b (0xe502 bytes) 000789 1780384522.318214 mapread 0x0211d1 thru 0x030850 (0xf680 bytes) 000790 1780384522.360680 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x2f9cbb3d (stale) Mirror 25: 0x2f9cbb3d (stale) Mirror 26: 0x2f9cbb3d (stale) Mirror 27: 0x2207e85d (stale) Mirror 28: 0x2207e85d (stale) Mirror 29: 0x2207e85d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6eb12f0b Mirror 30: 0x6eb12f0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000791 1780384523.325137 mapwrite 0x006d95 thru 0x007b58 (0xdc4 bytes) 000793 1780384523.557602 MIRROR_RESYNC 8 mirrors 000794 1780384525.025069 read 0x018ce8 thru 0x01dd15 (0x502e bytes) 000796 1780384525.037284 write 0x0069c1 thru 0x01326c (0xc8ac bytes) 000797 1780384525.117632 trunc from 0x040000 to 0x0148a0 (0x2b75f bytes) 000798 1780384525.166028 write 0x00e368 thru 0x01824c (0x9ee5 bytes) 000800 1780384525.172441 mapwrite 0x01981f thru 0x027512 (0xdcf4 bytes) 000801 1780384525.301326 mapwrite 0x010f70 thru 0x01a980 (0x9a11 bytes) 000803 1780384525.360846 mapwrite 0x021f3e thru 0x030656 (0xe719 bytes) 000804 1780384525.491186 write 0x0391b5 thru 0x03ffff (0x6e4b bytes) 000805 1780384525.492519 write 0x013036 thru 0x01e47a (0xb445 bytes) 000806 1780384525.496552 mapread 0x024afb thru 0x0290a5 (0x45ab bytes) 000807 1780384525.498824 write 0x007d8a thru 0x00d1c0 (0x5437 bytes) 000808 1780384525.506940 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0xcb57f108 (stale) Mirror 26: 0xcb57f108 (stale) Mirror 27: 0xcb57f108 (stale) Mirror 28: 0xcb57f108 (stale) Mirror 29: 0xcb57f108 (stale) Mirror 30: 0xcb57f108 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0xcb57f108 (stale) Mirror 26: 0xcb57f108 (stale) Mirror 27: 0xcb57f108 (stale) Mirror 28: 0xcb57f108 (stale) Mirror 29: 0xcb57f108 (stale) Mirror 30: 0xcb57f108 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000809 1780384526.099737 mapwrite 0x0080da thru 0x011e23 (0x9d4a bytes) 000810 1780384526.230725 trunc from 0x040000 to 0x030d29 (0xf2d6 bytes) 000811 1780384526.253345 read 0x00cf68 thru 0x01406e (0x7107 bytes) 000812 1780384526.262274 mapwrite 0x00275e thru 0x011e0f (0xf6b2 bytes) 000813 1780384526.335456 mapwrite 0x0288fa thru 0x02afe2 (0x26e9 bytes) 000814 1780384526.394566 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30d2a): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d2a): Mirror 25: 0xa9a281ac (stale) Mirror 26: 0xa9a281ac (stale) Mirror 27: 0xa9a281ac (stale) Mirror 28: 0xa9a281ac (stale) Mirror 29: 0xa9a281ac (stale) Mirror 30: 0xa9a281ac (stale) CRC-32 checksum value for chunk [0, 0x30d2a): Mirror 1: 0xc563239 Mirror 31: 0xc563239 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d2a: skipped 000815 1780384527.235376 write 0x000316 thru 0x006abd (0x67a8 bytes) 000821 1780384527.361542 read 0x0112a2 thru 0x019561 (0x82c0 bytes) 000822 1780384527.384751 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d2a): Mirror 25: 0xa9a281ac (stale) Mirror 26: 0xa9a281ac (stale) Mirror 27: 0xa9a281ac (stale) Mirror 29: 0xa9a281ac (stale) Mirror 30: 0xa9a281ac (stale) Mirror 31: 0xc563239 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d2a): Mirror 25: 0xa9a281ac (stale) Mirror 26: 0xa9a281ac (stale) Mirror 27: 0xa9a281ac (stale) Mirror 29: 0xa9a281ac (stale) Mirror 30: 0xa9a281ac (stale) Mirror 31: 0xc563239 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d2a: skipped 000824 1780384527.953980 write 0x01943e thru 0x0229c6 (0x9589 bytes) 000825 1780384527.968424 trunc from 0x030d2a to 0x03d7fa (0xcad1 bytes) 000833 1780384528.009043 trunc from 0x03d7fb to 0x03caf7 (0xd03 bytes) 000834 1780384528.027868 mapread 0x013b37 thru 0x017e1b (0x42e5 bytes) 000835 1780384528.044157 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3caf8): Mirror 25: 0x8f4b4ac0 (stale) Mirror 26: 0x8f4b4ac0 (stale) Mirror 27: 0x8f4b4ac0 (stale) Mirror 29: 0x8f4b4ac0 (stale) Mirror 30: 0x8f4b4ac0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3caf8): Mirror 25: 0x8f4b4ac0 (stale) Mirror 26: 0x8f4b4ac0 (stale) Mirror 27: 0x8f4b4ac0 (stale) Mirror 29: 0x8f4b4ac0 (stale) Mirror 30: 0x8f4b4ac0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3caf8: skipped 000836 1780384528.552304 read 0x034e86 thru 0x039625 (0x47a0 bytes) 000840 1780384528.576496 mapread 0x020b29 thru 0x024a4d (0x3f25 bytes) 000841 1780384528.619247 mapwrite 0x02bbaf thru 0x035bd7 (0xa029 bytes) 000842 1780384528.727005 mapwrite 0x01711d thru 0x024caf (0xdb93 bytes) 000843 1780384528.835187 MIRROR_RESYNC 6 mirrors 000846 1780384530.101923 mapwrite 0x006a99 thru 0x01364b (0xcbb3 bytes) 000851 1780384530.301201 read 0x016212 thru 0x0207cf (0xa5be bytes) 000852 1780384530.328070 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3caf8): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3caf8): Mirror 25: 0x512556e1 (stale) Mirror 26: 0x512556e1 (stale) Mirror 27: 0x512556e1 (stale) Mirror 29: 0x512556e1 (stale) Mirror 30: 0x512556e1 (stale) CRC-32 checksum value for chunk [0, 0x3caf8): Mirror 1: 0x59acaede Mirror 31: 0x59acaede lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3caf8: skipped 000854 1780384531.441736 mapread 0x007677 thru 0x017081 (0xfa0b bytes) 000855 1780384531.491667 read 0x00f737 thru 0x01dc4c (0xe516 bytes) 000857 1780384531.513165 mapread 0x01d88f thru 0x024dc4 (0x7536 bytes) 000858 1780384531.563940 mapread 0x0217e6 thru 0x02b46d (0x9c88 bytes) 000859 1780384531.584856 read 0x00cbd6 thru 0x018852 (0xbc7d bytes) 000860 1780384531.590289 write 0x01800d thru 0x01a631 (0x2625 bytes) 000862 1780384531.710679 read 0x017bcc thru 0x026992 (0xedc7 bytes) 000870 1780384531.716576 trunc from 0x03caf8 to 0x017549 (0x255ae bytes) 000871 1780384531.789852 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1754a): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1754a): Mirror 25: 0x4d17fb78 (stale) Mirror 26: 0x4d17fb78 (stale) Mirror 27: 0x4d17fb78 (stale) Mirror 29: 0x4d17fb78 (stale) Mirror 30: 0x4d17fb78 (stale) Mirror 31: 0xacf05629 (stale) CRC-32 checksum value for chunk [0, 0x1754a): Mirror 1: 0xacf05629 Mirror 32: 0xacf05629 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1754a: skipped 000872 1780384532.799642 mapwrite 0x017427 thru 0x01858d (0x1167 bytes) 000874 1780384532.990961 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1858e): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1858e): Mirror 25: 0x32fb36c2 (stale) Mirror 26: 0x32fb36c2 (stale) Mirror 27: 0x32fb36c2 (stale) Mirror 29: 0x32fb36c2 (stale) Mirror 30: 0x32fb36c2 (stale) Mirror 31: 0x4365ca4f (stale) Mirror 32: 0xa18f7c1a (stale) CRC-32 checksum value for chunk [0, 0x1858e): Mirror 1: 0x97dc1382 Mirror 33: 0x97dc1382 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1858e: skipped truncating to largest ever: 0x3f9e6 000875 1780384534.141305 trunc from 0x01858e to 0x03f9e5 (0x27458 bytes) 000878 1780384534.245925 write 0x0116b0 thru 0x014919 (0x326a bytes) 000879 1780384534.266635 mapread 0x03290a thru 0x03a3dd (0x7ad4 bytes) 000881 1780384534.286698 MIRROR_RESYNC 9 mirrors 000882 1780384536.049976 trunc from 0x03f9e6 to 0x0061ab (0x3983a bytes) 000884 1780384536.175699 mapread 0x000121 thru 0x0061ab (0x608b bytes) 000885 1780384536.231997 trunc from 0x0061ac to 0x037bdc (0x31a31 bytes) 000886 1780384536.264207 mapwrite 0x02f17e thru 0x0359d6 (0x6859 bytes) 000889 1780384536.403329 mapread 0x00f173 thru 0x01582a (0x66b8 bytes) 000891 1780384536.449879 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x37bdd): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bdd): Mirror 25: 0x4383fbb7 (stale) Mirror 26: 0x4383fbb7 (stale) Mirror 27: 0x4383fbb7 (stale) Mirror 29: 0x4383fbb7 (stale) Mirror 30: 0x4383fbb7 (stale) Mirror 31: 0x4383fbb7 (stale) Mirror 32: 0x4383fbb7 (stale) Mirror 33: 0x4383fbb7 (stale) CRC-32 checksum value for chunk [0, 0x37bdd): Mirror 1: 0xf27607eb Mirror 34: 0xf27607eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bdd: skipped 000894 1780384537.481796 trunc from 0x037bdd to 0x01e905 (0x192d7 bytes) 000895 1780384537.615068 trunc from 0x01e906 to 0x0058aa (0x1905b bytes) 000896 1780384537.672334 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x58ab): Mirror 25: 0xcd9e7b29 (stale) Mirror 27: 0xcd9e7b29 (stale) Mirror 29: 0xcd9e7b29 (stale) Mirror 30: 0xcd9e7b29 (stale) Mirror 31: 0xcd9e7b29 (stale) Mirror 32: 0xcd9e7b29 (stale) Mirror 33: 0xcd9e7b29 (stale) Mirror 34: 0xcd9e7b29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x58ab): Mirror 25: 0xcd9e7b29 (stale) Mirror 27: 0xcd9e7b29 (stale) Mirror 29: 0xcd9e7b29 (stale) Mirror 30: 0xcd9e7b29 (stale) Mirror 31: 0xcd9e7b29 (stale) Mirror 32: 0xcd9e7b29 (stale) Mirror 33: 0xcd9e7b29 (stale) Mirror 34: 0xcd9e7b29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x58ab: skipped 000898 1780384538.261105 mapread 0x00502e thru 0x0058aa (0x87d bytes) 000899 1780384538.272871 mapread 0x003526 thru 0x0058aa (0x2385 bytes) 000900 1780384538.298302 mapwrite 0x034fa2 thru 0x0392bb (0x431a bytes) 000901 1780384538.399962 write 0x021a79 thru 0x0229ab (0xf33 bytes) 000904 1780384538.412025 trunc from 0x0392bc to 0x00db7a (0x2b741 bytes) 000905 1780384538.432926 read 0x004548 thru 0x009d33 (0x57ec bytes) 000910 1780384538.446155 trunc from 0x00db7b to 0x0295ed (0x1ba73 bytes) 000912 1780384538.474210 trunc from 0x0295ee to 0x002c42 (0x269ab bytes) 000913 1780384538.492517 mapread 0x0017a9 thru 0x002c42 (0x149a bytes) 000915 1780384538.494174 mapread 0x001006 thru 0x002c42 (0x1c3d bytes) 000917 1780384538.505230 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2c43): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c43): Mirror 25: 0x1baf31f6 (stale) Mirror 27: 0x1baf31f6 (stale) Mirror 29: 0x1baf31f6 (stale) Mirror 30: 0x1baf31f6 (stale) Mirror 31: 0x1baf31f6 (stale) Mirror 32: 0x1baf31f6 (stale) Mirror 33: 0x1baf31f6 (stale) Mirror 34: 0x1baf31f6 (stale) CRC-32 checksum value for chunk [0, 0x2c43): Mirror 1: 0x1baf31f6 Mirror 35: 0x1baf31f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c43: skipped 000919 1780384539.224447 MIRROR_RESYNC 10 mirrors 000921 1780384540.500759 trunc from 0x002c43 to 0x00899f (0x5d5d bytes) 000922 1780384540.618801 trunc from 0x0089a0 to 0x008f32 (0x593 bytes) 000923 1780384540.658613 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x8f33): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f33): Mirror 25: 0xa38d30a4 (stale) Mirror 27: 0xa38d30a4 (stale) Mirror 29: 0xa38d30a4 (stale) Mirror 30: 0xa38d30a4 (stale) Mirror 31: 0xa38d30a4 (stale) Mirror 32: 0xa38d30a4 (stale) Mirror 33: 0xa38d30a4 (stale) Mirror 34: 0xa38d30a4 (stale) Mirror 35: 0xa38d30a4 (stale) CRC-32 checksum value for chunk [0, 0x8f33): Mirror 1: 0xa38d30a4 Mirror 36: 0xa38d30a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f33: skipped 000924 1780384541.626583 read 0x006fc7 thru 0x008f32 (0x1f6c bytes) 000925 1780384541.644491 trunc from 0x008f33 to 0x0210b2 (0x18180 bytes) 000927 1780384541.797638 mapread 0x016824 thru 0x01df87 (0x7764 bytes) 000928 1780384541.817109 trunc from 0x0210b3 to 0x008d0b (0x183a7 bytes) 000929 1780384541.842903 mapread 0x003238 thru 0x007ac4 (0x488d bytes) 000930 1780384541.860174 trunc from 0x008d0c to 0x02783c (0x1eb31 bytes) 000933 1780384541.879230 mapread 0x013d22 thru 0x016b8d (0x2e6c bytes) 000936 1780384541.890204 mapread 0x00edda thru 0x01681d (0x7a44 bytes) 000940 1780384541.894062 mapwrite 0x01f65a thru 0x02ec21 (0xf5c8 bytes) 000941 1780384542.010851 MIRROR_RESYNC 11 mirrors 000943 1780384543.936507 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000945 1780384544.637869 mapread 0x001a9e thru 0x00c3ae (0xa911 bytes) 000946 1780384544.679253 mapwrite 0x01b8e1 thru 0x01f25b (0x397b bytes) 000947 1780384544.971930 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2ec22): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec22): Mirror 25: 0x267a4978 (stale) Mirror 27: 0x267a4978 (stale) Mirror 29: 0x267a4978 (stale) Mirror 30: 0x267a4978 (stale) Mirror 32: 0x267a4978 (stale) Mirror 33: 0x267a4978 (stale) Mirror 34: 0x267a4978 (stale) Mirror 35: 0x267a4978 (stale) Mirror 36: 0x267a4978 (stale) CRC-32 checksum value for chunk [0, 0x2ec22): Mirror 1: 0xceecab8a Mirror 37: 0xceecab8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec22: skipped 000948 1780384546.241149 MIRROR_RESYNC 11 mirrors 000949 1780384547.759663 write 0x00c4d7 thru 0x01ad04 (0xe82e bytes) 000952 1780384547.845738 write 0x0028c3 thru 0x005ef4 (0x3632 bytes) 000955 1780384547.854167 read 0x0151d8 thru 0x022814 (0xd63d bytes) 000956 1780384547.869513 trunc from 0x02ec22 to 0x038aea (0x9ec9 bytes) 000957 1780384547.903333 write 0x011aa1 thru 0x01561c (0x3b7c bytes) 000958 1780384547.904725 mapwrite 0x02dc20 thru 0x02de2c (0x20d bytes) 000959 1780384548.005975 write 0x004d2f thru 0x01372a (0xe9fc bytes) 000961 1780384548.008471 mapread 0x035095 thru 0x038aea (0x3a56 bytes) 000964 1780384548.028112 write 0x001a99 thru 0x003db1 (0x2319 bytes) 000965 1780384548.031144 mapread 0x019003 thru 0x026a2c (0xda2a bytes) 000967 1780384548.048439 mapwrite 0x022112 thru 0x02f2aa (0xd199 bytes) 000968 1780384548.159981 trunc from 0x038aeb to 0x03aa2d (0x1f43 bytes) 000969 1780384548.199657 write 0x002b60 thru 0x009e17 (0x72b8 bytes) 000970 1780384548.201071 write 0x03ab70 thru 0x03ffff (0x5490 bytes) 000973 1780384548.210159 mapwrite 0x03f55a thru 0x03f852 (0x2f9 bytes) 000975 1780384548.308679 mapwrite 0x006e65 thru 0x0099dd (0x2b79 bytes) 000979 1780384548.373045 mapread 0x02b36a thru 0x03835d (0xcff4 bytes) 000981 1780384548.376664 mapwrite 0x027ac6 thru 0x02b30d (0x3848 bytes) 000982 1780384548.428702 read 0x01af2e thru 0x028bb9 (0xdc8c bytes) 000985 1780384548.436818 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0xc781aae9 (stale) Mirror 27: 0xc781aae9 (stale) Mirror 29: 0xc781aae9 (stale) Mirror 30: 0xc781aae9 (stale) Mirror 32: 0xc781aae9 (stale) Mirror 33: 0xc781aae9 (stale) Mirror 34: 0xc781aae9 (stale) Mirror 35: 0xc781aae9 (stale) Mirror 36: 0xc781aae9 (stale) Mirror 37: 0xc781aae9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfe85ba83 Mirror 38: 0xfe85ba83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000986 1780384549.801834 mapread 0x00c075 thru 0x01002e (0x3fba bytes) 000987 1780384549.864179 read 0x014d01 thru 0x014f3c (0x23c bytes) 000990 1780384549.877709 trunc from 0x040000 to 0x016dcb (0x29234 bytes) 000991 1780384550.005061 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x16dcc): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16dcc): Mirror 25: 0xed79fee6 (stale) Mirror 27: 0xed79fee6 (stale) Mirror 29: 0xed79fee6 (stale) Mirror 30: 0xed79fee6 (stale) Mirror 32: 0xed79fee6 (stale) Mirror 33: 0xed79fee6 (stale) Mirror 34: 0xed79fee6 (stale) Mirror 35: 0xed79fee6 (stale) Mirror 36: 0xed79fee6 (stale) Mirror 37: 0xed79fee6 (stale) Mirror 38: 0xb22ed6c0 (stale) CRC-32 checksum value for chunk [0, 0x16dcc): Mirror 1: 0xb22ed6c0 Mirror 39: 0xb22ed6c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16dcc: skipped 000994 1780384551.366031 mapread 0x009401 thru 0x016dcb (0xd9cb bytes) 000998 1780384551.445198 mapread 0x012169 thru 0x016dcb (0x4c63 bytes) 000999 1780384551.462678 mapwrite 0x02ae07 thru 0x03621e (0xb418 bytes) All operations completed A-OK! PASS 70a (129s) == sanity-flr test 71: check mirror extend parent fid ==== 03:15:56 (1780384556) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 03:15:58 (1780384558) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269027 s, 7.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180015 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 03:16:06 (1780384566) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8201696 .. Extending file size to 6856128 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8660800 .. Extending file size to 8258944 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56741 56742 56744 56745 56747 done 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) 192.168.204.146@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg446-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:17:26 (1780384646) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 279584 .. Extending file size to 4321984 .. failed Extending file size to 5020704 .. Extending file size to 5075712 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6921920 .. failed Extending file size to 6486368 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8454976 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9168928 .. Extending file size to 9050048 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58369 58370 58371 58373 58375 done 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) 192.168.204.146@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg446-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:18:47 (1780384727) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118152 s, 17.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.359396 s, 5.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9331ed89 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:19:03 (1780384743) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118014 s, 17.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 03:19:09 (1780384749) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ea:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:19:17 (1780384757) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201694 s, 10.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:19:23 (1780384763) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20508 s, 10.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:19:31 (1780384771) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215838 s, 9.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:20:01 (1780384801) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg446-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968447 s, 10.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:20:54 (1780384854) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg446-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191166 s, 11.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:21:48 (1780384908) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg446-server mds-ost sync done. Creating new pool test_204f oleg446-server: Pool lustre.test_204f created Adding targets to pool oleg446-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg446-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235971 s, 8.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg446-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg446-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg446-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:23:01 (1780384981) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:23:08 (1780384988) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:23:16 (1780384996) oleg446-server: Pool lustre.test_206 created Adding targets to pool oleg446-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg446-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg446-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg446-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg446-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:23:37 (1780385017) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683753 s, 15.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379712 s, 27.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x742dbd2d Mirror 3: 0x742dbd2d PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:23:44 (1780385024) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:23:47 (1780385027) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:23:50 (1780385030) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0947674 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2570000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1380000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd60000 Read this (4 bytes): BBBB 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) PASS 209a (29s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:24:19 (1780385059) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0311578 s, 263 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:24:24 (1780385064) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05656 s, 18.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834447 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-02 03:24:25.000000000 -0400 Modify: 2026-06-02 03:24:25.000000000 -0400 Change: 2026-06-02 03:24:26.000000000 -0400 Birth: 2026-06-02 03:24:26.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:24:30 (1780385070) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:24:32 (1780385072) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.357814 s, 114 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0373566 s, 110 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 2 03:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x211:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 2 03:24 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:24:38 (1780385078) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:24:44 (1780385084) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1832 sec =========== 03:24:50 (1780385090) === sanity-flr: start cleanup 03:24:51 (1780385091) === === sanity-flr: finish cleanup 03:24:57 (1780385097) ===