-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 01:22:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg149-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:23:19 (1788585799) === oleg149-client.virtnet: executing check_config_client /mnt/lustre oleg149-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg149-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8cae10a8d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cae10a8d800.idle_timeout=debug disable quota as required oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:23:58 (1788585838) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7646] == sanity-flr test 0a: lfs mirror create with -N option == 01:24:02 (1788585842) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:24:21 (1788585861) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:24:25 (1788585865) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:24:29 (1788585869) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:24:58 (1788585898) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:25:03 (1788585903) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:25:07 (1788585907) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8cae10a8d800.stats=clear osc.lustre-OST0001-osc-ffff8cae10a8d800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:25:16 (1788585916) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:25:28 (1788585928) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:25:36 (1788585936) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:25:40 (1788585940) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:25:47 (1788585947) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 01:25:57 (1788585957) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:26:05 (1788585965) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:26:10 (1788585970) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:26:19 (1788585979) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:26:31 (1788585991) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:26:38 (1788585998) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.07374 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180381 s, 5.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:26:49 (1788586009) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.206599 s, 35.5 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:26:58 (1788586018) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:27:10 (1788586030) Stopping /mnt/lustre-ost1 (opts:) on oleg149-server oleg149-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg149-server: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg149-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg149-server oleg149-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg149-server: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg149-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:28:33 (1788586113) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.557403 s, 18.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg149-server oleg149-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg149-server: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg149-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:29:19 (1788586159) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg149-server oleg149-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg149-server: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg149-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg149-server oleg149-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg149-server: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg149-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:30:47 (1788586247) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.67163 s, 11.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 01:30 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.490s user 0m0.001s sys 0m0.201s Stopping /mnt/lustre-ost1 (opts:) on oleg149-server oleg149-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cae10a8d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.306s user 0m0.001s sys 0m0.150s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg149-server: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg149-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg149-server oleg149-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.875s user 0m0.001s sys 0m0.185s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg149-server: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg149-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg149-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cae10a8d800.ost_server_uuid in FULL state after 0 sec PASS 33b (117s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:32:45 (1788586365) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:32:50 (1788586370) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (6s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:32:55 (1788586375) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 01:32:59 (1788586379) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 01:33:10 (1788586390) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.750805 s, 16.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:33:24 (1788586404) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039465 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418898 s, 25.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:33:32 (1788586412) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:33:42 (1788586422) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000853545 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00278213 s, 368 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00164989 s, 621 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00156169 s, 656 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 01:33:54 (1788586434) create test files with size 15739268 .. 4ca2bd123cc895f3b2ee009b30121720 - 145cf89f620e3207ca89cd110cddccc1 - 999610e5767d667025412d5f0bf8e24d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg149-client.virtnet /mnt/lustre (opts:) Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 01:34:21 (1788586461) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:34:39 (1788586479) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:34:47 (1788586487) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26707 s, 3.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721657 s, 14.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 01:35:00 (1788586500) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165767 s, 12.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.41192 s, 10.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190431 s, 11.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 01:35:20 (1788586520) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 01:35:24 (1788586524) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:35:27 (1788586527) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279112 s, 37.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 01:35:36 (1788586536) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:35:40 (1788586540) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:35:52 (1788586552) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.403942 s, 26.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg149-client.virtnet /mnt/lustre (opts:) Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:36:06 (1788586566) llite.lustre-ffff8cae2104e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.313591 s, 33.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg149-client.virtnet /mnt/lustre (opts:) Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8cae384a4000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:36:21 (1788586581) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:37:07 (1788586627) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:37:49 (1788586669) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 01:37:54 (1788586674) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.534528 s, 5.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.189744 s, 16.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf29b31c8 (nosync,stale) Mirror 2: 0xf29b31c8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: deae83bc8a7c8025814361534189f402 - mirror 1 checksum: 8067042cfab8ff6859850d0da9f4945d - mirror 2 checksum: deae83bc8a7c8025814361534189f402 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T01:37:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T01:37:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: deae83bc8a7c8025814361534189f402 - mirror 1 checksum: 8067042cfab8ff6859850d0da9f4945d - mirror 2 checksum: 8067042cfab8ff6859850d0da9f4945d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T01:37:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T01:37:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:38:11 (1788586691) Failing mds1 on oleg149-server Stopping /mnt/lustre-mds1 (opts:) on oleg149-server 01:38:20 (1788586700) shut down facet: mds1 facet_host: oleg149-server facet_failover_host: oleg149-server Failover mds1 to oleg149-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg149-server: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-MDT0000 01:38:55 (1788586735) targets are mounted 01:38:55 (1788586735) facet_failover done oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504119 s, 20.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:39:17 (1788586757) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:39:21 (1788586761) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:39:26 (1788586766) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0210967 s, 777 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:39:36 (1788586776) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:39:40 (1788586780) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0189001 s, 217 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.10065 s, 40.7 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:39:51 (1788586791) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:40:17 (1788586817) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:40:50 (1788586850) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:41:15 (1788586875) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:41:41 (1788586901) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 01:41:49 (1788586909) mirror create pid 54359 mirror split pid 54361 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54359 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54361 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 01:42:58 (1788586978) Seed set to 6980 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x13989 000002 1788586980.958167 trunc from 00000000 to 0x013988 (0x13989 bytes) 000003 1788586980.977237 punch from 0x00db9d to 0x0132f1 (0x5755 bytes) 000005 1788586980.983359 mapread 0x00f163 thru 0x013988 (0x4826 bytes) 000007 1788586981.006305 mapwrite 0x029385 thru 0x02f966 (0x65e2 bytes) 000010 1788586981.136000 mapwrite 0x001a36 thru 0x00f1de (0xd7a9 bytes) 000011 1788586981.330469 trunc from 0x02f967 to 0x009c54 (0x25d12 bytes) 000012 1788586981.352132 read 0x006c4e thru 0x009c54 (0x3007 bytes) 000013 1788586981.352806 mapread 0x00065f thru 0x0036c7 (0x3069 bytes) 000014 1788586981.362012 punch from 0x008547 to 0x009c54 (0x170e bytes) 000016 1788586981.368466 mapwrite 0x01da2b thru 0x0214be (0x3a94 bytes) 000017 1788586981.462508 read 0x01fbf7 thru 0x0214be (0x18c8 bytes) 000018 1788586981.463706 mapread 0x007502 thru 0x0154d3 (0xdfd2 bytes) 000020 1788586981.510166 MIRROR_EXTEND to 2 mirrors 000021 1788586982.569429 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000023 1788586982.748470 mapread 0x006c51 thru 0x007904 (0xcb4 bytes) 000024 1788586982.760213 punch from 0x010d69 to 0x012e8f (0x2127 bytes) truncating to largest ever: 0x2df0f 000025 1788586982.779103 trunc from 0x0214bf to 0x02df0e (0xca50 bytes) 000026 1788586982.795887 read 0x01dee2 thru 0x02d7c1 (0xf8e0 bytes) 000027 1788586982.813452 mapread 0x019a3e thru 0x01f422 (0x59e5 bytes) 000028 1788586982.822611 mapread 0x018374 thru 0x027556 (0xf1e3 bytes) 000030 1788586982.825362 write 0x021739 thru 0x023fcc (0x2894 bytes) 000032 1788586982.827824 mapwrite 0x038399 thru 0x03ffff (0x7c67 bytes) 000033 1788586982.956422 trunc from 0x040000 to 0x026b23 (0x194dc bytes) 000034 1788586982.991100 trunc from 0x026b24 to 0x016d26 (0xfdfd bytes) 000035 1788586983.014782 mapread 0x00fdf7 thru 0x0169fd (0x6c07 bytes) 000037 1788586983.024955 mapread 0x00da81 thru 0x00e140 (0x6c0 bytes) 000038 1788586983.026279 trunc from 0x016d27 to 0x021d10 (0xafea bytes) 000040 1788586983.041719 punch from 0x01e16f to 0x021d10 (0x3ba2 bytes) 000043 1788586983.048069 trunc from 0x021d11 to 0x020e22 (0xeee bytes) truncating to largest ever: 0x3d3ec 000048 1788586983.069853 trunc from 0x020e23 to 0x03d3eb (0x1c5c9 bytes) 000050 1788586983.098415 mapwrite 0x028b0a thru 0x0382fb (0xf7f2 bytes) 000051 1788586983.250944 write 0x037bb5 thru 0x03ffff (0x844b bytes) 000052 1788586983.257464 mapread 0x03aa94 thru 0x03c523 (0x1a90 bytes) 000053 1788586983.262563 write 0x0342eb thru 0x038900 (0x4616 bytes) 000054 1788586983.265769 write 0x01e703 thru 0x0279d2 (0x92d0 bytes) 000057 1788586983.286539 read 0x03030c thru 0x03e96b (0xe660 bytes) 000058 1788586983.288021 mapwrite 0x0216cb thru 0x02e997 (0xd2cd bytes) 000059 1788586983.476328 write 0x03a3b9 thru 0x03ac95 (0x8dd bytes) 000060 1788586983.479084 punch from 0x012f65 to 0x02293b (0xf9d7 bytes) 000062 1788586983.493897 punch from 0x020357 to 0x02a358 (0xa002 bytes) 000064 1788586983.500075 mapread 0x03d45d thru 0x03ffff (0x2ba3 bytes) 000066 1788586983.501477 write 0x024bfc thru 0x02aaaa (0x5eaf bytes) 000068 1788586983.514757 mapwrite 0x037c2e thru 0x03ffff (0x83d2 bytes) 000069 1788586983.582197 mapread 0x038331 thru 0x039622 (0x12f2 bytes) 000071 1788586983.584901 write 0x02e1af thru 0x039398 (0xb1ea bytes) 000072 1788586983.586915 mapread 0x007cb5 thru 0x00c866 (0x4bb2 bytes) 000073 1788586983.587953 write 0x007092 thru 0x013f07 (0xce76 bytes) 000074 1788586983.590809 punch from 0x035d89 to 0x03f1b5 (0x942d bytes) 000076 1788586983.614414 write 0x012096 thru 0x013414 (0x137f bytes) 000077 1788586983.615926 punch from 0x029640 to 0x03577b (0xc13c bytes) 000078 1788586983.626288 read 0x031d55 thru 0x036299 (0x4545 bytes) 000080 1788586983.638071 trunc from 0x040000 to 0x0077a4 (0x3885b bytes) 000081 1788586983.660431 punch from 0x0057c4 to 0x0077a4 (0x1fe1 bytes) 000083 1788586983.667474 mapwrite 0x023caa thru 0x02de3c (0xa193 bytes) 000084 1788586983.878403 read 0x010ca5 thru 0x01ecd5 (0xe031 bytes) 000085 1788586983.890425 write 0x012291 thru 0x01f693 (0xd403 bytes) 000086 1788586983.891693 mapread 0x01911f thru 0x028fa3 (0xfe85 bytes) 000087 1788586983.892535 mapwrite 0x027df3 thru 0x037268 (0xf476 bytes) 000090 1788586984.079532 write 0x021d1a thru 0x022c67 (0xf4e bytes) 000092 1788586984.083593 write 0x004777 thru 0x00d5eb (0x8e75 bytes) 000093 1788586984.085982 read 0x01a7f6 thru 0x01d65a (0x2e65 bytes) 000095 1788586984.087143 punch from 0x003835 to 0x0077d5 (0x3fa1 bytes) 000096 1788586984.102273 mapwrite 0x01acf6 thru 0x021f37 (0x7242 bytes) 000098 1788586984.206157 mapread 0x0319cf thru 0x0354fa (0x3b2c bytes) 000100 1788586984.207482 mapwrite 0x0171da thru 0x019456 (0x227d bytes) 000101 1788586984.255495 write 0x000f0f thru 0x00596a (0x4a5c bytes) 000102 1788586984.263130 write 0x003414 thru 0x005757 (0x2344 bytes) 000104 1788586984.264794 trunc from 0x037269 to 0x0067e2 (0x30a86 bytes) 000107 1788586984.291264 trunc from 0x0067e3 to 0x0135f7 (0xce15 bytes) 000109 1788586984.312294 trunc from 0x0135f8 to 0x01d24b (0x9c54 bytes) 000110 1788586984.331262 write 0x000abe thru 0x00318a (0x26cd bytes) 000111 1788586984.332293 mapwrite 0x011bd1 thru 0x0130bf (0x14ef bytes) 000112 1788586984.397419 mapwrite 0x027230 thru 0x02ea38 (0x7809 bytes) truncating to largest ever: 0x3f821 000113 1788586984.470487 trunc from 0x02ea39 to 0x03f820 (0x10de8 bytes) 000115 1788586984.490523 read 0x001489 thru 0x010c4f (0xf7c7 bytes) 000116 1788586984.501482 MIRROR_EXTEND to 2 mirrors 000117 1788586985.693236 read 0x015293 thru 0x022acf (0xd83d bytes) 000118 1788586985.729297 punch from 0x00cf5c to 0x01bd35 (0xedda bytes) 000119 1788586985.894475 read 0x0218ff thru 0x02772a (0x5e2c bytes) 000120 1788586985.924347 mapread 0x029add thru 0x0351f2 (0xb716 bytes) 000123 1788586985.926484 read 0x037ee0 thru 0x03b403 (0x3524 bytes) 000128 1788586985.932097 mapread 0x02d890 thru 0x038892 (0xb003 bytes) 000129 1788586985.933120 read 0x035b28 thru 0x03be29 (0x6302 bytes) 000130 1788586985.934564 mapwrite 0x039c3c thru 0x03dec3 (0x4288 bytes) 000131 1788586986.008357 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000133 1788586986.144201 read 0x015d76 thru 0x016939 (0xbc4 bytes) 000135 1788586986.150332 punch from 0x030fd9 to 0x03421c (0x3244 bytes) 000137 1788586986.159033 write 0x03d7b1 thru 0x03ffff (0x284f bytes) 000138 1788586986.165412 punch from 0x01fe1a to 0x0294b9 (0x96a0 bytes) 000139 1788586986.175411 mapwrite 0x02220b thru 0x022cb1 (0xaa7 bytes) 000140 1788586986.223080 read 0x018f87 thru 0x026d73 (0xdded bytes) 000141 1788586986.240216 mapread 0x02e4d1 thru 0x0306e7 (0x2217 bytes) 000142 1788586986.241087 punch from 0x014f00 to 0x015bd5 (0xcd6 bytes) 000143 1788586986.247459 write 0x019f08 thru 0x01d853 (0x394c bytes) 000144 1788586986.248552 write 0x01e6a9 thru 0x022720 (0x4078 bytes) 000145 1788586986.251051 mapwrite 0x024cc1 thru 0x0300f7 (0xb437 bytes) 000149 1788586986.321011 mapwrite 0x024daf thru 0x028617 (0x3869 bytes) 000150 1788586986.362468 punch from 0x0301d1 to 0x036fb8 (0x6de8 bytes) 000152 1788586986.368365 write 0x03f8be thru 0x03ffff (0x742 bytes) 000153 1788586986.372753 trunc from 0x040000 to 0x01f6fb (0x20904 bytes) 000154 1788586986.393694 mapread 0x01e410 thru 0x01f6fb (0x12ec bytes) 000155 1788586986.394520 mapwrite 0x021c3a thru 0x027e67 (0x622e bytes) 000156 1788586986.495562 write 0x03f63b thru 0x03ffff (0x9c5 bytes) 000157 1788586986.497900 mapread 0x03f8e0 thru 0x03ffff (0x720 bytes) 000158 1788586986.498686 mapread 0x02100d thru 0x024cf5 (0x3ce9 bytes) 000159 1788586986.500147 mapread 0x03a4af thru 0x03ffff (0x5b51 bytes) 000161 1788586986.534526 read 0x03bfc3 thru 0x03ffff (0x403d bytes) 000162 1788586986.536014 punch from 0x00c72e to 0x012ff5 (0x68c8 bytes) 000163 1788586986.547320 punch from 0x0363f0 to 0x03aa71 (0x4682 bytes) 000164 1788586986.553406 write 0x00b01c thru 0x00d477 (0x245c bytes) 000166 1788586986.591122 punch from 0x03093a to 0x036b36 (0x61fd bytes) 000167 1788586986.602393 trunc from 0x040000 to 0x008259 (0x37da6 bytes) 000168 1788586986.623285 read 0x007a58 thru 0x008259 (0x802 bytes) 000169 1788586986.623918 write 0x011e08 thru 0x01b023 (0x921c bytes) 000171 1788586986.627562 trunc from 0x01b024 to 0x0113c6 (0x9c5d bytes) 000173 1788586986.651095 mapwrite 0x017cb5 thru 0x019f1e (0x226a bytes) 000174 1788586986.734008 mapread 0x00ed55 thru 0x012ee7 (0x4193 bytes) 000175 1788586986.735285 write 0x013d22 thru 0x01aac2 (0x6da1 bytes) 000176 1788586986.738012 mapwrite 0x027174 thru 0x02ea9b (0x7928 bytes) 000177 1788586986.856481 write 0x00bb92 thru 0x00f252 (0x36c1 bytes) 000180 1788586986.863358 write 0x0227d8 thru 0x02f0ed (0xc916 bytes) 000182 1788586986.866299 trunc from 0x02f0ee to 0x01dda0 (0x1134d bytes) 000183 1788586986.901291 mapread 0x00e7d6 thru 0x01121e (0x2a49 bytes) 000184 1788586986.903057 read 0x01a888 thru 0x01dda0 (0x3519 bytes) 000186 1788586986.904637 write 0x00125d thru 0x00bf9d (0xad41 bytes) 000187 1788586986.914219 MIRROR_EXTEND to 2 mirrors 000188 1788586988.009964 mapwrite 0x030b17 thru 0x0399d5 (0x8ebf bytes) 000189 1788586988.124875 mapread 0x02be17 thru 0x02e62a (0x2814 bytes) 000190 1788586988.125557 punch from 0x019f87 to 0x01d4d9 (0x3553 bytes) 000193 1788586988.131295 mapwrite 0x00e3ca thru 0x0106ee (0x2325 bytes) 000195 1788586988.161337 mapread 0x03239e thru 0x0399d5 (0x7638 bytes) 000198 1788586988.162714 write 0x03a60c thru 0x03ffff (0x59f4 bytes) 000199 1788586988.163494 mapwrite 0x030546 thru 0x032f7d (0x2a38 bytes) 000200 1788586988.205866 mapwrite 0x0200aa thru 0x02ae6c (0xadc3 bytes) 000201 1788586988.264250 punch from 0x01bf8d to 0x0294e1 (0xd555 bytes) 000203 1788586988.270047 mapwrite 0x03ef9e thru 0x03ffff (0x1062 bytes) 000204 1788586988.304256 read 0x010b87 thru 0x0198e9 (0x8d63 bytes) 000206 1788586988.312210 mapread 0x011607 thru 0x01b492 (0x9e8c bytes) 000207 1788586988.323132 mapread 0x03f52c thru 0x03ffff (0xad4 bytes) 000208 1788586988.334195 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000209 1788586988.603754 punch from 0x023747 to 0x02cdc5 (0x967f bytes) 000211 1788586988.610658 punch from 0x0300c8 to 0x03803e (0x7f77 bytes) 000212 1788586988.621844 MIRROR_EXTEND to 2 mirrors 000214 1788586990.014401 mapread 0x0128e7 thru 0x01efc6 (0xc6e0 bytes) 000215 1788586990.076652 punch from 0x00269a to 0x0037f7 (0x115e bytes) 000218 1788586990.165506 trunc from 0x040000 to 0x02abd3 (0x1542c bytes) 000219 1788586990.192713 mapread 0x01d447 thru 0x028a07 (0xb5c1 bytes) 000220 1788586990.213979 write 0x005bf7 thru 0x007a2d (0x1e37 bytes) 000222 1788586990.224419 read 0x00dfd4 thru 0x016496 (0x84c3 bytes) 000226 1788586990.254319 mapwrite 0x003dae thru 0x004597 (0x7ea bytes) 000227 1788586990.362065 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2abd4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2abd4): Mirror 2: 0xdbe9314b (stale) CRC-32 checksum value for chunk [0, 0x2abd4): Mirror 1: 0x14dc8f2a Mirror 3: 0x14dc8f2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2abd4: skipped 000228 1788586991.797983 punch from 0x018854 to 0x01c528 (0x3cd5 bytes) 000229 1788586991.868079 mapwrite 0x03080c thru 0x03aacf (0xa2c4 bytes) 000231 1788586991.958046 punch from 0x0188b2 to 0x01f71c (0x6e6b bytes) 000233 1788586991.967917 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3aad0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aad0): Mirror 2: 0xcb57272a (stale) Mirror 3: 0x8a0f8b84 (stale) CRC-32 checksum value for chunk [0, 0x3aad0): Mirror 1: 0xebd0315 Mirror 4: 0xebd0315 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aad0: skipped 000235 1788586993.371927 read 0x01c4ab thru 0x02379a (0x72f0 bytes) 000236 1788586993.389018 mapwrite 0x0229a0 thru 0x0289c7 (0x6028 bytes) 000238 1788586993.570936 write 0x0157bd thru 0x016828 (0x106c bytes) 000239 1788586993.582105 mapread 0x03715c thru 0x03aacf (0x3974 bytes) 000240 1788586993.595429 MIRROR_RESYNC 4 mirrors 000241 1788586995.443032 mapwrite 0x037bdc thru 0x039c3b (0x2060 bytes) 000242 1788586995.601420 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3aad0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aad0): Mirror 2: 0x32d5e765 (stale) Mirror 3: 0x32d5e765 (stale) Mirror 4: 0x32d5e765 (stale) CRC-32 checksum value for chunk [0, 0x3aad0): Mirror 1: 0xa09a9a62 Mirror 5: 0xa09a9a62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aad0: skipped 000243 1788586996.635115 read 0x036a9e thru 0x03aacf (0x4032 bytes) 000244 1788586996.650072 mapread 0x01cbd5 thru 0x02a048 (0xd474 bytes) 000245 1788586996.704307 mapread 0x004467 thru 0x013a6b (0xf605 bytes) 000246 1788586996.745555 trunc from 0x03aad0 to 0x002dd6 (0x37cf9 bytes) 000247 1788586996.857403 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd7): Mirror 2: 0xd02fa75f (stale) Mirror 4: 0xd02fa75f (stale) Mirror 5: 0xd02fa75f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd7): Mirror 2: 0xd02fa75f (stale) Mirror 4: 0xd02fa75f (stale) Mirror 5: 0xd02fa75f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd7: skipped 000252 1788586997.060253 mapwrite 0x027a0e thru 0x02c49a (0x4a8d bytes) 000253 1788586997.136408 mapwrite 0x039d84 thru 0x03ffb1 (0x622e bytes) 000254 1788586997.265248 trunc from 0x03ffb2 to 0x022b20 (0x1d491 bytes) 000255 1788586997.288327 mapread 0x01a43c thru 0x022b20 (0x86e5 bytes) 000256 1788586997.308036 mapread 0x018fc9 thru 0x01a0ca (0x1102 bytes) 000257 1788586997.309191 punch from 0x00b5a6 to 0x00cdc8 (0x1823 bytes) 000258 1788586997.315379 read 0x0088d2 thru 0x014aec (0xc21b bytes) 000259 1788586997.324168 trunc from 0x022b21 to 0x023c44 (0x1124 bytes) 000260 1788586997.348599 punch from 0x022eea to 0x02336b (0x482 bytes) 000261 1788586997.356439 mapwrite 0x026810 thru 0x0290b1 (0x28a2 bytes) 000262 1788586997.441241 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x290b2): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x290b2): Mirror 2: 0x33484cc5 (stale) Mirror 4: 0x33484cc5 (stale) Mirror 5: 0x33484cc5 (stale) CRC-32 checksum value for chunk [0, 0x290b2): Mirror 1: 0x8728fe20 Mirror 6: 0x8728fe20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x290b2: skipped 000265 1788586998.754010 mapread 0x016ea1 thru 0x019e06 (0x2f66 bytes) 000267 1788586998.812530 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x290b2): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x290b2): Mirror 2: 0x33484cc5 (stale) Mirror 4: 0x33484cc5 (stale) Mirror 5: 0x33484cc5 (stale) CRC-32 checksum value for chunk [0, 0x290b2): Mirror 1: 0x8728fe20 Mirror 6: 0x8728fe20 Mirror 7: 0x8728fe20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x290b2: skipped 000269 1788587000.173912 mapwrite 0x01a253 thru 0x0242d9 (0xa087 bytes) 000273 1788587000.491930 mapread 0x010df4 thru 0x020b16 (0xfd23 bytes) 000275 1788587000.504743 mapread 0x02409c thru 0x0290b1 (0x5016 bytes) 000276 1788587000.506294 read 0x015216 thru 0x024d44 (0xfb2f bytes) 000277 1788587000.507472 mapwrite 0x037abb thru 0x03ffff (0x8545 bytes) 000278 1788587000.752825 MIRROR_RESYNC 6 mirrors 000279 1788587003.468779 trunc from 0x040000 to 0x01cf11 (0x230ee bytes) 000280 1788587003.615755 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1cf12): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf12): Mirror 2: 0xa30ecbe2 (stale) Mirror 4: 0xa30ecbe2 (stale) Mirror 5: 0xa30ecbe2 (stale) Mirror 6: 0xa30ecbe2 (stale) Mirror 7: 0xa30ecbe2 (stale) CRC-32 checksum value for chunk [0, 0x1cf12): Mirror 1: 0xa30ecbe2 Mirror 8: 0xa30ecbe2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf12: skipped 000281 1788587004.968869 read 0x013b61 thru 0x019e0e (0x62ae bytes) 000282 1788587005.010721 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1cf12): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf12): Mirror 2: 0xa30ecbe2 (stale) Mirror 4: 0xa30ecbe2 (stale) Mirror 5: 0xa30ecbe2 (stale) Mirror 6: 0xa30ecbe2 (stale) Mirror 7: 0xa30ecbe2 (stale) CRC-32 checksum value for chunk [0, 0x1cf12): Mirror 1: 0xa30ecbe2 Mirror 8: 0xa30ecbe2 Mirror 9: 0xa30ecbe2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf12: skipped 000283 1788587006.413788 mapwrite 0x032847 thru 0x0381d5 (0x598f bytes) 000285 1788587006.635751 read 0x02204c thru 0x024c51 (0x2c06 bytes) 000287 1788587006.645566 read 0x019b05 thru 0x025d52 (0xc24e bytes) 000288 1788587006.659315 read 0x00b0a5 thru 0x00b278 (0x1d4 bytes) 000289 1788587006.660872 trunc from 0x0381d6 to 0x020740 (0x17a95 bytes) 000290 1788587006.698707 MIRROR_RESYNC 8 mirrors 000292 1788587009.165093 mapread 0x007d2e thru 0x011fd2 (0xa2a5 bytes) 000293 1788587009.191458 punch from 0x01732a to 0x020740 (0x9417 bytes) 000294 1788587009.318980 read 0x01f115 thru 0x020740 (0x162c bytes) 000295 1788587009.329370 mapwrite 0x016d11 thru 0x01c20e (0x54fe bytes) 000298 1788587009.528825 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x20741): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20741): Mirror 2: 0x6bcd4d1e (stale) Mirror 4: 0x6bcd4d1e (stale) Mirror 5: 0x6bcd4d1e (stale) Mirror 6: 0x6bcd4d1e (stale) Mirror 7: 0x6bcd4d1e (stale) Mirror 8: 0x6bcd4d1e (stale) Mirror 9: 0x6bcd4d1e (stale) CRC-32 checksum value for chunk [0, 0x20741): Mirror 1: 0x136b7192 Mirror 10: 0x136b7192 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20741: skipped 000300 1788587010.901151 read 0x00920a thru 0x013f36 (0xad2d bytes) 000301 1788587010.920904 trunc from 0x020741 to 0x02a1ac (0x9a6c bytes) 000302 1788587011.072912 write 0x018f8e thru 0x027e9d (0xef10 bytes) 000304 1788587011.092039 read 0x020b49 thru 0x022602 (0x1aba bytes) 000305 1788587011.108196 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a1ad): Mirror 2: 0x57d6fc63 (stale) Mirror 4: 0x57d6fc63 (stale) Mirror 6: 0x57d6fc63 (stale) Mirror 7: 0x57d6fc63 (stale) Mirror 8: 0x57d6fc63 (stale) Mirror 9: 0x57d6fc63 (stale) Mirror 10: 0x3337d978 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1ad): Mirror 2: 0x57d6fc63 (stale) Mirror 4: 0x57d6fc63 (stale) Mirror 6: 0x57d6fc63 (stale) Mirror 7: 0x57d6fc63 (stale) Mirror 8: 0x57d6fc63 (stale) Mirror 9: 0x57d6fc63 (stale) Mirror 10: 0x3337d978 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1ad: skipped 000308 1788587011.942783 mapread 0x027be2 thru 0x027d3f (0x15e bytes) 000310 1788587011.955428 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2a1ad): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1ad): Mirror 2: 0x57d6fc63 (stale) Mirror 4: 0x57d6fc63 (stale) Mirror 6: 0x57d6fc63 (stale) Mirror 7: 0x57d6fc63 (stale) Mirror 8: 0x57d6fc63 (stale) Mirror 9: 0x57d6fc63 (stale) Mirror 10: 0x3337d978 (stale) CRC-32 checksum value for chunk [0, 0x2a1ad): Mirror 1: 0x48f26dff Mirror 11: 0x48f26dff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1ad: skipped 000313 1788587013.329862 MIRROR_RESYNC 9 mirrors 000314 1788587016.260522 write 0x02356c thru 0x029d7f (0x6814 bytes) 000315 1788587016.372228 punch from 0x01d0aa to 0x022d7b (0x5cd2 bytes) 000316 1788587016.388935 mapwrite 0x00af5d thru 0x00f649 (0x46ed bytes) 000317 1788587016.469345 trunc from 0x02a1ad to 0x00b672 (0x1eb3a bytes) 000318 1788587016.492623 mapwrite 0x0294a7 thru 0x0378e3 (0xe43d bytes) 000321 1788587016.606854 punch from 0x0162f9 to 0x021903 (0xb60b bytes) 000322 1788587016.612603 mapread 0x0156c8 thru 0x01fcbd (0xa5f6 bytes) 000323 1788587016.641911 MIRROR_RESYNC 9 mirrors 000326 1788587019.768293 trunc from 0x0378e4 to 0x023711 (0x141d2 bytes) 000327 1788587019.905246 trunc from 0x023712 to 0x02921f (0x5b0e bytes) 000328 1788587019.944040 mapwrite 0x008622 thru 0x009094 (0xa73 bytes) 000329 1788587020.005977 mapread 0x0096c2 thru 0x014c45 (0xb584 bytes) 000330 1788587020.029798 write 0x01fa06 thru 0x02ee4c (0xf447 bytes) 000333 1788587020.041761 read 0x013725 thru 0x013ff1 (0x8cd bytes) 000336 1788587020.044119 read 0x01f695 thru 0x02261f (0x2f8b bytes) 000337 1788587020.044802 read 0x021295 thru 0x0263e4 (0x5150 bytes) 000338 1788587020.045906 punch from 0x00a04b to 0x013115 (0x90cb bytes) 000339 1788587020.061525 write 0x035609 thru 0x03ffff (0xa9f7 bytes) 000340 1788587020.063127 read 0x01ffaf thru 0x02eb9e (0xebf0 bytes) 000341 1788587020.063802 punch from 0x0247ab to 0x0342ce (0xfb24 bytes) 000342 1788587020.084086 punch from 0x028cde to 0x034bcf (0xbef2 bytes) 000343 1788587020.089685 mapread 0x01f53d thru 0x028608 (0x90cc bytes) 000344 1788587020.103331 trunc from 0x040000 to 0x033617 (0xc9e8 bytes) 000345 1788587020.134790 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33618): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33618): Mirror 2: 0x7800c06a (stale) Mirror 4: 0x7800c06a (stale) Mirror 6: 0x7800c06a (stale) Mirror 7: 0x7800c06a (stale) Mirror 8: 0x7800c06a (stale) Mirror 9: 0x7800c06a (stale) Mirror 10: 0x7800c06a (stale) Mirror 11: 0x7800c06a (stale) CRC-32 checksum value for chunk [0, 0x33618): Mirror 1: 0x1539cdf6 Mirror 12: 0x1539cdf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33618: skipped 000346 1788587021.842399 punch from 0x01cfa0 to 0x023ffa (0x705b bytes) 000348 1788587021.977813 write 0x028756 thru 0x033c6b (0xb516 bytes) 000349 1788587021.992499 mapwrite 0x02bba2 thru 0x02e0b1 (0x2510 bytes) 000351 1788587022.071446 punch from 0x000efe to 0x0095b2 (0x86b5 bytes) 000353 1788587022.080319 punch from 0x003859 to 0x00633b (0x2ae3 bytes) 000354 1788587022.088422 punch from 0x02d58d to 0x033c6b (0x66df bytes) 000355 1788587022.099885 punch from 0x02bfe3 to 0x0310a0 (0x50be bytes) 000357 1788587022.110478 punch from 0x00355a to 0x006580 (0x3027 bytes) 000358 1788587022.119843 read 0x008ecc thru 0x013e4b (0xaf80 bytes) 000359 1788587022.134542 punch from 0x015cd8 to 0x018ec1 (0x31ea bytes) 000360 1788587022.160354 trunc from 0x033c6c to 0x007e85 (0x2bde6 bytes) 000361 1788587022.196256 punch from 0x004273 to 0x007e85 (0x3c13 bytes) 000362 1788587022.201972 write 0x01db79 thru 0x02aae1 (0xcf69 bytes) 000364 1788587022.206233 trunc from 0x02aae2 to 0x007f03 (0x22bde bytes) 000366 1788587022.241470 MIRROR_RESYNC 10 mirrors 000367 1788587024.149268 trunc from 0x007f04 to 0x00baac (0x3ba9 bytes) 000368 1788587024.243849 mapwrite 0x024f7c thru 0x028e1d (0x3ea2 bytes) 000370 1788587024.347916 mapread 0x022767 thru 0x028e1d (0x66b7 bytes) 000371 1788587024.349787 trunc from 0x028e1e to 0x0186bb (0x10762 bytes) 000372 1788587024.379248 trunc from 0x0186bc to 0x03cd96 (0x246db bytes) 000373 1788587024.402490 trunc from 0x03cd97 to 0x03d6ba (0x924 bytes) 000374 1788587024.428360 punch from 0x029e8c to 0x02d330 (0x34a5 bytes) 000375 1788587024.453370 punch from 0x01d14b to 0x02ae87 (0xdd3d bytes) 000377 1788587024.475522 punch from 0x026e08 to 0x02bd6b (0x4f64 bytes) 000379 1788587024.480899 write 0x01bd41 thru 0x028cf6 (0xcfb6 bytes) 000381 1788587024.493005 punch from 0x020d18 to 0x02fe8b (0xf174 bytes) 000382 1788587024.518560 read 0x011530 thru 0x01aa66 (0x9537 bytes) 000385 1788587024.539468 mapread 0x01f70b thru 0x021f1b (0x2811 bytes) 000386 1788587024.548865 punch from 0x011607 to 0x01bbc7 (0xa5c1 bytes) 000387 1788587024.553566 punch from 0x017f9c to 0x018aea (0xb4f bytes) 000389 1788587024.559444 mapread 0x028dfd thru 0x02f720 (0x6924 bytes) 000390 1788587024.568906 mapread 0x001197 thru 0x00d968 (0xc7d2 bytes) 000391 1788587024.583795 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d6bb): Mirror 2: 0xbe61f454 (stale) Mirror 4: 0xbe61f454 (stale) Mirror 6: 0xbe61f454 (stale) Mirror 7: 0xbe61f454 (stale) Mirror 8: 0xbe61f454 (stale) Mirror 9: 0xbe61f454 (stale) Mirror 10: 0xbe61f454 (stale) Mirror 11: 0xbe61f454 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6bb): Mirror 2: 0xbe61f454 (stale) Mirror 4: 0xbe61f454 (stale) Mirror 6: 0xbe61f454 (stale) Mirror 7: 0xbe61f454 (stale) Mirror 8: 0xbe61f454 (stale) Mirror 9: 0xbe61f454 (stale) Mirror 10: 0xbe61f454 (stale) Mirror 11: 0xbe61f454 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6bb: skipped 000392 1788587025.288496 mapread 0x01a5ce thru 0x01c9dc (0x240f bytes) 000394 1788587025.309978 punch from 0x0034bb to 0x00abe0 (0x7726 bytes) 000396 1788587025.319113 write 0x009600 thru 0x00c2af (0x2cb0 bytes) 000397 1788587025.333227 trunc from 0x03d6bb to 0x00c643 (0x31077 bytes) 000398 1788587025.358671 write 0x004e3f thru 0x006adc (0x1c9e bytes) 000401 1788587025.374231 punch from 0x0058b9 to 0x008206 (0x294e bytes) 000402 1788587025.404467 write 0x028e12 thru 0x02b20f (0x23fe bytes) 000403 1788587025.409483 write 0x0317a1 thru 0x039bb6 (0x8416 bytes) 000406 1788587025.419919 trunc from 0x039bb7 to 0x0214e2 (0x186d4 bytes) 000410 1788587025.453481 write 0x01d02e thru 0x024ffb (0x7fce bytes) 000411 1788587025.464071 trunc from 0x024ffc to 0x0031fb (0x21e00 bytes) 000412 1788587025.496618 mapread 0x000de0 thru 0x0031fb (0x241c bytes) 000413 1788587025.505620 read 0x001c28 thru 0x0031fb (0x15d4 bytes) 000414 1788587025.508441 write 0x02a5fd thru 0x0381a5 (0xdba9 bytes) 000417 1788587025.520755 write 0x02f144 thru 0x03abb1 (0xba6e bytes) 000418 1788587025.523557 mapwrite 0x0175f4 thru 0x01be79 (0x4886 bytes) 000420 1788587025.599461 read 0x02b253 thru 0x038de6 (0xdb94 bytes) 000421 1788587025.600802 mapwrite 0x00aeb3 thru 0x015c30 (0xad7e bytes) 000422 1788587025.706112 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3abb2): Mirror 2: 0x15cebf46 (stale) Mirror 6: 0x15cebf46 (stale) Mirror 7: 0x15cebf46 (stale) Mirror 8: 0x15cebf46 (stale) Mirror 9: 0x15cebf46 (stale) Mirror 10: 0x15cebf46 (stale) Mirror 11: 0x15cebf46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3abb2): Mirror 2: 0x15cebf46 (stale) Mirror 6: 0x15cebf46 (stale) Mirror 7: 0x15cebf46 (stale) Mirror 8: 0x15cebf46 (stale) Mirror 9: 0x15cebf46 (stale) Mirror 10: 0x15cebf46 (stale) Mirror 11: 0x15cebf46 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3abb2: skipped 000424 1788587026.524943 read 0x035dfe thru 0x03abb1 (0x4db4 bytes) 000426 1788587026.535867 punch from 0x02b28b to 0x03605a (0xadd0 bytes) 000428 1788587026.542232 read 0x00143e thru 0x00ecf0 (0xd8b3 bytes) 000429 1788587026.556743 mapwrite 0x019993 thru 0x01fd6d (0x63db bytes) 000430 1788587026.657560 punch from 0x034f51 to 0x038040 (0x30f0 bytes) 000431 1788587026.666399 mapread 0x0308ac thru 0x0345f4 (0x3d49 bytes) 000432 1788587026.680198 write 0x010488 thru 0x0123cb (0x1f44 bytes) 000434 1788587026.686368 mapwrite 0x02fc20 thru 0x036378 (0x6759 bytes) 000435 1788587026.886268 read 0x01488b thru 0x017546 (0x2cbc bytes) 000437 1788587026.897776 MIRROR_RESYNC 8 mirrors 000438 1788587032.870376 write 0x03b797 thru 0x03ffff (0x4869 bytes) 000439 1788587032.963783 read 0x00d2e4 thru 0x01390d (0x662a bytes) 000441 1788587032.973497 trunc from 0x040000 to 0x0164b7 (0x29b48 bytes) 000442 1788587032.994512 punch from 0x012063 to 0x0164b7 (0x4455 bytes) 000443 1788587032.999498 punch from 0x00cac9 to 0x0164b7 (0x99ef bytes) 000446 1788587033.015342 MIRROR_RESYNC 8 mirrors 000447 1788587035.899665 write 0x0294ad thru 0x029fd2 (0xb26 bytes) 000448 1788587036.054029 trunc from 0x029fd3 to 0x0065be (0x23a14 bytes) 000449 1788587036.090389 punch from 0x0051ae to 0x006532 (0x1385 bytes) 000451 1788587036.101787 read 0x002a88 thru 0x0065be (0x3b37 bytes) 000452 1788587036.124408 read 0x005e6e thru 0x0065be (0x751 bytes) 000453 1788587036.125349 read 0x001967 thru 0x0056b2 (0x3d4c bytes) 000460 1788587036.152632 write 0x014a70 thru 0x017eb8 (0x3449 bytes) 000461 1788587036.160563 mapwrite 0x036af8 thru 0x03cf54 (0x645d bytes) 000462 1788587036.363006 read 0x03af4f thru 0x03cf54 (0x2006 bytes) 000463 1788587036.365371 punch from 0x02475b to 0x0339b4 (0xf25a bytes) 000464 1788587036.379304 read 0x016f33 thru 0x01a22d (0x32fb bytes) 000465 1788587036.396893 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cf55): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf55): Mirror 2: 0xef526b1e (stale) Mirror 6: 0xef526b1e (stale) Mirror 7: 0xef526b1e (stale) Mirror 8: 0xef526b1e (stale) Mirror 9: 0xef526b1e (stale) Mirror 10: 0xef526b1e (stale) Mirror 11: 0xef526b1e (stale) CRC-32 checksum value for chunk [0, 0x3cf55): Mirror 1: 0x293174fc Mirror 12: 0x293174fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf55: skipped 000466 1788587038.376107 mapwrite 0x006425 thru 0x00de78 (0x7a54 bytes) 000470 1788587038.657872 write 0x03b16b thru 0x03b6fd (0x593 bytes) 000473 1788587038.667991 punch from 0x00baec to 0x017abb (0xbfd0 bytes) 000477 1788587038.682469 mapwrite 0x010943 thru 0x01b3e1 (0xaa9f bytes) 000479 1788587038.781475 punch from 0x010189 to 0x01084b (0x6c3 bytes) 000480 1788587038.789907 punch from 0x03056f to 0x03cf54 (0xc9e6 bytes) 000481 1788587038.796502 write 0x020efa thru 0x028b8a (0x7c91 bytes) 000482 1788587038.805712 trunc from 0x03cf55 to 0x01a179 (0x22ddb bytes) 000483 1788587038.840984 trunc from 0x01a17a to 0x039512 (0x1f399 bytes) 000484 1788587038.871649 punch from 0x034e74 to 0x039512 (0x469f bytes) 000485 1788587038.881171 read 0x010b92 thru 0x0163cb (0x583a bytes) 000486 1788587038.888015 punch from 0x020cb8 to 0x026f15 (0x625e bytes) 000487 1788587038.896269 mapread 0x01cb18 thru 0x01d931 (0xe1a bytes) 000488 1788587038.909984 read 0x01d02b thru 0x027160 (0xa136 bytes) 000489 1788587038.923802 punch from 0x00be39 to 0x0106b4 (0x487c bytes) 000490 1788587038.932069 mapread 0x000b96 thru 0x0102db (0xf746 bytes) 000491 1788587038.982455 read 0x03593a thru 0x039512 (0x3bd9 bytes) 000492 1788587038.983152 mapwrite 0x018fd5 thru 0x0285a3 (0xf5cf bytes) 000493 1788587039.081495 trunc from 0x039513 to 0x0354d8 (0x403a bytes) 000497 1788587039.109369 mapwrite 0x0379ba thru 0x037bae (0x1f5 bytes) 000498 1788587039.346780 write 0x00ddaf thru 0x01ae5d (0xd0af bytes) 000499 1788587039.352402 punch from 0x00ffba to 0x0141fe (0x4245 bytes) 000500 1788587039.398119 mapwrite 0x03e4a2 thru 0x03ffff (0x1b5e bytes) 000501 1788587039.530649 read 0x037b3e thru 0x03ffff (0x84c2 bytes) 000502 1788587039.531461 write 0x0171af thru 0x01e8dd (0x772f bytes) 000503 1788587039.541254 trunc from 0x040000 to 0x02c165 (0x13e9a bytes) 000504 1788587039.567117 trunc from 0x02c166 to 0x017a04 (0x14761 bytes) 000505 1788587039.591479 trunc from 0x017a05 to 0x031ace (0x1a0ca bytes) 000506 1788587039.623256 trunc from 0x031acf to 0x00aa90 (0x2703e bytes) 000507 1788587039.645412 read 0x0052a8 thru 0x00aa90 (0x57e9 bytes) 000508 1788587039.647310 trunc from 0x00aa91 to 0x0003eb (0xa6a5 bytes) 000510 1788587039.673415 mapwrite 0x009709 thru 0x00f897 (0x618f bytes) 000511 1788587039.798485 mapwrite 0x013087 thru 0x0169d6 (0x3950 bytes) 000512 1788587039.869787 write 0x017d6f thru 0x01fbe8 (0x7e7a bytes) 000513 1788587039.871252 mapread 0x00aa96 thru 0x018ae2 (0xe04d bytes) 000514 1788587039.874480 mapwrite 0x02f8dc thru 0x03260b (0x2d30 bytes) 000515 1788587039.952470 mapread 0x022e7f thru 0x0283f4 (0x5576 bytes) 000518 1788587039.956957 punch from 0x004403 to 0x009c7a (0x5878 bytes) 000519 1788587039.962252 mapread 0x015d4b thru 0x01f261 (0x9517 bytes) 000520 1788587039.968494 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3260c): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3260c): Mirror 2: 0x27783803 (stale) Mirror 6: 0x27783803 (stale) Mirror 7: 0x27783803 (stale) Mirror 8: 0x27783803 (stale) Mirror 9: 0x27783803 (stale) Mirror 10: 0x27783803 (stale) Mirror 11: 0x27783803 (stale) Mirror 12: 0x58c2dedb (stale) CRC-32 checksum value for chunk [0, 0x3260c): Mirror 1: 0x3be90da0 Mirror 13: 0x3be90da0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3260c: skipped 000521 1788587041.424630 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3260c): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3260c): Mirror 2: 0x27783803 (stale) Mirror 6: 0x27783803 (stale) Mirror 7: 0x27783803 (stale) Mirror 8: 0x27783803 (stale) Mirror 9: 0x27783803 (stale) Mirror 10: 0x27783803 (stale) Mirror 11: 0x27783803 (stale) Mirror 12: 0x58c2dedb (stale) CRC-32 checksum value for chunk [0, 0x3260c): Mirror 1: 0x3be90da0 Mirror 13: 0x3be90da0 Mirror 14: 0x3be90da0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3260c: skipped 000522 1788587043.177273 trunc from 0x03260c to 0x00a0f8 (0x28513 bytes) 000524 1788587043.286804 read 0x005ab1 thru 0x00a0f8 (0x4648 bytes) 000525 1788587043.296272 write 0x0266ce thru 0x026798 (0xcb bytes) 000526 1788587043.298275 read 0x0197a0 thru 0x020c8a (0x74eb bytes) 000527 1788587043.312842 MIRROR_RESYNC 11 mirrors 000528 1788587046.992018 MIRROR_EXTEND to 12 mirrors 000529 1788587047.785860 mapread 0x00d953 thru 0x019242 (0xb8f0 bytes) 000530 1788587047.842268 MIRROR_EXTEND to 13 mirrors 000531 1788587049.025666 read 0x009153 thru 0x018c36 (0xfae4 bytes) 000532 1788587049.038285 write 0x0317e3 thru 0x03d4c4 (0xbce2 bytes) 000534 1788587049.126786 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4c5): Mirror 2: 0xecf8166f (stale) Mirror 6: 0xecf8166f (stale) Mirror 7: 0xecf8166f (stale) Mirror 8: 0xecf8166f (stale) Mirror 9: 0xecf8166f (stale) Mirror 10: 0xecf8166f (stale) Mirror 11: 0xecf8166f (stale) Mirror 13: 0xecf8166f (stale) Mirror 14: 0xecf8166f (stale) Mirror 15: 0xecf8166f (stale) Mirror 16: 0xecf8166f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4c5): Mirror 2: 0xecf8166f (stale) Mirror 6: 0xecf8166f (stale) Mirror 7: 0xecf8166f (stale) Mirror 8: 0xecf8166f (stale) Mirror 9: 0xecf8166f (stale) Mirror 10: 0xecf8166f (stale) Mirror 11: 0xecf8166f (stale) Mirror 13: 0xecf8166f (stale) Mirror 14: 0xecf8166f (stale) Mirror 15: 0xecf8166f (stale) Mirror 16: 0xecf8166f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4c5: skipped 000535 1788587050.082475 mapwrite 0x006fdb thru 0x010701 (0x9727 bytes) 000536 1788587050.270080 read 0x026adb thru 0x035a48 (0xef6e bytes) 000537 1788587050.334045 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3d4c5): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4c5): Mirror 2: 0xecf8166f (stale) Mirror 6: 0xecf8166f (stale) Mirror 7: 0xecf8166f (stale) Mirror 8: 0xecf8166f (stale) Mirror 9: 0xecf8166f (stale) Mirror 10: 0xecf8166f (stale) Mirror 11: 0xecf8166f (stale) Mirror 13: 0xecf8166f (stale) Mirror 14: 0xecf8166f (stale) Mirror 15: 0xecf8166f (stale) Mirror 16: 0xecf8166f (stale) CRC-32 checksum value for chunk [0, 0x3d4c5): Mirror 1: 0x92d22035 Mirror 17: 0x92d22035 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4c5: skipped 000538 1788587052.393046 mapread 0x02feb0 thru 0x035657 (0x57a8 bytes) 000539 1788587052.433431 mapwrite 0x01c346 thru 0x01f77e (0x3439 bytes) 000541 1788587052.672972 mapwrite 0x03e710 thru 0x03ffff (0x18f0 bytes) 000543 1788587052.782063 read 0x00f670 thru 0x017529 (0x7eba bytes) 000546 1788587052.793201 write 0x03b500 thru 0x03ffff (0x4b00 bytes) 000547 1788587052.794247 read 0x039272 thru 0x03927c (0xb bytes) 000550 1788587052.797126 write 0x031e0b thru 0x03ed05 (0xcefb bytes) 000551 1788587052.809185 punch from 0x0362b2 to 0x03a8b7 (0x4606 bytes) 000553 1788587052.823533 mapread 0x03aa10 thru 0x03ffff (0x55f0 bytes) 000554 1788587052.830549 write 0x01b40f thru 0x025cfb (0xa8ed bytes) 000556 1788587052.832662 trunc from 0x040000 to 0x03ecdf (0x1320 bytes) 000557 1788587052.859070 mapread 0x027a75 thru 0x02c3a0 (0x492c bytes) 000558 1788587052.866392 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ece0): Mirror 2: 0xc28f3e09 (stale) Mirror 6: 0xc28f3e09 (stale) Mirror 8: 0xc28f3e09 (stale) Mirror 9: 0xc28f3e09 (stale) Mirror 10: 0xc28f3e09 (stale) Mirror 11: 0xc28f3e09 (stale) Mirror 13: 0xc28f3e09 (stale) Mirror 14: 0xc28f3e09 (stale) Mirror 15: 0xc28f3e09 (stale) Mirror 16: 0xc28f3e09 (stale) Mirror 17: 0x83f419db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ece0): Mirror 2: 0xc28f3e09 (stale) Mirror 6: 0xc28f3e09 (stale) Mirror 8: 0xc28f3e09 (stale) Mirror 9: 0xc28f3e09 (stale) Mirror 10: 0xc28f3e09 (stale) Mirror 11: 0xc28f3e09 (stale) Mirror 13: 0xc28f3e09 (stale) Mirror 14: 0xc28f3e09 (stale) Mirror 15: 0xc28f3e09 (stale) Mirror 16: 0xc28f3e09 (stale) Mirror 17: 0x83f419db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ece0: skipped 000559 1788587053.737031 mapread 0x0208e0 thru 0x02d537 (0xcc58 bytes) 000560 1788587053.795348 read 0x030b65 thru 0x03a722 (0x9bbe bytes) 000561 1788587053.805284 read 0x03bac0 thru 0x03ecdf (0x3220 bytes) 000562 1788587053.822449 mapread 0x027e00 thru 0x031896 (0x9a97 bytes) 000563 1788587053.823488 punch from 0x03c2e0 to 0x03ecdf (0x2a00 bytes) 000565 1788587053.837173 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ece0): Mirror 2: 0xc28f3e09 (stale) Mirror 6: 0xc28f3e09 (stale) Mirror 8: 0xc28f3e09 (stale) Mirror 9: 0xc28f3e09 (stale) Mirror 10: 0xc28f3e09 (stale) Mirror 11: 0xc28f3e09 (stale) Mirror 14: 0xc28f3e09 (stale) Mirror 15: 0xc28f3e09 (stale) Mirror 16: 0xc28f3e09 (stale) Mirror 17: 0x83f419db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ece0): Mirror 2: 0xc28f3e09 (stale) Mirror 6: 0xc28f3e09 (stale) Mirror 8: 0xc28f3e09 (stale) Mirror 9: 0xc28f3e09 (stale) Mirror 10: 0xc28f3e09 (stale) Mirror 11: 0xc28f3e09 (stale) Mirror 14: 0xc28f3e09 (stale) Mirror 15: 0xc28f3e09 (stale) Mirror 16: 0xc28f3e09 (stale) Mirror 17: 0x83f419db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ece0: skipped 000566 1788587055.001305 mapwrite 0x013c94 thru 0x01be05 (0x8172 bytes) 000568 1788587055.085269 mapread 0x03160c thru 0x037bcf (0x65c4 bytes) 000570 1788587055.122397 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ece0): Mirror 2: 0xc28f3e09 (stale) Mirror 6: 0xc28f3e09 (stale) Mirror 8: 0xc28f3e09 (stale) Mirror 9: 0xc28f3e09 (stale) Mirror 11: 0xc28f3e09 (stale) Mirror 14: 0xc28f3e09 (stale) Mirror 15: 0xc28f3e09 (stale) Mirror 16: 0xc28f3e09 (stale) Mirror 17: 0x83f419db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ece0): Mirror 2: 0xc28f3e09 (stale) Mirror 6: 0xc28f3e09 (stale) Mirror 8: 0xc28f3e09 (stale) Mirror 9: 0xc28f3e09 (stale) Mirror 11: 0xc28f3e09 (stale) Mirror 14: 0xc28f3e09 (stale) Mirror 15: 0xc28f3e09 (stale) Mirror 16: 0xc28f3e09 (stale) Mirror 17: 0x83f419db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ece0: skipped 000572 1788587056.024254 punch from 0x02b1a3 to 0x02d3da (0x2238 bytes) 000573 1788587056.030088 read 0x02f373 thru 0x03c384 (0xd012 bytes) 000574 1788587056.037390 mapread 0x02246f thru 0x02dc29 (0xb7bb bytes) 000576 1788587056.061299 read 0x0065a9 thru 0x00c803 (0x625b bytes) 000577 1788587056.073425 mapread 0x02740a thru 0x03580d (0xe404 bytes) 000578 1788587056.078607 trunc from 0x03ece0 to 0x02cead (0x11e32 bytes) 000580 1788587056.099103 punch from 0x00c0da to 0x019214 (0xd13b bytes) 000581 1788587056.105535 mapread 0x00a475 thru 0x0160ec (0xbc78 bytes) 000582 1788587056.172490 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 2: 0x8e159e46 (stale) Mirror 6: 0x8e159e46 (stale) Mirror 8: 0x8e159e46 (stale) Mirror 9: 0x8e159e46 (stale) Mirror 11: 0x8e159e46 (stale) Mirror 14: 0x8e159e46 (stale) Mirror 15: 0x8e159e46 (stale) Mirror 16: 0x8e159e46 (stale) Mirror 17: 0xa4054c40 (stale) CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 1: 0xa0e3ff93 Mirror 18: 0xa0e3ff93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ceae: skipped 000584 1788587058.014126 punch from 0x0134cb to 0x01b845 (0x837b bytes) 000587 1788587058.106120 MIRROR_RESYNC 11 mirrors 000588 1788587062.710766 punch from 0x010c7b to 0x012289 (0x160f bytes) 000589 1788587062.781191 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 2: 0x1eec134d (stale) Mirror 6: 0x1eec134d (stale) Mirror 8: 0x1eec134d (stale) Mirror 9: 0x1eec134d (stale) Mirror 11: 0x1eec134d (stale) Mirror 14: 0x1eec134d (stale) Mirror 15: 0x1eec134d (stale) Mirror 16: 0x1eec134d (stale) Mirror 17: 0x1eec134d (stale) Mirror 18: 0x1eec134d (stale) CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 1: 0x1eec134d Mirror 19: 0x1eec134d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ceae: skipped 000590 1788587064.520819 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 2: 0x1eec134d (stale) Mirror 6: 0x1eec134d (stale) Mirror 8: 0x1eec134d (stale) Mirror 9: 0x1eec134d (stale) Mirror 11: 0x1eec134d (stale) Mirror 14: 0x1eec134d (stale) Mirror 15: 0x1eec134d (stale) Mirror 17: 0x1eec134d (stale) Mirror 18: 0x1eec134d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 2: 0x1eec134d (stale) Mirror 6: 0x1eec134d (stale) Mirror 8: 0x1eec134d (stale) Mirror 9: 0x1eec134d (stale) Mirror 11: 0x1eec134d (stale) Mirror 14: 0x1eec134d (stale) Mirror 15: 0x1eec134d (stale) Mirror 17: 0x1eec134d (stale) Mirror 18: 0x1eec134d (stale) CRC-32 checksum value for chunk [0, 0x2ceae): Mirror 1: 0x1eec134d Mirror 19: 0x1eec134d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ceae: skipped 000591 1788587065.920944 trunc from 0x02ceae to 0x012ac9 (0x1a3e4 bytes) 000592 1788587066.084263 write 0x02df43 thru 0x032725 (0x47e3 bytes) 000594 1788587066.086241 read 0x01d75a thru 0x0208f7 (0x319e bytes) 000595 1788587066.100916 mapwrite 0x01f4a0 thru 0x029289 (0x9dea bytes) 000597 1788587066.206728 mapread 0x00f558 thru 0x01cffa (0xdaa3 bytes) 000598 1788587066.228761 read 0x013164 thru 0x01664f (0x34ec bytes) 000602 1788587066.234494 punch from 0x0040fa to 0x0081d1 (0x40d8 bytes) 000604 1788587066.242083 punch from 0x00761c to 0x00bcb6 (0x469b bytes) 000606 1788587066.249802 write 0x03f0a7 thru 0x03ffff (0xf59 bytes) 000607 1788587066.256778 write 0x03d8f8 thru 0x03ffff (0x2708 bytes) 000608 1788587066.269414 mapread 0x009052 thru 0x016979 (0xd928 bytes) 000609 1788587066.273869 read 0x00969f thru 0x00e6f4 (0x5056 bytes) 000610 1788587066.276018 punch from 0x011a7b to 0x01eef8 (0xd47e bytes) 000611 1788587066.299294 write 0x02754f thru 0x0350df (0xdb91 bytes) 000612 1788587066.301913 trunc from 0x040000 to 0x0095e4 (0x36a1b bytes) 000614 1788587066.335484 write 0x02c316 thru 0x03a72a (0xe415 bytes) 000615 1788587066.337323 mapread 0x0015b3 thru 0x00f52a (0xdf78 bytes) 000616 1788587066.354895 mapread 0x002fd6 thru 0x00d1c6 (0xa1f1 bytes) 000617 1788587066.357139 trunc from 0x03a72b to 0x03932f (0x13fb bytes) 000619 1788587066.386850 read 0x012032 thru 0x01da4a (0xba19 bytes) 000620 1788587066.390723 write 0x01096e thru 0x01ea86 (0xe119 bytes) 000621 1788587066.393716 trunc from 0x039330 to 0x0365ee (0x2d41 bytes) 000623 1788587066.438170 mapread 0x024720 thru 0x02ae07 (0x66e8 bytes) 000624 1788587066.440014 mapread 0x016172 thru 0x018a17 (0x28a6 bytes) 000626 1788587066.444082 mapread 0x02e3ff thru 0x02e99f (0x5a1 bytes) 000627 1788587066.446019 mapread 0x01d10e thru 0x0219f2 (0x48e5 bytes) 000628 1788587066.448029 mapread 0x00ca8f thru 0x0162ff (0x9871 bytes) 000629 1788587066.449158 punch from 0x03286b to 0x033a65 (0x11fb bytes) 000630 1788587066.494848 write 0x02564a thru 0x02aa86 (0x543d bytes) 000631 1788587066.496986 punch from 0x00ffe7 to 0x0184f2 (0x850c bytes) 000634 1788587066.536185 read 0x017b5f thru 0x01db19 (0x5fbb bytes) 000635 1788587066.560710 mapwrite 0x0328ff thru 0x036486 (0x3b88 bytes) 000636 1788587066.725663 mapwrite 0x016c8b thru 0x023eb0 (0xd226 bytes) 000642 1788587066.786069 read 0x025a7d thru 0x0290f9 (0x367d bytes) 000647 1788587066.791188 trunc from 0x0365ef to 0x03d453 (0x6e65 bytes) 000648 1788587066.813553 trunc from 0x03d454 to 0x0369bb (0x6a98 bytes) 000649 1788587066.833024 trunc from 0x0369bc to 0x02f924 (0x7097 bytes) 000651 1788587066.855792 write 0x019197 thru 0x01a06f (0xed9 bytes) 000652 1788587066.856858 punch from 0x02aab2 to 0x02f924 (0x4e73 bytes) 000653 1788587066.874420 MIRROR_RESYNC 11 mirrors 000654 1788587072.451827 mapwrite 0x009986 thru 0x00ffd6 (0x6651 bytes) 000655 1788587072.615367 mapwrite 0x005765 thru 0x013b19 (0xe3b5 bytes) 000657 1788587072.678181 mapread 0x00a53b thru 0x012abd (0x8583 bytes) 000660 1788587072.680290 mapwrite 0x01507e thru 0x01f982 (0xa905 bytes) 000661 1788587072.738941 mapread 0x011e1b thru 0x016505 (0x46eb bytes) 000662 1788587072.745541 MIRROR_RESYNC 11 mirrors 000664 1788587077.044357 trunc from 0x02f925 to 0x008527 (0x273fd bytes) 000666 1788587077.129527 trunc from 0x008528 to 0x0096c0 (0x1199 bytes) 000668 1788587077.157474 mapread 0x00194d thru 0x004f88 (0x363c bytes) 000669 1788587077.175220 write 0x00fb8f thru 0x01bc92 (0xc104 bytes) 000670 1788587077.177814 write 0x00b6f6 thru 0x012601 (0x6f0c bytes) 000671 1788587077.187619 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bc93): Mirror 2: 0x946e615c (stale) Mirror 6: 0x946e615c (stale) Mirror 8: 0x946e615c (stale) Mirror 11: 0x946e615c (stale) Mirror 14: 0x946e615c (stale) Mirror 15: 0x946e615c (stale) Mirror 17: 0x946e615c (stale) Mirror 18: 0x946e615c (stale) Mirror 19: 0x946e615c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc93): Mirror 2: 0x946e615c (stale) Mirror 6: 0x946e615c (stale) Mirror 8: 0x946e615c (stale) Mirror 11: 0x946e615c (stale) Mirror 14: 0x946e615c (stale) Mirror 15: 0x946e615c (stale) Mirror 17: 0x946e615c (stale) Mirror 18: 0x946e615c (stale) Mirror 19: 0x946e615c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc93: skipped 000672 1788587077.749330 read 0x00d12e thru 0x010444 (0x3317 bytes) 000674 1788587077.759888 write 0x00b378 thru 0x01642c (0xb0b5 bytes) 000676 1788587077.775892 trunc from 0x01bc93 to 0x030c2e (0x14f9c bytes) 000677 1788587077.797015 write 0x00b4ed thru 0x00d764 (0x2278 bytes) 000680 1788587077.801336 write 0x010f0e thru 0x016051 (0x5144 bytes) 000682 1788587077.803169 trunc from 0x030c2f to 0x024387 (0xc8a7 bytes) 000683 1788587077.825509 read 0x009423 thru 0x0118f6 (0x84d4 bytes) 000684 1788587077.843833 trunc from 0x024388 to 0x00f5f8 (0x14d8f bytes) 000685 1788587077.869935 mapwrite 0x00029a thru 0x0047c8 (0x452f bytes) 000686 1788587077.917353 read 0x00f5e3 thru 0x00f5f8 (0x16 bytes) 000687 1788587077.918009 punch from 0x0071ce to 0x00f5f8 (0x842b bytes) 000688 1788587077.923953 read 0x0074a2 thru 0x00d5cc (0x612b bytes) 000689 1788587077.932935 read 0x001470 thru 0x00f5f8 (0xe189 bytes) 000690 1788587077.933810 trunc from 0x00f5f9 to 0x031830 (0x22238 bytes) 000691 1788587077.949226 trunc from 0x031831 to 0x02ad0f (0x6b21 bytes) 000692 1788587077.966230 mapread 0x00e851 thru 0x00f3aa (0xb5a bytes) 000693 1788587077.972239 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2ad10): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad10): Mirror 2: 0x810ba06 (stale) Mirror 6: 0x810ba06 (stale) Mirror 8: 0x810ba06 (stale) Mirror 11: 0x810ba06 (stale) Mirror 14: 0x810ba06 (stale) Mirror 15: 0x810ba06 (stale) Mirror 17: 0x810ba06 (stale) Mirror 18: 0x810ba06 (stale) Mirror 19: 0x810ba06 (stale) CRC-32 checksum value for chunk [0, 0x2ad10): Mirror 1: 0x66117877 Mirror 20: 0x66117877 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad10: skipped 000694 1788587079.162212 write 0x03dc86 thru 0x03ffff (0x237a bytes) 000695 1788587079.244257 mapwrite 0x02d3ce thru 0x03b903 (0xe536 bytes) 000696 1788587079.319238 punch from 0x0359ce to 0x03ffff (0xa632 bytes) 000697 1788587079.329753 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbd840acf (stale) Mirror 6: 0xbd840acf (stale) Mirror 8: 0xbd840acf (stale) Mirror 11: 0xbd840acf (stale) Mirror 14: 0xbd840acf (stale) Mirror 15: 0xbd840acf (stale) Mirror 17: 0xbd840acf (stale) Mirror 18: 0xbd840acf (stale) Mirror 19: 0xbd840acf (stale) Mirror 20: 0x4a03a4a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa2c95fb0 Mirror 21: 0xa2c95fb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000698 1788587080.610971 MIRROR_RESYNC 12 mirrors 000699 1788587084.496132 trunc from 0x040000 to 0x02d29f (0x12d60 bytes) 000700 1788587084.617769 mapwrite 0x003428 thru 0x0132f0 (0xfec9 bytes) 000701 1788587084.715765 punch from 0x0117ba to 0x017d39 (0x6580 bytes) 000703 1788587084.721876 read 0x01a5ed thru 0x01d112 (0x2b26 bytes) 000704 1788587084.729691 write 0x03683b thru 0x03ffff (0x97c5 bytes) 000705 1788587084.731708 mapwrite 0x032647 thru 0x03d59a (0xaf54 bytes) 000707 1788587084.779530 punch from 0x0240c4 to 0x02edb4 (0xacf1 bytes) 000708 1788587084.783566 write 0x014af9 thru 0x0168ab (0x1db3 bytes) 000709 1788587084.791364 punch from 0x02fbcf to 0x03abfc (0xb02e bytes) 000710 1788587084.800832 trunc from 0x040000 to 0x035190 (0xae6f bytes) 000711 1788587084.830144 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x35191): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35191): Mirror 2: 0xc8e82ed4 (stale) Mirror 6: 0xc8e82ed4 (stale) Mirror 8: 0xc8e82ed4 (stale) Mirror 11: 0xc8e82ed4 (stale) Mirror 14: 0xc8e82ed4 (stale) Mirror 15: 0xc8e82ed4 (stale) Mirror 17: 0xc8e82ed4 (stale) Mirror 18: 0xc8e82ed4 (stale) Mirror 19: 0xc8e82ed4 (stale) Mirror 20: 0xc8e82ed4 (stale) Mirror 21: 0xc8e82ed4 (stale) CRC-32 checksum value for chunk [0, 0x35191): Mirror 1: 0xa1271a8f Mirror 22: 0xa1271a8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35191: skipped 000713 1788587086.438135 mapread 0x02c26c thru 0x02f815 (0x35aa bytes) 000714 1788587086.481651 mapread 0x000f2b thru 0x006235 (0x530b bytes) 000715 1788587086.551322 punch from 0x008c11 to 0x011259 (0x8649 bytes) 000716 1788587086.689213 trunc from 0x035191 to 0x021a2d (0x13763 bytes) 000720 1788587086.736056 mapwrite 0x0232cf thru 0x0323b0 (0xf0e2 bytes) 000721 1788587086.944902 read 0x010448 thru 0x01bc3f (0xb7f8 bytes) 000723 1788587086.963662 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x323b1): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323b1): Mirror 2: 0x8d1c7f08 (stale) Mirror 6: 0x8d1c7f08 (stale) Mirror 8: 0x8d1c7f08 (stale) Mirror 11: 0x8d1c7f08 (stale) Mirror 14: 0x8d1c7f08 (stale) Mirror 15: 0x8d1c7f08 (stale) Mirror 17: 0x8d1c7f08 (stale) Mirror 18: 0x8d1c7f08 (stale) Mirror 19: 0x8d1c7f08 (stale) Mirror 20: 0x8d1c7f08 (stale) Mirror 21: 0x8d1c7f08 (stale) Mirror 22: 0x744d2447 (stale) CRC-32 checksum value for chunk [0, 0x323b1): Mirror 1: 0xd68d9608 Mirror 23: 0xd68d9608 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323b1: skipped 000724 1788587088.587503 mapwrite 0x03d792 thru 0x03ffff (0x286e bytes) 000725 1788587088.848538 mapread 0x02b049 thru 0x0399e0 (0xe998 bytes) 000726 1788587088.870628 trunc from 0x040000 to 0x03d352 (0x2cad bytes) 000727 1788587088.905212 write 0x0272c8 thru 0x031a59 (0xa792 bytes) 000728 1788587088.908145 mapwrite 0x00ed8b thru 0x018025 (0x929b bytes) 000729 1788587089.041817 read 0x036b6b thru 0x03d352 (0x67e8 bytes) 000730 1788587089.043499 mapread 0x010314 thru 0x016fa3 (0x6c90 bytes) 000731 1788587089.044839 mapwrite 0x02ec56 thru 0x03ea07 (0xfdb2 bytes) 000732 1788587089.153380 mapread 0x01f9e9 thru 0x02506a (0x5682 bytes) 000733 1788587089.165190 mapwrite 0x033849 thru 0x0358f0 (0x20a8 bytes) 000734 1788587089.236760 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3ea08): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea08): Mirror 2: 0x4ba4e3ff (stale) Mirror 6: 0x4ba4e3ff (stale) Mirror 8: 0x4ba4e3ff (stale) Mirror 11: 0x4ba4e3ff (stale) Mirror 14: 0x4ba4e3ff (stale) Mirror 15: 0x4ba4e3ff (stale) Mirror 17: 0x4ba4e3ff (stale) Mirror 18: 0x4ba4e3ff (stale) Mirror 19: 0x4ba4e3ff (stale) Mirror 20: 0x4ba4e3ff (stale) Mirror 21: 0x4ba4e3ff (stale) Mirror 22: 0xb2e0369 (stale) Mirror 23: 0x9b4618a0 (stale) CRC-32 checksum value for chunk [0, 0x3ea08): Mirror 1: 0x634c5836 Mirror 24: 0x634c5836 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea08: skipped 000735 1788587091.168953 trunc from 0x03ea08 to 0x030357 (0xe6b0 bytes) 000736 1788587091.378533 write 0x0010f3 thru 0x00757e (0x648c bytes) 000737 1788587091.406340 read 0x0165ac thru 0x01cdad (0x6802 bytes) 000738 1788587091.421252 write 0x025804 thru 0x033349 (0xdb46 bytes) 000739 1788587091.443936 punch from 0x013d6b to 0x01e0dd (0xa373 bytes) 000740 1788587091.484930 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3334a): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3334a): Mirror 2: 0xe0be519c (stale) Mirror 6: 0xe0be519c (stale) Mirror 8: 0xe0be519c (stale) Mirror 11: 0xe0be519c (stale) Mirror 14: 0xe0be519c (stale) Mirror 15: 0xe0be519c (stale) Mirror 17: 0xe0be519c (stale) Mirror 18: 0xe0be519c (stale) Mirror 19: 0xe0be519c (stale) Mirror 20: 0xe0be519c (stale) Mirror 21: 0xe0be519c (stale) Mirror 22: 0xbb67af40 (stale) Mirror 23: 0x69968a36 (stale) Mirror 24: 0x76687736 (stale) CRC-32 checksum value for chunk [0, 0x3334a): Mirror 1: 0xbb70f5bd Mirror 25: 0xbb70f5bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3334a: skipped 000743 1788587093.814985 write 0x006813 thru 0x00713b (0x929 bytes) 000744 1788587093.920031 read 0x005f04 thru 0x00b8ef (0x59ec bytes) 000745 1788587093.928968 mapread 0x02cb98 thru 0x033349 (0x67b2 bytes) 000746 1788587093.949433 punch from 0x025c63 to 0x0274f6 (0x1894 bytes) 000747 1788587093.959074 mapwrite 0x026c12 thru 0x02ea02 (0x7df1 bytes) 000748 1788587094.020360 punch from 0x01bbd3 to 0x01cfd0 (0x13fe bytes) 000750 1788587094.026711 trunc from 0x03334a to 0x036ba9 (0x3860 bytes) 000751 1788587094.051184 mapwrite 0x0126dd thru 0x0188c9 (0x61ed bytes) 000752 1788587094.104433 mapread 0x0189e3 thru 0x018a28 (0x46 bytes) 000754 1788587094.109713 mapread 0x027698 thru 0x0322ad (0xac16 bytes) 000755 1788587094.112388 trunc from 0x036baa to 0x00d0d5 (0x29ad4 bytes) 000756 1788587094.137879 MIRROR_RESYNC 16 mirrors 000757 1788587097.274512 mapwrite 0x02019c thru 0x02101f (0xe84 bytes) 000758 1788587097.473772 MIRROR_RESYNC 16 mirrors 000760 1788587101.460955 punch from 0x013c34 to 0x01bcdc (0x80a9 bytes) 000761 1788587101.554103 mapread 0x01693f thru 0x017946 (0x1008 bytes) 000764 1788587101.573493 mapread 0x003870 thru 0x013635 (0xfdc6 bytes) 000765 1788587101.602051 MIRROR_SPLIT mirror 10 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21020): Mirror 2: 0x7ec08299 (stale) Mirror 6: 0x7ec08299 (stale) Mirror 8: 0x7ec08299 (stale) Mirror 11: 0x7ec08299 (stale) Mirror 14: 0x7ec08299 (stale) Mirror 15: 0x7ec08299 (stale) Mirror 17: 0x7ec08299 (stale) Mirror 18: 0x7ec08299 (stale) Mirror 19: 0x7ec08299 (stale) Mirror 21: 0x7ec08299 (stale) Mirror 22: 0x7ec08299 (stale) Mirror 23: 0x7ec08299 (stale) Mirror 24: 0x7ec08299 (stale) Mirror 25: 0x7ec08299 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21020): Mirror 2: 0x7ec08299 (stale) Mirror 6: 0x7ec08299 (stale) Mirror 8: 0x7ec08299 (stale) Mirror 11: 0x7ec08299 (stale) Mirror 14: 0x7ec08299 (stale) Mirror 15: 0x7ec08299 (stale) Mirror 17: 0x7ec08299 (stale) Mirror 18: 0x7ec08299 (stale) Mirror 19: 0x7ec08299 (stale) Mirror 21: 0x7ec08299 (stale) Mirror 22: 0x7ec08299 (stale) Mirror 23: 0x7ec08299 (stale) Mirror 24: 0x7ec08299 (stale) Mirror 25: 0x7ec08299 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21020: skipped 000768 1788587102.572947 mapwrite 0x031987 thru 0x03ffff (0xe679 bytes) 000770 1788587102.703814 write 0x03e98a thru 0x03ffff (0x1676 bytes) 000771 1788587102.705113 mapwrite 0x01cd03 thru 0x0224d9 (0x57d7 bytes) 000772 1788587102.778897 write 0x03bcce thru 0x03ffff (0x4332 bytes) 000773 1788587102.780480 mapwrite 0x01d658 thru 0x0225e5 (0x4f8e bytes) 000774 1788587102.836273 read 0x033e84 thru 0x03fe5a (0xbfd7 bytes) 000775 1788587102.837064 mapread 0x02c024 thru 0x033566 (0x7543 bytes) 000778 1788587102.840499 punch from 0x032cc1 to 0x0331af (0x4ef bytes) 000779 1788587102.846508 read 0x031525 thru 0x03e1cb (0xcca7 bytes) 000780 1788587102.867412 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbf789662 (stale) Mirror 6: 0xbf789662 (stale) Mirror 8: 0xbf789662 (stale) Mirror 11: 0xbf789662 (stale) Mirror 14: 0xbf789662 (stale) Mirror 15: 0xbf789662 (stale) Mirror 17: 0xbf789662 (stale) Mirror 18: 0xbf789662 (stale) Mirror 19: 0xbf789662 (stale) Mirror 21: 0xbf789662 (stale) Mirror 22: 0xbf789662 (stale) Mirror 23: 0xbf789662 (stale) Mirror 24: 0xbf789662 (stale) Mirror 25: 0xbf789662 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf8835771 Mirror 26: 0xf8835771 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1788587105.474937 mapwrite 0x018805 thru 0x01e568 (0x5d64 bytes) 000782 1788587105.917819 read 0x01fa1e thru 0x02abad (0xb190 bytes) 000784 1788587105.942890 punch from 0x018b94 to 0x0239eb (0xae58 bytes) 000786 1788587105.958755 write 0x028f6a thru 0x0318ba (0x8951 bytes) 000788 1788587105.976082 mapread 0x02b29f thru 0x033c38 (0x899a bytes) 000792 1788587106.027667 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbf789662 (stale) Mirror 6: 0xbf789662 (stale) Mirror 8: 0xbf789662 (stale) Mirror 11: 0xbf789662 (stale) Mirror 14: 0xbf789662 (stale) Mirror 15: 0xbf789662 (stale) Mirror 17: 0xbf789662 (stale) Mirror 18: 0xbf789662 (stale) Mirror 19: 0xbf789662 (stale) Mirror 21: 0xbf789662 (stale) Mirror 23: 0xbf789662 (stale) Mirror 24: 0xbf789662 (stale) Mirror 25: 0xbf789662 (stale) Mirror 26: 0xf8835771 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbf789662 (stale) Mirror 6: 0xbf789662 (stale) Mirror 8: 0xbf789662 (stale) Mirror 11: 0xbf789662 (stale) Mirror 14: 0xbf789662 (stale) Mirror 15: 0xbf789662 (stale) Mirror 17: 0xbf789662 (stale) Mirror 18: 0xbf789662 (stale) Mirror 19: 0xbf789662 (stale) Mirror 21: 0xbf789662 (stale) Mirror 23: 0xbf789662 (stale) Mirror 24: 0xbf789662 (stale) Mirror 25: 0xbf789662 (stale) Mirror 26: 0xf8835771 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000795 1788587107.678333 mapread 0x0208e2 thru 0x021040 (0x75f bytes) 000796 1788587107.725607 write 0x02f3c0 thru 0x03b02c (0xbc6d bytes) 000797 1788587107.752517 mapwrite 0x01057a thru 0x019fc2 (0x9a49 bytes) 000798 1788587108.030791 MIRROR_RESYNC 15 mirrors 000799 1788587116.994384 MIRROR_EXTEND to 16 mirrors 000802 1788587118.325934 punch from 0x02d7f9 to 0x03ab5f (0xd367 bytes) 000803 1788587118.500009 read 0x01fcca thru 0x020d0d (0x1044 bytes) 000804 1788587118.517938 mapread 0x0322b5 thru 0x03392a (0x1676 bytes) 000806 1788587118.530153 trunc from 0x040000 to 0x000b5d (0x3f4a2 bytes) 000808 1788587118.563956 trunc from 0x000b5e to 0x02194f (0x20df2 bytes) 000809 1788587118.601736 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21950): Mirror 2: 0x47c3e0b4 (stale) Mirror 6: 0x47c3e0b4 (stale) Mirror 8: 0x47c3e0b4 (stale) Mirror 11: 0x47c3e0b4 (stale) Mirror 14: 0x47c3e0b4 (stale) Mirror 15: 0x47c3e0b4 (stale) Mirror 17: 0x47c3e0b4 (stale) Mirror 18: 0x47c3e0b4 (stale) Mirror 19: 0x47c3e0b4 (stale) Mirror 21: 0x47c3e0b4 (stale) Mirror 23: 0x47c3e0b4 (stale) Mirror 24: 0x47c3e0b4 (stale) Mirror 25: 0x47c3e0b4 (stale) Mirror 26: 0x47c3e0b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21950): Mirror 2: 0x47c3e0b4 (stale) Mirror 6: 0x47c3e0b4 (stale) Mirror 8: 0x47c3e0b4 (stale) Mirror 11: 0x47c3e0b4 (stale) Mirror 14: 0x47c3e0b4 (stale) Mirror 15: 0x47c3e0b4 (stale) Mirror 17: 0x47c3e0b4 (stale) Mirror 18: 0x47c3e0b4 (stale) Mirror 19: 0x47c3e0b4 (stale) Mirror 21: 0x47c3e0b4 (stale) Mirror 23: 0x47c3e0b4 (stale) Mirror 24: 0x47c3e0b4 (stale) Mirror 25: 0x47c3e0b4 (stale) Mirror 26: 0x47c3e0b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21950: skipped 000810 1788587119.807148 write 0x00da94 thru 0x01ca31 (0xef9e bytes) 000811 1788587119.825183 MIRROR_RESYNC 15 mirrors 000812 1788587125.129662 trunc from 0x021950 to 0x0193b8 (0x8597 bytes) 000814 1788587125.244420 read 0x00dfd4 thru 0x0113fe (0x342b bytes) 000815 1788587125.250519 write 0x03eaee thru 0x03ffff (0x1512 bytes) 000816 1788587125.251398 mapread 0x03f332 thru 0x03ffff (0xcce bytes) 000817 1788587125.252219 mapread 0x007537 thru 0x00896d (0x1437 bytes) 000819 1788587125.265291 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2f439880 (stale) Mirror 8: 0x2f439880 (stale) Mirror 11: 0x2f439880 (stale) Mirror 14: 0x2f439880 (stale) Mirror 15: 0x2f439880 (stale) Mirror 17: 0x2f439880 (stale) Mirror 18: 0x2f439880 (stale) Mirror 19: 0x2f439880 (stale) Mirror 21: 0x2f439880 (stale) Mirror 23: 0x2f439880 (stale) Mirror 24: 0x2f439880 (stale) Mirror 25: 0x2f439880 (stale) Mirror 26: 0x2f439880 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2f439880 (stale) Mirror 8: 0x2f439880 (stale) Mirror 11: 0x2f439880 (stale) Mirror 14: 0x2f439880 (stale) Mirror 15: 0x2f439880 (stale) Mirror 17: 0x2f439880 (stale) Mirror 18: 0x2f439880 (stale) Mirror 19: 0x2f439880 (stale) Mirror 21: 0x2f439880 (stale) Mirror 23: 0x2f439880 (stale) Mirror 24: 0x2f439880 (stale) Mirror 25: 0x2f439880 (stale) Mirror 26: 0x2f439880 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000820 1788587126.370372 write 0x01352d thru 0x01f3d4 (0xbea8 bytes) 000821 1788587126.385738 mapread 0x03fed0 thru 0x03ffff (0x130 bytes) 000826 1788587126.398147 write 0x020c84 thru 0x026415 (0x5792 bytes) 000828 1788587126.418820 trunc from 0x040000 to 0x018a3e (0x275c1 bytes) 000830 1788587126.443558 mapread 0x000867 thru 0x00e1a4 (0xd93e bytes) 000831 1788587126.472430 punch from 0x00d6ad to 0x010e67 (0x37bb bytes) 000832 1788587126.493383 mapread 0x00033f thru 0x00aa52 (0xa714 bytes) 000833 1788587126.503524 MIRROR_RESYNC 14 mirrors 000834 1788587132.733895 mapwrite 0x032e0e thru 0x035b50 (0x2d43 bytes) 000835 1788587132.996837 mapread 0x02f9c5 thru 0x035b50 (0x618c bytes) 000839 1788587133.008045 punch from 0x026732 to 0x02b14c (0x4a1b bytes) 000841 1788587133.015859 mapwrite 0x01d9bf thru 0x0265cc (0x8c0e bytes) 000842 1788587133.131078 mapread 0x000809 thru 0x010490 (0xfc88 bytes) 000843 1788587133.157044 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35b51): Mirror 6: 0x9b081499 (stale) Mirror 8: 0x9b081499 (stale) Mirror 14: 0x9b081499 (stale) Mirror 15: 0x9b081499 (stale) Mirror 17: 0x9b081499 (stale) Mirror 18: 0x9b081499 (stale) Mirror 19: 0x9b081499 (stale) Mirror 21: 0x9b081499 (stale) Mirror 23: 0x9b081499 (stale) Mirror 24: 0x9b081499 (stale) Mirror 25: 0x9b081499 (stale) Mirror 26: 0x9b081499 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b51): Mirror 6: 0x9b081499 (stale) Mirror 8: 0x9b081499 (stale) Mirror 14: 0x9b081499 (stale) Mirror 15: 0x9b081499 (stale) Mirror 17: 0x9b081499 (stale) Mirror 18: 0x9b081499 (stale) Mirror 19: 0x9b081499 (stale) Mirror 21: 0x9b081499 (stale) Mirror 23: 0x9b081499 (stale) Mirror 24: 0x9b081499 (stale) Mirror 25: 0x9b081499 (stale) Mirror 26: 0x9b081499 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b51: skipped 000844 1788587134.170142 mapread 0x000a3f thru 0x00e535 (0xdaf7 bytes) 000845 1788587134.198761 mapwrite 0x01c39d thru 0x026931 (0xa595 bytes) 000847 1788587134.272264 trunc from 0x035b51 to 0x03c267 (0x6717 bytes) 000848 1788587134.289844 mapwrite 0x0145bc thru 0x022bec (0xe631 bytes) 000851 1788587134.347224 punch from 0x0319b7 to 0x03a173 (0x87bd bytes) 000853 1788587134.360992 MIRROR_RESYNC 13 mirrors 000854 1788587139.398077 punch from 0x007213 to 0x008f85 (0x1d73 bytes) 000855 1788587139.471266 trunc from 0x03c268 to 0x01c214 (0x20053 bytes) 000856 1788587139.493469 trunc from 0x01c215 to 0x03c50f (0x202fb bytes) 000857 1788587139.515962 mapread 0x016c4f thru 0x01761e (0x9d0 bytes) 000859 1788587139.530887 write 0x03cccf thru 0x03ffff (0x3331 bytes) 000860 1788587139.537921 mapread 0x02ead1 thru 0x034291 (0x57c1 bytes) 000861 1788587139.548481 mapwrite 0x01efb0 thru 0x01fdac (0xdfd bytes) 000862 1788587139.627305 mapread 0x012256 thru 0x01d9dc (0xb787 bytes) 000864 1788587139.639181 write 0x020f1f thru 0x0279c9 (0x6aab bytes) 000865 1788587139.640708 read 0x0367b5 thru 0x038e6d (0x26b9 bytes) 000867 1788587139.647180 mapwrite 0x02fdd8 thru 0x03ba8d (0xbcb6 bytes) 000869 1788587139.714850 mapwrite 0x00d03f thru 0x01be07 (0xedc9 bytes) 000870 1788587139.768890 punch from 0x0031a8 to 0x008657 (0x54b0 bytes) 000876 1788587139.777445 punch from 0x01f83d to 0x02b356 (0xbb1a bytes) 000878 1788587139.783840 punch from 0x0247b9 to 0x025b97 (0x13df bytes) 000880 1788587139.794644 write 0x039c68 thru 0x03ffff (0x6398 bytes) 000881 1788587139.796152 punch from 0x0000aa to 0x00b12c (0xb083 bytes) 000882 1788587139.814501 read 0x034b46 thru 0x03da25 (0x8ee0 bytes) 000883 1788587139.815594 mapwrite 0x03ae39 thru 0x03ffff (0x51c7 bytes) 000884 1788587139.907082 punch from 0x0188fb to 0x0252d2 (0xc9d8 bytes) 000885 1788587139.924038 mapread 0x025bcd thru 0x033322 (0xd756 bytes) 000886 1788587139.960890 write 0x00afc6 thru 0x01853c (0xd577 bytes) 000887 1788587139.963891 punch from 0x00533f to 0x0054ac (0x16e bytes) 000888 1788587140.012555 punch from 0x0383c2 to 0x039921 (0x1560 bytes) 000889 1788587140.028875 mapwrite 0x008138 thru 0x00b8d1 (0x379a bytes) 000892 1788587140.268826 write 0x018ff5 thru 0x022064 (0x9070 bytes) 000894 1788587140.272747 read 0x0290d2 thru 0x02beec (0x2e1b bytes) 000895 1788587140.274021 mapread 0x021dd4 thru 0x02c414 (0xa641 bytes) 000896 1788587140.277184 trunc from 0x040000 to 0x0004f7 (0x3fb08 bytes) 000897 1788587140.305550 trunc from 0x0004f8 to 0x031e09 (0x31912 bytes) 000898 1788587140.331619 mapread 0x003076 thru 0x00540f (0x239a bytes) 000899 1788587140.349371 mapwrite 0x037490 thru 0x03ffff (0x8b70 bytes) 000900 1788587140.563864 mapread 0x022b09 thru 0x02f031 (0xc529 bytes) 000901 1788587140.567709 read 0x02e210 thru 0x035f07 (0x7cf8 bytes) 000902 1788587140.578868 MIRROR_RESYNC 13 mirrors 000904 1788587144.733217 punch from 0x03e906 to 0x03ffff (0x16fa bytes) 000905 1788587144.812523 mapwrite 0x02c428 thru 0x03522b (0x8e04 bytes) 000906 1788587144.906195 punch from 0x03ac20 to 0x03ffff (0x53e0 bytes) 000907 1788587144.911701 read 0x02a03c thru 0x02a13e (0x103 bytes) 000908 1788587144.912343 punch from 0x0121c8 to 0x016a1e (0x4857 bytes) 000910 1788587144.918294 write 0x03f992 thru 0x03ffff (0x66e bytes) 000911 1788587144.923855 trunc from 0x040000 to 0x0214e9 (0x1eb16 bytes) 000912 1788587144.958550 mapread 0x013e6e thru 0x017f16 (0x40a9 bytes) 000915 1788587144.981519 mapwrite 0x010739 thru 0x01936b (0x8c33 bytes) 000916 1788587145.046476 mapwrite 0x019cc5 thru 0x01ff54 (0x6290 bytes) 000917 1788587145.148099 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x214ea): Mirror 6: 0x4aec72a1 (stale) Mirror 8: 0x4aec72a1 (stale) Mirror 14: 0x4aec72a1 (stale) Mirror 15: 0x4aec72a1 (stale) Mirror 17: 0x4aec72a1 (stale) Mirror 18: 0x4aec72a1 (stale) Mirror 19: 0x4aec72a1 (stale) Mirror 21: 0x4aec72a1 (stale) Mirror 23: 0x4aec72a1 (stale) Mirror 25: 0x4aec72a1 (stale) Mirror 26: 0x4aec72a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214ea): Mirror 6: 0x4aec72a1 (stale) Mirror 8: 0x4aec72a1 (stale) Mirror 14: 0x4aec72a1 (stale) Mirror 15: 0x4aec72a1 (stale) Mirror 17: 0x4aec72a1 (stale) Mirror 18: 0x4aec72a1 (stale) Mirror 19: 0x4aec72a1 (stale) Mirror 21: 0x4aec72a1 (stale) Mirror 23: 0x4aec72a1 (stale) Mirror 25: 0x4aec72a1 (stale) Mirror 26: 0x4aec72a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214ea: skipped 000918 1788587146.255986 write 0x036679 thru 0x03c98a (0x6312 bytes) 000921 1788587146.260134 trunc from 0x03c98b to 0x036e6f (0x5b1b bytes) 000922 1788587146.284148 punch from 0x017976 to 0x02698a (0xf015 bytes) 000923 1788587146.293326 mapwrite 0x0041f5 thru 0x00c982 (0x878e bytes) 000924 1788587146.352190 mapread 0x006d63 thru 0x012d96 (0xc034 bytes) 000925 1788587146.359891 mapread 0x01d69f thru 0x0219c5 (0x4327 bytes) 000926 1788587146.368596 trunc from 0x036e70 to 0x01fd17 (0x17158 bytes) 000928 1788587146.381479 mapread 0x00d118 thru 0x0183c8 (0xb2b1 bytes) 000929 1788587146.382682 write 0x00002a thru 0x00c322 (0xc2f9 bytes) 000930 1788587146.390739 mapread 0x009ba8 thru 0x00db90 (0x3fe9 bytes) 000931 1788587146.391798 write 0x01dd3a thru 0x0258d6 (0x7b9d bytes) 000932 1788587146.393664 mapread 0x00ac4b thru 0x00c998 (0x1d4e bytes) 000935 1788587146.395960 write 0x024262 thru 0x0321f9 (0xdf98 bytes) 000936 1788587146.397943 trunc from 0x0321fa to 0x006b63 (0x2b696 bytes) 000937 1788587146.419176 trunc from 0x006b64 to 0x018a4c (0x11ee9 bytes) 000938 1788587146.454977 mapread 0x001c1a thru 0x004244 (0x262b bytes) 000939 1788587146.455890 write 0x018c97 thru 0x0285ed (0xf957 bytes) 000942 1788587146.492269 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x285ee): Mirror 6: 0x94a5f86c (stale) Mirror 14: 0x94a5f86c (stale) Mirror 15: 0x94a5f86c (stale) Mirror 17: 0x94a5f86c (stale) Mirror 18: 0x94a5f86c (stale) Mirror 19: 0x94a5f86c (stale) Mirror 21: 0x94a5f86c (stale) Mirror 23: 0x94a5f86c (stale) Mirror 25: 0x94a5f86c (stale) Mirror 26: 0x94a5f86c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285ee): Mirror 6: 0x94a5f86c (stale) Mirror 14: 0x94a5f86c (stale) Mirror 15: 0x94a5f86c (stale) Mirror 17: 0x94a5f86c (stale) Mirror 18: 0x94a5f86c (stale) Mirror 19: 0x94a5f86c (stale) Mirror 21: 0x94a5f86c (stale) Mirror 23: 0x94a5f86c (stale) Mirror 25: 0x94a5f86c (stale) Mirror 26: 0x94a5f86c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285ee: skipped 000943 1788587147.273244 mapread 0x02705c thru 0x0285ed (0x1592 bytes) 000945 1788587147.291475 punch from 0x021170 to 0x0285ed (0x747e bytes) 000946 1788587147.320870 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x285ee): Mirror 6: 0x94a5f86c (stale) Mirror 14: 0x94a5f86c (stale) Mirror 15: 0x94a5f86c (stale) Mirror 17: 0x94a5f86c (stale) Mirror 18: 0x94a5f86c (stale) Mirror 19: 0x94a5f86c (stale) Mirror 21: 0x94a5f86c (stale) Mirror 23: 0x94a5f86c (stale) Mirror 25: 0x94a5f86c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285ee): Mirror 6: 0x94a5f86c (stale) Mirror 14: 0x94a5f86c (stale) Mirror 15: 0x94a5f86c (stale) Mirror 17: 0x94a5f86c (stale) Mirror 18: 0x94a5f86c (stale) Mirror 19: 0x94a5f86c (stale) Mirror 21: 0x94a5f86c (stale) Mirror 23: 0x94a5f86c (stale) Mirror 25: 0x94a5f86c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285ee: skipped 000947 1788587148.127404 punch from 0x01cae8 to 0x020d64 (0x427d bytes) 000949 1788587148.148457 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x285ee): Mirror 6: 0x94a5f86c (stale) Mirror 14: 0x94a5f86c (stale) Mirror 15: 0x94a5f86c (stale) Mirror 17: 0x94a5f86c (stale) Mirror 18: 0x94a5f86c (stale) Mirror 19: 0x94a5f86c (stale) Mirror 23: 0x94a5f86c (stale) Mirror 25: 0x94a5f86c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285ee): Mirror 6: 0x94a5f86c (stale) Mirror 14: 0x94a5f86c (stale) Mirror 15: 0x94a5f86c (stale) Mirror 17: 0x94a5f86c (stale) Mirror 18: 0x94a5f86c (stale) Mirror 19: 0x94a5f86c (stale) Mirror 23: 0x94a5f86c (stale) Mirror 25: 0x94a5f86c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285ee: skipped 000950 1788587149.065518 MIRROR_RESYNC 9 mirrors 000951 1788587152.341867 punch from 0x00caf7 to 0x01490b (0x7e15 bytes) 000952 1788587152.437925 read 0x009d95 thru 0x01497b (0xabe7 bytes) 000953 1788587152.452913 mapwrite 0x01962a thru 0x01e0a5 (0x4a7c bytes) 000954 1788587152.544454 mapread 0x01a6b8 thru 0x01ec67 (0x45b0 bytes) 000955 1788587152.546935 trunc from 0x0285ee to 0x00e684 (0x19f69 bytes) 000957 1788587152.580935 read 0x00cde7 thru 0x00e684 (0x189e bytes) 000958 1788587152.582452 read 0x0042b3 thru 0x00e684 (0xa3d2 bytes) 000961 1788587152.594544 read 0x001172 thru 0x00e684 (0xd513 bytes) 000962 1788587152.595965 write 0x03b2df thru 0x03ffff (0x4d21 bytes) 000963 1788587152.597946 mapwrite 0x007e82 thru 0x00ee40 (0x6fbf bytes) 000964 1788587152.673854 mapread 0x01cdcc thru 0x026271 (0x94a6 bytes) 000965 1788587152.694093 punch from 0x01e755 to 0x02dc71 (0xf51d bytes) 000966 1788587152.700047 trunc from 0x040000 to 0x032ef2 (0xd10d bytes) 000967 1788587152.740698 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x32ef3): Mirror 6: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ef3): Mirror 6: 0x43822859 (stale) Mirror 14: 0x43822859 (stale) Mirror 15: 0x43822859 (stale) Mirror 17: 0x43822859 (stale) Mirror 18: 0x43822859 (stale) Mirror 19: 0x43822859 (stale) Mirror 23: 0x43822859 (stale) Mirror 25: 0x43822859 (stale) CRC-32 checksum value for chunk [0, 0x32ef3): Mirror 1: 0x87e8d253 Mirror 26: 0x87e8d253 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ef3: skipped 000968 1788587154.249068 write 0x024073 thru 0x0329bb (0xe949 bytes) 000971 1788587154.428854 mapwrite 0x008704 thru 0x00ba88 (0x3385 bytes) 000972 1788587154.521635 write 0x03e6c6 thru 0x03ffff (0x193a bytes) 000973 1788587154.524366 trunc from 0x040000 to 0x019eb9 (0x26146 bytes) 000974 1788587154.551101 punch from 0x01280b to 0x019eb9 (0x76af bytes) 000975 1788587154.556106 mapread 0x002361 thru 0x00c37c (0xa01c bytes) 000976 1788587154.563938 mapread 0x00b56b thru 0x015b1e (0xa5b4 bytes) 000977 1788587154.574198 mapread 0x013e6c thru 0x016290 (0x2425 bytes) 000978 1788587154.575250 punch from 0x008fc5 to 0x0151cf (0xc20b bytes) 000979 1788587154.589263 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x19eba): Mirror 6: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19eba): Mirror 6: 0x807ef409 (stale) Mirror 14: 0x807ef409 (stale) Mirror 15: 0x807ef409 (stale) Mirror 17: 0x807ef409 (stale) Mirror 18: 0x807ef409 (stale) Mirror 19: 0x807ef409 (stale) Mirror 23: 0x807ef409 (stale) Mirror 25: 0x807ef409 (stale) Mirror 26: 0x74818c27 (stale) CRC-32 checksum value for chunk [0, 0x19eba): Mirror 1: 0x2cad54c Mirror 27: 0x2cad54c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19eba: skipped 000980 1788587156.203487 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x19eba): Mirror 6: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19eba): Mirror 6: 0x807ef409 (stale) Mirror 14: 0x807ef409 (stale) Mirror 15: 0x807ef409 (stale) Mirror 17: 0x807ef409 (stale) Mirror 18: 0x807ef409 (stale) Mirror 19: 0x807ef409 (stale) Mirror 23: 0x807ef409 (stale) Mirror 25: 0x807ef409 (stale) Mirror 26: 0x74818c27 (stale) CRC-32 checksum value for chunk [0, 0x19eba): Mirror 1: 0x2cad54c Mirror 27: 0x2cad54c Mirror 28: 0x2cad54c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19eba: skipped 000982 1788587157.298231 punch from 0x008ec7 to 0x016555 (0xd68f bytes) 000983 1788587157.386862 write 0x03b683 thru 0x03de6d (0x27eb bytes) 000985 1788587157.388771 trunc from 0x03de6e to 0x029cc4 (0x141a9 bytes) 000986 1788587157.414461 punch from 0x026829 to 0x029cc4 (0x349c bytes) 000987 1788587157.424008 MIRROR_RESYNC 12 mirrors 000989 1788587160.142066 mapwrite 0x03b084 thru 0x03ffff (0x4f7c bytes) 000993 1788587160.384135 mapwrite 0x032bfd thru 0x034758 (0x1b5c bytes) 000994 1788587160.456193 mapread 0x010bc8 thru 0x01f321 (0xe75a bytes) 000995 1788587160.508472 MIRROR_RESYNC 12 mirrors 000996 1788587164.582344 trunc from 0x040000 to 0x0162d8 (0x29d27 bytes) 000998 1788587164.694061 MIRROR_RESYNC 12 mirrors 000999 1788587167.580439 mapwrite 0x00b145 thru 0x014f61 (0x9e1d bytes) All operations completed A-OK! PASS 70a (195s) == sanity-flr test 71: check mirror extend parent fid ==== 01:46:13 (1788587173) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 01:46:17 (1788587177) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.405851 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32878 s, 3.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 01:46:27 (1788587187) Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre2 Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5690240 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2611456 .. failed Extending file size to 4635136 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6245600 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8512544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7187840 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7353856 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8209312 .. Extending file size to 8217408 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8535360 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8756576 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8725152 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57412 57414 57415 57418 57419 failed 192.168.201.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg149-client.virtnet /mnt/lustre2 (opts:) 192.168.201.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg149-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:47:52 (1788587272) Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre2 Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4718912 .. Extending file size to 1994912 .. Extending file size to 7405600 .. Extending file size to 5569376 .. Extending file size to 7673344 .. Extending file size to 8789088 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8004640 .. Extending file size to 9092192 .. Extending file size to 8907808 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59076 59077 59078 59082 59083 failed 192.168.201.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg149-client.virtnet /mnt/lustre2 (opts:) 192.168.201.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg149-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:49:10 (1788587350) Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108172 s, 19.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.326288 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe62377ca (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg149-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:49:32 (1788587372) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157069 s, 13.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 01:49:41 (1788587381) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:49:50 (1788587390) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119421 s, 17.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:49:58 (1788587398) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.331249 s, 6.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:50:09 (1788587409) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.467334 s, 4.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } PASS 204c (38s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:50:46 (1788587446) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg149-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19705 s, 5.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:51:52 (1788587512) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg149-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287805 s, 7.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (71s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:53:04 (1788587584) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg149-server mds-ost sync done. Creating new pool test_204f oleg149-server: Pool lustre.test_204f created Adding targets to pool oleg149-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg149-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272318 s, 7.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg149-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg149-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg149-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:54:30 (1788587670) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:54:41 (1788587681) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:54:52 (1788587692) oleg149-server: Pool lustre.test_206 created Adding targets to pool oleg149-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg149-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg149-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg149-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg149-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:55:20 (1788587720) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821913 s, 12.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495451 s, 21.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3c8a75a1 Mirror 3: 0x3c8a75a1 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:55:34 (1788587734) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:55:39 (1788587739) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:55:44 (1788587744) Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.133421 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x810000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1140000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1140000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1140000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25a0000 Read this (4 bytes): BBBB 192.168.201.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg149-client.virtnet /mnt/lustre2 (opts:) PASS 209a (46s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:56:31 (1788587791) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0334312 s, 245 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:56:40 (1788587800) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118515 s, 8.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x203 lmm_fid: [0x200000404:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x203 lmm_fid: [0x200000404:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834435 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 01:56:42.000000000 -0400 Modify: 2026-09-05 01:56:43.000000000 -0400 Change: 2026-09-05 01:56:44.000000000 -0400 Birth: 2026-09-05 01:56:42.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:56:52 (1788587812) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:56:57 (1788587817) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.675602 s, 60.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0867366 s, 47.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 01:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x205:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 01:57 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:57:10 (1788587830) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:57:21 (1788587841) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 2073 sec =========== 01:57:34 (1788587854) === sanity-flr: start cleanup 01:57:36 (1788587856) === === sanity-flr: finish cleanup 01:57:43 (1788587863) ===