-----============= acceptance-small: sanity-flr ============----- Tue Sep 8 01:21:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg160-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:21:40 (1788844900) === oleg160-client.virtnet: executing check_config_client /mnt/lustre oleg160-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg160-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff927447ed4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff927447ed4800.idle_timeout=debug disable quota as required oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:22:23 (1788844943) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8235] == sanity-flr test 0a: lfs mirror create with -N option == 01:22:25 (1788844945) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:22:43 (1788844963) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:22:47 (1788844967) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:22:51 (1788844971) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:23:15 (1788844995) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:23:18 (1788844998) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:23:21 (1788845001) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff927447ed4800.stats=clear osc.lustre-OST0001-osc-ffff927447ed4800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:23:30 (1788845010) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:23:41 (1788845021) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:23:49 (1788845029) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:23:52 (1788845032) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:24:00 (1788845040) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 01:24:11 (1788845051) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:24:19 (1788845059) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:24:27 (1788845067) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:24:34 (1788845074) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:24:45 (1788845085) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:24:52 (1788845092) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.82495 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100829 s, 10.4 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:25:04 (1788845104) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.108292 s, 29.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:25:12 (1788845112) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:25:24 (1788845124) Stopping /mnt/lustre-ost1 (opts:) on oleg160-server oleg160-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg160-server: oleg160-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg160-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg160-server oleg160-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg160-server: oleg160-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg160-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (84s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:26:48 (1788845208) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.088328 s, 23.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg160-server oleg160-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg160-server: oleg160-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg160-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:27:30 (1788845250) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg160-server oleg160-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg160-server: oleg160-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg160-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg160-server oleg160-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg160-server: oleg160-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg160-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (92s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:29:01 (1788845341) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.06925 s, 17.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 8 01:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.046s user 0m0.002s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg160-server oleg160-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927447ed4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.178s user 0m0.002s sys 0m0.193s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg160-server: oleg160-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg160-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg160-server oleg160-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.205s user 0m0.000s sys 0m0.202s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg160-server: oleg160-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg160-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg160-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927447ed4800.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:30:41 (1788845441) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:30:45 (1788845445) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:30:49 (1788845449) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 01:30:53 (1788845453) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 01:31:00 (1788845460) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.721388 s, 17.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:31:14 (1788845474) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423555 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322612 s, 32.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:31:22 (1788845482) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:31:31 (1788845491) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129016 s, 794 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00181096 s, 565 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00259089 s, 395 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.002527 s, 405 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 01:31:42 (1788845502) create test files with size 15745513 .. 2d7ee12cee1a38f4636e0fd8def394b9 - 643cfd45980248bf9d31fe2eabc61f5a - 494a539997b4e7a2b7e4503e7d801576 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg160-client.virtnet /mnt/lustre (opts:) Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 01:32:08 (1788845528) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:32:21 (1788845541) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:32:29 (1788845549) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836278 s, 12.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659401 s, 15.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 01:32:41 (1788845561) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113566 s, 18.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.438735 s, 9.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188786 s, 11.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 01:33:00 (1788845580) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:33:04 (1788845584) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:33:08 (1788845588) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290191 s, 36.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 01:33:16 (1788845596) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.335418 s, 9.4 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:33:27 (1788845607) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:33:38 (1788845618) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.455948 s, 23.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg160-client.virtnet /mnt/lustre (opts:) Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:33:54 (1788845634) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff92746175f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.420202 s, 25.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg160-client.virtnet /mnt/lustre (opts:) Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9274607ae000.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:34:15 (1788845655) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:35:02 (1788845702) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:35:41 (1788845741) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:35:45 (1788845745) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.219052 s, 14.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140806 s, 22.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe4f07876 (nosync,stale) Mirror 2: 0xe4f07876 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1a5e1c580010770bcf39dd4787c5dd97 - mirror 1 checksum: 5354f009b37dbc0c31054f883d0cbbbd - mirror 2 checksum: 1a5e1c580010770bcf39dd4787c5dd97 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T01:35:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T01:35:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1a5e1c580010770bcf39dd4787c5dd97 - mirror 1 checksum: 5354f009b37dbc0c31054f883d0cbbbd - mirror 2 checksum: 5354f009b37dbc0c31054f883d0cbbbd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T01:35:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T01:35:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:35:55 (1788845755) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306369 s, 34.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:36:04 (1788845764) Failing mds1 on oleg160-server Stopping /mnt/lustre-mds1 (opts:) on oleg160-server 01:36:18 (1788845778) shut down facet: mds1 facet_host: oleg160-server facet_failover_host: oleg160-server Failover mds1 to oleg160-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 01:36:52 (1788845812) targets are mounted 01:36:52 (1788845812) facet_failover done oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515125 s, 20.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:37:09 (1788845829) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0469462 s, 436 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.022931 s, 893 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:37:20 (1788845840) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0616995 s, 332 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0198748 s, 1.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:37:30 (1788845850) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0173146 s, 946 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:37:39 (1788845859) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0774258 s, 265 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:37:49 (1788845869) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0138337 s, 296 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0588577 s, 69.6 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:37:59 (1788845879) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:38:25 (1788845905) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:38:55 (1788845935) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:39:19 (1788845959) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:39:44 (1788845984) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 01:39:53 (1788845993) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57595 mirror split pid 57596 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57595 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57596 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 01:41:02 (1788846062) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6065 000001 1788846066.059150 write 0x020129 thru 0x02943d (0x9315 bytes) 000002 1788846066.074191 trunc from 0x018f9f to 0x02561f (0xc681 bytes) 000003 1788846066.095211 mapwrite 0x036708 thru 0x03ffff (0x98f8 bytes) truncating to largest ever: 0x12e20 000004 1788846066.198387 trunc from 0x040000 to 0x012e1f (0x2d1e0 bytes) 000005 1788846066.253177 mapread 0x003b3b thru 0x00e3ae (0xa874 bytes) 000007 1788846066.283940 mapwrite 0x025798 thru 0x03383f (0xe0a8 bytes) 000008 1788846066.443111 mapread 0x02b940 thru 0x032b3b (0x71fc bytes) 000009 1788846066.447783 trunc from 0x01baca to 0x02b4fc (0xfa33 bytes) 000010 1788846066.459469 mapwrite 0x026191 thru 0x027237 (0x10a7 bytes) truncating to largest ever: 0x3dd21 000012 1788846066.518219 trunc from 0x033840 to 0x03dd20 (0xa4e1 bytes) 000014 1788846066.553149 trunc from 0x031f0b to 0x03aba4 (0x8c9a bytes) 000016 1788846066.565032 write 0x032b1d thru 0x03598f (0x2e73 bytes) 000017 1788846066.576053 trunc from 0x03dd21 to 0x032bc0 (0xb160 bytes) 000022 1788846066.602849 trunc from 0x032bc1 to 0x03d1d9 (0xa619 bytes) 000024 1788846066.638056 read 0x01a2cd thru 0x029a5f (0xf793 bytes) 000025 1788846066.664450 trunc from 0x033929 to 0x03d1d9 (0x98b1 bytes) 000026 1788846066.674590 mapread 0x03c7fe thru 0x03d1d9 (0x9dc bytes) 000028 1788846066.707171 mapwrite 0x00179d thru 0x0052ca (0x3b2e bytes) 000029 1788846066.778177 trunc from 0x031b12 to 0x036322 (0x4811 bytes) 000030 1788846066.792289 trunc from 0x02ddc3 to 0x035e3e (0x807c bytes) 000031 1788846066.808942 trunc from 0x03864f to 0x03d1d9 (0x4b8b bytes) 000032 1788846066.815120 trunc from 0x03a922 to 0x03d1d9 (0x28b8 bytes) 000034 1788846066.834981 write 0x012781 thru 0x015d43 (0x35c3 bytes) 000036 1788846066.856251 mapread 0x03bc5e thru 0x03d1d9 (0x157c bytes) 000037 1788846066.886960 fallocate 0x01714b thru 0x01d56e (0x6424 bytes) 000038 1788846066.897874 trunc from 0x0290e9 to 0x033e38 (0xad50 bytes) 000039 1788846066.906833 trunc from 0x038a21 to 0x03a90d (0x1eed bytes) 000040 1788846066.911944 write 0x01ae29 thru 0x01fe0e (0x4fe6 bytes) 000041 1788846066.913069 mapwrite 0x014d81 thru 0x0205da (0xb85a bytes) 000043 1788846067.000965 MIRROR_EXTEND to 2 mirrors 000044 1788846068.531302 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000045 1788846068.675461 mapread 0x01a419 thru 0x023009 (0x8bf1 bytes) 000046 1788846068.693753 trunc from 0x03d1da to 0x03d57f (0x3a6 bytes) 000047 1788846068.734130 trunc from 0x03d580 to 0x039666 (0x3f19 bytes) 000048 1788846068.771331 fallocate 0x01a033 thru 0x01cecf (0x2e9d bytes) 000049 1788846068.776403 trunc from 0x039667 to 0x03393f (0x5d27 bytes) 000050 1788846068.819316 mapwrite 0x01d8e8 thru 0x01edde (0x14f7 bytes) 000052 1788846068.886272 trunc from 0x0265b3 to 0x02888d (0x22db bytes) fallocating to largest ever: 0x3ffdf 000053 1788846068.904837 fallocate 0x0349ce thru 0x03ffde (0xb611 bytes) 000054 1788846068.912050 write 0x026bde thru 0x031a95 (0xaeb8 bytes) 000055 1788846068.933284 mapread 0x0248ab thru 0x02ee8c (0xa5e2 bytes) 000056 1788846068.934204 punch from 0x011d91 to 0x01ba67 (0x9cd7 bytes) 000058 1788846068.956685 trunc from 0x01f7f7 to 0x02e68a (0xee94 bytes) 000059 1788846068.985658 mapread 0x0047b9 thru 0x00d80a (0x9052 bytes) 000060 1788846069.017020 write 0x031a8d thru 0x03d142 (0xb6b6 bytes) 000061 1788846069.035640 punch from 0x021922 to 0x02eb82 (0xd261 bytes) 000062 1788846069.045680 trunc from 0x03ffdf to 0x03fd1c (0x2c2 bytes) 000063 1788846069.081229 trunc from 0x03fd1d to 0x0087ff (0x3751d bytes) 000064 1788846069.117910 mapwrite 0x008520 thru 0x015cb4 (0xd795 bytes) 000065 1788846069.201283 trunc from 0x015cb5 to 0x012b94 (0x3120 bytes) 000066 1788846069.238736 trunc from 0x00c886 to 0x012b94 (0x630f bytes) 000067 1788846069.244765 read 0x005046 thru 0x00f91a (0xa8d5 bytes) 000070 1788846069.268551 read 0x006736 thru 0x00ed13 (0x85de bytes) 000071 1788846069.270733 read 0x004be1 thru 0x00824d (0x366d bytes) 000072 1788846069.272896 mapwrite 0x036fc5 thru 0x03959e (0x25da bytes) 000074 1788846069.360195 punch from 0x003442 to 0x00b8b8 (0x8477 bytes) 000075 1788846069.375729 trunc from 0x03959f to 0x02896e (0x10c30 bytes) 000076 1788846069.412767 write 0x027bfd thru 0x02993e (0x1d42 bytes) 000077 1788846069.416194 write 0x0099df thru 0x0115aa (0x7bcc bytes) 000078 1788846069.432416 trunc from 0x02993f to 0x03c215 (0x128d7 bytes) 000079 1788846069.460160 mapwrite 0x03c55a thru 0x03d544 (0xfeb bytes) 000080 1788846069.550476 mapwrite 0x03ef2d thru 0x03ffff (0x10d3 bytes) 000083 1788846069.638886 mapwrite 0x02bf44 thru 0x032f49 (0x7006 bytes) 000084 1788846069.809203 punch from 0x021235 to 0x02197b (0x747 bytes) 000086 1788846069.818206 trunc from 0x02be1e to 0x02db8c (0x1d6f bytes) 000087 1788846069.846975 fallocate 0x00a589 thru 0x019e13 (0xf88b bytes) 000088 1788846069.862007 trunc from 0x010c2c to 0x014246 (0x361b bytes) 000089 1788846069.882998 mapwrite 0x0057f8 thru 0x012a8b (0xd294 bytes) 000090 1788846069.993084 write 0x00a667 thru 0x01039c (0x5d36 bytes) 000091 1788846069.994306 trunc from 0x0165ee to 0x021f49 (0xb95c bytes) 000092 1788846070.003798 mapwrite 0x0005c7 thru 0x008788 (0x81c2 bytes) 000093 1788846070.051037 trunc from 0x040000 to 0x002ed2 (0x3d12d bytes) 000094 1788846070.067594 punch from 0x000116 to 0x002ed2 (0x2dbd bytes) 000095 1788846070.074708 punch from 0x000aa0 to 0x002dd8 (0x2339 bytes) 000096 1788846070.079822 fallocate 0x010ec3 thru 0x0159a9 (0x4ae7 bytes) 000097 1788846070.086164 trunc from 0x010885 to 0x013024 (0x27a0 bytes) 000098 1788846070.091564 fallocate 0x00c772 thru 0x017904 (0xb193 bytes) 000100 1788846070.097992 mapread 0x0153c4 thru 0x0159a9 (0x5e6 bytes) 000101 1788846070.103209 trunc from 0x0159aa to 0x02e2eb (0x18942 bytes) 000102 1788846070.117407 trunc from 0x02e2ec to 0x0353f3 (0x7108 bytes) 000103 1788846070.133815 fallocate 0x01e662 thru 0x02d6f0 (0xf08f bytes) 000104 1788846070.139853 trunc from 0x00289a to 0x009723 (0x6e8a bytes) 000105 1788846070.147782 mapwrite 0x036768 thru 0x03c7e0 (0x6079 bytes) 000106 1788846070.273181 mapwrite 0x006c26 thru 0x0110d1 (0xa4ac bytes) 000107 1788846070.382906 trunc from 0x01a4e3 to 0x028c1c (0xe73a bytes) 000108 1788846070.388884 trunc from 0x02bdf3 to 0x039ff8 (0xe206 bytes) 000109 1788846070.394204 punch from 0x028473 to 0x031fa2 (0x9b30 bytes) 000110 1788846070.399005 fallocate 0x026cb6 thru 0x0336e5 (0xca30 bytes) 000111 1788846070.404717 write 0x032d27 thru 0x03b3b3 (0x868d bytes) 000112 1788846070.413435 write 0x00388f thru 0x00da1e (0xa190 bytes) 000113 1788846070.420624 mapread 0x01bd14 thru 0x02b6e1 (0xf9ce bytes) 000114 1788846070.524453 punch from 0x002dd0 to 0x00c63f (0x9870 bytes) 000115 1788846070.550942 fallocate 0x021732 thru 0x028c5e (0x752d bytes) 000116 1788846070.556715 mapread 0x019553 thru 0x026e3f (0xd8ed bytes) 000117 1788846070.572818 trunc from 0x015946 to 0x01c96f (0x702a bytes) 000118 1788846070.581270 mapwrite 0x01a795 thru 0x01b316 (0xb82 bytes) 000119 1788846070.612828 punch from 0x01dde7 to 0x0247cd (0x69e7 bytes) 000120 1788846070.622032 punch from 0x02253c to 0x02ff28 (0xd9ed bytes) 000121 1788846070.629100 mapread 0x00980f thru 0x00db6d (0x435f bytes) 000122 1788846070.665033 trunc from 0x03c080 to 0x03c7e0 (0x761 bytes) 000124 1788846070.678652 mapread 0x015078 thru 0x01766b (0x25f4 bytes) 000125 1788846070.723393 trunc from 0x03c7e1 to 0x0039e1 (0x38dff bytes) 000126 1788846070.743673 read 0x00224b thru 0x00339b (0x1151 bytes) 000128 1788846070.753265 trunc from 0x003072 to 0x0039e1 (0x970 bytes) 000129 1788846070.759876 write 0x02391d thru 0x024b26 (0x120a bytes) 000132 1788846070.764700 read 0x00ed11 thru 0x0140ed (0x53dd bytes) 000133 1788846070.776236 read 0x0198a1 thru 0x0237d8 (0x9f38 bytes) 000135 1788846070.783861 MIRROR_EXTEND to 2 mirrors 000136 1788846071.739224 fallocate 0x01a128 thru 0x027d38 (0xdc11 bytes) 000137 1788846071.774046 punch from 0x0159a0 to 0x015abf (0x120 bytes) 000138 1788846071.781046 trunc from 0x015a42 to 0x01a7bd (0x4d7c bytes) 000139 1788846071.802691 mapread 0x0091e2 thru 0x00a6ec (0x150b bytes) 000140 1788846071.814122 mapread 0x012b67 thru 0x01d3e4 (0xa87e bytes) 000141 1788846071.833594 write 0x005a26 thru 0x0157d8 (0xfdb3 bytes) 000142 1788846071.846697 mapwrite 0x00187f thru 0x010000 (0xe782 bytes) 000143 1788846071.883900 read 0x00533e thru 0x008ec1 (0x3b84 bytes) 000144 1788846071.884558 fallocate 0x01ed96 thru 0x021c8a (0x2ef5 bytes) 000145 1788846071.891429 mapread 0x018120 thru 0x01f551 (0x7432 bytes) 000146 1788846071.897161 punch from 0x010628 to 0x01e1d0 (0xdba9 bytes) 000147 1788846071.917846 write 0x028015 thru 0x02eec8 (0x6eb4 bytes) 000148 1788846071.921922 mapread 0x004bf3 thru 0x00d671 (0x8a7f bytes) 000149 1788846071.934988 MIRROR_RESYNC 2 mirrors 000150 1788846073.047040 trunc from 0x026c17 to 0x0276dc (0xac6 bytes) 000151 1788846073.083959 mapwrite 0x038b7d thru 0x03ffff (0x7483 bytes) 000152 1788846073.147173 punch from 0x002404 to 0x007ee7 (0x5ae4 bytes) 000153 1788846073.153078 trunc from 0x040000 to 0x00ea5f (0x315a0 bytes) 000154 1788846073.168224 read 0x005987 thru 0x00ab73 (0x51ed bytes) 000155 1788846073.179678 mapread 0x0059e0 thru 0x0077cf (0x1df0 bytes) 000156 1788846073.180666 mapwrite 0x027325 thru 0x02e09d (0x6d79 bytes) 000157 1788846073.226463 fallocate 0x01a8ce thru 0x01cd44 (0x2477 bytes) 000158 1788846073.236901 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000159 1788846073.381208 trunc from 0x02e09e to 0x00e7fd (0x1f8a0 bytes) 000160 1788846073.404671 fallocate 0x008258 thru 0x01101a (0x8dc3 bytes) 000161 1788846073.409877 mapwrite 0x010794 thru 0x019dff (0x966c bytes) 000163 1788846073.461314 punch from 0x000470 to 0x00ef01 (0xea92 bytes) 000164 1788846073.466412 fallocate 0x01c15f thru 0x0278ef (0xb791 bytes) 000166 1788846073.472545 read 0x004c37 thru 0x00848e (0x3858 bytes) 000167 1788846073.486902 read 0x00a2d8 thru 0x015f50 (0xbc79 bytes) 000168 1788846073.501961 read 0x01c1c7 thru 0x0228b1 (0x66eb bytes) 000169 1788846073.502626 trunc from 0x01cc45 to 0x02392e (0x6cea bytes) 000170 1788846073.508035 mapread 0x01d89e thru 0x0278ef (0xa052 bytes) 000171 1788846073.521703 read 0x014f6e thru 0x0157fa (0x88d bytes) 000172 1788846073.522611 punch from 0x0047c7 to 0x008b16 (0x4350 bytes) 000173 1788846073.532826 trunc from 0x01684c to 0x023e33 (0xd5e8 bytes) 000176 1788846073.543975 trunc from 0x0278f0 to 0x0187d8 (0xf117 bytes) 000177 1788846073.569777 trunc from 0x0187d9 to 0x03e053 (0x2587b bytes) 000178 1788846073.587920 mapwrite 0x0049f9 thru 0x011663 (0xcc6b bytes) 000179 1788846073.657685 read 0x0146f6 thru 0x014e85 (0x790 bytes) 000180 1788846073.658276 mapread 0x01d2d8 thru 0x026fea (0x9d13 bytes) 000181 1788846073.659190 write 0x0375c3 thru 0x03ffff (0x8a3d bytes) 000183 1788846073.665664 fallocate 0x01fd30 thru 0x02af54 (0xb225 bytes) 000184 1788846073.672889 trunc from 0x007b99 to 0x011453 (0x98bb bytes) 000185 1788846073.680045 read 0x0392d3 thru 0x03ffff (0x6d2d bytes) 000187 1788846073.682277 punch from 0x03363d to 0x03ffff (0xc9c3 bytes) 000188 1788846073.701572 write 0x0267ef thru 0x02db26 (0x7338 bytes) 000189 1788846073.704771 mapwrite 0x03103b thru 0x033ea1 (0x2e67 bytes) 000190 1788846073.741360 fallocate 0x01cf03 thru 0x0299ae (0xcaac bytes) 000191 1788846073.747339 punch from 0x020efa to 0x02a360 (0x9467 bytes) 000192 1788846073.755091 trunc from 0x01551d to 0x01d71e (0x8202 bytes) 000194 1788846073.761930 trunc from 0x040000 to 0x01dfef (0x22010 bytes) 000195 1788846073.779883 mapread 0x003918 thru 0x00c0d5 (0x87be bytes) 000196 1788846073.780957 trunc from 0x01dff0 to 0x01cd32 (0x12bd bytes) 000197 1788846073.796469 punch from 0x001780 to 0x00ca6f (0xb2f0 bytes) 000199 1788846073.803513 write 0x029bf1 thru 0x034635 (0xaa45 bytes) fallocating to largest ever: 0x40000 000201 1788846073.807361 fallocate 0x03e681 thru 0x03ffff (0x197f bytes) 000202 1788846073.816417 MIRROR_EXTEND to 2 mirrors 000203 1788846074.520295 punch from 0x03f995 to 0x03ffff (0x66b bytes) 000204 1788846074.584803 read 0x034939 thru 0x03ffff (0xb6c7 bytes) 000205 1788846074.602629 read 0x02ce28 thru 0x037840 (0xaa19 bytes) 000206 1788846074.632235 mapread 0x0022b5 thru 0x0094fe (0x724a bytes) 000207 1788846074.633189 mapwrite 0x00bf00 thru 0x015a0f (0x9b10 bytes) 000209 1788846074.686736 read 0x012d6e thru 0x0150be (0x2351 bytes) 000210 1788846074.687839 mapwrite 0x02704e thru 0x032695 (0xb648 bytes) 000211 1788846074.724664 mapwrite 0x008fd5 thru 0x011426 (0x8452 bytes) 000212 1788846074.758062 mapwrite 0x001876 thru 0x008b80 (0x730b bytes) 000213 1788846074.790429 fallocate 0x013af5 thru 0x01c77f (0x8c8b bytes) 000215 1788846074.797122 mapwrite 0x03677a thru 0x03ffff (0x9886 bytes) 000216 1788846074.825477 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbbbe9d28 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfeaf6067 Mirror 3: 0xfeaf6067 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000217 1788846076.054577 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbbbe9d28 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfeaf6067 Mirror 3: 0xfeaf6067 Mirror 4: 0xfeaf6067 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000218 1788846076.959422 mapwrite 0x000eb2 thru 0x00c369 (0xb4b8 bytes) 000219 1788846077.123699 mapread 0x013db2 thru 0x0211b1 (0xd400 bytes) 000220 1788846077.143693 punch from 0x028d49 to 0x02e334 (0x55ec bytes) 000221 1788846077.154372 mapwrite 0x031ba8 thru 0x03a5a8 (0x8a01 bytes) 000222 1788846077.283329 mapwrite 0x039106 thru 0x03ffff (0x6efa bytes) 000223 1788846077.330745 MIRROR_RESYNC 4 mirrors 000224 1788846078.353415 punch from 0x0300fe to 0x034440 (0x4343 bytes) 000225 1788846078.402000 trunc from 0x01488d to 0x022581 (0xdcf5 bytes) 000226 1788846078.413804 MIRROR_RESYNC 4 mirrors 000227 1788846079.613929 write 0x03aea1 thru 0x03b117 (0x277 bytes) 000228 1788846079.658144 fallocate 0x020178 thru 0x02fe07 (0xfc90 bytes) 000229 1788846079.670711 trunc from 0x040000 to 0x01d5d1 (0x22a2e bytes) 000231 1788846079.715923 trunc from 0x01d5d2 to 0x01df4b (0x97a bytes) 000232 1788846079.740645 read 0x00abf2 thru 0x010910 (0x5d1f bytes) 000233 1788846079.755040 fallocate 0x00d118 thru 0x00fe45 (0x2d2e bytes) 000234 1788846079.767179 fallocate 0x01923e thru 0x020c40 (0x7a03 bytes) 000236 1788846079.774040 trunc from 0x01df4c to 0x0242fc (0x63b1 bytes) 000237 1788846079.793479 mapwrite 0x02abdb thru 0x0377d7 (0xcbfd bytes) 000238 1788846079.853971 fallocate 0x03d914 thru 0x03dcb3 (0x3a0 bytes) 000239 1788846079.868644 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x377d8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377d8): Mirror 2: 0x3543143b (stale) Mirror 3: 0x3543143b (stale) Mirror 4: 0x3543143b (stale) CRC-32 checksum value for chunk [0, 0x377d8): Mirror 1: 0x4a7d8c26 Mirror 5: 0x4a7d8c26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377d8: skipped 000240 1788846080.720759 fallocate 0x0273cd thru 0x02982e (0x2462 bytes) 000243 1788846080.769881 trunc from 0x0377d8 to 0x010513 (0x272c4 bytes) 000244 1788846080.800985 trunc from 0x010514 to 0x002739 (0xddda bytes) 000245 1788846080.824198 read 0x00195b thru 0x002739 (0xddf bytes) 000247 1788846080.831748 mapread 0x002711 thru 0x002739 (0x29 bytes) 000248 1788846080.832743 mapread 0x000a6f thru 0x002739 (0x1ccb bytes) 000249 1788846080.839375 fallocate 0x03532e thru 0x0373cf (0x20a2 bytes) 000250 1788846080.845165 mapread 0x019524 thru 0x0255d6 (0xc0b3 bytes) 000251 1788846080.872638 trunc from 0x0373d0 to 0x0258a5 (0x11b2a bytes) 000252 1788846080.899329 mapwrite 0x03c611 thru 0x03f618 (0x3008 bytes) 000253 1788846080.960876 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f619): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f619): Mirror 2: 0x7b439d34 (stale) Mirror 3: 0x7b439d34 (stale) Mirror 4: 0x7b439d34 (stale) Mirror 5: 0xd809c515 (stale) CRC-32 checksum value for chunk [0, 0x3f619): Mirror 1: 0x2a59499f Mirror 6: 0x2a59499f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f619: skipped 000254 1788846082.280117 write 0x004a91 thru 0x00f8e1 (0xae51 bytes) 000255 1788846082.384697 fallocate 0x006620 thru 0x0130b9 (0xca9a bytes) 000256 1788846082.401769 read 0x014dae thru 0x016c20 (0x1e73 bytes) 000257 1788846082.410313 mapwrite 0x0103f7 thru 0x01b9f8 (0xb602 bytes) 000259 1788846082.519546 trunc from 0x02407d to 0x0303e5 (0xc369 bytes) 000261 1788846082.529356 trunc from 0x03f619 to 0x0127f5 (0x2ce23 bytes) 000263 1788846082.562570 trunc from 0x0127f6 to 0x017ec9 (0x56d4 bytes) 000264 1788846082.582023 trunc from 0x017eca to 0x02a2fe (0x12435 bytes) 000266 1788846082.645056 write 0x007fef thru 0x016c56 (0xec68 bytes) 000267 1788846082.655856 mapread 0x01515f thru 0x0157b3 (0x655 bytes) 000268 1788846082.657727 write 0x00a8d0 thru 0x00de72 (0x35a3 bytes) 000269 1788846082.664096 MIRROR_RESYNC 6 mirrors 000270 1788846084.373773 trunc from 0x02a2ff to 0x00a923 (0x1f9db bytes) 000271 1788846084.419956 fallocate 0x012a2c thru 0x01b60a (0x8bdf bytes) 000272 1788846084.428049 mapread 0x019f84 thru 0x01a4ac (0x529 bytes) 000273 1788846084.443673 read 0x014e05 thru 0x018376 (0x3572 bytes) 000274 1788846084.452123 mapwrite 0x008f38 thru 0x0112f6 (0x83bf bytes) 000275 1788846084.492895 read 0x00deae thru 0x017c26 (0x9d79 bytes) 000276 1788846084.493691 mapwrite 0x002787 thru 0x006ff8 (0x4872 bytes) 000277 1788846084.511447 mapread 0x014f01 thru 0x0184f9 (0x35f9 bytes) 000278 1788846084.512317 trunc from 0x01b60b to 0x01315f (0x84ab bytes) 000279 1788846084.530724 fallocate 0x0016d1 thru 0x004f82 (0x38b2 bytes) 000280 1788846084.536580 mapwrite 0x02a809 thru 0x02f2b0 (0x4aa8 bytes) 000281 1788846084.579717 fallocate 0x020a38 thru 0x026aa9 (0x6072 bytes) 000282 1788846084.585621 mapwrite 0x03d347 thru 0x03ffff (0x2cb9 bytes) 000283 1788846084.632284 write 0x036040 thru 0x03ffff (0x9fc0 bytes) 000284 1788846084.634091 punch from 0x0083f0 to 0x015c27 (0xd838 bytes) 000285 1788846084.640297 read 0x03341b thru 0x039161 (0x5d47 bytes) 000286 1788846084.640949 mapread 0x019231 thru 0x01f289 (0x6059 bytes) 000287 1788846084.642395 mapread 0x00b87c thru 0x01869e (0xce23 bytes) 000288 1788846084.649978 mapwrite 0x0328bb thru 0x03ffff (0xd745 bytes) 000289 1788846084.671432 write 0x00c4fb thru 0x00e48e (0x1f94 bytes) 000290 1788846084.672605 read 0x010073 thru 0x01ceed (0xce7b bytes) 000291 1788846084.673635 punch from 0x02fbd2 to 0x03c571 (0xc9a0 bytes) 000292 1788846084.679408 mapwrite 0x038db8 thru 0x03ffff (0x7248 bytes) 000293 1788846084.707711 trunc from 0x0244d8 to 0x03059f (0xc0c8 bytes) 000294 1788846084.713630 trunc from 0x00c31a to 0x013cdc (0x79c3 bytes) 000295 1788846084.724851 MIRROR_RESYNC 6 mirrors 000296 1788846086.188705 fallocate 0x029bd4 thru 0x035902 (0xbd2f bytes) 000297 1788846086.221117 punch from 0x02999d to 0x02e78a (0x4dee bytes) 000298 1788846086.226733 mapwrite 0x01120e thru 0x020441 (0xf234 bytes) 000299 1788846086.297957 mapread 0x01f711 thru 0x02439f (0x4c8f bytes) 000302 1788846086.299861 write 0x039e9b thru 0x03d74c (0x38b2 bytes) 000304 1788846086.316793 trunc from 0x000f57 to 0x0059db (0x4a85 bytes) 000305 1788846086.330090 write 0x0268e8 thru 0x0312dd (0xa9f6 bytes) 000307 1788846086.353883 read 0x01c842 thru 0x02691a (0xa0d9 bytes) 000308 1788846086.361534 write 0x02ac86 thru 0x02c624 (0x199f bytes) 000309 1788846086.363281 fallocate 0x033d0c thru 0x03fe3f (0xc134 bytes) 000310 1788846086.369568 fallocate 0x007502 thru 0x00ef56 (0x7a55 bytes) 000311 1788846086.375599 read 0x0096f9 thru 0x00eb61 (0x5469 bytes) 000312 1788846086.411119 trunc from 0x040000 to 0x011bb7 (0x2e448 bytes) 000313 1788846086.465419 trunc from 0x011bb8 to 0x03b463 (0x298ac bytes) 000315 1788846086.513372 MIRROR_RESYNC 6 mirrors 000317 1788846088.857922 punch from 0x009f00 to 0x01876e (0xe86f bytes) 000318 1788846088.891716 mapwrite 0x0385b7 thru 0x03ffff (0x7a49 bytes) 000319 1788846088.944280 trunc from 0x040000 to 0x03cfde (0x3021 bytes) 000320 1788846088.963822 punch from 0x02a442 to 0x0370ac (0xcc6b bytes) 000321 1788846088.987569 punch from 0x01fe67 to 0x024295 (0x442f bytes) 000322 1788846088.999417 trunc from 0x033c94 to 0x033d2f (0x9c bytes) 000324 1788846089.020306 read 0x00138e thru 0x007784 (0x63f7 bytes) 000326 1788846089.068089 mapread 0x006200 thru 0x0079f1 (0x17f2 bytes) 000327 1788846089.078415 MIRROR_RESYNC 6 mirrors 000328 1788846090.832439 mapread 0x0390bb thru 0x03972b (0x671 bytes) 000329 1788846090.842441 trunc from 0x03cfdf to 0x0338e1 (0x96fd bytes) 000330 1788846090.961475 fallocate 0x038cff thru 0x039c70 (0xf72 bytes) 000331 1788846090.983427 trunc from 0x022259 to 0x02806f (0x5e17 bytes) 000332 1788846090.997530 mapwrite 0x0369bc thru 0x038391 (0x19d6 bytes) 000333 1788846091.120052 trunc from 0x038392 to 0x030f53 (0x743e bytes) 000334 1788846091.146528 write 0x037471 thru 0x03ca68 (0x55f8 bytes) 000335 1788846091.160052 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ca69): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca69): Mirror 2: 0x587ab7ac (stale) Mirror 3: 0x587ab7ac (stale) Mirror 4: 0x587ab7ac (stale) Mirror 5: 0x587ab7ac (stale) Mirror 6: 0x587ab7ac (stale) CRC-32 checksum value for chunk [0, 0x3ca69): Mirror 1: 0xc7ad5c45 Mirror 7: 0xc7ad5c45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca69: skipped 000336 1788846092.664459 fallocate 0x00507b thru 0x010a09 (0xb98f bytes) 000337 1788846092.716725 read 0x000ab2 thru 0x005341 (0x4890 bytes) 000338 1788846092.736296 punch from 0x023531 to 0x02f303 (0xbdd3 bytes) 000339 1788846092.747953 mapread 0x02c784 thru 0x039f7f (0xd7fc bytes) 000340 1788846092.812626 trunc from 0x03ca69 to 0x026459 (0x1660f bytes) 000341 1788846092.859590 mapwrite 0x03c323 thru 0x03ffff (0x3cdd bytes) 000343 1788846092.995069 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x476468c2 (stale) Mirror 4: 0x476468c2 (stale) Mirror 5: 0x476468c2 (stale) Mirror 6: 0x476468c2 (stale) Mirror 7: 0x3bb9fa43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x476468c2 (stale) Mirror 4: 0x476468c2 (stale) Mirror 5: 0x476468c2 (stale) Mirror 6: 0x476468c2 (stale) Mirror 7: 0x3bb9fa43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000346 1788846093.638474 read 0x0188b5 thru 0x022f69 (0xa6b5 bytes) 000347 1788846093.656993 mapread 0x0093ef thru 0x009bc5 (0x7d7 bytes) 000348 1788846093.662467 mapread 0x013e0b thru 0x020045 (0xc23b bytes) 000349 1788846093.684417 trunc from 0x040000 to 0x03ddb5 (0x224a bytes) 000350 1788846093.717267 mapread 0x02f834 thru 0x03cf8b (0xd758 bytes) 000351 1788846093.756548 write 0x01cb18 thru 0x02c196 (0xf67f bytes) 000352 1788846093.758241 punch from 0x014349 to 0x01f3f0 (0xb0a8 bytes) 000353 1788846093.779481 punch from 0x001e9e to 0x006981 (0x4ae4 bytes) 000354 1788846093.793256 mapread 0x007096 thru 0x00cf7d (0x5ee8 bytes) 000355 1788846093.815353 read 0x0305a8 thru 0x033bb5 (0x360e bytes) 000356 1788846093.816030 trunc from 0x02d3f1 to 0x038da9 (0xb9b9 bytes) 000357 1788846093.825461 trunc from 0x03ddb6 to 0x013b3a (0x2a27b bytes) 000358 1788846093.859189 mapwrite 0x0108d0 thru 0x01827a (0x79ab bytes) 000359 1788846093.945376 fallocate 0x000874 thru 0x00bf95 (0xb722 bytes) 000360 1788846093.958377 mapwrite 0x0307fe thru 0x03d6a5 (0xcea8 bytes) 000361 1788846094.072828 trunc from 0x004102 to 0x013bf6 (0xfaf5 bytes) 000362 1788846094.107479 MIRROR_RESYNC 6 mirrors 000363 1788846096.288656 mapwrite 0x016462 thru 0x024027 (0xdbc6 bytes) 000364 1788846096.509923 fallocate 0x023b02 thru 0x02ea9c (0xaf9b bytes) 000365 1788846096.517765 mapwrite 0x021100 thru 0x02d981 (0xc882 bytes) 000366 1788846096.591444 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d6a6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6a6): Mirror 3: 0xeffb37c5 (stale) Mirror 4: 0xeffb37c5 (stale) Mirror 5: 0xeffb37c5 (stale) Mirror 6: 0xeffb37c5 (stale) Mirror 7: 0xeffb37c5 (stale) CRC-32 checksum value for chunk [0, 0x3d6a6): Mirror 1: 0x2864cacf Mirror 8: 0x2864cacf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6a6: skipped 000367 1788846098.405245 write 0x03a7e6 thru 0x03e6df (0x3efa bytes) 000368 1788846098.477771 trunc from 0x03e6e0 to 0x034750 (0x9f8f bytes) 000369 1788846098.534140 read 0x033de4 thru 0x034750 (0x96d bytes) 000370 1788846098.542454 punch from 0x032adb to 0x034750 (0x1c76 bytes) 000371 1788846098.548612 trunc from 0x034751 to 0x0177d9 (0x1cf77 bytes) 000372 1788846098.573490 mapread 0x001051 thru 0x007c58 (0x6c08 bytes) 000374 1788846098.640453 read 0x00120d thru 0x00e99a (0xd78e bytes) 000375 1788846098.653990 MIRROR_RESYNC 7 mirrors 000377 1788846100.333374 punch from 0x00a171 to 0x016204 (0xc094 bytes) 000378 1788846100.382007 read 0x01720a thru 0x0177d9 (0x5d0 bytes) 000380 1788846100.388727 mapread 0x00804f thru 0x016026 (0xdfd8 bytes) 000381 1788846100.424479 mapread 0x000d92 thru 0x00c85f (0xbace bytes) 000382 1788846100.440944 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x177da): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x177da): Mirror 3: 0xf980106f (stale) Mirror 4: 0xf980106f (stale) Mirror 5: 0xf980106f (stale) Mirror 6: 0xf980106f (stale) Mirror 7: 0xf980106f (stale) Mirror 8: 0xf980106f (stale) CRC-32 checksum value for chunk [0, 0x177da): Mirror 1: 0x20cfda94 Mirror 9: 0x20cfda94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x177da: skipped 000383 1788846102.161385 fallocate 0x0179fb thru 0x01adea (0x33f0 bytes) 000384 1788846102.239625 read 0x0100ba thru 0x0177d9 (0x7720 bytes) 000385 1788846102.264006 punch from 0x000535 to 0x00a25d (0x9d29 bytes) 000386 1788846102.270486 trunc from 0x017476 to 0x0177d9 (0x364 bytes) 000387 1788846102.280440 trunc from 0x0177da to 0x01da52 (0x6279 bytes) 000389 1788846102.312954 punch from 0x003cb3 to 0x007ce0 (0x402e bytes) 000390 1788846102.318673 fallocate 0x00da85 thru 0x01184a (0x3dc6 bytes) 000391 1788846102.324956 trunc from 0x01da53 to 0x00c11e (0x11934 bytes) 000392 1788846102.354995 trunc from 0x0022df to 0x003ae2 (0x1804 bytes) 000393 1788846102.360322 trunc from 0x0077c0 to 0x0094af (0x1cf0 bytes) 000394 1788846102.370120 MIRROR_RESYNC 8 mirrors 000395 1788846104.147092 mapwrite 0x0063d0 thru 0x0131ef (0xce20 bytes) 000396 1788846104.367178 read 0x00f7b3 thru 0x0131ef (0x3a3d bytes) 000397 1788846104.367931 fallocate 0x0289ac thru 0x02adf2 (0x2447 bytes) 000398 1788846104.381203 fallocate 0x03e130 thru 0x03ffff (0x1ed0 bytes) 000399 1788846104.394319 punch from 0x024f7d to 0x025c4d (0xcd1 bytes) 000400 1788846104.409270 trunc from 0x040000 to 0x03715b (0x8ea4 bytes) 000401 1788846104.462128 fallocate 0x03e0fa thru 0x03ffff (0x1f06 bytes) 000402 1788846104.469732 trunc from 0x03715c to 0x0046c7 (0x32a94 bytes) 000403 1788846104.565025 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x46c8): Mirror 3: 0x23948209 (stale) Mirror 4: 0x23948209 (stale) Mirror 6: 0x23948209 (stale) Mirror 7: 0x23948209 (stale) Mirror 8: 0x23948209 (stale) Mirror 9: 0x23948209 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x46c8): Mirror 3: 0x23948209 (stale) Mirror 4: 0x23948209 (stale) Mirror 6: 0x23948209 (stale) Mirror 7: 0x23948209 (stale) Mirror 8: 0x23948209 (stale) Mirror 9: 0x23948209 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x46c8: skipped 000404 1788846105.320439 trunc from 0x0046c8 to 0x031679 (0x2cfb2 bytes) 000405 1788846105.360672 trunc from 0x03167a to 0x009baf (0x27aca bytes) 000406 1788846105.398008 trunc from 0x003aca to 0x009baf (0x60e6 bytes) 000407 1788846105.404112 read 0x003078 thru 0x009baf (0x6b38 bytes) 000408 1788846105.442166 fallocate 0x034e76 thru 0x03d766 (0x88f1 bytes) 000409 1788846105.453778 write 0x01a790 thru 0x01ef07 (0x4778 bytes) 000411 1788846105.470134 trunc from 0x01ef08 to 0x028e06 (0x9eff bytes) 000412 1788846105.518290 punch from 0x01a319 to 0x0264ce (0xc1b6 bytes) 000413 1788846105.551191 fallocate 0x00076c thru 0x007259 (0x6aee bytes) 000414 1788846105.565180 mapwrite 0x01ba8e thru 0x0260ce (0xa641 bytes) 000415 1788846105.671985 trunc from 0x028e07 to 0x035a76 (0xcc70 bytes) 000416 1788846105.693616 fallocate 0x02d06c thru 0x03c7eb (0xf780 bytes) 000417 1788846105.713014 trunc from 0x0151f2 to 0x019e50 (0x4c5f bytes) 000418 1788846105.734414 write 0x03413f thru 0x03ffff (0xbec1 bytes) 000420 1788846105.744312 fallocate 0x0372fb thru 0x03c7d2 (0x54d8 bytes) 000421 1788846105.760188 mapread 0x0364b1 thru 0x03baa5 (0x55f5 bytes) 000423 1788846105.764449 mapread 0x01747e thru 0x02119d (0x9d20 bytes) 000424 1788846105.782094 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000425 1788846106.257034 fallocate 0x012912 thru 0x018435 (0x5b24 bytes) 000426 1788846106.270266 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc519c9a0 Mirror 9: 0xc519c9a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000428 1788846107.952637 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc519c9a0 Mirror 9: 0xc519c9a0 Mirror 10: 0xc519c9a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000429 1788846109.624773 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc519c9a0 Mirror 9: 0xc519c9a0 Mirror 10: 0xc519c9a0 Mirror 11: 0xc519c9a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000431 1788846111.105863 mapwrite 0x03adb6 thru 0x03ffff (0x524a bytes) 000432 1788846111.270182 read 0x0186f6 thru 0x01ae68 (0x2773 bytes) 000433 1788846111.284096 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xc519c9a0 (stale) Mirror 10: 0xc519c9a0 (stale) Mirror 11: 0xc519c9a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xc519c9a0 (stale) Mirror 10: 0xc519c9a0 (stale) Mirror 11: 0xc519c9a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000434 1788846112.167979 mapwrite 0x015de2 thru 0x01afc8 (0x51e7 bytes) 000435 1788846112.253346 fallocate 0x0243cc thru 0x02aebd (0x6af2 bytes) 000436 1788846112.259457 fallocate 0x0197d6 thru 0x0224d0 (0x8cfb bytes) 000437 1788846112.276023 punch from 0x034fd2 to 0x03be70 (0x6e9f bytes) 000438 1788846112.286447 punch from 0x01f3ff to 0x02a360 (0xaf62 bytes) 000439 1788846112.294427 mapread 0x01e096 thru 0x022b75 (0x4ae0 bytes) 000440 1788846112.304651 trunc from 0x02b38b to 0x034b36 (0x97ac bytes) 000441 1788846112.310974 mapread 0x018f74 thru 0x01995a (0x9e7 bytes) 000442 1788846112.313239 mapread 0x031c38 thru 0x03ffff (0xe3c8 bytes) 000443 1788846112.353206 write 0x00d0e3 thru 0x0117a0 (0x46be bytes) 000444 1788846112.377863 mapread 0x03c04b thru 0x03ffff (0x3fb5 bytes) 000445 1788846112.378954 read 0x00226e thru 0x007331 (0x50c4 bytes) 000446 1788846112.399096 fallocate 0x0205bf thru 0x026647 (0x6089 bytes) 000447 1788846112.405315 mapwrite 0x01b4a4 thru 0x02662d (0xb18a bytes) 000449 1788846112.445674 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xc519c9a0 (stale) Mirror 10: 0xc519c9a0 (stale) Mirror 11: 0xc519c9a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5bdd94e Mirror 12: 0xf5bdd94e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000450 1788846113.812936 mapwrite 0x016ec7 thru 0x02533d (0xe477 bytes) 000451 1788846113.972138 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xc519c9a0 (stale) Mirror 10: 0xc519c9a0 (stale) Mirror 12: 0xf5bdd94e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xc519c9a0 (stale) Mirror 10: 0xc519c9a0 (stale) Mirror 12: 0xf5bdd94e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000452 1788846114.633336 mapwrite 0x01fe75 thru 0x02a670 (0xa7fc bytes) 000453 1788846114.761871 trunc from 0x040000 to 0x00c2c4 (0x33d3b bytes) 000454 1788846114.840969 read 0x004169 thru 0x00c2c4 (0x815c bytes) 000455 1788846114.854279 punch from 0x00ac11 to 0x00c2c4 (0x16b4 bytes) 000457 1788846114.882302 MIRROR_RESYNC 8 mirrors 000458 1788846116.488451 trunc from 0x0079c3 to 0x00c2c4 (0x4902 bytes) 000459 1788846116.531002 trunc from 0x00b511 to 0x00c2c4 (0xdb4 bytes) 000460 1788846116.541962 punch from 0x005a8b to 0x00c2c4 (0x683a bytes) 000461 1788846116.549011 punch from 0x000ef4 to 0x004cfc (0x3e09 bytes) 000462 1788846116.557655 mapwrite 0x0040a7 thru 0x00e245 (0xa19f bytes) 000463 1788846116.635212 mapwrite 0x028711 thru 0x02ced1 (0x47c1 bytes) 000464 1788846116.690231 write 0x022bae thru 0x031a77 (0xeeca bytes) 000465 1788846116.706030 MIRROR_RESYNC 8 mirrors 000466 1788846119.460841 mapwrite 0x0120a9 thru 0x018350 (0x62a8 bytes) 000467 1788846119.641533 mapwrite 0x029985 thru 0x03767d (0xdcf9 bytes) 000468 1788846119.703581 write 0x020e5b thru 0x0244f1 (0x3697 bytes) 000469 1788846119.709836 fallocate 0x00153e thru 0x009f77 (0x8a3a bytes) 000470 1788846119.716243 mapwrite 0x002790 thru 0x00336c (0xbdd bytes) 000472 1788846119.747681 mapread 0x0256cd thru 0x02f49b (0x9dcf bytes) 000473 1788846119.749922 trunc from 0x03767e to 0x00ab8a (0x2caf3 bytes) 000474 1788846119.771912 trunc from 0x001d8a to 0x00439f (0x2616 bytes) 000475 1788846119.779328 mapread 0x006fc8 thru 0x00ab8a (0x3bc3 bytes) 000477 1788846119.782096 trunc from 0x000d8e to 0x008c8d (0x7f00 bytes) 000478 1788846119.788798 read 0x00a955 thru 0x00ab8a (0x236 bytes) 000479 1788846119.789706 fallocate 0x00d2a3 thru 0x017f10 (0xac6e bytes) 000481 1788846119.795485 trunc from 0x017f11 to 0x01df31 (0x6021 bytes) 000482 1788846119.823622 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1df32): Mirror 6: 0x50c8b464 (stale) Mirror 7: 0x50c8b464 (stale) Mirror 8: 0x50c8b464 (stale) Mirror 9: 0x50c8b464 (stale) Mirror 10: 0x50c8b464 (stale) Mirror 12: 0x50c8b464 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1df32): Mirror 6: 0x50c8b464 (stale) Mirror 7: 0x50c8b464 (stale) Mirror 8: 0x50c8b464 (stale) Mirror 9: 0x50c8b464 (stale) Mirror 10: 0x50c8b464 (stale) Mirror 12: 0x50c8b464 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1df32: skipped 000484 1788846120.310607 fallocate 0x01f719 thru 0x01ff5b (0x843 bytes) 000485 1788846120.315907 read 0x002fbb thru 0x004182 (0x11c8 bytes) 000486 1788846120.322786 trunc from 0x01ff5c to 0x0251f8 (0x529d bytes) 000488 1788846120.347793 mapwrite 0x019757 thru 0x01e18d (0x4a37 bytes) 000489 1788846120.384591 mapwrite 0x0328de thru 0x03ffff (0xd722 bytes) 000490 1788846120.451022 punch from 0x00e127 to 0x0158f4 (0x77ce bytes) 000491 1788846120.458325 punch from 0x0207e8 to 0x02ae30 (0xa649 bytes) 000492 1788846120.463544 read 0x01a25f thru 0x0202ca (0x606c bytes) 000493 1788846120.477863 punch from 0x020476 to 0x02866c (0x81f7 bytes) 000494 1788846120.486701 fallocate 0x008cd0 thru 0x00b2c6 (0x25f7 bytes) 000496 1788846120.507767 MIRROR_RESYNC 7 mirrors 000497 1788846122.930255 punch from 0x03b26b to 0x03e070 (0x2e06 bytes) 000498 1788846122.984710 mapwrite 0x020793 thru 0x02d31f (0xcb8d bytes) 000499 1788846123.053077 trunc from 0x040000 to 0x026ffb (0x19004 bytes) 000500 1788846123.076103 write 0x00e177 thru 0x01c4a8 (0xe332 bytes) 000502 1788846123.086816 read 0x023942 thru 0x026ffb (0x36ba bytes) 000503 1788846123.087965 fallocate 0x03ffb7 thru 0x03ffff (0x49 bytes) 000504 1788846123.098298 fallocate 0x0314f1 thru 0x03f5c7 (0xe0d7 bytes) 000505 1788846123.129523 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x3be39f00 (stale) Mirror 7: 0x3be39f00 (stale) Mirror 8: 0x3be39f00 (stale) Mirror 9: 0x3be39f00 (stale) Mirror 10: 0x3be39f00 (stale) Mirror 12: 0x3be39f00 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf351cd79 Mirror 13: 0xf351cd79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000506 1788846124.418061 read 0x01ed84 thru 0x0236e8 (0x4965 bytes) 000507 1788846124.461276 mapwrite 0x02a026 thru 0x037d5c (0xdd37 bytes) 000508 1788846124.638692 mapwrite 0x0275f5 thru 0x030964 (0x9370 bytes) 000509 1788846124.676744 trunc from 0x024a97 to 0x03081d (0xbd87 bytes) 000510 1788846124.696120 mapwrite 0x029e09 thru 0x038aca (0xecc2 bytes) 000511 1788846124.748985 mapwrite 0x03063a thru 0x038352 (0x7d19 bytes) 000512 1788846124.771293 fallocate 0x0262b2 thru 0x02f653 (0x93a2 bytes) 000513 1788846124.776995 trunc from 0x040000 to 0x03b0ce (0x4f31 bytes) 000514 1788846124.842231 trunc from 0x00e626 to 0x01901c (0xa9f7 bytes) 000515 1788846124.847941 mapwrite 0x000475 thru 0x0059c1 (0x554d bytes) 000516 1788846124.873992 trunc from 0x03b0cf to 0x03653a (0x4b94 bytes) 000517 1788846124.892270 write 0x01640b thru 0x025b0c (0xf702 bytes) 000518 1788846124.899950 trunc from 0x01aa17 to 0x02202b (0x7615 bytes) 000519 1788846124.915680 mapwrite 0x01f8a2 thru 0x02f398 (0xfaf7 bytes) 000520 1788846124.946478 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3653b): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3653b): Mirror 6: 0x5077010d (stale) Mirror 7: 0x5077010d (stale) Mirror 8: 0x5077010d (stale) Mirror 9: 0x5077010d (stale) Mirror 10: 0x5077010d (stale) Mirror 12: 0x5077010d (stale) Mirror 13: 0x615b6d69 (stale) CRC-32 checksum value for chunk [0, 0x3653b): Mirror 1: 0xd64a0472 Mirror 14: 0xd64a0472 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3653b: skipped 000521 1788846126.776415 trunc from 0x03653b to 0x006341 (0x301f9 bytes) 000522 1788846126.878768 write 0x01a917 thru 0x01dabd (0x31a7 bytes) 000523 1788846126.882708 fallocate 0x016173 thru 0x01778b (0x1619 bytes) 000524 1788846126.889125 punch from 0x00ad96 to 0x00fee9 (0x5154 bytes) 000525 1788846126.899225 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dabe): Mirror 7: 0xe8486580 (stale) Mirror 8: 0xe8486580 (stale) Mirror 9: 0xe8486580 (stale) Mirror 10: 0xe8486580 (stale) Mirror 12: 0xe8486580 (stale) Mirror 13: 0x7d7f95d1 (stale) Mirror 14: 0xab2b4dfb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dabe): Mirror 7: 0xe8486580 (stale) Mirror 8: 0xe8486580 (stale) Mirror 9: 0xe8486580 (stale) Mirror 10: 0xe8486580 (stale) Mirror 12: 0xe8486580 (stale) Mirror 13: 0x7d7f95d1 (stale) Mirror 14: 0xab2b4dfb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dabe: skipped 000526 1788846127.492640 punch from 0x010754 to 0x01562c (0x4ed9 bytes) 000527 1788846127.498758 read 0x01a815 thru 0x01dabd (0x32a9 bytes) 000528 1788846127.510168 write 0x01f369 thru 0x0202ff (0xf97 bytes) 000529 1788846127.515944 punch from 0x00eea6 to 0x01e23d (0xf398 bytes) 000530 1788846127.521427 trunc from 0x017d84 to 0x019f7d (0x21fa bytes) 000531 1788846127.527006 mapwrite 0x0101e7 thru 0x0186f5 (0x850f bytes) 000532 1788846127.568795 punch from 0x013e1c to 0x0202ff (0xc4e4 bytes) 000534 1788846127.575016 mapwrite 0x038698 thru 0x03ffff (0x7968 bytes) 000535 1788846127.653181 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3be39f00 (stale) Mirror 8: 0x3be39f00 (stale) Mirror 10: 0x3be39f00 (stale) Mirror 12: 0x3be39f00 (stale) Mirror 13: 0xf351cd79 (stale) Mirror 14: 0xef3c6f70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3be39f00 (stale) Mirror 8: 0x3be39f00 (stale) Mirror 10: 0x3be39f00 (stale) Mirror 12: 0x3be39f00 (stale) Mirror 13: 0xf351cd79 (stale) Mirror 14: 0xef3c6f70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000536 1788846128.215381 read 0x00d51b thru 0x012c0d (0x56f3 bytes) 000537 1788846128.229804 fallocate 0x007f4a thru 0x0094a0 (0x1557 bytes) 000538 1788846128.236358 mapread 0x014fde thru 0x01a286 (0x52a9 bytes) 000539 1788846128.274887 write 0x030c30 thru 0x035acc (0x4e9d bytes) 000540 1788846128.296847 fallocate 0x02025d thru 0x023993 (0x3737 bytes) 000541 1788846128.303933 write 0x00eaca thru 0x011789 (0x2cc0 bytes) 000542 1788846128.306232 trunc from 0x040000 to 0x038c29 (0x73d6 bytes) 000543 1788846128.332063 read 0x0354a9 thru 0x038c29 (0x3781 bytes) 000545 1788846128.357164 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c2a): Mirror 8: 0x79a3030f (stale) Mirror 10: 0x79a3030f (stale) Mirror 12: 0x79a3030f (stale) Mirror 13: 0x79acbda0 (stale) Mirror 14: 0x2212f458 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c2a): Mirror 8: 0x79a3030f (stale) Mirror 10: 0x79a3030f (stale) Mirror 12: 0x79a3030f (stale) Mirror 13: 0x79acbda0 (stale) Mirror 14: 0x2212f458 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c2a: skipped 000547 1788846128.875879 trunc from 0x038c2a to 0x018003 (0x20c26 bytes) 000548 1788846128.901069 punch from 0x0048ba to 0x008799 (0x3ee0 bytes) 000549 1788846128.914662 trunc from 0x0129d4 to 0x01564c (0x2c79 bytes) 000550 1788846128.928445 punch from 0x012bde to 0x018003 (0x5426 bytes) 000551 1788846128.944408 mapwrite 0x00b4c4 thru 0x019329 (0xde66 bytes) 000552 1788846129.058812 fallocate 0x0319cf thru 0x03c801 (0xae33 bytes) 000553 1788846129.070732 mapwrite 0x0161e3 thru 0x01c728 (0x6546 bytes) 000554 1788846129.131019 mapwrite 0x031a52 thru 0x03f2f7 (0xd8a6 bytes) 000555 1788846129.250447 read 0x029ef3 thru 0x038791 (0xe89f bytes) 000556 1788846129.263841 trunc from 0x01df87 to 0x0210bc (0x3136 bytes) 000557 1788846129.271135 fallocate 0x02ace9 thru 0x033f1c (0x9234 bytes) 000558 1788846129.304150 mapwrite 0x0128ee thru 0x01c0ce (0x97e1 bytes) 000559 1788846129.339170 trunc from 0x03f2f8 to 0x016911 (0x289e6 bytes) 000560 1788846129.398201 read 0x0001d0 thru 0x00b22e (0xb05f bytes) 000561 1788846129.410466 mapread 0x00c95a thru 0x0109f8 (0x409f bytes) 000562 1788846129.414858 mapwrite 0x03218e thru 0x03b4af (0x9322 bytes) 000563 1788846129.568084 mapwrite 0x015c40 thru 0x025545 (0xf906 bytes) 000564 1788846129.635655 punch from 0x03a031 to 0x03b4af (0x147f bytes) 000565 1788846129.645077 punch from 0x02025a to 0x020cda (0xa81 bytes) 000566 1788846129.658860 fallocate 0x00d793 thru 0x01b329 (0xdb97 bytes) 000567 1788846129.681054 write 0x03d689 thru 0x03eed2 (0x184a bytes) 000568 1788846129.692782 fallocate 0x0321fd thru 0x038843 (0x6647 bytes) 000569 1788846129.710622 trunc from 0x01417b to 0x021a8d (0xd913 bytes) 000570 1788846129.721961 read 0x01f2f8 thru 0x027a52 (0x875b bytes) 000571 1788846129.740877 punch from 0x032f8b to 0x038731 (0x57a7 bytes) 000572 1788846129.764595 punch from 0x01c8b4 to 0x01f300 (0x2a4d bytes) 000573 1788846129.779955 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3eed3): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eed3): Mirror 8: 0xaeee53e6 (stale) Mirror 10: 0xaeee53e6 (stale) Mirror 12: 0xaeee53e6 (stale) Mirror 13: 0x88469804 (stale) Mirror 14: 0x6508799a (stale) CRC-32 checksum value for chunk [0, 0x3eed3): Mirror 1: 0x1d0e8e33 Mirror 15: 0x1d0e8e33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eed3: skipped 000574 1788846131.435721 write 0x02840f thru 0x0372ec (0xeede bytes) 000575 1788846131.508823 punch from 0x0292a1 to 0x033810 (0xa570 bytes) 000577 1788846131.527656 read 0x005c9e thru 0x00d3e8 (0x774b bytes) 000579 1788846131.538436 trunc from 0x0363ce to 0x03eed2 (0x8b05 bytes) 000580 1788846131.545824 fallocate 0x0173d0 thru 0x01cd71 (0x59a2 bytes) 000581 1788846131.551463 fallocate 0x00de07 thru 0x017b46 (0x9d40 bytes) 000582 1788846131.562711 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eed3): Mirror 10: 0xaeee53e6 (stale) Mirror 12: 0xaeee53e6 (stale) Mirror 13: 0x88469804 (stale) Mirror 14: 0x6508799a (stale) Mirror 15: 0x1d0e8e33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eed3): Mirror 10: 0xaeee53e6 (stale) Mirror 12: 0xaeee53e6 (stale) Mirror 13: 0x88469804 (stale) Mirror 14: 0x6508799a (stale) Mirror 15: 0x1d0e8e33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eed3: skipped 000583 1788846132.325919 trunc from 0x029a3c to 0x039429 (0xf9ee bytes) 000584 1788846132.332866 trunc from 0x03eed3 to 0x01b03c (0x23e96 bytes) 000586 1788846132.354484 mapread 0x013583 thru 0x01ae74 (0x78f2 bytes) 000587 1788846132.396166 trunc from 0x00b6e7 to 0x00f96d (0x4287 bytes) 000588 1788846132.408037 mapwrite 0x0220c7 thru 0x02d81e (0xb758 bytes) 000589 1788846132.498866 read 0x006812 thru 0x01313d (0xc92c bytes) 000590 1788846132.515528 trunc from 0x00b8d7 to 0x00f2fd (0x3a27 bytes) 000592 1788846132.528001 mapwrite 0x0178c5 thru 0x020511 (0x8c4d bytes) 000593 1788846132.555874 trunc from 0x0228fd to 0x02862b (0x5d2f bytes) 000594 1788846132.563851 trunc from 0x02d81f to 0x0057fc (0x28022 bytes) 000595 1788846132.598314 trunc from 0x000a20 to 0x0057fc (0x4ddd bytes) 000596 1788846132.610157 fallocate 0x03afb9 thru 0x03ffff (0x5047 bytes) 000597 1788846132.616156 mapread 0x00acbc thru 0x00f34c (0x4691 bytes) 000598 1788846132.637555 mapread 0x00b4f2 thru 0x00ed91 (0x38a0 bytes) 000599 1788846132.649670 write 0x0073b5 thru 0x015d7e (0xe9ca bytes) 000600 1788846132.666537 trunc from 0x035e55 to 0x03ffff (0xa1ab bytes) 000601 1788846132.687795 trunc from 0x040000 to 0x036f49 (0x90b6 bytes) 000602 1788846132.708360 mapwrite 0x019640 thru 0x01cef6 (0x38b7 bytes) 000603 1788846132.771066 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36f4a): Mirror 10: 0x7a5823a6 (stale) Mirror 13: 0x352a414c (stale) Mirror 14: 0xd4a24c9b (stale) Mirror 15: 0x35637950 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f4a): Mirror 10: 0x7a5823a6 (stale) Mirror 13: 0x352a414c (stale) Mirror 14: 0xd4a24c9b (stale) Mirror 15: 0x35637950 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f4a: skipped 000604 1788846133.218922 trunc from 0x0360d7 to 0x036f49 (0xe73 bytes) 000605 1788846133.224836 read 0x0069d3 thru 0x01632b (0xf959 bytes) 000606 1788846133.235729 trunc from 0x030e29 to 0x032c29 (0x1e01 bytes) 000607 1788846133.240942 read 0x02a19b thru 0x02d4f6 (0x335c bytes) 000608 1788846133.263077 read 0x00011f thru 0x00d41b (0xd2fd bytes) 000609 1788846133.269730 MIRROR_RESYNC 5 mirrors 000610 1788846134.537606 mapwrite 0x0274d0 thru 0x02ff35 (0x8a66 bytes) 000611 1788846134.634679 mapwrite 0x013416 thru 0x015063 (0x1c4e bytes) 000612 1788846134.659723 punch from 0x023ecd to 0x02d33f (0x9473 bytes) 000613 1788846134.665176 trunc from 0x01cd23 to 0x02b77b (0xea59 bytes) 000614 1788846134.677442 trunc from 0x036f4a to 0x03e762 (0x7819 bytes) 000615 1788846134.737612 fallocate 0x03cd0b thru 0x03ffff (0x32f5 bytes) 000616 1788846134.752289 read 0x0281bd thru 0x02ea13 (0x6857 bytes) 000617 1788846134.770651 read 0x01391e thru 0x0157d5 (0x1eb8 bytes) 000618 1788846134.771699 mapwrite 0x01c39e thru 0x026098 (0x9cfb bytes) 000619 1788846134.824885 trunc from 0x006510 to 0x00fe39 (0x992a bytes) 000620 1788846134.832577 mapwrite 0x0120c3 thru 0x01801b (0x5f59 bytes) 000621 1788846134.862839 read 0x00ac82 thru 0x018931 (0xdcb0 bytes) 000622 1788846134.879447 write 0x003ab9 thru 0x0082e7 (0x482f bytes) 000623 1788846134.882381 fallocate 0x039ca6 thru 0x03ffff (0x635a bytes) 000624 1788846134.890066 fallocate 0x037c79 thru 0x03ffff (0x8387 bytes) 000625 1788846134.898655 mapread 0x028622 thru 0x029dba (0x1799 bytes) 000626 1788846134.900975 mapread 0x027188 thru 0x02cd7f (0x5bf8 bytes) 000627 1788846134.904041 trunc from 0x031f9a to 0x03ed69 (0xcdd0 bytes) 000628 1788846134.913232 mapwrite 0x032b72 thru 0x0391df (0x666e bytes) 000630 1788846135.001918 write 0x029113 thru 0x02bf34 (0x2e22 bytes) 000632 1788846135.004676 write 0x01d159 thru 0x02c67b (0xf523 bytes) 000633 1788846135.007450 punch from 0x0265ad to 0x033d4f (0xd7a3 bytes) 000634 1788846135.028293 mapread 0x0231fb thru 0x032183 (0xef89 bytes) 000635 1788846135.101717 trunc from 0x040000 to 0x02eab2 (0x1154d bytes) 000637 1788846135.120089 trunc from 0x02eab3 to 0x02748e (0x7624 bytes) 000639 1788846135.141681 write 0x02b3b8 thru 0x03410c (0x8d55 bytes) 000640 1788846135.144258 trunc from 0x02fe5a to 0x03410c (0x42b3 bytes) 000641 1788846135.158783 write 0x01fed1 thru 0x023a96 (0x3bc6 bytes) 000642 1788846135.159954 mapwrite 0x022775 thru 0x03141c (0xeca8 bytes) 000643 1788846135.216098 trunc from 0x017dfd to 0x01ea40 (0x6c44 bytes) 000644 1788846135.222426 trunc from 0x03410d to 0x028335 (0xbdd7 bytes) 000645 1788846135.243064 trunc from 0x0053ef to 0x008db7 (0x39c9 bytes) 000646 1788846135.250005 mapread 0x0267a9 thru 0x028335 (0x1b8d bytes) 000647 1788846135.251783 fallocate 0x0283f9 thru 0x035b6a (0xd772 bytes) 000648 1788846135.257604 read 0x012219 thru 0x018028 (0x5e10 bytes) 000649 1788846135.268347 write 0x008517 thru 0x00930c (0xdf6 bytes) 000650 1788846135.269885 punch from 0x00da47 to 0x01d324 (0xf8de bytes) 000651 1788846135.276383 trunc from 0x0168ed to 0x020b1a (0xa22e bytes) 000652 1788846135.282577 read 0x02796a thru 0x028335 (0x9cc bytes) 000654 1788846135.284934 mapread 0x0120cc thru 0x014982 (0x28b7 bytes) 000656 1788846135.298443 trunc from 0x028336 to 0x02aa3c (0x2707 bytes) 000657 1788846135.313208 trunc from 0x02aa3d to 0x01cf39 (0xdb03 bytes) 000658 1788846135.334443 trunc from 0x01cf3a to 0x03a71c (0x1d7e3 bytes) 000659 1788846135.349420 mapwrite 0x03c6d7 thru 0x03ffff (0x3929 bytes) 000660 1788846135.389972 fallocate 0x0198d6 thru 0x022089 (0x87b4 bytes) 000661 1788846135.396389 punch from 0x01496d to 0x01e5c4 (0x9c58 bytes) 000663 1788846135.401925 mapwrite 0x02cc40 thru 0x02d1ef (0x5b0 bytes) 000664 1788846135.414060 punch from 0x034b3c to 0x034e64 (0x329 bytes) 000665 1788846135.419264 read 0x02b645 thru 0x0344ad (0x8e69 bytes) 000666 1788846135.429861 mapread 0x027ce2 thru 0x02a78e (0x2aad bytes) 000667 1788846135.430847 punch from 0x022f7e to 0x02bdbb (0x8e3e bytes) 000668 1788846135.436622 trunc from 0x040000 to 0x007189 (0x38e76 bytes) 000669 1788846135.457160 fallocate 0x005d4a thru 0x0065cc (0x883 bytes) 000670 1788846135.466733 MIRROR_RESYNC 5 mirrors 000671 1788846136.516840 mapread 0x00234e thru 0x007189 (0x4e3c bytes) 000672 1788846136.540427 trunc from 0x000949 to 0x007189 (0x6841 bytes) 000673 1788846136.573738 punch from 0x002a68 to 0x0051fb (0x2794 bytes) 000674 1788846136.587183 MIRROR_RESYNC 5 mirrors 000675 1788846137.944441 write 0x003859 thru 0x01115f (0xd907 bytes) 000676 1788846138.015449 mapwrite 0x011561 thru 0x016f4d (0x59ed bytes) 000677 1788846138.134194 MIRROR_RESYNC 5 mirrors 000678 1788846139.160897 trunc from 0x016f4e to 0x014b4d (0x2400 bytes) 000680 1788846139.278691 mapwrite 0x005a81 thru 0x01041f (0xa99f bytes) 000681 1788846139.340649 trunc from 0x014b4e to 0x00d0ad (0x7aa0 bytes) 000682 1788846139.393870 mapwrite 0x01a2ab thru 0x02784a (0xd5a0 bytes) 000684 1788846139.518711 write 0x0344c3 thru 0x03e462 (0x9fa0 bytes) 000685 1788846139.521223 punch from 0x028f90 to 0x03243e (0x94af bytes) 000686 1788846139.527843 fallocate 0x03424f thru 0x03ffff (0xbdb1 bytes) 000687 1788846139.550837 fallocate 0x03b24a thru 0x03ffff (0x4db6 bytes) 000688 1788846139.561564 punch from 0x03820c to 0x03d3a0 (0x5195 bytes) 000689 1788846139.583454 mapread 0x00d7d0 thru 0x019112 (0xb943 bytes) 000691 1788846139.647993 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xb661e3a0 (stale) Mirror 13: 0xb661e3a0 (stale) Mirror 14: 0xb661e3a0 (stale) Mirror 15: 0xb661e3a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x81acaf87 Mirror 16: 0x81acaf87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000692 1788846140.997891 trunc from 0x03edc3 to 0x03ffff (0x123d bytes) 000693 1788846141.042333 trunc from 0x02e3a3 to 0x03d65e (0xf2bc bytes) 000694 1788846141.047048 mapread 0x01d380 thru 0x0257fa (0x847b bytes) 000695 1788846141.066944 trunc from 0x028030 to 0x03673c (0xe70d bytes) 000696 1788846141.072922 read 0x02fa30 thru 0x0354e9 (0x5aba bytes) 000697 1788846141.083665 fallocate 0x009c67 thru 0x00d9ee (0x3d88 bytes) 000698 1788846141.088973 mapread 0x01f17c thru 0x02efd7 (0xfe5c bytes) 000699 1788846141.097595 punch from 0x02ea75 to 0x037a75 (0x9001 bytes) 000700 1788846141.102881 write 0x037af5 thru 0x03ffff (0x850b bytes) 000701 1788846141.119107 read 0x01eb1e thru 0x02bea6 (0xd389 bytes) 000703 1788846141.121413 read 0x02dcf1 thru 0x02ed9b (0x10ab bytes) 000704 1788846141.142361 mapread 0x00d16b thru 0x011f2c (0x4dc2 bytes) 000705 1788846141.143320 mapread 0x01e3d1 thru 0x02b7c6 (0xd3f6 bytes) 000706 1788846141.144410 fallocate 0x00dd63 thru 0x01c0b5 (0xe353 bytes) 000707 1788846141.148471 trunc from 0x040000 to 0x00de58 (0x321a7 bytes) 000708 1788846141.165069 trunc from 0x008c7e to 0x00de58 (0x51db bytes) 000709 1788846141.171555 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xde59): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xde59): Mirror 10: 0xd7d8dcd4 (stale) Mirror 13: 0xd7d8dcd4 (stale) Mirror 14: 0xd7d8dcd4 (stale) Mirror 15: 0xd7d8dcd4 (stale) Mirror 16: 0xf7800470 (stale) CRC-32 checksum value for chunk [0, 0xde59): Mirror 1: 0x69c15bd4 Mirror 17: 0x69c15bd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xde59: skipped 000710 1788846142.226942 fallocate 0x030f6e thru 0x03c499 (0xb52c bytes) 000711 1788846142.280860 punch from 0x004215 to 0x0099a0 (0x578c bytes) 000712 1788846142.298719 read 0x003b00 thru 0x00b75c (0x7c5d bytes) 000713 1788846142.323881 write 0x0362c4 thru 0x03ed50 (0x8a8d bytes) 000714 1788846142.339102 trunc from 0x0065df to 0x006d8a (0x7ac bytes) 000715 1788846142.349345 fallocate 0x032ae2 thru 0x0334ae (0x9cd bytes) 000717 1788846142.374146 mapread 0x037713 thru 0x03ed50 (0x763e bytes) 000718 1788846142.388728 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed51): Mirror 13: 0x90741e9c (stale) Mirror 14: 0x90741e9c (stale) Mirror 15: 0x90741e9c (stale) Mirror 16: 0x672e8c99 (stale) Mirror 17: 0x807bab6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed51): Mirror 13: 0x90741e9c (stale) Mirror 14: 0x90741e9c (stale) Mirror 15: 0x90741e9c (stale) Mirror 16: 0x672e8c99 (stale) Mirror 17: 0x807bab6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed51: skipped 000719 1788846142.994979 read 0x02adea thru 0x02dd65 (0x2f7c bytes) 000720 1788846143.003368 mapread 0x00ca70 thru 0x0158e1 (0x8e72 bytes) 000721 1788846143.034694 trunc from 0x03ed51 to 0x01c34b (0x22a05 bytes) 000722 1788846143.076150 mapwrite 0x03222e thru 0x039a23 (0x77f6 bytes) 000723 1788846143.136849 mapwrite 0x002475 thru 0x00736d (0x4ef9 bytes) 000724 1788846143.177264 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39a24): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a24): Mirror 13: 0xf24f5a04 (stale) Mirror 14: 0xf24f5a04 (stale) Mirror 15: 0xf24f5a04 (stale) Mirror 16: 0xa4d3f8e7 (stale) Mirror 17: 0x572270 (stale) CRC-32 checksum value for chunk [0, 0x39a24): Mirror 1: 0xaa24d384 Mirror 18: 0xaa24d384 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a24: skipped 000726 1788846144.411931 punch from 0x032e71 to 0x032ef2 (0x82 bytes) 000727 1788846144.492863 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39a24): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a24): Mirror 13: 0xf24f5a04 (stale) Mirror 14: 0xf24f5a04 (stale) Mirror 15: 0xf24f5a04 (stale) Mirror 16: 0xa4d3f8e7 (stale) Mirror 17: 0x572270 (stale) Mirror 18: 0xaa24d384 (stale) CRC-32 checksum value for chunk [0, 0x39a24): Mirror 1: 0x962964cc Mirror 19: 0x962964cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a24: skipped 000728 1788846145.930219 write 0x002cd4 thru 0x0039c7 (0xcf4 bytes) 000729 1788846146.080982 MIRROR_RESYNC 8 mirrors 000730 1788846148.911187 mapread 0x009618 thru 0x010c68 (0x7651 bytes) 000732 1788846148.967468 fallocate 0x0199ac thru 0x02325d (0x98b2 bytes) 000733 1788846149.055103 mapwrite 0x01a1d8 thru 0x01f5e5 (0x540e bytes) 000734 1788846149.123885 mapwrite 0x026118 thru 0x034279 (0xe162 bytes) 000735 1788846149.203529 punch from 0x038fb4 to 0x039a23 (0xa70 bytes) 000736 1788846149.210150 mapread 0x009e9c thru 0x00a770 (0x8d5 bytes) 000737 1788846149.224598 read 0x004050 thru 0x00be3d (0x7dee bytes) 000738 1788846149.225241 mapwrite 0x0008d3 thru 0x007ff9 (0x7727 bytes) 000739 1788846149.284956 write 0x019877 thru 0x01b717 (0x1ea1 bytes) 000740 1788846149.286177 mapwrite 0x0338e9 thru 0x03ffff (0xc717 bytes) 000741 1788846149.342769 trunc from 0x040000 to 0x01569c (0x2a963 bytes) 000742 1788846149.361207 fallocate 0x011e57 thru 0x013289 (0x1433 bytes) 000743 1788846149.369201 trunc from 0x01569d to 0x035424 (0x1fd88 bytes) 000744 1788846149.387222 trunc from 0x02b3c1 to 0x033e5a (0x8a9a bytes) 000745 1788846149.394113 trunc from 0x035425 to 0x01037d (0x250a7 bytes) 000746 1788846149.417454 trunc from 0x009a8c to 0x01037d (0x68f2 bytes) 000748 1788846149.424247 fallocate 0x035c11 thru 0x037451 (0x1841 bytes) 000749 1788846149.430592 write 0x024de3 thru 0x0328fb (0xdb19 bytes) 000750 1788846149.445046 fallocate 0x031fe0 thru 0x0393d4 (0x73f5 bytes) 000751 1788846149.453220 mapread 0x027aae thru 0x0302d5 (0x8828 bytes) 000752 1788846149.460797 MIRROR_RESYNC 8 mirrors 000753 1788846151.577987 mapread 0x0119ef thru 0x011f2b (0x53d bytes) 000754 1788846151.586605 read 0x02f177 thru 0x037451 (0x82db bytes) 000755 1788846151.598376 punch from 0x001262 to 0x0073d0 (0x616f bytes) 000756 1788846151.636464 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37452): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37452): Mirror 13: 0xaa8c9f80 (stale) Mirror 14: 0xaa8c9f80 (stale) Mirror 15: 0xaa8c9f80 (stale) Mirror 16: 0xaa8c9f80 (stale) Mirror 17: 0xaa8c9f80 (stale) Mirror 18: 0xaa8c9f80 (stale) Mirror 19: 0xaa8c9f80 (stale) CRC-32 checksum value for chunk [0, 0x37452): Mirror 1: 0xdef469ce Mirror 20: 0xdef469ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37452: skipped 000757 1788846152.819300 trunc from 0x0190aa to 0x019b50 (0xaa7 bytes) 000758 1788846152.876924 mapwrite 0x0120b9 thru 0x0161e9 (0x4131 bytes) 000760 1788846152.918284 mapread 0x027cfb thru 0x0286cf (0x9d5 bytes) 000762 1788846152.937782 trunc from 0x02885e to 0x02bf9a (0x373d bytes) 000763 1788846152.948371 MIRROR_RESYNC 9 mirrors 000764 1788846156.989837 MIRROR_RESYNC 9 mirrors 000765 1788846157.967827 mapwrite 0x028b23 thru 0x037630 (0xeb0e bytes) 000766 1788846158.095449 mapread 0x01ca1e thru 0x01f42d (0x2a10 bytes) 000769 1788846158.128687 read 0x02049e thru 0x029d6b (0x98ce bytes) 000770 1788846158.132080 write 0x028b63 thru 0x029aaf (0xf4d bytes) 000771 1788846158.134645 fallocate 0x0071b5 thru 0x01122f (0xa07b bytes) 000772 1788846158.151533 trunc from 0x03375c to 0x037630 (0x3ed5 bytes) 000773 1788846158.163989 mapwrite 0x014234 thru 0x01e764 (0xa531 bytes) 000774 1788846158.224329 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37631): Mirror 14: 0x2e7de6a1 (stale) Mirror 15: 0x2e7de6a1 (stale) Mirror 16: 0x2e7de6a1 (stale) Mirror 17: 0x2e7de6a1 (stale) Mirror 18: 0x2e7de6a1 (stale) Mirror 19: 0x2e7de6a1 (stale) Mirror 20: 0x2e7de6a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37631): Mirror 14: 0x2e7de6a1 (stale) Mirror 15: 0x2e7de6a1 (stale) Mirror 16: 0x2e7de6a1 (stale) Mirror 17: 0x2e7de6a1 (stale) Mirror 18: 0x2e7de6a1 (stale) Mirror 19: 0x2e7de6a1 (stale) Mirror 20: 0x2e7de6a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37631: skipped 000775 1788846159.002651 read 0x007705 thru 0x0151f6 (0xdaf2 bytes) 000776 1788846159.020384 write 0x02aea4 thru 0x034bdf (0x9d3c bytes) 000777 1788846159.038484 read 0x00472a thru 0x00f52e (0xae05 bytes) 000778 1788846159.053034 trunc from 0x02af48 to 0x02fb41 (0x4bfa bytes) 000779 1788846159.074386 trunc from 0x037631 to 0x01c5f6 (0x1b03a bytes) 000781 1788846159.108141 trunc from 0x01c5f7 to 0x01ed6d (0x2777 bytes) 000782 1788846159.147000 trunc from 0x01ed6e to 0x02c4ff (0xd792 bytes) 000783 1788846159.181395 punch from 0x025543 to 0x02c4ff (0x6fbd bytes) 000784 1788846159.188459 trunc from 0x00195e to 0x00feff (0xe5a2 bytes) 000785 1788846159.208200 mapwrite 0x03a2ce thru 0x03f70c (0x543f bytes) 000786 1788846159.308382 punch from 0x0026f9 to 0x004db7 (0x26bf bytes) 000787 1788846159.322034 trunc from 0x01291d to 0x018d03 (0x63e7 bytes) 000788 1788846159.331069 read 0x01ac85 thru 0x01d82e (0x2baa bytes) 000789 1788846159.360788 fallocate 0x019f0b thru 0x028d84 (0xee7a bytes) 000791 1788846159.369796 trunc from 0x03f70d to 0x02e687 (0x11085 bytes) 000792 1788846159.396355 mapwrite 0x0374bf thru 0x03ffff (0x8b41 bytes) 000793 1788846159.508201 trunc from 0x020b07 to 0x026dce (0x62c8 bytes) 000794 1788846159.515562 mapread 0x00092a thru 0x009c37 (0x930e bytes) 000795 1788846159.523422 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x61ff76f2 (stale) Mirror 15: 0x61ff76f2 (stale) Mirror 16: 0x61ff76f2 (stale) Mirror 17: 0x61ff76f2 (stale) Mirror 18: 0x61ff76f2 (stale) Mirror 19: 0x61ff76f2 (stale) Mirror 20: 0x61ff76f2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xac63a205 Mirror 21: 0xac63a205 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000796 1788846160.670475 punch from 0x03c4cd to 0x03ffff (0x3b33 bytes) 000798 1788846160.766541 fallocate 0x02b135 thru 0x02cb05 (0x19d1 bytes) 000801 1788846160.776037 punch from 0x00b3d4 to 0x015507 (0xa134 bytes) 000802 1788846160.782005 write 0x03fc60 thru 0x03ffff (0x3a0 bytes) 000803 1788846160.799261 mapwrite 0x00cef5 thru 0x00f90d (0x2a19 bytes) 000804 1788846160.860087 punch from 0x02cad0 to 0x03073f (0x3c70 bytes) 000805 1788846160.866257 write 0x022b7c thru 0x028cfc (0x6181 bytes) 000806 1788846160.877113 fallocate 0x02db50 thru 0x032ac3 (0x4f74 bytes) 000807 1788846160.883106 trunc from 0x0395a7 to 0x03ffff (0x6a59 bytes) 000808 1788846160.893466 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x61ff76f2 (stale) Mirror 15: 0x61ff76f2 (stale) Mirror 16: 0x61ff76f2 (stale) Mirror 17: 0x61ff76f2 (stale) Mirror 18: 0x61ff76f2 (stale) Mirror 19: 0x61ff76f2 (stale) Mirror 20: 0x61ff76f2 (stale) Mirror 21: 0xac63a205 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5d4f4dfd Mirror 22: 0x5d4f4dfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1788846162.354112 mapread 0x02e019 thru 0x034740 (0x6728 bytes) 000811 1788846162.430948 read 0x010cf9 thru 0x01ad56 (0xa05e bytes) 000812 1788846162.451959 mapwrite 0x00963a thru 0x00a5d3 (0xf9a bytes) 000813 1788846162.593923 write 0x0288a9 thru 0x02ef1d (0x6675 bytes) 000814 1788846162.609091 mapread 0x019db3 thru 0x01f47b (0x56c9 bytes) 000815 1788846162.619361 mapwrite 0x00efc4 thru 0x01edc7 (0xfe04 bytes) 000816 1788846162.641374 trunc from 0x03335f to 0x034ec9 (0x1b6b bytes) 000817 1788846162.646006 trunc from 0x0184ae to 0x028172 (0xfcc5 bytes) 000818 1788846162.651147 trunc from 0x040000 to 0x0166e2 (0x2991d bytes) 000819 1788846162.675448 mapwrite 0x02f866 thru 0x036ae2 (0x727d bytes) 000820 1788846162.717479 mapread 0x00891f thru 0x0154a1 (0xcb83 bytes) 000822 1788846162.719204 mapwrite 0x0330bf thru 0x034c87 (0x1bc9 bytes) 000823 1788846162.731963 fallocate 0x007c25 thru 0x0138aa (0xbc86 bytes) 000824 1788846162.740346 mapread 0x022555 thru 0x02d828 (0xb2d4 bytes) 000825 1788846162.751982 trunc from 0x036ae3 to 0x000da2 (0x35d40 bytes) 000826 1788846162.767273 trunc from 0x000b3a to 0x000da2 (0x269 bytes) 000827 1788846162.780321 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xda3): Mirror 14: 0xf77e5ad5 (stale) Mirror 15: 0xf77e5ad5 (stale) Mirror 16: 0xf77e5ad5 (stale) Mirror 17: 0xf77e5ad5 (stale) Mirror 19: 0xf77e5ad5 (stale) Mirror 20: 0xf77e5ad5 (stale) Mirror 21: 0xf77e5ad5 (stale) Mirror 22: 0xf77e5ad5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xda3): Mirror 14: 0xf77e5ad5 (stale) Mirror 15: 0xf77e5ad5 (stale) Mirror 16: 0xf77e5ad5 (stale) Mirror 17: 0xf77e5ad5 (stale) Mirror 19: 0xf77e5ad5 (stale) Mirror 20: 0xf77e5ad5 (stale) Mirror 21: 0xf77e5ad5 (stale) Mirror 22: 0xf77e5ad5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xda3: skipped 000828 1788846163.167390 read 0x000151 thru 0x000da2 (0xc52 bytes) 000829 1788846163.172272 read 0x0008eb thru 0x000da2 (0x4b8 bytes) 000831 1788846163.174753 write 0x02fcb1 thru 0x037c09 (0x7f59 bytes) 000832 1788846163.183250 trunc from 0x037c0a to 0x00c462 (0x2b7a7 bytes) 000834 1788846163.199429 punch from 0x00965f to 0x00c462 (0x2e04 bytes) 000835 1788846163.205315 write 0x014e24 thru 0x01d688 (0x8865 bytes) 000836 1788846163.210183 trunc from 0x012349 to 0x0192e7 (0x6f9f bytes) 000838 1788846163.228457 MIRROR_RESYNC 9 mirrors 000839 1788846164.813733 write 0x036cc3 thru 0x03b188 (0x44c6 bytes) 000840 1788846164.854089 punch from 0x004e28 to 0x00fe63 (0xb03c bytes) 000841 1788846164.860898 punch from 0x037dea to 0x03b188 (0x339f bytes) 000842 1788846164.873190 trunc from 0x03b189 to 0x025155 (0x16033 bytes) 000843 1788846164.894380 trunc from 0x025156 to 0x0206a6 (0x4aaf bytes) 000844 1788846164.906447 mapwrite 0x01b796 thru 0x01d4cf (0x1d3a bytes) 000845 1788846164.957427 punch from 0x015aa1 to 0x0206a6 (0xac06 bytes) 000846 1788846164.966561 trunc from 0x0154a2 to 0x0206a6 (0xb205 bytes) 000847 1788846164.972099 mapread 0x015e90 thru 0x01f5dd (0x974e bytes) 000848 1788846165.002753 mapwrite 0x0051df thru 0x0117b7 (0xc5d9 bytes) 000850 1788846165.084471 mapread 0x00744c thru 0x007deb (0x9a0 bytes) 000851 1788846165.085474 write 0x03f8c6 thru 0x03ffff (0x73a bytes) 000852 1788846165.087423 fallocate 0x01f2cc thru 0x029830 (0xa565 bytes) 000853 1788846165.104376 MIRROR_RESYNC 9 mirrors 000854 1788846167.055160 trunc from 0x01a27c to 0x01dc96 (0x3a1b bytes) 000855 1788846167.098038 fallocate 0x03d734 thru 0x03ffff (0x28cc bytes) 000856 1788846167.108053 punch from 0x02a969 to 0x02cdcb (0x2463 bytes) 000857 1788846167.113771 read 0x02ed78 thru 0x03c0f8 (0xd381 bytes) 000858 1788846167.132166 trunc from 0x013023 to 0x01a825 (0x7803 bytes) 000859 1788846167.139153 trunc from 0x02e529 to 0x02fbde (0x16b6 bytes) 000860 1788846167.155653 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x686ccfb6 (stale) Mirror 15: 0x686ccfb6 (stale) Mirror 16: 0x686ccfb6 (stale) Mirror 17: 0x686ccfb6 (stale) Mirror 19: 0x686ccfb6 (stale) Mirror 20: 0x686ccfb6 (stale) Mirror 21: 0x686ccfb6 (stale) Mirror 22: 0x686ccfb6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x686ccfb6 Mirror 23: 0x686ccfb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000861 1788846168.525832 mapwrite 0x018486 thru 0x01a86b (0x23e6 bytes) 000862 1788846168.655753 MIRROR_RESYNC 10 mirrors 000863 1788846171.290110 trunc from 0x040000 to 0x029121 (0x16ede bytes) 000864 1788846171.359309 trunc from 0x027282 to 0x029121 (0x1ea0 bytes) 000866 1788846171.366172 mapwrite 0x00f72e thru 0x01c7f7 (0xd0ca bytes) 000867 1788846171.476895 trunc from 0x029122 to 0x00fff7 (0x1912a bytes) 000868 1788846171.521174 fallocate 0x03901d thru 0x03f11c (0x6100 bytes) 000869 1788846171.531825 punch from 0x0062ce to 0x00ca02 (0x6735 bytes) 000870 1788846171.548153 punch from 0x0009af to 0x00fc9e (0xf2f0 bytes) 000871 1788846171.572828 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xfff8): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfff8): Mirror 14: 0x426ff060 (stale) Mirror 15: 0x426ff060 (stale) Mirror 16: 0x426ff060 (stale) Mirror 17: 0x426ff060 (stale) Mirror 19: 0x426ff060 (stale) Mirror 20: 0x426ff060 (stale) Mirror 21: 0x426ff060 (stale) Mirror 22: 0x426ff060 (stale) Mirror 23: 0x426ff060 (stale) CRC-32 checksum value for chunk [0, 0xfff8): Mirror 1: 0x4114ccf4 Mirror 24: 0x4114ccf4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfff8: skipped 000872 1788846172.962178 mapread 0x000199 thru 0x005d58 (0x5bc0 bytes) 000873 1788846173.029076 mapread 0x00e0f0 thru 0x00fff7 (0x1f08 bytes) 000874 1788846173.073922 write 0x02c87c thru 0x02dc13 (0x1398 bytes) 000875 1788846173.157407 trunc from 0x02dc14 to 0x019624 (0x145ef bytes) 000876 1788846173.186796 mapread 0x0074cd thru 0x00e93e (0x7472 bytes) 000877 1788846173.199252 mapread 0x012260 thru 0x01471f (0x24c0 bytes) 000878 1788846173.216321 write 0x014b3f thru 0x0237bf (0xec81 bytes) 000879 1788846173.220898 mapread 0x00d5c1 thru 0x016eb3 (0x98f3 bytes) 000880 1788846173.228105 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x237c0): Mirror 14: 0x66daa9b9 (stale) Mirror 15: 0x66daa9b9 (stale) Mirror 16: 0x66daa9b9 (stale) Mirror 17: 0x66daa9b9 (stale) Mirror 19: 0x66daa9b9 (stale) Mirror 21: 0x66daa9b9 (stale) Mirror 22: 0x66daa9b9 (stale) Mirror 23: 0x66daa9b9 (stale) Mirror 24: 0x1cfcf899 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x237c0): Mirror 14: 0x66daa9b9 (stale) Mirror 15: 0x66daa9b9 (stale) Mirror 16: 0x66daa9b9 (stale) Mirror 17: 0x66daa9b9 (stale) Mirror 19: 0x66daa9b9 (stale) Mirror 21: 0x66daa9b9 (stale) Mirror 22: 0x66daa9b9 (stale) Mirror 23: 0x66daa9b9 (stale) Mirror 24: 0x1cfcf899 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x237c0: skipped 000881 1788846174.160662 mapwrite 0x036095 thru 0x036d60 (0xccc bytes) 000882 1788846174.251643 trunc from 0x036d61 to 0x03fe67 (0x9107 bytes) 000883 1788846174.283748 write 0x011608 thru 0x012e28 (0x1821 bytes) 000884 1788846174.302300 mapread 0x03215a thru 0x03fe67 (0xdd0e bytes) 000885 1788846174.330174 fallocate 0x0237f0 thru 0x029dee (0x65ff bytes) 000886 1788846174.335336 read 0x0127fa thru 0x01ec45 (0xc44c bytes) 000887 1788846174.347977 mapwrite 0x02071e thru 0x02ac45 (0xa528 bytes) 000888 1788846174.387689 MIRROR_RESYNC 10 mirrors 000889 1788846177.753962 read 0x01f3dd thru 0x02dcf8 (0xe91c bytes) 000890 1788846177.765093 fallocate 0x035b73 thru 0x03ffff (0xa48d bytes) 000891 1788846177.817721 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x38127884 (stale) Mirror 15: 0x38127884 (stale) Mirror 16: 0x38127884 (stale) Mirror 17: 0x38127884 (stale) Mirror 19: 0x38127884 (stale) Mirror 21: 0x38127884 (stale) Mirror 22: 0x38127884 (stale) Mirror 23: 0x38127884 (stale) Mirror 24: 0x38127884 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x38127884 Mirror 25: 0x38127884 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000892 1788846178.984168 read 0x013165 thru 0x0223e7 (0xf283 bytes) 000893 1788846179.014723 fallocate 0x01c099 thru 0x01c403 (0x36b bytes) 000894 1788846179.092133 fallocate 0x00906f thru 0x0135ed (0xa57f bytes) 000896 1788846179.108995 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x38127884 (stale) Mirror 15: 0x38127884 (stale) Mirror 16: 0x38127884 (stale) Mirror 17: 0x38127884 (stale) Mirror 19: 0x38127884 (stale) Mirror 21: 0x38127884 (stale) Mirror 22: 0x38127884 (stale) Mirror 23: 0x38127884 (stale) Mirror 24: 0x38127884 (stale) Mirror 25: 0x38127884 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x38127884 Mirror 26: 0x38127884 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000898 1788846180.563166 trunc from 0x040000 to 0x03ea01 (0x15fe bytes) 000899 1788846180.632925 punch from 0x011559 to 0x019414 (0x7ebc bytes) 000900 1788846180.638040 trunc from 0x03ea02 to 0x01718a (0x27877 bytes) 000901 1788846180.661713 mapread 0x011059 thru 0x013fd7 (0x2f7f bytes) 000902 1788846180.681549 punch from 0x002c2c to 0x0039bf (0xd94 bytes) 000903 1788846180.686233 trunc from 0x004b00 to 0x00876c (0x3c6d bytes) 000904 1788846180.691787 write 0x032ade thru 0x03ceac (0xa3cf bytes) 000905 1788846180.699000 MIRROR_RESYNC 12 mirrors 000906 1788846184.226713 write 0x021526 thru 0x024d2d (0x3808 bytes) 000907 1788846184.295618 mapwrite 0x02855d thru 0x0322d2 (0x9d76 bytes) 000908 1788846184.355722 mapwrite 0x016c83 thru 0x025ce8 (0xf066 bytes) 000909 1788846184.436717 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3cead): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cead): Mirror 14: 0xc2e26d8f (stale) Mirror 15: 0xc2e26d8f (stale) Mirror 16: 0xc2e26d8f (stale) Mirror 17: 0xc2e26d8f (stale) Mirror 19: 0xc2e26d8f (stale) Mirror 21: 0xc2e26d8f (stale) Mirror 22: 0xc2e26d8f (stale) Mirror 23: 0xc2e26d8f (stale) Mirror 24: 0xc2e26d8f (stale) Mirror 25: 0xc2e26d8f (stale) Mirror 26: 0xc2e26d8f (stale) CRC-32 checksum value for chunk [0, 0x3cead): Mirror 1: 0xcde412ac Mirror 27: 0xcde412ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cead: skipped 000910 1788846185.677294 write 0x019ddc thru 0x02006a (0x628f bytes) 000911 1788846185.744406 fallocate 0x03e2bb thru 0x03ffff (0x1d45 bytes) 000912 1788846185.749968 mapread 0x00f691 thru 0x0138cf (0x423f bytes) 000914 1788846185.774013 write 0x03eaa5 thru 0x03ffff (0x155b bytes) 000915 1788846185.779755 mapread 0x0299bb thru 0x03482e (0xae74 bytes) 000916 1788846185.805683 trunc from 0x040000 to 0x01c54f (0x23ab0 bytes) 000917 1788846185.849117 mapwrite 0x015a6a thru 0x0234be (0xda55 bytes) 000918 1788846185.900675 write 0x02588a thru 0x02e281 (0x89f8 bytes) 000919 1788846185.902254 mapread 0x01213b thru 0x01506e (0x2f34 bytes) 000920 1788846185.904537 read 0x00da1e thru 0x01435b (0x693e bytes) 000921 1788846185.905768 mapread 0x01f7eb thru 0x02e281 (0xea97 bytes) 000922 1788846185.920215 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e282): Mirror 15: 0xf1692b2a (stale) Mirror 16: 0xf1692b2a (stale) Mirror 17: 0xf1692b2a (stale) Mirror 19: 0xf1692b2a (stale) Mirror 21: 0xf1692b2a (stale) Mirror 22: 0xf1692b2a (stale) Mirror 23: 0xf1692b2a (stale) Mirror 24: 0xf1692b2a (stale) Mirror 25: 0xf1692b2a (stale) Mirror 26: 0xf1692b2a (stale) Mirror 27: 0xc7d29c85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e282): Mirror 15: 0xf1692b2a (stale) Mirror 16: 0xf1692b2a (stale) Mirror 17: 0xf1692b2a (stale) Mirror 19: 0xf1692b2a (stale) Mirror 21: 0xf1692b2a (stale) Mirror 22: 0xf1692b2a (stale) Mirror 23: 0xf1692b2a (stale) Mirror 24: 0xf1692b2a (stale) Mirror 25: 0xf1692b2a (stale) Mirror 26: 0xf1692b2a (stale) Mirror 27: 0xc7d29c85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e282: skipped 000923 1788846186.741136 trunc from 0x025da3 to 0x02e281 (0x84df bytes) 000924 1788846186.747896 fallocate 0x03b49d thru 0x03ffff (0x4b63 bytes) 000925 1788846186.753882 read 0x028c42 thru 0x02fed2 (0x7291 bytes) 000926 1788846186.763636 punch from 0x03c974 to 0x03ffff (0x368c bytes) 000927 1788846186.774663 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x39712a4d (stale) Mirror 16: 0x39712a4d (stale) Mirror 17: 0x39712a4d (stale) Mirror 19: 0x39712a4d (stale) Mirror 21: 0x39712a4d (stale) Mirror 23: 0x39712a4d (stale) Mirror 24: 0x39712a4d (stale) Mirror 25: 0x39712a4d (stale) Mirror 26: 0x39712a4d (stale) Mirror 27: 0xcb98cf7d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x39712a4d (stale) Mirror 16: 0x39712a4d (stale) Mirror 17: 0x39712a4d (stale) Mirror 19: 0x39712a4d (stale) Mirror 21: 0x39712a4d (stale) Mirror 23: 0x39712a4d (stale) Mirror 24: 0x39712a4d (stale) Mirror 25: 0x39712a4d (stale) Mirror 26: 0x39712a4d (stale) Mirror 27: 0xcb98cf7d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000928 1788846187.632791 mapread 0x011e53 thru 0x019cb3 (0x7e61 bytes) 000929 1788846187.657645 mapwrite 0x005494 thru 0x0065be (0x112b bytes) 000931 1788846187.685885 punch from 0x027d09 to 0x036100 (0xe3f8 bytes) 000932 1788846187.692592 trunc from 0x032267 to 0x03f8f0 (0xd68a bytes) 000933 1788846187.698589 trunc from 0x040000 to 0x036768 (0x9897 bytes) 000934 1788846187.730574 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36769): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36769): Mirror 15: 0x3f4ed4c5 (stale) Mirror 16: 0x3f4ed4c5 (stale) Mirror 17: 0x3f4ed4c5 (stale) Mirror 19: 0x3f4ed4c5 (stale) Mirror 21: 0x3f4ed4c5 (stale) Mirror 23: 0x3f4ed4c5 (stale) Mirror 24: 0x3f4ed4c5 (stale) Mirror 25: 0x3f4ed4c5 (stale) Mirror 26: 0x3f4ed4c5 (stale) Mirror 27: 0x295ba2e2 (stale) CRC-32 checksum value for chunk [0, 0x36769): Mirror 1: 0x699e436c Mirror 28: 0x699e436c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36769: skipped 000935 1788846189.193592 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36769): Mirror 15: 0x3f4ed4c5 (stale) Mirror 16: 0x3f4ed4c5 (stale) Mirror 17: 0x3f4ed4c5 (stale) Mirror 19: 0x3f4ed4c5 (stale) Mirror 21: 0x3f4ed4c5 (stale) Mirror 24: 0x3f4ed4c5 (stale) Mirror 25: 0x3f4ed4c5 (stale) Mirror 26: 0x3f4ed4c5 (stale) Mirror 27: 0x295ba2e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36769): Mirror 15: 0x3f4ed4c5 (stale) Mirror 16: 0x3f4ed4c5 (stale) Mirror 17: 0x3f4ed4c5 (stale) Mirror 19: 0x3f4ed4c5 (stale) Mirror 21: 0x3f4ed4c5 (stale) Mirror 24: 0x3f4ed4c5 (stale) Mirror 25: 0x3f4ed4c5 (stale) Mirror 26: 0x3f4ed4c5 (stale) Mirror 27: 0x295ba2e2 (stale) CRC-32 checksum value for chunk [0, 0x36769): Mirror 1: 0x699e436c Mirror 28: 0x699e436c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36769: skipped 000936 1788846190.138895 mapread 0x030000 thru 0x036768 (0x6769 bytes) 000937 1788846190.180037 trunc from 0x01dc61 to 0x0211cb (0x356b bytes) 000938 1788846190.253964 fallocate 0x01af42 thru 0x02268f (0x774e bytes) 000939 1788846190.277835 trunc from 0x00f731 to 0x012890 (0x3160 bytes) 000940 1788846190.290438 fallocate 0x01a6e8 thru 0x023229 (0x8b42 bytes) 000941 1788846190.297990 punch from 0x02c8d8 to 0x02cebe (0x5e7 bytes) 000942 1788846190.304293 write 0x020b10 thru 0x02247d (0x196e bytes) 000943 1788846190.323775 fallocate 0x003d12 thru 0x01345c (0xf74b bytes) 000944 1788846190.331606 write 0x00d7d5 thru 0x01a5af (0xcddb bytes) 000945 1788846190.353820 trunc from 0x023045 to 0x025ba7 (0x2b63 bytes) 000946 1788846190.359060 mapread 0x02fd39 thru 0x036768 (0x6a30 bytes) 000947 1788846190.402376 write 0x0200f5 thru 0x027274 (0x7180 bytes) 000948 1788846190.409285 write 0x03c981 thru 0x03d08c (0x70c bytes) 000949 1788846190.410548 punch from 0x01868f to 0x01b3b3 (0x2d25 bytes) 000950 1788846190.477968 mapread 0x01bd7d thru 0x029ba8 (0xde2c bytes) 000951 1788846190.492813 trunc from 0x03d08d to 0x0061ee (0x36e9e bytes) 000952 1788846190.595579 mapread 0x005bf4 thru 0x0061ee (0x5fb bytes) 000954 1788846190.618944 trunc from 0x0061ef to 0x00efde (0x8df0 bytes) 000955 1788846190.662911 trunc from 0x0022c3 to 0x00efde (0xcd1c bytes) 000956 1788846190.675758 mapread 0x007ec0 thru 0x00efde (0x711f bytes) 000958 1788846190.688374 trunc from 0x00efdf to 0x003ccf (0xb30f bytes) 000959 1788846190.755418 punch from 0x001e25 to 0x003ccf (0x1eab bytes) 000960 1788846190.764006 mapread 0x00211f thru 0x003759 (0x163b bytes) 000961 1788846190.774795 trunc from 0x003cd0 to 0x000946 (0x3389 bytes) 000962 1788846190.832575 fallocate 0x004ce0 thru 0x012328 (0xd649 bytes) 000963 1788846190.856773 trunc from 0x00e521 to 0x012328 (0x3e08 bytes) 000964 1788846190.869703 read 0x0014c2 thru 0x00a95a (0x9499 bytes) 000965 1788846190.888305 trunc from 0x0100dd to 0x011016 (0xf3a bytes) 000966 1788846190.904298 read 0x0086b7 thru 0x012328 (0x9c72 bytes) 000967 1788846190.925974 mapwrite 0x015955 thru 0x01c381 (0x6a2d bytes) 000968 1788846191.011952 punch from 0x0140d6 to 0x01580b (0x1736 bytes) 000969 1788846191.019055 trunc from 0x01c382 to 0x01dc16 (0x1895 bytes) 000972 1788846191.083204 mapwrite 0x031879 thru 0x032fde (0x1766 bytes) 000973 1788846191.164401 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 15: 0x9b98446d (stale) Mirror 16: 0x9b98446d (stale) Mirror 17: 0x9b98446d (stale) Mirror 19: 0x9b98446d (stale) Mirror 21: 0x9b98446d (stale) Mirror 24: 0x9b98446d (stale) Mirror 25: 0x9b98446d (stale) Mirror 26: 0x9b98446d (stale) Mirror 27: 0xb33b0777 (stale) Mirror 28: 0x1d5dd757 (stale) CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 1: 0x5bd6003b Mirror 29: 0x5bd6003b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fdf: skipped 000975 1788846192.434529 mapread 0x02553a thru 0x0307ae (0xb275 bytes) 000976 1788846192.529921 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 15: 0x9b98446d (stale) Mirror 16: 0x9b98446d (stale) Mirror 17: 0x9b98446d (stale) Mirror 19: 0x9b98446d (stale) Mirror 21: 0x9b98446d (stale) Mirror 24: 0x9b98446d (stale) Mirror 25: 0x9b98446d (stale) Mirror 27: 0xb33b0777 (stale) Mirror 28: 0x1d5dd757 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 15: 0x9b98446d (stale) Mirror 16: 0x9b98446d (stale) Mirror 17: 0x9b98446d (stale) Mirror 19: 0x9b98446d (stale) Mirror 21: 0x9b98446d (stale) Mirror 24: 0x9b98446d (stale) Mirror 25: 0x9b98446d (stale) Mirror 27: 0xb33b0777 (stale) Mirror 28: 0x1d5dd757 (stale) CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 1: 0x5bd6003b Mirror 29: 0x5bd6003b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fdf: skipped 000977 1788846194.126920 read 0x02ad22 thru 0x032fde (0x82bd bytes) 000978 1788846194.160883 trunc from 0x002e54 to 0x007ca2 (0x4e4f bytes) 000979 1788846194.222374 read 0x02eb2c thru 0x032453 (0x3928 bytes) 000981 1788846194.241862 mapread 0x02a742 thru 0x031f00 (0x77bf bytes) 000982 1788846194.259183 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 15: 0x9b98446d (stale) Mirror 16: 0x9b98446d (stale) Mirror 17: 0x9b98446d (stale) Mirror 19: 0x9b98446d (stale) Mirror 21: 0x9b98446d (stale) Mirror 24: 0x9b98446d (stale) Mirror 25: 0x9b98446d (stale) Mirror 28: 0x1d5dd757 (stale) Mirror 29: 0x5bd6003b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fdf): Mirror 15: 0x9b98446d (stale) Mirror 16: 0x9b98446d (stale) Mirror 17: 0x9b98446d (stale) Mirror 19: 0x9b98446d (stale) Mirror 21: 0x9b98446d (stale) Mirror 24: 0x9b98446d (stale) Mirror 25: 0x9b98446d (stale) Mirror 28: 0x1d5dd757 (stale) Mirror 29: 0x5bd6003b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fdf: skipped 000983 1788846195.130283 punch from 0x017469 to 0x024698 (0xd230 bytes) 000985 1788846195.139228 mapread 0x0047ec thru 0x014205 (0xfa1a bytes) 000986 1788846195.172204 mapread 0x028aae thru 0x029fc7 (0x151a bytes) 000987 1788846195.186878 trunc from 0x02495a to 0x028608 (0x3caf bytes) 000988 1788846195.193792 punch from 0x032567 to 0x032fde (0xa78 bytes) 000989 1788846195.200200 trunc from 0x032fdf to 0x03e92a (0xb94c bytes) 000990 1788846195.260408 fallocate 0x03f596 thru 0x03ffff (0xa6a bytes) 000991 1788846195.266391 punch from 0x0270ff to 0x02dc27 (0x6b29 bytes) 000992 1788846195.272162 write 0x00789f thru 0x00f66d (0x7dcf bytes) 000993 1788846195.274408 trunc from 0x040000 to 0x010270 (0x2fd8f bytes) 000994 1788846195.299236 mapread 0x003c16 thru 0x00cb32 (0x8f1d bytes) 000995 1788846195.300177 punch from 0x002aac to 0x00b482 (0x89d7 bytes) 000996 1788846195.317031 read 0x007c2f thru 0x0096ec (0x1abe bytes) 000997 1788846195.333051 MIRROR_RESYNC 10 mirrors 000998 1788846198.005354 mapwrite 0x00fdb7 thru 0x012148 (0x2392 bytes) 000999 1788846198.087853 trunc from 0x00db6c to 0x0115ff (0x3a94 bytes) 001000 1788846198.095816 trunc from 0x012149 to 0x03e115 (0x2bfcd bytes) All operations completed A-OK! PASS 70a (142s) == sanity-flr test 71: check mirror extend parent fid ==== 01:43:24 (1788846204) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x7d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 01:43:36 (1788846216) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168049 s, 12.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11384 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2aa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2aa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 01:43:45 (1788846225) Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre2 Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4496640 .. Extending file size to 8627264 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9120192 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9035680 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61310 61311 61312 61313 61315 done 192.168.201.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg160-client.virtnet /mnt/lustre2 (opts:) 192.168.201.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg160-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:45:09 (1788846309) Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre2 Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4489088 .. Extending file size to 5859328 .. Extending file size to 8307584 .. Extending file size to 7760896 .. Extending file size to 8285312 .. Extending file size to 8463456 .. Extending file size to 8686528 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8861952 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9163552 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8865760 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62645 62646 62647 62648 62649 done 192.168.201.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg160-client.virtnet /mnt/lustre2 (opts:) 192.168.201.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg160-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:46:35 (1788846395) Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1268864 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2080 1269224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25580 3581440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17836 3589184 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43416 7170624 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1268864 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2080 1269224 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25580 3581440 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17836 3589184 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43416 7170624 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134851 s, 15.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334149 s, 6.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf90b362a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg160-client.virtnet /mnt/lustre2 (opts:) PASS 200c (25s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:47:03 (1788846423) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180536 s, 11.6 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 01:47:12 (1788846432) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x187:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000400:0x2dd:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:47:21 (1788846441) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0847032 s, 24.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:47:32 (1788846452) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0917544 s, 22.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x56:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x56:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x57:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:47:40 (1788846460) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140713 s, 14.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e1:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:48:16 (1788846496) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg160-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111628 s, 9.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x59:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x59:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:49:15 (1788846555) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg160-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103202 s, 20.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:50:11 (1788846611) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg160-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg160-server: Pool lustre.test_204f created Adding targets to pool oleg160-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg160-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139261 s, 15.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg160-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg160-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg160-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:51:27 (1788846687) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:51:35 (1788846695) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:51:45 (1788846705) oleg160-server: Pool lustre.test_206 created Adding targets to pool oleg160-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg160-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg160-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg160-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg160-server: Pool lustre.test_206 destroyed PASS 206 (33s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:52:18 (1788846738) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126634 s, 8.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409155 s, 25.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x462517d6 Mirror 3: 0x462517d6 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:52:33 (1788846753) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:52:37 (1788846757) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:52:42 (1788846762) Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0797342 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1670000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1130000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1130000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1130000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x800000 Read this (4 bytes): BBBB 192.168.201.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg160-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:53:23 (1788846803) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0157585 s, 520 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:53:33 (1788846813) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666628 s, 15.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1a4 lmm_fid: [0x200000405:0x1a4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1a4 lmm_fid: [0x200000405:0x1a4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ce:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611556 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 01:53:35.000000000 -0400 Modify: 2026-09-08 01:53:35.000000000 -0400 Change: 2026-09-08 01:53:37.000000000 -0400 Birth: 2026-09-08 01:53:34.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:53:45 (1788846825) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829695 s, 12.6 MB/s oleg160-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg160-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg160-server: O/280000400/d11/2eb: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:54:11 (1788846851) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.231735 s, 177 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0483239 s, 84.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 01:54 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 01:54 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:54:20 (1788846860) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:54:28 (1788846868) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1993 sec =========== 01:54:37 (1788846877) === sanity-flr: start cleanup 01:54:39 (1788846879) === === sanity-flr: finish cleanup 01:54:50 (1788846890) ===