-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 23:28:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg229-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:28:57 (1788838137) === oleg229-client.virtnet: executing check_config_client /mnt/lustre oleg229-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg229-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98da51360800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98da51360800.idle_timeout=debug disable quota as required oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:29:39 (1788838179) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8207] == sanity-flr test 0a: lfs mirror create with -N option == 23:29:41 (1788838181) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:30:01 (1788838201) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:30:05 (1788838205) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:30:09 (1788838209) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:30:33 (1788838233) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:30:36 (1788838236) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:30:40 (1788838240) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98da51360800.stats=clear osc.lustre-OST0001-osc-ffff98da51360800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:30:47 (1788838247) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:30:58 (1788838258) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:31:05 (1788838265) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:31:09 (1788838269) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:31:16 (1788838276) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 23:31:26 (1788838286) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:31:34 (1788838294) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:31:42 (1788838302) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:31:50 (1788838310) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:32:01 (1788838321) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:32:08 (1788838328) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.920091 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781737 s, 13.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:32:18 (1788838338) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.14558 s, 36.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:32:26 (1788838346) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:32:37 (1788838357) Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg229-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (80s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:33:57 (1788838437) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.32368 s, 13.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:34:49 (1788838489) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg229-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:36:20 (1788838580) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.20556 s, 16.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 23:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.047s user 0m0.002s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98da51360800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.131s user 0m0.000s sys 0m0.130s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg229-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg229-server oleg229-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.596s user 0m0.000s sys 0m0.171s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg229-server: oleg229-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg229-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg229-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98da51360800.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:37:58 (1788838678) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:38:01 (1788838681) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:38:05 (1788838685) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:38:08 (1788838688) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:38:16 (1788838696) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.661601 s, 19.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:38:32 (1788838712) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494426 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607084 s, 17.3 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:38:42 (1788838722) PASS 36c (11s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:38:53 (1788838733) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000919491 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00160484 s, 638 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103513 s, 989 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0017592 s, 582 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 23:39:07 (1788838747) create test files with size 15747917 .. 277e988724fae31aabe826c6e4b3e19d - 00444920703d21a4224103d88cb5a294 - 91d811dd5aab1595033750a057e6f7f3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg229-client.virtnet /mnt/lustre (opts:) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 23:39:37 (1788838777) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:39:50 (1788838790) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:39:58 (1788838798) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145681 s, 7.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055497 s, 18.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 23:40:07 (1788838807) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0578679 s, 36.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.290514 s, 14.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11045 s, 19.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 23:40:24 (1788838824) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 23:40:28 (1788838828) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:40:33 (1788838833) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239556 s, 43.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 23:40:41 (1788838841) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.295619 s, 10.6 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:40:51 (1788838851) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:41:01 (1788838861) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.371504 s, 28.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg229-client.virtnet /mnt/lustre (opts:) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:41:13 (1788838873) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff98da60bb5000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.260957 s, 40.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg229-client.virtnet /mnt/lustre (opts:) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff98da42c8d000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:41:28 (1788838888) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:41:58 (1788838918) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:42:33 (1788838953) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 23:42:37 (1788838957) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.258858 s, 12.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.195833 s, 16.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1cc71017 (nosync,stale) Mirror 2: 0x1cc71017 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 334bbd62dd4405348a51a7dc82c3e75b - mirror 1 checksum: f9ec80271c7b1b92a82c2cfb5671676d - mirror 2 checksum: 334bbd62dd4405348a51a7dc82c3e75b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:42:39-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:42:39-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 334bbd62dd4405348a51a7dc82c3e75b - mirror 1 checksum: f9ec80271c7b1b92a82c2cfb5671676d - mirror 2 checksum: f9ec80271c7b1b92a82c2cfb5671676d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:42:39-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:42:39-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:42:47 (1788838967) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564008 s, 18.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:42:54 (1788838974) Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 23:43:07 (1788838987) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 23:43:40 (1788839020) targets are mounted 23:43:40 (1788839020) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255138 s, 41.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:43:55 (1788839035) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0371803 s, 551 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0202683 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:44:04 (1788839044) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0841228 s, 243 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00598132 s, 3.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:44:13 (1788839053) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00734761 s, 2.2 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:44:20 (1788839060) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0853533 s, 240 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:44:27 (1788839067) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00353039 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0351603 s, 116 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:44:35 (1788839075) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:44:55 (1788839095) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:45:22 (1788839122) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:45:46 (1788839146) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:46:09 (1788839169) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 23:46:16 (1788839176) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57566 mirror split pid 57567 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57566 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57567 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 23:47:25 (1788839245) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 9248 000001 1788839249.019263 mapwrite 0x006196 thru 0x00f69d (0x9508 bytes) 000002 1788839249.095921 write 0x03b2ec thru 0x03ffff (0x4d14 bytes) 000003 1788839249.112922 mapread 0x02e0bc thru 0x0332d0 (0x5215 bytes) zero_range to largest ever: 0x40000 000004 1788839249.129805 trunc from 0x037d94 to 0x03ffff (0x826c bytes) 000005 1788839249.166707 trunc from 0x040000 to 0x00ea67 (0x31598 bytes) 000006 1788839249.220858 fallocate 0x00456e thru 0x013c5c (0xf6ef bytes) 000007 1788839249.233051 read 0x00e67d thru 0x00ea67 (0x3eb bytes) 000009 1788839249.234000 mapread 0x0091a1 thru 0x00cc08 (0x3a68 bytes) 000010 1788839249.234741 mapread 0x000a90 thru 0x0073fe (0x696f bytes) 000011 1788839249.241966 trunc from 0x00846d to 0x00b366 (0x2efa bytes) 000012 1788839249.266222 trunc from 0x00ea68 to 0x00eb12 (0xab bytes) 000014 1788839249.292035 fallocate 0x000b6f thru 0x000e49 (0x2db bytes) 000015 1788839249.302566 trunc from 0x0031d7 to 0x00a3bd (0x71e7 bytes) 000016 1788839249.308420 mapwrite 0x03809b thru 0x03ffff (0x7f65 bytes) 000017 1788839249.408410 write 0x03f41b thru 0x03ffff (0xbe5 bytes) 000018 1788839249.409341 trunc from 0x040000 to 0x03c2ad (0x3d52 bytes) 000019 1788839249.456402 fallocate 0x004b9d thru 0x011219 (0xc67d bytes) 000020 1788839249.462597 punch from 0x020f18 to 0x029110 (0x81f9 bytes) 000021 1788839249.467430 write 0x0037a3 thru 0x0101ad (0xca0b bytes) 000023 1788839249.475979 mapwrite 0x0287af thru 0x035328 (0xcb7a bytes) 000024 1788839249.526727 mapread 0x02229d thru 0x029470 (0x71d4 bytes) 000025 1788839249.528605 trunc from 0x03c2ae to 0x0028e8 (0x399c5 bytes) 000026 1788839249.545322 MIRROR_EXTEND to 2 mirrors 000027 1788839250.050997 fallocate 0x0289be thru 0x0369f9 (0xe03c bytes) 000028 1788839250.092893 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000029 1788839250.237903 mapread 0x032ee6 thru 0x033984 (0xa9f bytes) 000030 1788839250.248997 mapwrite 0x036671 thru 0x03a03a (0x39ca bytes) 000032 1788839250.302448 mapread 0x0253e2 thru 0x03136b (0xbf8a bytes) 000033 1788839250.317359 mapwrite 0x0103ee thru 0x01b732 (0xb345 bytes) 000034 1788839250.352282 punch from 0x01f7f9 to 0x02c0c5 (0xc8cd bytes) 000035 1788839250.357570 write 0x01cf8a thru 0x029147 (0xc1be bytes) 000036 1788839250.366142 trunc from 0x03a03b to 0x00e589 (0x2bab1 bytes) 000037 1788839250.402430 MIRROR_EXTEND to 2 mirrors 000038 1788839251.162303 mapwrite 0x036c89 thru 0x03ffff (0x9377 bytes) 000039 1788839251.271938 punch from 0x00720f to 0x00f7a7 (0x8599 bytes) 000040 1788839251.283839 MIRROR_RESYNC 2 mirrors 000041 1788839251.875179 trunc from 0x001c7a to 0x00b628 (0x99af bytes) 000042 1788839251.929618 mapread 0x038cfd thru 0x039a02 (0xd06 bytes) 000043 1788839251.956614 trunc from 0x00277b to 0x0038af (0x1135 bytes) 000045 1788839251.968834 punch from 0x02d1a5 to 0x02d2bf (0x11b bytes) 000046 1788839251.979151 trunc from 0x040000 to 0x0313dd (0xec22 bytes) 000047 1788839252.056131 trunc from 0x01a6a0 to 0x02402f (0x9990 bytes) 000048 1788839252.067198 trunc from 0x00a74d to 0x013b83 (0x9437 bytes) 000051 1788839252.075789 mapread 0x02bcc1 thru 0x02df16 (0x2256 bytes) 000052 1788839252.087550 fallocate 0x008fc1 thru 0x015663 (0xc6a3 bytes) 000053 1788839252.096676 mapwrite 0x018747 thru 0x0276b7 (0xef71 bytes) 000054 1788839252.148223 mapwrite 0x00ac69 thru 0x00ee37 (0x41cf bytes) 000056 1788839252.178178 read 0x026fa0 thru 0x0313dd (0xa43e bytes) 000057 1788839252.178691 trunc from 0x0313de to 0x01e6f6 (0x12ce7 bytes) 000058 1788839252.201668 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e6f7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e6f7): Mirror 2: 0x225057c3 (stale) CRC-32 checksum value for chunk [0, 0x1e6f7): Mirror 1: 0x7731ddda Mirror 3: 0x7731ddda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e6f7: skipped 000059 1788839253.410081 mapread 0x017b99 thru 0x01a2cf (0x2737 bytes) 000060 1788839253.461123 read 0x000581 thru 0x009cd2 (0x9752 bytes) 000061 1788839253.481392 mapwrite 0x01d0fe thru 0x02064d (0x3550 bytes) 000062 1788839253.589788 fallocate 0x03174f thru 0x03c7b1 (0xb063 bytes) 000064 1788839253.601678 trunc from 0x03c7b2 to 0x007460 (0x35351 bytes) 000065 1788839253.622329 trunc from 0x004bf5 to 0x007460 (0x286c bytes) 000066 1788839253.630261 mapread 0x0018de thru 0x007460 (0x5b83 bytes) 000067 1788839253.640558 write 0x009322 thru 0x012177 (0x8e56 bytes) 000068 1788839253.648392 MIRROR_RESYNC 3 mirrors 000069 1788839254.361450 punch from 0x001e6c to 0x00f0f7 (0xd28c bytes) 000071 1788839254.413252 punch from 0x0023d0 to 0x00929b (0x6ecc bytes) 000072 1788839254.425159 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x12178): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12178): Mirror 2: 0xd668ce0 (stale) Mirror 3: 0xd668ce0 (stale) CRC-32 checksum value for chunk [0, 0x12178): Mirror 1: 0xcf9a0dff Mirror 4: 0xcf9a0dff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12178: skipped 000073 1788839255.271366 fallocate 0x00cb2b thru 0x0194ed (0xc9c3 bytes) 000074 1788839255.320930 write 0x0094f9 thru 0x016bd5 (0xd6dd bytes) 000075 1788839255.336123 write 0x01e39a thru 0x01f6f2 (0x1359 bytes) 000076 1788839255.337333 punch from 0x00328c to 0x00cdb5 (0x9b2a bytes) 000077 1788839255.353200 trunc from 0x00898d to 0x017395 (0xea09 bytes) 000078 1788839255.366840 MIRROR_RESYNC 4 mirrors 000080 1788839256.207379 trunc from 0x01f6f3 to 0x03416e (0x14a7c bytes) 000081 1788839256.276655 write 0x018034 thru 0x022b36 (0xab03 bytes) 000082 1788839256.302116 read 0x00b17b thru 0x0117bd (0x6643 bytes) 000083 1788839256.326054 fallocate 0x012838 thru 0x017e9f (0x5668 bytes) 000084 1788839256.333087 trunc from 0x03416f to 0x039e97 (0x5d29 bytes) 000085 1788839256.360565 fallocate 0x01f348 thru 0x02119b (0x1e54 bytes) 000086 1788839256.369368 trunc from 0x0049ff to 0x00cbad (0x81af bytes) 000087 1788839256.381533 fallocate 0x02ed3c thru 0x030833 (0x1af8 bytes) 000088 1788839256.395120 mapread 0x00642d thru 0x00a43d (0x4011 bytes) 000089 1788839256.439620 mapread 0x011f4b thru 0x01cdfd (0xaeb3 bytes) 000090 1788839256.460149 mapwrite 0x034583 thru 0x03e1c6 (0x9c44 bytes) 000091 1788839256.563015 read 0x025b96 thru 0x031329 (0xb794 bytes) 000092 1788839256.598198 mapread 0x0023bd thru 0x0122dd (0xff21 bytes) 000093 1788839256.603114 fallocate 0x01cb93 thru 0x02382c (0x6c9a bytes) 000094 1788839256.610207 trunc from 0x022d1f to 0x02e919 (0xbbfb bytes) 000096 1788839256.617890 trunc from 0x033ede to 0x03d32e (0x9451 bytes) 000097 1788839256.637396 read 0x00429b thru 0x006776 (0x24dc bytes) 000098 1788839256.641582 MIRROR_RESYNC 4 mirrors 000099 1788839257.829326 MIRROR_EXTEND to 5 mirrors 000100 1788839258.823665 read 0x01c78d thru 0x0245b5 (0x7e29 bytes) 000102 1788839258.846623 trunc from 0x03e1c7 to 0x034d94 (0x9432 bytes) 000103 1788839258.964525 mapread 0x02eb20 thru 0x034d94 (0x6275 bytes) 000104 1788839259.022234 MIRROR_RESYNC 5 mirrors 000105 1788839260.450740 trunc from 0x034d95 to 0x00462f (0x30765 bytes) 000106 1788839260.484331 punch from 0x004609 to 0x00462f (0x27 bytes) 000107 1788839260.490791 mapread 0x000517 thru 0x00462f (0x4119 bytes) 000110 1788839260.515547 fallocate 0x0222be thru 0x028ec9 (0x6c0c bytes) 000111 1788839260.526666 mapwrite 0x0134a6 thru 0x02064c (0xd1a7 bytes) 000112 1788839260.608467 punch from 0x00aaa5 to 0x01810f (0xd66b bytes) 000113 1788839260.615711 trunc from 0x019a3e to 0x01e3ae (0x4971 bytes) 000114 1788839260.626440 trunc from 0x0163a4 to 0x02064c (0xa2a9 bytes) 000115 1788839260.634311 fallocate 0x024751 thru 0x02fbfd (0xb4ad bytes) 000117 1788839260.641461 trunc from 0x02064d to 0x02d174 (0xcb28 bytes) 000118 1788839260.661881 trunc from 0x02d175 to 0x017a17 (0x1575d bytes) 000119 1788839260.684482 read 0x00157a thru 0x004f55 (0x39dc bytes) 000120 1788839260.685673 trunc from 0x017a18 to 0x03b457 (0x23a40 bytes) 000121 1788839260.702541 mapread 0x00e4f9 thru 0x00f784 (0x128c bytes) 000123 1788839260.717122 punch from 0x02ccd6 to 0x035388 (0x86b3 bytes) 000124 1788839260.723706 fallocate 0x004b35 thru 0x012a46 (0xdf12 bytes) 000125 1788839260.730723 write 0x03fc70 thru 0x03ffff (0x390 bytes) 000126 1788839260.738605 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x71cf2cca (stale) Mirror 3: 0x71cf2cca (stale) Mirror 4: 0x71cf2cca (stale) Mirror 5: 0x71cf2cca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x21cf1045 Mirror 6: 0x21cf1045 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000127 1788839261.938908 fallocate 0x01b5fe thru 0x0225cf (0x6fd2 bytes) 000128 1788839262.020816 fallocate 0x01398f thru 0x015233 (0x18a5 bytes) 000129 1788839262.073564 fallocate 0x02b28a thru 0x0314a1 (0x6218 bytes) 000130 1788839262.092547 read 0x036d02 thru 0x03ffff (0x92fe bytes) 000131 1788839262.113700 trunc from 0x01ffda to 0x021970 (0x1997 bytes) 000133 1788839262.129824 trunc from 0x013e85 to 0x01862c (0x47a8 bytes) 000134 1788839262.144648 fallocate 0x02b831 thru 0x03ace0 (0xf4b0 bytes) 000136 1788839262.150808 mapread 0x007c03 thru 0x012fa1 (0xb39f bytes) 000137 1788839262.190827 read 0x01eff9 thru 0x020ab3 (0x1abb bytes) 000138 1788839262.218983 MIRROR_RESYNC 6 mirrors 000139 1788839263.418453 MIRROR_RESYNC 6 mirrors 000140 1788839264.315425 read 0x0063d3 thru 0x00beeb (0x5b19 bytes) 000141 1788839264.333113 read 0x03efae thru 0x03ffff (0x1052 bytes) 000142 1788839264.357438 mapwrite 0x016c97 thru 0x024ac2 (0xde2c bytes) 000143 1788839264.547830 trunc from 0x040000 to 0x00b17b (0x34e84 bytes) 000144 1788839264.579356 mapwrite 0x0004e9 thru 0x00b01c (0xab34 bytes) 000145 1788839264.711014 fallocate 0x023e75 thru 0x02999a (0x5b26 bytes) 000146 1788839264.737243 mapwrite 0x01be2e thru 0x024cb7 (0x8e8a bytes) 000147 1788839264.940122 mapread 0x000a7f thru 0x00c22f (0xb7b1 bytes) 000148 1788839264.952395 punch from 0x0193f7 to 0x01b388 (0x1f92 bytes) 000149 1788839264.961371 trunc from 0x02999b to 0x025f68 (0x3a32 bytes) 000150 1788839264.987783 fallocate 0x03babe thru 0x03ffff (0x4542 bytes) 000151 1788839265.023571 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25f69): Mirror 2: 0xede5dbd7 (stale) Mirror 3: 0xede5dbd7 (stale) Mirror 4: 0xede5dbd7 (stale) Mirror 5: 0xede5dbd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25f69): Mirror 2: 0xede5dbd7 (stale) Mirror 3: 0xede5dbd7 (stale) Mirror 4: 0xede5dbd7 (stale) Mirror 5: 0xede5dbd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25f69: skipped 000152 1788839265.646345 mapread 0x00970c thru 0x014a15 (0xb30a bytes) 000153 1788839265.696711 read 0x01201b thru 0x02070d (0xe6f3 bytes) 000155 1788839265.709799 mapread 0x009c3f thru 0x0118fa (0x7cbc bytes) 000156 1788839265.714567 read 0x01a516 thru 0x01edca (0x48b5 bytes) 000157 1788839265.717978 trunc from 0x00d125 to 0x01562e (0x850a bytes) 000158 1788839265.730028 trunc from 0x025f69 to 0x03225f (0xc2f7 bytes) 000159 1788839265.787111 mapread 0x01a417 thru 0x01f0c7 (0x4cb1 bytes) 000160 1788839265.788142 trunc from 0x02cd83 to 0x03225f (0x54dd bytes) 000161 1788839265.811824 trunc from 0x013031 to 0x01abb3 (0x7b83 bytes) 000162 1788839265.825371 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32260): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32260): Mirror 2: 0x6e7f9c19 (stale) Mirror 3: 0x6e7f9c19 (stale) Mirror 4: 0x6e7f9c19 (stale) Mirror 5: 0x6e7f9c19 (stale) CRC-32 checksum value for chunk [0, 0x32260): Mirror 1: 0xa7aa616d Mirror 6: 0xa7aa616d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32260: skipped 000163 1788839267.142295 fallocate 0x021cc8 thru 0x02e362 (0xc69b bytes) 000164 1788839267.193118 trunc from 0x01116c to 0x018d35 (0x7bca bytes) 000165 1788839267.206687 fallocate 0x031c20 thru 0x03c209 (0xa5ea bytes) 000166 1788839267.227542 write 0x0181d3 thru 0x0249c7 (0xc7f5 bytes) 000167 1788839267.252420 mapwrite 0x0009c1 thru 0x00823c (0x787c bytes) 000168 1788839267.368026 fallocate 0x01ec46 thru 0x02c5f0 (0xd9ab bytes) 000169 1788839267.380675 trunc from 0x029650 to 0x0374fc (0xdead bytes) 000171 1788839267.398019 mapread 0x02c5e4 thru 0x034392 (0x7daf bytes) 000172 1788839267.448531 trunc from 0x03c20a to 0x00fd8c (0x2c47d bytes) 000173 1788839267.502664 mapread 0x001709 thru 0x00ec7e (0xd576 bytes) 000174 1788839267.503959 trunc from 0x00fd8d to 0x03c45e (0x2c6d2 bytes) 000175 1788839267.527398 mapwrite 0x009896 thru 0x010aa5 (0x7210 bytes) 000177 1788839267.574145 read 0x00ee36 thru 0x01879b (0x9966 bytes) 000179 1788839267.575309 mapread 0x0245c3 thru 0x0342c7 (0xfd05 bytes) 000180 1788839267.623270 write 0x005665 thru 0x006e9c (0x1838 bytes) 000181 1788839267.624358 mapread 0x024aa0 thru 0x0339aa (0xef0b bytes) 000182 1788839267.629452 trunc from 0x03c45f to 0x00afec (0x31472 bytes) 000183 1788839267.686237 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xafed): Mirror 3: 0x180c060b (stale) Mirror 4: 0x180c060b (stale) Mirror 5: 0x180c060b (stale) Mirror 6: 0xdd99a076 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xafed): Mirror 3: 0x180c060b (stale) Mirror 4: 0x180c060b (stale) Mirror 5: 0x180c060b (stale) Mirror 6: 0xdd99a076 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xafed: skipped 000184 1788839268.122657 trunc from 0x00afed to 0x032bb4 (0x27bc8 bytes) 000185 1788839268.151437 trunc from 0x032a5a to 0x032bb4 (0x15b bytes) 000186 1788839268.160129 mapwrite 0x007a11 thru 0x00e7af (0x6d9f bytes) 000187 1788839268.209405 mapread 0x029fb9 thru 0x030802 (0x684a bytes) 000188 1788839268.226665 fallocate 0x033ae5 thru 0x035c5c (0x2178 bytes) 000190 1788839268.235596 read 0x015fb0 thru 0x01711f (0x1170 bytes) 000191 1788839268.248468 mapwrite 0x01c7f8 thru 0x01dc02 (0x140b bytes) 000193 1788839268.275866 punch from 0x00e774 to 0x015dcf (0x765c bytes) 000194 1788839268.282327 read 0x03405a thru 0x035c5c (0x1c03 bytes) 000195 1788839268.292388 trunc from 0x006e85 to 0x00703b (0x1b7 bytes) 000196 1788839268.297347 mapread 0x0258eb thru 0x03341d (0xdb33 bytes) 000197 1788839268.298693 read 0x026d11 thru 0x02b458 (0x4748 bytes) 000198 1788839268.300298 write 0x03351d thru 0x03c014 (0x8af8 bytes) 000199 1788839268.302288 mapwrite 0x003d35 thru 0x013c09 (0xfed5 bytes) 000200 1788839268.333202 trunc from 0x021999 to 0x0240b0 (0x2718 bytes) 000201 1788839268.339429 fallocate 0x012630 thru 0x01eca7 (0xc678 bytes) 000202 1788839268.344706 read 0x013167 thru 0x016c3e (0x3ad8 bytes) 000203 1788839268.345306 trunc from 0x03c015 to 0x028eb5 (0x1315f bytes) 000204 1788839268.374408 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x28eb6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28eb6): Mirror 3: 0xbf2f6792 (stale) Mirror 4: 0xbf2f6792 (stale) Mirror 5: 0xbf2f6792 (stale) Mirror 6: 0x537047c6 (stale) CRC-32 checksum value for chunk [0, 0x28eb6): Mirror 1: 0xc77e91f4 Mirror 7: 0xc77e91f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28eb6: skipped 000205 1788839269.869039 MIRROR_RESYNC 6 mirrors 000206 1788839271.766622 mapread 0x0278b0 thru 0x028eb5 (0x1606 bytes) 000207 1788839271.781055 read 0x0111d7 thru 0x01d7d5 (0xc5ff bytes) 000208 1788839271.818402 write 0x028887 thru 0x031544 (0x8cbe bytes) 000209 1788839271.938980 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31545): Mirror 3: 0x2ce418af (stale) Mirror 4: 0x2ce418af (stale) Mirror 5: 0x2ce418af (stale) Mirror 6: 0x2ce418af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31545): Mirror 3: 0x2ce418af (stale) Mirror 4: 0x2ce418af (stale) Mirror 5: 0x2ce418af (stale) Mirror 6: 0x2ce418af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31545: skipped 000210 1788839272.695297 read 0x0127c9 thru 0x01fc35 (0xd46d bytes) 000211 1788839272.705722 trunc from 0x02e086 to 0x031544 (0x34bf bytes) 000212 1788839272.713013 write 0x03d786 thru 0x03ffff (0x287a bytes) 000213 1788839272.722401 write 0x01988b thru 0x020027 (0x679d bytes) 000214 1788839272.731666 write 0x00f84d thru 0x0101d8 (0x98c bytes) 000215 1788839272.748476 punch from 0x000d68 to 0x00e31e (0xd5b7 bytes) 000216 1788839272.754952 mapread 0x00820f thru 0x012da3 (0xab95 bytes) 000217 1788839272.778816 fallocate 0x018baa thru 0x0204cf (0x7926 bytes) 000218 1788839272.785513 trunc from 0x01eb7c to 0x020c61 (0x20e6 bytes) 000219 1788839272.808272 punch from 0x025b7d to 0x02a9aa (0x4e2e bytes) 000220 1788839272.813432 mapread 0x0306c5 thru 0x037e77 (0x77b3 bytes) 000221 1788839272.829228 MIRROR_RESYNC 5 mirrors 000222 1788839275.367288 trunc from 0x016980 to 0x025e26 (0xf4a7 bytes) 000223 1788839275.434371 trunc from 0x040000 to 0x025244 (0x1adbb bytes) 000224 1788839275.475170 mapread 0x018697 thru 0x025244 (0xcbae bytes) 000225 1788839275.525717 fallocate 0x01a9a5 thru 0x01fc34 (0x5290 bytes) 000226 1788839275.542063 write 0x0104b9 thru 0x01cd22 (0xc86a bytes) 000227 1788839275.552971 trunc from 0x025245 to 0x036410 (0x111cc bytes) 000228 1788839275.572917 mapread 0x01c437 thru 0x01fe4c (0x3a16 bytes) 000229 1788839275.580132 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36411): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36411): Mirror 3: 0x25e5c70b (stale) Mirror 4: 0x25e5c70b (stale) Mirror 5: 0x25e5c70b (stale) Mirror 6: 0x25e5c70b (stale) CRC-32 checksum value for chunk [0, 0x36411): Mirror 1: 0x4d09833c Mirror 7: 0x4d09833c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36411: skipped 000230 1788839276.834241 fallocate 0x03e1af thru 0x03ffff (0x1e51 bytes) 000231 1788839276.893372 mapwrite 0x029088 thru 0x0375e6 (0xe55f bytes) 000232 1788839276.982671 fallocate 0x03d889 thru 0x03ffff (0x2777 bytes) 000233 1788839276.990559 trunc from 0x0330e9 to 0x03c9a0 (0x98b8 bytes) 000234 1788839277.013656 fallocate 0x0053b0 thru 0x00c00c (0x6c5d bytes) 000235 1788839277.029285 read 0x012728 thru 0x01ffd1 (0xd8aa bytes) 000236 1788839277.052808 read 0x025d22 thru 0x0271d4 (0x14b3 bytes) 000237 1788839277.053383 trunc from 0x037bf2 to 0x039399 (0x17a8 bytes) 000238 1788839277.068994 punch from 0x033a91 to 0x03619a (0x270a bytes) 000239 1788839277.077147 write 0x007869 thru 0x0150af (0xd847 bytes) 000240 1788839277.094950 fallocate 0x0202cd thru 0x026bda (0x690e bytes) 000241 1788839277.102815 read 0x00a90e thru 0x012fe6 (0x86d9 bytes) 000242 1788839277.104857 write 0x03a5bf thru 0x03ffff (0x5a41 bytes) 000243 1788839277.116473 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x405cf892 (stale) Mirror 4: 0x405cf892 (stale) Mirror 5: 0x405cf892 (stale) Mirror 6: 0x405cf892 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x405cf892 (stale) Mirror 4: 0x405cf892 (stale) Mirror 5: 0x405cf892 (stale) Mirror 6: 0x405cf892 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000244 1788839277.630588 read 0x00c2da thru 0x0136b1 (0x73d8 bytes) 000245 1788839277.640738 mapread 0x022153 thru 0x027b9a (0x5a48 bytes) 000246 1788839277.666490 mapwrite 0x004b54 thru 0x00fa3d (0xaeea bytes) 000247 1788839277.723006 mapwrite 0x0188a8 thru 0x025df4 (0xd54d bytes) 000248 1788839277.778045 fallocate 0x0331e4 thru 0x03f378 (0xc195 bytes) 000249 1788839277.790093 write 0x0302ee thru 0x032a1f (0x2732 bytes) 000250 1788839277.816006 trunc from 0x00b415 to 0x00ffc6 (0x4bb2 bytes) 000252 1788839277.831558 mapwrite 0x0340f0 thru 0x03ffff (0xbf10 bytes) 000253 1788839277.921373 mapwrite 0x015171 thru 0x01fa8f (0xa91f bytes) 000254 1788839277.975780 punch from 0x00934e to 0x00d16c (0x3e1f bytes) 000256 1788839277.988257 read 0x009d79 thru 0x013f51 (0xa1d9 bytes) 000257 1788839278.043072 punch from 0x020950 to 0x02ea1c (0xe0cd bytes) 000259 1788839278.065997 trunc from 0x0248de to 0x0346ed (0xfe10 bytes) 000260 1788839278.087732 trunc from 0x040000 to 0x017726 (0x288d9 bytes) 000261 1788839278.146411 mapread 0x015dec thru 0x017726 (0x193b bytes) 000262 1788839278.147510 mapwrite 0x03ff24 thru 0x03ffff (0xdc bytes) 000263 1788839278.259427 fallocate 0x037968 thru 0x03ffff (0x8698 bytes) 000264 1788839278.270979 write 0x02fadb thru 0x03c6d9 (0xcbff bytes) 000265 1788839278.288026 read 0x02c25f thru 0x038e5b (0xcbfd bytes) 000267 1788839278.301563 punch from 0x029c0d to 0x0368ac (0xcca0 bytes) 000268 1788839278.332968 write 0x03ae4d thru 0x03ffff (0x51b3 bytes) 000269 1788839278.334097 write 0x024fa4 thru 0x026150 (0x11ad bytes) 000270 1788839278.337467 read 0x016430 thru 0x02627d (0xfe4e bytes) 000271 1788839278.359619 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x405cf892 (stale) Mirror 5: 0x405cf892 (stale) Mirror 6: 0x405cf892 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x405cf892 (stale) Mirror 5: 0x405cf892 (stale) Mirror 6: 0x405cf892 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000272 1788839278.952696 MIRROR_RESYNC 4 mirrors 000273 1788839280.788949 write 0x00e711 thru 0x01674f (0x803f bytes) 000274 1788839280.842321 punch from 0x01723a to 0x019bd2 (0x2999 bytes) 000276 1788839280.852449 fallocate 0x023b71 thru 0x0269df (0x2e6f bytes) 000277 1788839280.859368 trunc from 0x024e84 to 0x032980 (0xdafd bytes) 000278 1788839280.877401 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1e4a8b78 (stale) Mirror 5: 0x1e4a8b78 (stale) Mirror 6: 0x1e4a8b78 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79d2c811 Mirror 7: 0x79d2c811 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000279 1788839282.392698 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1e4a8b78 (stale) Mirror 6: 0x1e4a8b78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1e4a8b78 (stale) Mirror 6: 0x1e4a8b78 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79d2c811 Mirror 7: 0x79d2c811 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1788839283.081116 mapread 0x02dcac thru 0x0306e7 (0x2a3c bytes) 000281 1788839283.170108 trunc from 0x040000 to 0x02d3ca (0x12c35 bytes) 000282 1788839283.279234 mapread 0x02c49e thru 0x02d3ca (0xf2d bytes) 000283 1788839283.288737 read 0x00831a thru 0x0154bd (0xd1a4 bytes) 000284 1788839283.301363 mapwrite 0x01f77d thru 0x02d174 (0xd9f8 bytes) 000285 1788839283.355687 mapread 0x02ae7a thru 0x02d3ca (0x2551 bytes) 000286 1788839283.357425 read 0x0220e3 thru 0x0225a3 (0x4c1 bytes) 000287 1788839283.369431 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d3cb): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d3cb): Mirror 4: 0x1f3bbc3b (stale) Mirror 6: 0x1f3bbc3b (stale) Mirror 7: 0xd0d4dd75 (stale) CRC-32 checksum value for chunk [0, 0x2d3cb): Mirror 1: 0x9b3a03f2 Mirror 8: 0x9b3a03f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d3cb: skipped 000288 1788839284.457034 mapwrite 0x03d2c2 thru 0x03ffff (0x2d3e bytes) 000289 1788839284.603413 write 0x01b1b2 thru 0x020d04 (0x5b53 bytes) 000290 1788839284.635576 write 0x01b331 thru 0x02a566 (0xf236 bytes) 000291 1788839284.638138 mapwrite 0x012541 thru 0x017723 (0x51e3 bytes) 000292 1788839284.693982 mapread 0x02ed19 thru 0x033947 (0x4c2f bytes) 000293 1788839284.712589 trunc from 0x040000 to 0x030267 (0xfd98 bytes) 000294 1788839284.758296 mapwrite 0x01655e thru 0x01d715 (0x71b8 bytes) 000295 1788839284.789312 trunc from 0x030268 to 0x018aa7 (0x177c0 bytes) 000296 1788839284.808664 write 0x01a311 thru 0x0250fd (0xaded bytes) 000297 1788839284.810376 trunc from 0x0250fe to 0x0102b8 (0x14e45 bytes) 000298 1788839284.836046 write 0x00295f thru 0x005bff (0x32a1 bytes) 000299 1788839284.854318 fallocate 0x032791 thru 0x03ffff (0xd86f bytes) 000300 1788839284.860104 trunc from 0x000f4f to 0x0022ec (0x139e bytes) 000301 1788839284.890905 trunc from 0x001e4f to 0x0023ee (0x5a0 bytes) 000302 1788839284.907658 punch from 0x009e80 to 0x00f672 (0x57f3 bytes) 000303 1788839284.912085 trunc from 0x0102b9 to 0x001b01 (0xe7b7 bytes) 000304 1788839284.945438 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b02): Mirror 6: 0x1c5830da (stale) Mirror 7: 0x1c5830da (stale) Mirror 8: 0x1c5830da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b02): Mirror 6: 0x1c5830da (stale) Mirror 7: 0x1c5830da (stale) Mirror 8: 0x1c5830da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b02: skipped 000305 1788839285.425608 trunc from 0x001843 to 0x001b01 (0x2bf bytes) 000306 1788839285.435448 mapread 0x000b85 thru 0x001b01 (0xf7d bytes) 000307 1788839285.464830 write 0x034a10 thru 0x03f40d (0xa9fe bytes) 000309 1788839285.483559 punch from 0x026a03 to 0x03095b (0x9f59 bytes) 000310 1788839285.501491 trunc from 0x03f40e to 0x01c392 (0x2307b bytes) 000311 1788839285.545596 fallocate 0x01638f thru 0x01be49 (0x5abb bytes) 000312 1788839285.560073 write 0x02bd1d thru 0x03a437 (0xe71b bytes) 000313 1788839285.561841 read 0x01e6d6 thru 0x02bf3a (0xd865 bytes) 000314 1788839285.578582 mapwrite 0x01f98e thru 0x0242d2 (0x4945 bytes) 000315 1788839285.629365 fallocate 0x01ab05 thru 0x02359c (0x8a98 bytes) 000316 1788839285.653347 read 0x015680 thru 0x02566b (0xffec bytes) 000317 1788839285.697590 mapwrite 0x004979 thru 0x00ef21 (0xa5a9 bytes) 000318 1788839285.741335 trunc from 0x030481 to 0x030773 (0x2f3 bytes) 000319 1788839285.746616 mapwrite 0x024998 thru 0x0284b9 (0x3b22 bytes) 000320 1788839285.792528 trunc from 0x00b661 to 0x016ad0 (0xb470 bytes) 000321 1788839285.801087 read 0x0369f6 thru 0x03a437 (0x3a42 bytes) 000323 1788839285.809393 mapread 0x028d93 thru 0x02e8c7 (0x5b35 bytes) 000324 1788839285.811929 mapread 0x025c37 thru 0x03461c (0xe9e6 bytes) 000325 1788839285.844471 punch from 0x00027a to 0x00fdee (0xfb75 bytes) 000326 1788839285.879961 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a438): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a438): Mirror 6: 0x2b05c56a (stale) Mirror 7: 0x528c5f91 (stale) Mirror 8: 0xaa1aa382 (stale) CRC-32 checksum value for chunk [0, 0x3a438): Mirror 1: 0x508f7ae9 Mirror 9: 0x508f7ae9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a438: skipped 000327 1788839287.538418 punch from 0x004d2a to 0x006554 (0x182b bytes) 000329 1788839287.658869 write 0x034ce7 thru 0x036ef3 (0x220d bytes) 000330 1788839287.709146 trunc from 0x03a438 to 0x025959 (0x14ade bytes) 000331 1788839287.792092 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2595a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2595a): Mirror 6: 0x5262656c (stale) Mirror 7: 0xf4e9840f (stale) Mirror 8: 0xe7434b89 (stale) Mirror 9: 0x1a46b8b5 (stale) CRC-32 checksum value for chunk [0, 0x2595a): Mirror 1: 0x1a46b8b5 Mirror 10: 0x1a46b8b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2595a: skipped 000332 1788839289.117364 MIRROR_RESYNC 6 mirrors 000333 1788839290.487678 mapread 0x0007ac thru 0x007b60 (0x73b5 bytes) 000335 1788839290.542119 mapwrite 0x02cb09 thru 0x0374a6 (0xa99e bytes) 000336 1788839290.767179 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x374a7): Mirror 7: 0x60fd41c8 (stale) Mirror 8: 0x60fd41c8 (stale) Mirror 9: 0x60fd41c8 (stale) Mirror 10: 0x60fd41c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374a7): Mirror 7: 0x60fd41c8 (stale) Mirror 8: 0x60fd41c8 (stale) Mirror 9: 0x60fd41c8 (stale) Mirror 10: 0x60fd41c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374a7: skipped 000337 1788839291.668464 trunc from 0x00370e to 0x00b308 (0x7bfb bytes) 000338 1788839291.676914 trunc from 0x0069e4 to 0x006ec9 (0x4e6 bytes) 000339 1788839291.689467 trunc from 0x004f64 to 0x011e3a (0xced7 bytes) 000340 1788839291.708350 MIRROR_RESYNC 5 mirrors 000341 1788839293.618537 trunc from 0x0374a7 to 0x03815a (0xcb4 bytes) 000342 1788839293.670793 trunc from 0x0363e5 to 0x03815a (0x1d76 bytes) 000343 1788839293.676477 trunc from 0x0309f2 to 0x0311bf (0x7ce bytes) 000344 1788839293.688543 MIRROR_RESYNC 5 mirrors 000345 1788839295.583858 trunc from 0x03231d to 0x03815a (0x5e3e bytes) 000346 1788839295.714630 fallocate 0x0200e9 thru 0x022153 (0x206b bytes) 000347 1788839295.726543 mapread 0x007089 thru 0x00d47e (0x63f6 bytes) 000348 1788839295.773261 mapwrite 0x0020f1 thru 0x00f8db (0xd7eb bytes) 000349 1788839295.829612 mapread 0x01ce2d thru 0x01d86c (0xa40 bytes) 000350 1788839295.851037 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3815b): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3815b): Mirror 7: 0x68143ef8 (stale) Mirror 8: 0x68143ef8 (stale) Mirror 9: 0x68143ef8 (stale) Mirror 10: 0x68143ef8 (stale) CRC-32 checksum value for chunk [0, 0x3815b): Mirror 1: 0x23d5186f Mirror 11: 0x23d5186f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3815b: skipped 000351 1788839296.913506 mapread 0x02881f thru 0x02a189 (0x196b bytes) 000352 1788839296.939516 write 0x004c1f thru 0x009434 (0x4816 bytes) 000353 1788839296.998800 punch from 0x01bc54 to 0x02689c (0xac49 bytes) 000354 1788839297.003301 read 0x01d4db thru 0x025bf8 (0x871e bytes) 000355 1788839297.009709 trunc from 0x017783 to 0x0218b6 (0xa134 bytes) 000356 1788839297.013397 punch from 0x023505 to 0x023606 (0x102 bytes) 000357 1788839297.019048 fallocate 0x002069 thru 0x004913 (0x28ab bytes) 000358 1788839297.023543 fallocate 0x015cbb thru 0x018a76 (0x2dbc bytes) 000359 1788839297.029342 mapread 0x026039 thru 0x027f3d (0x1f05 bytes) 000360 1788839297.040432 punch from 0x02b0d0 to 0x031aec (0x6a1d bytes) 000361 1788839297.046391 punch from 0x0181e1 to 0x0208cf (0x86ef bytes) 000362 1788839297.053346 read 0x01c8a8 thru 0x020de3 (0x453c bytes) 000363 1788839297.063278 fallocate 0x01e964 thru 0x024470 (0x5b0d bytes) 000365 1788839297.066842 mapread 0x02a6a7 thru 0x0322cf (0x7c29 bytes) 000366 1788839297.067570 trunc from 0x03815b to 0x03e7c7 (0x666d bytes) 000367 1788839297.078401 mapwrite 0x037c2a thru 0x03bf6b (0x4342 bytes) 000368 1788839297.107437 trunc from 0x03202c to 0x03e7c7 (0xc79c bytes) 000369 1788839297.116960 write 0x039c77 thru 0x03ffff (0x6389 bytes) 000370 1788839297.128198 mapwrite 0x001c60 thru 0x00a75e (0x8aff bytes) 000371 1788839297.177974 write 0x02644c thru 0x02e737 (0x82ec bytes) 000372 1788839297.179113 punch from 0x02f7dc to 0x03b2b2 (0xbad7 bytes) 000373 1788839297.192534 trunc from 0x040000 to 0x03e36c (0x1c93 bytes) 000374 1788839297.228326 trunc from 0x03e36d to 0x012bf4 (0x2b778 bytes) 000375 1788839297.252261 write 0x003853 thru 0x0077ec (0x3f9a bytes) 000376 1788839297.253481 mapread 0x005f8a thru 0x0101e4 (0xa25b bytes) 000377 1788839297.254436 trunc from 0x012bf5 to 0x0119f4 (0x1200 bytes) 000378 1788839297.267018 write 0x01b061 thru 0x025351 (0xa2f1 bytes) 000379 1788839297.268141 mapread 0x016897 thru 0x018796 (0x1f00 bytes) 000380 1788839297.279231 write 0x02cc09 thru 0x037d45 (0xb13d bytes) 000381 1788839297.280464 trunc from 0x037d46 to 0x029490 (0xe8b5 bytes) 000382 1788839297.303829 trunc from 0x029491 to 0x006e61 (0x2262f bytes) 000383 1788839297.323480 trunc from 0x000b0c to 0x002ed7 (0x23cc bytes) 000384 1788839297.329430 trunc from 0x006e62 to 0x00f052 (0x81f1 bytes) 000385 1788839297.351356 write 0x003355 thru 0x006870 (0x351c bytes) 000387 1788839297.354421 mapread 0x004622 thru 0x00f052 (0xaa31 bytes) 000388 1788839297.411470 mapread 0x00bfa8 thru 0x00f052 (0x30ab bytes) 000389 1788839297.413329 punch from 0x003045 to 0x00f052 (0xc00e bytes) 000390 1788839297.442580 mapread 0x002560 thru 0x009fef (0x7a90 bytes) 000391 1788839297.571644 read 0x00d6c9 thru 0x00f052 (0x198a bytes) 000392 1788839297.610676 punch from 0x000b9d to 0x006f17 (0x637b bytes) 000393 1788839297.635401 punch from 0x005f31 to 0x009b97 (0x3c67 bytes) 000394 1788839297.666542 fallocate 0x00dae8 thru 0x018917 (0xae30 bytes) 000395 1788839297.685413 trunc from 0x00bd80 to 0x00f052 (0x32d3 bytes) 000396 1788839297.719441 mapread 0x00cac1 thru 0x00f052 (0x2592 bytes) 000398 1788839297.773258 write 0x017530 thru 0x01ac7b (0x374c bytes) 000399 1788839297.774474 fallocate 0x03218a thru 0x038d8a (0x6c01 bytes) 000400 1788839297.783245 read 0x00e54f thru 0x01ac7b (0xc72d bytes) 000403 1788839297.823315 mapread 0x01a34e thru 0x01ac7b (0x92e bytes) 000404 1788839297.827504 trunc from 0x01ac7c to 0x017fc7 (0x2cb4 bytes) 000405 1788839297.886188 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x17fc8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17fc8): Mirror 7: 0x1b4fa557 (stale) Mirror 8: 0x1b4fa557 (stale) Mirror 9: 0x1b4fa557 (stale) Mirror 10: 0x1b4fa557 (stale) Mirror 11: 0xaef5d290 (stale) CRC-32 checksum value for chunk [0, 0x17fc8): Mirror 1: 0x82064052 Mirror 12: 0x82064052 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17fc8: skipped 000407 1788839299.416212 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17fc8): Mirror 7: 0x1b4fa557 (stale) Mirror 8: 0x1b4fa557 (stale) Mirror 9: 0x1b4fa557 (stale) Mirror 10: 0x1b4fa557 (stale) Mirror 11: 0xaef5d290 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17fc8): Mirror 7: 0x1b4fa557 (stale) Mirror 8: 0x1b4fa557 (stale) Mirror 9: 0x1b4fa557 (stale) Mirror 10: 0x1b4fa557 (stale) Mirror 11: 0xaef5d290 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17fc8: skipped 000408 1788839299.952277 read 0x004446 thru 0x00d225 (0x8de0 bytes) 000410 1788839299.961114 fallocate 0x00f4f8 thru 0x01e69a (0xf1a3 bytes) 000411 1788839299.980119 mapwrite 0x0169fb thru 0x01a59a (0x3ba0 bytes) 000412 1788839300.118403 mapread 0x001774 thru 0x003dc0 (0x264d bytes) 000414 1788839300.148406 mapwrite 0x0151b5 thru 0x02441e (0xf26a bytes) 000415 1788839300.314289 mapread 0x012876 thru 0x0208ca (0xe055 bytes) 000416 1788839300.318343 trunc from 0x0113a8 to 0x01763f (0x6298 bytes) 000417 1788839300.339590 fallocate 0x00bb95 thru 0x01619f (0xa60b bytes) 000419 1788839300.352796 mapwrite 0x01742e thru 0x01c3c1 (0x4f94 bytes) 000420 1788839300.436898 trunc from 0x018f53 to 0x019a74 (0xb22 bytes) 000421 1788839300.450028 read 0x00c108 thru 0x01b76d (0xf666 bytes) 000423 1788839300.465195 write 0x027319 thru 0x028d9f (0x1a87 bytes) 000424 1788839300.466195 read 0x022df1 thru 0x028d9f (0x5faf bytes) 000425 1788839300.476060 fallocate 0x010ed4 thru 0x011554 (0x681 bytes) 000429 1788839300.515208 fallocate 0x029e43 thru 0x030312 (0x64d0 bytes) 000430 1788839300.537112 trunc from 0x028da0 to 0x02d986 (0x4be7 bytes) 000431 1788839300.586931 punch from 0x024998 to 0x026304 (0x196d bytes) 000432 1788839300.619356 trunc from 0x00f7c1 to 0x011651 (0x1e91 bytes) 000433 1788839300.662892 punch from 0x01a856 to 0x02212d (0x78d8 bytes) 000435 1788839300.699058 punch from 0x018a52 to 0x01c3bb (0x396a bytes) 000437 1788839300.718802 trunc from 0x02d987 to 0x02ce74 (0xb12 bytes) 000438 1788839300.757514 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ce75): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce75): Mirror 7: 0xaa0b3c70 (stale) Mirror 8: 0xaa0b3c70 (stale) Mirror 9: 0xaa0b3c70 (stale) Mirror 10: 0xaa0b3c70 (stale) Mirror 11: 0x59cc80ab (stale) CRC-32 checksum value for chunk [0, 0x2ce75): Mirror 1: 0x63418465 Mirror 12: 0x63418465 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce75: skipped 000439 1788839302.297728 read 0x000bac thru 0x00f2dc (0xe731 bytes) 000440 1788839302.334136 mapread 0x026b23 thru 0x02ce74 (0x6352 bytes) 000441 1788839302.376635 write 0x0279a7 thru 0x0316e7 (0x9d41 bytes) 000443 1788839302.439503 MIRROR_RESYNC 7 mirrors 000444 1788839304.923341 punch from 0x00e529 to 0x015e4a (0x7922 bytes) 000446 1788839304.952147 read 0x02ab8f thru 0x0316e7 (0x6b59 bytes) 000447 1788839304.965005 fallocate 0x026967 thru 0x030e58 (0xa4f2 bytes) 000449 1788839304.976625 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x316e8): Mirror 7: 0xb982774d (stale) Mirror 9: 0xb982774d (stale) Mirror 10: 0xb982774d (stale) Mirror 11: 0xb982774d (stale) Mirror 12: 0xb982774d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x316e8): Mirror 7: 0xb982774d (stale) Mirror 9: 0xb982774d (stale) Mirror 10: 0xb982774d (stale) Mirror 11: 0xb982774d (stale) Mirror 12: 0xb982774d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x316e8: skipped 000450 1788839305.508704 write 0x00a055 thru 0x00cb56 (0x2b02 bytes) 000451 1788839305.529575 mapwrite 0x022d42 thru 0x024714 (0x19d3 bytes) 000452 1788839305.602272 mapread 0x021a86 thru 0x02cdb7 (0xb332 bytes) 000453 1788839305.603593 punch from 0x007635 to 0x0124dd (0xaea9 bytes) 000454 1788839305.617673 mapread 0x0067f6 thru 0x015446 (0xec51 bytes) 000455 1788839305.637741 trunc from 0x009b97 to 0x012032 (0x849c bytes) 000456 1788839305.653038 trunc from 0x0316e8 to 0x016d12 (0x1a9d5 bytes) 000457 1788839305.674599 read 0x00a6be thru 0x013ba3 (0x94e6 bytes) 000458 1788839305.687432 trunc from 0x004592 to 0x01026a (0xbcd9 bytes) 000459 1788839305.693448 punch from 0x005e69 to 0x00ac66 (0x4dfe bytes) 000460 1788839305.708039 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x16d13): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16d13): Mirror 7: 0x1cd6f79f (stale) Mirror 9: 0x1cd6f79f (stale) Mirror 10: 0x1cd6f79f (stale) Mirror 11: 0x1cd6f79f (stale) Mirror 12: 0x1cd6f79f (stale) CRC-32 checksum value for chunk [0, 0x16d13): Mirror 1: 0x1cd6f79f Mirror 13: 0x1cd6f79f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16d13: skipped 000461 1788839306.566156 mapread 0x01497e thru 0x016d12 (0x2395 bytes) 000462 1788839306.605657 mapwrite 0x02af11 thru 0x02e2cf (0x33bf bytes) 000463 1788839306.702435 trunc from 0x02e2d0 to 0x031d5a (0x3a8b bytes) 000464 1788839306.758738 trunc from 0x00411f to 0x004c4b (0xb2d bytes) 000465 1788839306.764820 fallocate 0x020ea3 thru 0x02ed5d (0xdebb bytes) 000466 1788839306.773838 fallocate 0x019d32 thru 0x01eb8a (0x4e59 bytes) 000467 1788839306.786332 read 0x02cf95 thru 0x031d5a (0x4dc6 bytes) 000468 1788839306.799496 trunc from 0x031d5b to 0x008bec (0x2916e bytes) 000469 1788839306.833164 read 0x00843b thru 0x008bec (0x7b2 bytes) 000470 1788839306.847452 mapwrite 0x03f963 thru 0x03ffff (0x69d bytes) 000471 1788839306.972287 punch from 0x021114 to 0x030062 (0xef4f bytes) 000472 1788839306.981850 read 0x0229c6 thru 0x0277c5 (0x4e00 bytes) 000473 1788839306.997501 punch from 0x037ac8 to 0x0382af (0x7e8 bytes) 000474 1788839307.004694 trunc from 0x01a953 to 0x01d6f6 (0x2da4 bytes) 000475 1788839307.008396 read 0x0037f7 thru 0x0053e7 (0x1bf1 bytes) 000476 1788839307.010143 punch from 0x01542d to 0x018a60 (0x3634 bytes) 000477 1788839307.014022 mapread 0x02e5dc thru 0x0312f7 (0x2d1c bytes) 000478 1788839307.023336 MIRROR_RESYNC 7 mirrors 000479 1788839308.445123 mapwrite 0x030157 thru 0x03db75 (0xda1f bytes) 000480 1788839308.586087 fallocate 0x03571d thru 0x03b4e3 (0x5dc7 bytes) 000481 1788839308.592412 punch from 0x0298d2 to 0x029fdf (0x70e bytes) 000482 1788839308.608883 MIRROR_RESYNC 7 mirrors 000483 1788839310.429142 trunc from 0x03cbaf to 0x03ffff (0x3451 bytes) 000484 1788839310.462142 trunc from 0x040000 to 0x0293e9 (0x16c16 bytes) 000485 1788839310.485840 fallocate 0x0005bd thru 0x0081ad (0x7bf1 bytes) 000486 1788839310.505094 read 0x016098 thru 0x016879 (0x7e2 bytes) 000488 1788839310.512671 write 0x00120f thru 0x0109ff (0xf7f1 bytes) 000489 1788839310.533267 read 0x017c9a thru 0x023ed2 (0xc239 bytes) 000490 1788839310.544276 read 0x007432 thru 0x011527 (0xa0f6 bytes) 000492 1788839310.545524 mapread 0x00df1a thru 0x014d99 (0x6e80 bytes) 000493 1788839310.546935 mapwrite 0x02dc0c thru 0x03c472 (0xe867 bytes) 000494 1788839310.611184 fallocate 0x00bd94 thru 0x00d45b (0x16c8 bytes) 000495 1788839310.617530 mapread 0x035eb1 thru 0x03b01a (0x516a bytes) 000497 1788839310.620298 mapwrite 0x00b2fa thru 0x019640 (0xe347 bytes) 000499 1788839310.643155 write 0x000d08 thru 0x006bfe (0x5ef7 bytes) 000500 1788839310.645455 mapwrite 0x023259 thru 0x02bf4c (0x8cf4 bytes) 000501 1788839310.666936 fallocate 0x01e00e thru 0x025f5e (0x7f51 bytes) 000502 1788839310.672077 mapread 0x012d33 thru 0x01df3a (0xb208 bytes) 000503 1788839310.674396 mapread 0x027fe0 thru 0x036292 (0xe2b3 bytes) 000504 1788839310.676314 trunc from 0x03c473 to 0x0153c4 (0x270ae bytes) 000505 1788839310.714249 mapread 0x0068dc thru 0x00f78d (0x8eb2 bytes) 000506 1788839310.723527 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x153c5): Mirror 7: 0xaac6b8ec (stale) Mirror 9: 0xaac6b8ec (stale) Mirror 10: 0xaac6b8ec (stale) Mirror 11: 0xaac6b8ec (stale) Mirror 12: 0xaac6b8ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x153c5): Mirror 7: 0xaac6b8ec (stale) Mirror 9: 0xaac6b8ec (stale) Mirror 10: 0xaac6b8ec (stale) Mirror 11: 0xaac6b8ec (stale) Mirror 12: 0xaac6b8ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x153c5: skipped 000507 1788839311.433178 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x153c5): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x153c5): Mirror 7: 0xaac6b8ec (stale) Mirror 9: 0xaac6b8ec (stale) Mirror 10: 0xaac6b8ec (stale) Mirror 11: 0xaac6b8ec (stale) Mirror 12: 0xaac6b8ec (stale) CRC-32 checksum value for chunk [0, 0x153c5): Mirror 1: 0x3a829590 Mirror 13: 0x3a829590 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x153c5: skipped 000508 1788839312.428493 mapread 0x002173 thru 0x00d0ec (0xaf7a bytes) 000509 1788839312.454420 write 0x005ac8 thru 0x012041 (0xc57a bytes) 000510 1788839312.541903 mapwrite 0x004944 thru 0x0104af (0xbb6c bytes) 000511 1788839312.578382 mapread 0x005bdd thru 0x00d998 (0x7dbc bytes) 000512 1788839312.582053 mapread 0x005706 thru 0x00bc5d (0x6558 bytes) 000513 1788839312.589219 MIRROR_RESYNC 7 mirrors 000514 1788839313.825403 mapwrite 0x00c776 thru 0x01bb35 (0xf3c0 bytes) 000515 1788839313.944161 read 0x005469 thru 0x008762 (0x32fa bytes) 000516 1788839313.962273 mapread 0x00a901 thru 0x010d2a (0x642a bytes) 000517 1788839313.967397 trunc from 0x0111ea to 0x01bb35 (0xa94c bytes) 000518 1788839313.978721 mapwrite 0x029659 thru 0x036c98 (0xd640 bytes) 000519 1788839314.092274 punch from 0x012454 to 0x0128f5 (0x4a2 bytes) 000520 1788839314.108047 read 0x01fcf0 thru 0x022cf9 (0x300a bytes) 000521 1788839314.125614 fallocate 0x038949 thru 0x03b749 (0x2e01 bytes) 000522 1788839314.130475 punch from 0x033110 to 0x035076 (0x1f67 bytes) 000523 1788839314.140011 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b74a): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b74a): Mirror 7: 0x62a535ca (stale) Mirror 9: 0x62a535ca (stale) Mirror 10: 0x62a535ca (stale) Mirror 11: 0x62a535ca (stale) Mirror 12: 0x62a535ca (stale) Mirror 13: 0x62a535ca (stale) CRC-32 checksum value for chunk [0, 0x3b74a): Mirror 1: 0x646c8f3b Mirror 14: 0x646c8f3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b74a: skipped 000524 1788839315.379326 MIRROR_RESYNC 8 mirrors 000525 1788839319.107481 MIRROR_EXTEND to 9 mirrors 000526 1788839320.667289 mapwrite 0x01098f thru 0x010f20 (0x592 bytes) 000527 1788839320.818970 punch from 0x0158b3 to 0x01aa45 (0x5193 bytes) 000528 1788839320.830230 MIRROR_RESYNC 9 mirrors 000529 1788839323.934413 write 0x0212d8 thru 0x027e6d (0x6b96 bytes) 000530 1788839323.983978 mapwrite 0x037930 thru 0x03d841 (0x5f12 bytes) 000531 1788839324.094841 mapwrite 0x004893 thru 0x00cc7e (0x83ec bytes) 000532 1788839324.155912 mapwrite 0x023fbe thru 0x02f111 (0xb154 bytes) 000533 1788839324.233920 trunc from 0x03d842 to 0x012cd6 (0x2ab6b bytes) 000534 1788839324.258639 punch from 0x000825 to 0x000bb3 (0x38f bytes) 000535 1788839324.271131 fallocate 0x003c58 thru 0x00fb21 (0xbeca bytes) 000536 1788839324.278693 write 0x02fc90 thru 0x03909d (0x940e bytes) 000537 1788839324.283935 mapread 0x001701 thru 0x001ee5 (0x7e5 bytes) 000538 1788839324.285122 trunc from 0x03909e to 0x036c1c (0x2481 bytes) 000539 1788839324.317976 punch from 0x00fcf5 to 0x019010 (0x931c bytes) 000540 1788839324.328657 trunc from 0x034fd5 to 0x03654e (0x157a bytes) 000541 1788839324.371861 trunc from 0x036c1d to 0x032647 (0x45d5 bytes) 000542 1788839324.427241 trunc from 0x032648 to 0x01cc89 (0x159be bytes) 000543 1788839324.471010 trunc from 0x01cc8a to 0x028957 (0xbcce bytes) 000544 1788839324.516613 mapwrite 0x02230f thru 0x02eea6 (0xcb98 bytes) 000545 1788839324.639916 read 0x01d32f thru 0x01fb4b (0x281d bytes) 000546 1788839324.641816 read 0x0028dd thru 0x004ee3 (0x2607 bytes) 000547 1788839324.642975 trunc from 0x02438d to 0x02dc4a (0x98be bytes) 000548 1788839324.652676 trunc from 0x02eea7 to 0x016232 (0x18c74 bytes) 000549 1788839324.682064 read 0x00d360 thru 0x0155a2 (0x8243 bytes) 000550 1788839324.694614 trunc from 0x005573 to 0x00dfe0 (0x8a6e bytes) 000551 1788839324.702948 fallocate 0x023cec thru 0x028283 (0x4598 bytes) 000552 1788839324.710893 mapwrite 0x00e43b thru 0x017b3b (0x9701 bytes) 000553 1788839324.762413 trunc from 0x0131bd to 0x01631f (0x3163 bytes) 000554 1788839324.776692 write 0x018a38 thru 0x02360e (0xabd7 bytes) 000555 1788839324.786932 punch from 0x021d4c to 0x026dc2 (0x5077 bytes) 000556 1788839324.827941 read 0x0095e0 thru 0x00b5af (0x1fd0 bytes) 000557 1788839324.846167 trunc from 0x028284 to 0x02c3f1 (0x416e bytes) 000558 1788839324.865148 mapwrite 0x01d8b2 thru 0x021931 (0x4080 bytes) 000559 1788839324.886629 trunc from 0x02c3f2 to 0x03a704 (0xe313 bytes) 000560 1788839324.904558 mapwrite 0x02f5a9 thru 0x038c6a (0x96c2 bytes) 000561 1788839324.939063 trunc from 0x03a705 to 0x00b16d (0x2f597 bytes) 000562 1788839324.961925 punch from 0x007f18 to 0x00b16d (0x3256 bytes) 000563 1788839324.973342 fallocate 0x03b55e thru 0x03d1de (0x1c81 bytes) 000564 1788839324.981113 mapwrite 0x0346f0 thru 0x03a8db (0x61ec bytes) 000565 1788839325.079353 mapread 0x00bd02 thru 0x01958d (0xd88c bytes) 000566 1788839325.246634 trunc from 0x033fb9 to 0x03a8db (0x6923 bytes) 000567 1788839325.254710 read 0x00fd98 thru 0x01af31 (0xb19a bytes) 000568 1788839325.289020 mapwrite 0x024a88 thru 0x031b15 (0xd08e bytes) 000569 1788839325.315026 punch from 0x00a59f to 0x0197ac (0xf20e bytes) 000570 1788839325.321845 read 0x00b721 thru 0x01624e (0xab2e bytes) 000571 1788839325.341042 mapwrite 0x003108 thru 0x0094c5 (0x63be bytes) 000572 1788839325.367170 read 0x02a5eb thru 0x03790e (0xd324 bytes) 000573 1788839325.367947 write 0x032400 thru 0x03af8b (0x8b8c bytes) 000574 1788839325.369257 mapwrite 0x01e83d thru 0x02479d (0x5f61 bytes) 000575 1788839325.398873 fallocate 0x0149d0 thru 0x0204e8 (0xbb19 bytes) 000576 1788839325.405987 mapwrite 0x0313fb thru 0x034657 (0x325d bytes) 000577 1788839325.430017 write 0x01f6f5 thru 0x026d70 (0x767c bytes) 000578 1788839325.432876 write 0x0246ea thru 0x033e23 (0xf73a bytes) 000579 1788839325.436145 trunc from 0x03af8c to 0x022f3b (0x18050 bytes) 000580 1788839325.491642 trunc from 0x022f3c to 0x01d648 (0x58f3 bytes) 000581 1788839325.523912 mapwrite 0x01c27d thru 0x027a45 (0xb7c9 bytes) 000583 1788839325.564449 write 0x02553a thru 0x0301fe (0xacc5 bytes) 000584 1788839325.569190 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301ff): Mirror 7: 0xe71f5a69 (stale) Mirror 10: 0xe71f5a69 (stale) Mirror 11: 0xe71f5a69 (stale) Mirror 12: 0xe71f5a69 (stale) Mirror 13: 0xe71f5a69 (stale) Mirror 14: 0xe71f5a69 (stale) Mirror 15: 0xe71f5a69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301ff): Mirror 7: 0xe71f5a69 (stale) Mirror 10: 0xe71f5a69 (stale) Mirror 11: 0xe71f5a69 (stale) Mirror 12: 0xe71f5a69 (stale) Mirror 13: 0xe71f5a69 (stale) Mirror 14: 0xe71f5a69 (stale) Mirror 15: 0xe71f5a69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301ff: skipped 000585 1788839326.192156 write 0x03f875 thru 0x03ffff (0x78b bytes) 000586 1788839326.213984 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x4b7ee1db (stale) Mirror 10: 0x4b7ee1db (stale) Mirror 11: 0x4b7ee1db (stale) Mirror 12: 0x4b7ee1db (stale) Mirror 13: 0x4b7ee1db (stale) Mirror 14: 0x4b7ee1db (stale) Mirror 15: 0x4b7ee1db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcea0867d Mirror 16: 0xcea0867d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1788839328.101089 fallocate 0x00777d thru 0x00d572 (0x5df6 bytes) 000588 1788839328.211912 fallocate 0x02f80f thru 0x031285 (0x1a77 bytes) 000589 1788839328.230102 read 0x02ba23 thru 0x03610f (0xa6ed bytes) 000590 1788839328.266884 punch from 0x0089b8 to 0x00997b (0xfc4 bytes) 000591 1788839328.274974 read 0x01c9e1 thru 0x023e52 (0x7472 bytes) 000592 1788839328.311290 trunc from 0x040000 to 0x02ccf4 (0x1330b bytes) 000593 1788839328.381611 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ccf5): Mirror 7: 0x149496ce (stale) Mirror 10: 0x149496ce (stale) Mirror 11: 0x149496ce (stale) Mirror 12: 0x149496ce (stale) Mirror 14: 0x149496ce (stale) Mirror 15: 0x149496ce (stale) Mirror 16: 0x869304c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ccf5): Mirror 7: 0x149496ce (stale) Mirror 10: 0x149496ce (stale) Mirror 11: 0x149496ce (stale) Mirror 12: 0x149496ce (stale) Mirror 14: 0x149496ce (stale) Mirror 15: 0x149496ce (stale) Mirror 16: 0x869304c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ccf5: skipped 000594 1788839329.454335 trunc from 0x02ccf5 to 0x00e185 (0x1eb6f bytes) 000595 1788839329.546050 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe186): Mirror 10: 0xd4c3b415 (stale) Mirror 11: 0xd4c3b415 (stale) Mirror 12: 0xd4c3b415 (stale) Mirror 14: 0xd4c3b415 (stale) Mirror 15: 0xd4c3b415 (stale) Mirror 16: 0x9684459f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe186): Mirror 10: 0xd4c3b415 (stale) Mirror 11: 0xd4c3b415 (stale) Mirror 12: 0xd4c3b415 (stale) Mirror 14: 0xd4c3b415 (stale) Mirror 15: 0xd4c3b415 (stale) Mirror 16: 0x9684459f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe186: skipped 000596 1788839330.328497 mapwrite 0x0324c6 thru 0x03ffff (0xdb3a bytes) 000597 1788839330.435151 mapwrite 0x01d994 thru 0x025729 (0x7d96 bytes) 000598 1788839330.510550 fallocate 0x01145a thru 0x01cb02 (0xb6a9 bytes) 000599 1788839330.522265 fallocate 0x00c4af thru 0x00e8da (0x242c bytes) 000600 1788839330.552953 mapread 0x013487 thru 0x018672 (0x51ec bytes) 000601 1788839330.582591 read 0x00bd9b thru 0x019ac5 (0xdd2b bytes) 000602 1788839330.598218 write 0x038701 thru 0x03ef83 (0x6883 bytes) 000603 1788839330.606525 punch from 0x00d2f3 to 0x013161 (0x5e6f bytes) 000604 1788839330.623268 write 0x01642d thru 0x01eb2f (0x8703 bytes) 000605 1788839330.624656 fallocate 0x019d53 thru 0x01c8a4 (0x2b52 bytes) 000606 1788839330.634317 fallocate 0x026dd6 thru 0x0341c6 (0xd3f1 bytes) 000607 1788839330.663898 fallocate 0x011f61 thru 0x019f1a (0x7fba bytes) 000608 1788839330.688082 write 0x03b2c9 thru 0x03d9f6 (0x272e bytes) 000609 1788839330.701220 mapwrite 0x0006a7 thru 0x0059a1 (0x52fb bytes) 000610 1788839330.808883 fallocate 0x019a49 thru 0x029847 (0xfdff bytes) 000611 1788839330.830172 trunc from 0x040000 to 0x0184c1 (0x27b3e bytes) 000612 1788839330.862408 trunc from 0x010bbc to 0x0184c1 (0x7906 bytes) 000613 1788839330.880014 MIRROR_RESYNC 7 mirrors 000614 1788839332.478861 MIRROR_EXTEND to 8 mirrors 000615 1788839333.083288 MIRROR_RESYNC 8 mirrors 000616 1788839333.842686 fallocate 0x02e47c thru 0x03b535 (0xd0ba bytes) 000617 1788839333.904996 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x184c2): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x184c2): Mirror 10: 0x39c53540 (stale) Mirror 11: 0x39c53540 (stale) Mirror 12: 0x39c53540 (stale) Mirror 14: 0x39c53540 (stale) Mirror 15: 0x39c53540 (stale) Mirror 16: 0x39c53540 (stale) Mirror 17: 0x39c53540 (stale) CRC-32 checksum value for chunk [0, 0x184c2): Mirror 1: 0x39c53540 Mirror 18: 0x39c53540 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x184c2: skipped 000619 1788839335.387031 trunc from 0x0109ad to 0x0171d7 (0x682b bytes) 000620 1788839335.535098 mapread 0x009909 thru 0x010385 (0x6a7d bytes) 000621 1788839335.580711 trunc from 0x0184c2 to 0x00c3de (0xc0e3 bytes) 000622 1788839335.635794 read 0x005f16 thru 0x00c1a4 (0x628f bytes) 000623 1788839335.638510 punch from 0x006641 to 0x00c3de (0x5d9e bytes) 000624 1788839335.658114 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc3df): Mirror 10: 0xf302663 (stale) Mirror 11: 0xf302663 (stale) Mirror 12: 0xf302663 (stale) Mirror 14: 0xf302663 (stale) Mirror 15: 0xf302663 (stale) Mirror 16: 0xf302663 (stale) Mirror 18: 0xf302663 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc3df): Mirror 10: 0xf302663 (stale) Mirror 11: 0xf302663 (stale) Mirror 12: 0xf302663 (stale) Mirror 14: 0xf302663 (stale) Mirror 15: 0xf302663 (stale) Mirror 16: 0xf302663 (stale) Mirror 18: 0xf302663 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc3df: skipped 000625 1788839336.912503 punch from 0x0034d2 to 0x00b8c8 (0x83f7 bytes) 000626 1788839336.929464 read 0x00a1c8 thru 0x00c3de (0x2217 bytes) 000627 1788839336.957088 mapread 0x000090 thru 0x00c3de (0xc34f bytes) 000628 1788839337.054351 mapread 0x001d02 thru 0x00b604 (0x9903 bytes) 000629 1788839337.055212 mapread 0x00050e thru 0x00c3de (0xbed1 bytes) 000630 1788839337.056046 read 0x00bf18 thru 0x00c3de (0x4c7 bytes) 000631 1788839337.081498 trunc from 0x00c3df to 0x01e56a (0x1218c bytes) 000632 1788839337.146089 MIRROR_RESYNC 8 mirrors 000633 1788839338.865018 punch from 0x00a42a to 0x00d509 (0x30e0 bytes) 000634 1788839338.903848 write 0x00b8c5 thru 0x00e85e (0x2f9a bytes) 000635 1788839338.913111 trunc from 0x011208 to 0x01e56a (0xd363 bytes) 000636 1788839338.919470 mapwrite 0x00a06f thru 0x00d4eb (0x347d bytes) 000638 1788839338.941676 trunc from 0x01e56b to 0x027cc7 (0x975d bytes) 000639 1788839338.957172 mapwrite 0x00bb80 thru 0x010268 (0x46e9 bytes) 000640 1788839338.995847 read 0x02504e thru 0x02790f (0x28c2 bytes) 000641 1788839339.002572 write 0x0335ce thru 0x03dcff (0xa732 bytes) 000642 1788839339.009803 MIRROR_RESYNC 8 mirrors 000643 1788839341.926029 trunc from 0x00fb2a to 0x01b72b (0xbc02 bytes) 000644 1788839341.979861 mapwrite 0x009b8b thru 0x00b387 (0x17fd bytes) 000646 1788839342.034196 punch from 0x0030a5 to 0x00ef59 (0xbeb5 bytes) 000648 1788839342.042425 mapread 0x019f0a thru 0x01c4f1 (0x25e8 bytes) 000649 1788839342.058418 fallocate 0x0160f0 thru 0x022feb (0xcefc bytes) 000650 1788839342.063131 fallocate 0x017e72 thru 0x01df37 (0x60c6 bytes) 000651 1788839342.068479 trunc from 0x0206f5 to 0x02e92d (0xe239 bytes) 000652 1788839342.078196 fallocate 0x023aae thru 0x02dd56 (0xa2a9 bytes) 000653 1788839342.083413 punch from 0x03a1aa to 0x03dcff (0x3b56 bytes) 000654 1788839342.098290 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3dd00): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd00): Mirror 10: 0xf779f148 (stale) Mirror 11: 0xf779f148 (stale) Mirror 12: 0xf779f148 (stale) Mirror 14: 0xf779f148 (stale) Mirror 15: 0xf779f148 (stale) Mirror 16: 0xf779f148 (stale) Mirror 18: 0xf779f148 (stale) CRC-32 checksum value for chunk [0, 0x3dd00): Mirror 1: 0x8a35324d Mirror 19: 0x8a35324d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd00: skipped 000655 1788839343.046117 trunc from 0x03dd00 to 0x00b215 (0x32aea bytes) 000656 1788839343.157458 mapread 0x007599 thru 0x008a99 (0x1501 bytes) 000657 1788839343.169513 trunc from 0x00b216 to 0x01b518 (0x10303 bytes) 000658 1788839343.200670 trunc from 0x006df0 to 0x014e45 (0xe056 bytes) 000659 1788839343.209965 fallocate 0x0372ba thru 0x03ae51 (0x3b98 bytes) 000661 1788839343.219723 read 0x0397d3 thru 0x03ae51 (0x167f bytes) 000662 1788839343.227410 mapwrite 0x011456 thru 0x016f3b (0x5ae6 bytes) 000663 1788839343.272153 trunc from 0x03ae52 to 0x013ede (0x26f73 bytes) 000664 1788839343.291113 punch from 0x00b338 to 0x013ede (0x8ba7 bytes) 000665 1788839343.297439 write 0x012a04 thru 0x01fa46 (0xd043 bytes) 000666 1788839343.308066 fallocate 0x0093cb thru 0x0177c6 (0xe3fc bytes) 000667 1788839343.315036 fallocate 0x01b788 thru 0x024d78 (0x95f1 bytes) 000668 1788839343.323442 trunc from 0x017745 to 0x019488 (0x1d44 bytes) 000669 1788839343.351528 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fa47): Mirror 10: 0xa82cc772 (stale) Mirror 11: 0xa82cc772 (stale) Mirror 14: 0xa82cc772 (stale) Mirror 15: 0xa82cc772 (stale) Mirror 16: 0xa82cc772 (stale) Mirror 18: 0xa82cc772 (stale) Mirror 19: 0xbadf168a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fa47): Mirror 10: 0xa82cc772 (stale) Mirror 11: 0xa82cc772 (stale) Mirror 14: 0xa82cc772 (stale) Mirror 15: 0xa82cc772 (stale) Mirror 16: 0xa82cc772 (stale) Mirror 18: 0xa82cc772 (stale) Mirror 19: 0xbadf168a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fa47: skipped 000670 1788839344.256241 mapwrite 0x03f607 thru 0x03ffff (0x9f9 bytes) 000671 1788839344.355861 trunc from 0x040000 to 0x0206c5 (0x1f93a bytes) 000672 1788839344.411913 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: 0xa669f46c (stale) Mirror 14: 0xa669f46c (stale) Mirror 15: 0xa669f46c (stale) Mirror 16: 0xa669f46c (stale) Mirror 18: 0xa669f46c (stale) Mirror 19: 0xee478aed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: 0xa669f46c (stale) Mirror 14: 0xa669f46c (stale) Mirror 15: 0xa669f46c (stale) Mirror 16: 0xa669f46c (stale) Mirror 18: 0xa669f46c (stale) Mirror 19: 0xee478aed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x206c6: skipped 000673 1788839345.382906 trunc from 0x00f4cc to 0x0174f1 (0x8026 bytes) 000674 1788839345.396915 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: 0xa669f46c (stale) Mirror 14: 0xa669f46c (stale) Mirror 15: 0xa669f46c (stale) Mirror 16: 0xa669f46c (stale) Mirror 18: 0xa669f46c (stale) Mirror 19: 0xee478aed (stale) CRC-32 checksum value for chunk [0, 0x206c6): Mirror 1: 0xe506a246 Mirror 20: 0xe506a246 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x206c6: skipped 000675 1788839346.715394 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: 0xa669f46c (stale) Mirror 14: 0xa669f46c (stale) Mirror 15: 0xa669f46c (stale) Mirror 16: 0xa669f46c (stale) Mirror 18: 0xa669f46c (stale) Mirror 19: 0xee478aed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: 0xa669f46c (stale) Mirror 14: 0xa669f46c (stale) Mirror 15: 0xa669f46c (stale) Mirror 16: 0xa669f46c (stale) Mirror 18: 0xa669f46c (stale) Mirror 19: 0xee478aed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x206c6: skipped 000676 1788839347.153670 punch from 0x00a541 to 0x014c6c (0xa72c bytes) 000677 1788839347.182427 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: 0xa669f46c (stale) Mirror 14: 0xa669f46c (stale) Mirror 15: 0xa669f46c (stale) Mirror 18: 0xa669f46c (stale) Mirror 19: 0xee478aed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x206c6): Mirror 10: 0xa669f46c (stale) Mirror 14: 0xa669f46c (stale) Mirror 15: 0xa669f46c (stale) Mirror 18: 0xa669f46c (stale) Mirror 19: 0xee478aed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x206c6: skipped 000678 1788839347.853385 fallocate 0x0287f5 thru 0x034349 (0xbb55 bytes) 000679 1788839347.864945 trunc from 0x01fb04 to 0x01fb34 (0x31 bytes) 000680 1788839347.873633 mapwrite 0x03967d thru 0x039bcb (0x54f bytes) 000681 1788839347.914876 MIRROR_RESYNC 6 mirrors 000682 1788839350.782912 trunc from 0x039bcc to 0x02c012 (0xdbb9 bytes) 000684 1788839350.888732 mapread 0x015908 thru 0x015f58 (0x651 bytes) 000685 1788839350.895624 punch from 0x013c9c to 0x01421a (0x57f bytes) 000686 1788839350.901489 punch from 0x00a6de to 0x00ba97 (0x13ba bytes) 000687 1788839350.907718 read 0x02b9e5 thru 0x02c012 (0x62e bytes) 000689 1788839350.917354 write 0x029448 thru 0x0326c6 (0x927f bytes) 000690 1788839350.929108 mapwrite 0x0119e7 thru 0x017da6 (0x63c0 bytes) 000691 1788839350.990374 punch from 0x0012ef to 0x004e11 (0x3b23 bytes) 000692 1788839351.003548 trunc from 0x0326c7 to 0x007f8a (0x2a73c bytes) 000694 1788839351.049612 fallocate 0x014c28 thru 0x019e0d (0x51e6 bytes) 000695 1788839351.054329 trunc from 0x007f8b to 0x001ead (0x60dd bytes) 000696 1788839351.086028 trunc from 0x001eae to 0x00f9f9 (0xdb4c bytes) 000697 1788839351.150670 MIRROR_RESYNC 6 mirrors 000698 1788839352.282581 read 0x00e9f3 thru 0x00f9f9 (0x1007 bytes) 000699 1788839352.287306 write 0x021444 thru 0x02529c (0x3e59 bytes) 000700 1788839352.338900 mapread 0x0171a8 thru 0x0190a3 (0x1efc bytes) 000701 1788839352.370722 fallocate 0x0084fa thru 0x00c51d (0x4024 bytes) 000702 1788839352.375813 trunc from 0x02529d to 0x022af5 (0x27a7 bytes) 000703 1788839352.443582 MIRROR_RESYNC 6 mirrors 000704 1788839353.675293 mapread 0x0080b6 thru 0x0159b1 (0xd8fc bytes) 000705 1788839353.713908 trunc from 0x008d4a to 0x015a12 (0xccc9 bytes) 000706 1788839353.748865 mapread 0x0088cc thru 0x009a62 (0x1197 bytes) 000707 1788839353.759282 write 0x01ff00 thru 0x025fcc (0x60cd bytes) 000708 1788839353.766343 mapwrite 0x03fe05 thru 0x03ffff (0x1fb bytes) 000709 1788839353.819049 read 0x013f18 thru 0x01ab1d (0x6c06 bytes) 000710 1788839353.830291 mapread 0x02e841 thru 0x03209b (0x385b bytes) 000713 1788839353.854955 MIRROR_RESYNC 6 mirrors 000714 1788839356.128847 mapread 0x0034fc thru 0x0109c1 (0xd4c6 bytes) 000715 1788839356.197415 read 0x03a8f6 thru 0x03f2c9 (0x49d4 bytes) 000716 1788839356.224418 trunc from 0x040000 to 0x03bcdc (0x4323 bytes) 000717 1788839356.356488 punch from 0x00c117 to 0x00e29b (0x2185 bytes) 000718 1788839356.375282 punch from 0x015aa9 to 0x0225da (0xcb32 bytes) 000719 1788839356.384846 trunc from 0x03bcdd to 0x01e552 (0x1d78a bytes) 000720 1788839356.448646 mapread 0x012f5d thru 0x01445f (0x1503 bytes) 000721 1788839356.476476 trunc from 0x0048ea to 0x014651 (0xfd68 bytes) 000722 1788839356.490352 mapread 0x01cc2b thru 0x01e552 (0x1928 bytes) 000723 1788839356.513459 mapwrite 0x036d63 thru 0x03ffff (0x929d bytes) 000724 1788839356.616618 trunc from 0x040000 to 0x0326eb (0xd914 bytes) 000726 1788839356.644456 trunc from 0x0326ec to 0x038205 (0x5b1a bytes) 000727 1788839356.698210 write 0x026428 thru 0x0339fb (0xd5d4 bytes) 000728 1788839356.729316 punch from 0x020a54 to 0x02dee8 (0xd495 bytes) 000729 1788839356.766830 mapwrite 0x016edd thru 0x020ddf (0x9f03 bytes) 000730 1788839356.850540 punch from 0x025080 to 0x032587 (0xd508 bytes) 000731 1788839356.857442 write 0x0243ea thru 0x028050 (0x3c67 bytes) 000732 1788839356.878337 fallocate 0x01d408 thru 0x0214ec (0x40e5 bytes) 000733 1788839356.892554 write 0x034474 thru 0x035c4d (0x17da bytes) 000734 1788839356.894931 mapwrite 0x005ce9 thru 0x00df3a (0x8252 bytes) 000735 1788839356.973668 read 0x01889b thru 0x022fb1 (0xa717 bytes) 000736 1788839356.980215 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38206): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38206): Mirror 10: 0x7d12decc (stale) Mirror 14: 0x7d12decc (stale) Mirror 15: 0x7d12decc (stale) Mirror 18: 0x7d12decc (stale) Mirror 19: 0x7d12decc (stale) CRC-32 checksum value for chunk [0, 0x38206): Mirror 1: 0xe7b47ac6 Mirror 20: 0xe7b47ac6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38206: skipped 000738 1788839359.552092 trunc from 0x038206 to 0x0040b1 (0x34154 bytes) 000739 1788839359.619648 read 0x003019 thru 0x0040b1 (0x1099 bytes) 000740 1788839359.630224 read 0x002c10 thru 0x0040b1 (0x14a2 bytes) 000741 1788839359.640284 trunc from 0x001407 to 0x002b4e (0x1748 bytes) 000742 1788839359.646157 mapwrite 0x02a88b thru 0x02fa72 (0x51e8 bytes) 000743 1788839359.738460 trunc from 0x01f1a0 to 0x02e376 (0xf1d7 bytes) 000744 1788839359.746932 mapread 0x0219b9 thru 0x02d4fc (0xbb44 bytes) 000745 1788839359.859139 trunc from 0x02fa73 to 0x024d3c (0xad36 bytes) 000746 1788839359.907323 mapwrite 0x031789 thru 0x03c6f6 (0xaf6e bytes) 000747 1788839360.008314 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c6f7): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6f7): Mirror 10: 0x4052d4bf (stale) Mirror 14: 0x4052d4bf (stale) Mirror 15: 0x4052d4bf (stale) Mirror 18: 0x4052d4bf (stale) Mirror 19: 0x4052d4bf (stale) Mirror 20: 0xbe4e8429 (stale) CRC-32 checksum value for chunk [0, 0x3c6f7): Mirror 1: 0xad4fe1af Mirror 21: 0xad4fe1af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6f7: skipped 000748 1788839361.877025 mapwrite 0x00552f thru 0x00e541 (0x9013 bytes) 000749 1788839362.072649 mapwrite 0x0178e5 thru 0x023778 (0xbe94 bytes) 000750 1788839362.200295 fallocate 0x0190cc thru 0x01d67a (0x45af bytes) 000751 1788839362.222636 write 0x03e2d0 thru 0x03ffff (0x1d30 bytes) 000752 1788839362.232884 fallocate 0x01b3bc thru 0x020398 (0x4fdd bytes) 000753 1788839362.258921 trunc from 0x014e6b to 0x01e596 (0x972c bytes) 000754 1788839362.286983 trunc from 0x006c49 to 0x00d720 (0x6ad8 bytes) 000755 1788839362.315092 write 0x0340f1 thru 0x03d4a0 (0x93b0 bytes) 000756 1788839362.368637 fallocate 0x03b594 thru 0x03ffff (0x4a6c bytes) 000757 1788839362.396678 punch from 0x039f7a to 0x03ae17 (0xe9e bytes) 000759 1788839362.436665 fallocate 0x00ce1a thru 0x016a6b (0x9c52 bytes) 000761 1788839362.458983 mapread 0x02d80b thru 0x034d72 (0x7568 bytes) 000762 1788839362.480012 trunc from 0x040000 to 0x02bb49 (0x144b6 bytes) 000764 1788839362.525263 fallocate 0x00f5e6 thru 0x01a1e6 (0xac01 bytes) 000765 1788839362.573376 MIRROR_RESYNC 8 mirrors 000766 1788839367.409641 trunc from 0x01e8e9 to 0x022cc7 (0x43df bytes) 000767 1788839367.484896 write 0x032985 thru 0x03ffff (0xd67b bytes) 000768 1788839367.512473 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xbc1d6bca (stale) Mirror 14: 0xbc1d6bca (stale) Mirror 15: 0xbc1d6bca (stale) Mirror 18: 0xbc1d6bca (stale) Mirror 19: 0xbc1d6bca (stale) Mirror 20: 0xbc1d6bca (stale) Mirror 21: 0xbc1d6bca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x69dd97e4 Mirror 22: 0x69dd97e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000769 1788839369.363519 mapwrite 0x00d6ac thru 0x01657d (0x8ed2 bytes) 000770 1788839369.517402 mapread 0x0328d4 thru 0x03d53f (0xac6c bytes) 000771 1788839369.542460 trunc from 0x040000 to 0x00e86c (0x31793 bytes) 000772 1788839369.620843 trunc from 0x00e86d to 0x02aaeb (0x1c27f bytes) 000773 1788839369.693051 trunc from 0x009272 to 0x00df13 (0x4ca2 bytes) 000774 1788839369.706097 mapread 0x01334f thru 0x01e08a (0xad3c bytes) 000775 1788839369.735156 read 0x01c1fe thru 0x0293ef (0xd1f2 bytes) 000776 1788839369.750205 fallocate 0x00687d thru 0x013721 (0xcea5 bytes) 000777 1788839369.762132 fallocate 0x02067d thru 0x024861 (0x41e5 bytes) 000778 1788839369.782736 mapread 0x027d65 thru 0x02aaeb (0x2d87 bytes) 000779 1788839369.793081 trunc from 0x02aaec to 0x005b03 (0x24fe8 bytes) 000781 1788839369.842235 trunc from 0x003e8d to 0x005b03 (0x1c77 bytes) 000782 1788839369.851173 mapwrite 0x036e48 thru 0x03c04f (0x5208 bytes) 000783 1788839369.923941 mapread 0x013a35 thru 0x01e646 (0xac12 bytes) 000784 1788839369.993486 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c050): Mirror 10: 0xbd3f1fdb (stale) Mirror 14: 0xbd3f1fdb (stale) Mirror 18: 0xbd3f1fdb (stale) Mirror 19: 0xbd3f1fdb (stale) Mirror 20: 0xbd3f1fdb (stale) Mirror 21: 0xbd3f1fdb (stale) Mirror 22: 0x3102afd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c050): Mirror 10: 0xbd3f1fdb (stale) Mirror 14: 0xbd3f1fdb (stale) Mirror 18: 0xbd3f1fdb (stale) Mirror 19: 0xbd3f1fdb (stale) Mirror 20: 0xbd3f1fdb (stale) Mirror 21: 0xbd3f1fdb (stale) Mirror 22: 0x3102afd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c050: skipped 000785 1788839370.544544 write 0x0393f3 thru 0x03ffff (0x6c0d bytes) 000786 1788839370.565792 MIRROR_RESYNC 8 mirrors 000787 1788839372.917339 fallocate 0x00bdd5 thru 0x00d581 (0x17ad bytes) 000788 1788839372.989663 write 0x00471b thru 0x004b65 (0x44b bytes) 000789 1788839373.028401 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x26c57c8d (stale) Mirror 14: 0x26c57c8d (stale) Mirror 19: 0x26c57c8d (stale) Mirror 20: 0x26c57c8d (stale) Mirror 21: 0x26c57c8d (stale) Mirror 22: 0x26c57c8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x26c57c8d (stale) Mirror 14: 0x26c57c8d (stale) Mirror 19: 0x26c57c8d (stale) Mirror 20: 0x26c57c8d (stale) Mirror 21: 0x26c57c8d (stale) Mirror 22: 0x26c57c8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000790 1788839373.744699 write 0x007798 thru 0x011ca1 (0xa50a bytes) 000791 1788839373.772661 trunc from 0x040000 to 0x002a02 (0x3d5fd bytes) 000792 1788839373.808152 trunc from 0x001139 to 0x002a02 (0x18ca bytes) 000793 1788839373.814281 punch from 0x000ec1 to 0x002a02 (0x1b42 bytes) 000794 1788839373.821426 punch from 0x000cc9 to 0x002a02 (0x1d3a bytes) 000795 1788839373.836152 mapwrite 0x020558 thru 0x02d626 (0xd0cf bytes) 000796 1788839374.007261 trunc from 0x0215c1 to 0x027dc0 (0x6800 bytes) 000797 1788839374.018523 mapread 0x00240b thru 0x01213b (0xfd31 bytes) 000798 1788839374.059706 write 0x01032d thru 0x01aa08 (0xa6dc bytes) 000799 1788839374.069635 fallocate 0x033e9e thru 0x039d40 (0x5ea3 bytes) 000800 1788839374.078235 punch from 0x02332d to 0x028ba2 (0x5876 bytes) 000801 1788839374.090373 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39d41): Mirror 14: 0x1c2d3d2a (stale) Mirror 19: 0x1c2d3d2a (stale) Mirror 20: 0x1c2d3d2a (stale) Mirror 21: 0x1c2d3d2a (stale) Mirror 22: 0x1c2d3d2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d41): Mirror 14: 0x1c2d3d2a (stale) Mirror 19: 0x1c2d3d2a (stale) Mirror 20: 0x1c2d3d2a (stale) Mirror 21: 0x1c2d3d2a (stale) Mirror 22: 0x1c2d3d2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d41: skipped 000802 1788839374.554601 write 0x03e82a thru 0x03ffff (0x17d6 bytes) 000803 1788839374.559896 fallocate 0x0034d1 thru 0x00ff37 (0xca67 bytes) 000804 1788839374.570692 trunc from 0x00d38e to 0x01352f (0x61a2 bytes) 000805 1788839374.580494 mapwrite 0x016899 thru 0x024945 (0xe0ad bytes) 000806 1788839374.680237 fallocate 0x038c55 thru 0x03ffff (0x73ab bytes) 000807 1788839374.688538 trunc from 0x01f09b to 0x020546 (0x14ac bytes) 000808 1788839374.699327 trunc from 0x0336af to 0x03a6d2 (0x7024 bytes) 000809 1788839374.721434 punch from 0x00480f to 0x008cbc (0x44ae bytes) 000810 1788839374.729152 trunc from 0x02c488 to 0x0305d6 (0x414f bytes) 000811 1788839374.736287 punch from 0x020e4a to 0x02b90f (0xaac6 bytes) 000812 1788839374.748129 fallocate 0x003727 thru 0x00854d (0x4e27 bytes) 000813 1788839374.775493 MIRROR_RESYNC 6 mirrors 000814 1788839377.115702 punch from 0x012c55 to 0x01a1a8 (0x7554 bytes) 000815 1788839377.155906 mapwrite 0x021408 thru 0x030b76 (0xf76f bytes) 000816 1788839377.201088 punch from 0x012133 to 0x0136eb (0x15b9 bytes) 000817 1788839377.204686 read 0x0281f4 thru 0x02ab3d (0x294a bytes) 000818 1788839377.205276 fallocate 0x0133af thru 0x021cc3 (0xe915 bytes) 000819 1788839377.209790 fallocate 0x03ba83 thru 0x03fa91 (0x400f bytes) 000820 1788839377.218439 MIRROR_RESYNC 6 mirrors 000821 1788839378.761619 mapread 0x02a0cb thru 0x02c47d (0x23b3 bytes) 000822 1788839378.774679 write 0x01f223 thru 0x02c98b (0xd769 bytes) 000823 1788839378.812866 mapread 0x035d5a thru 0x03ffff (0xa2a6 bytes) 000824 1788839378.829921 punch from 0x032c73 to 0x03ffff (0xd38d bytes) 000825 1788839378.844714 read 0x03d545 thru 0x03ffff (0x2abb bytes) 000826 1788839378.854872 trunc from 0x040000 to 0x03b397 (0x4c68 bytes) 000827 1788839378.882718 read 0x030038 thru 0x039cb8 (0x9c81 bytes) 000828 1788839378.913028 write 0x027474 thru 0x02f26b (0x7df8 bytes) 000829 1788839378.918704 mapread 0x01cc64 thru 0x027656 (0xa9f3 bytes) 000830 1788839378.927462 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b398): Mirror 14: 0xa66ba571 (stale) Mirror 19: 0xa66ba571 (stale) Mirror 20: 0xa66ba571 (stale) Mirror 21: 0xa66ba571 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b398): Mirror 14: 0xa66ba571 (stale) Mirror 19: 0xa66ba571 (stale) Mirror 20: 0xa66ba571 (stale) Mirror 21: 0xa66ba571 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b398: skipped 000831 1788839379.809660 trunc from 0x03b398 to 0x018afd (0x2289a bytes) 000832 1788839379.881848 trunc from 0x00ec41 to 0x018afd (0x9ebd bytes) 000833 1788839379.892081 read 0x0019ad thru 0x008852 (0x6ea6 bytes) 000834 1788839379.908909 mapwrite 0x02bf75 thru 0x0382fd (0xc389 bytes) 000835 1788839380.061146 read 0x01d100 thru 0x0225af (0x54b0 bytes) 000837 1788839380.091148 punch from 0x01002a to 0x01a98c (0xa963 bytes) 000838 1788839380.109562 fallocate 0x025f53 thru 0x02b295 (0x5343 bytes) 000839 1788839380.116841 read 0x0176ec thru 0x025bc2 (0xe4d7 bytes) 000840 1788839380.136483 trunc from 0x0382fe to 0x007b99 (0x30764 bytes) 000841 1788839380.172402 fallocate 0x03a1b2 thru 0x03ffff (0x5e4e bytes) 000842 1788839380.178447 write 0x03ef19 thru 0x03ffff (0x10e7 bytes) 000843 1788839380.179434 write 0x02546d thru 0x03470e (0xf2a2 bytes) 000844 1788839380.192706 write 0x00446a thru 0x005280 (0xe17 bytes) 000845 1788839380.194545 mapread 0x02a1fe thru 0x033445 (0x9248 bytes) 000846 1788839380.195855 punch from 0x025722 to 0x025f98 (0x877 bytes) 000847 1788839380.218198 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x9786cfc3 (stale) Mirror 20: 0x9786cfc3 (stale) Mirror 21: 0x9786cfc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x9786cfc3 (stale) Mirror 20: 0x9786cfc3 (stale) Mirror 21: 0x9786cfc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000848 1788839380.784449 read 0x02ccd9 thru 0x0325b1 (0x58d9 bytes) 000849 1788839380.795484 trunc from 0x040000 to 0x031ffb (0xe004 bytes) 000850 1788839380.815043 mapread 0x02c46a thru 0x031ffb (0x5b92 bytes) 000852 1788839380.816813 trunc from 0x031ffc to 0x007ef3 (0x2a108 bytes) 000853 1788839380.834629 fallocate 0x031f12 thru 0x03ffff (0xe0ee bytes) 000854 1788839380.840338 fallocate 0x0338ed thru 0x03ea68 (0xb17c bytes) 000855 1788839380.846329 mapwrite 0x01586f thru 0x02422f (0xe9c1 bytes) 000856 1788839380.895093 trunc from 0x018c00 to 0x024b9d (0xbf9e bytes) 000858 1788839380.903621 trunc from 0x008522 to 0x00b56d (0x304c bytes) 000859 1788839380.909392 read 0x001d9f thru 0x006937 (0x4b99 bytes) 000860 1788839380.931975 write 0x02f6da thru 0x03db4b (0xe472 bytes) 000861 1788839380.937786 MIRROR_RESYNC 4 mirrors 000862 1788839382.205499 mapread 0x01dc98 thru 0x02181c (0x3b85 bytes) 000863 1788839382.230616 mapwrite 0x01de79 thru 0x025e07 (0x7f8f bytes) 000864 1788839382.330924 fallocate 0x018777 thru 0x025d64 (0xd5ee bytes) 000865 1788839382.343090 MIRROR_RESYNC 4 mirrors 000866 1788839384.140340 fallocate 0x0280d5 thru 0x03597c (0xd8a8 bytes) 000867 1788839384.172482 punch from 0x006e86 to 0x00f5b8 (0x8733 bytes) 000868 1788839384.179611 mapread 0x002ef0 thru 0x00fc5c (0xcd6d bytes) 000869 1788839384.204342 mapwrite 0x01d937 thru 0x02ab35 (0xd1ff bytes) 000870 1788839384.264466 fallocate 0x02666f thru 0x0349e4 (0xe376 bytes) 000871 1788839384.270446 mapwrite 0x01218b thru 0x01449d (0x2313 bytes) 000872 1788839384.300443 write 0x00f657 thru 0x010ab6 (0x1460 bytes) 000873 1788839384.302336 fallocate 0x032305 thru 0x03e969 (0xc665 bytes) 000874 1788839384.308151 punch from 0x031ea7 to 0x03868a (0x67e4 bytes) 000875 1788839384.315138 mapread 0x03ec30 thru 0x03ffff (0x13d0 bytes) 000876 1788839384.325448 mapread 0x019775 thru 0x01e73f (0x4fcb bytes) 000878 1788839384.328028 trunc from 0x006930 to 0x0166c8 (0xfd99 bytes) 000879 1788839384.340243 write 0x02a1ef thru 0x0324e2 (0x82f4 bytes) 000882 1788839384.352116 fallocate 0x03d33d thru 0x03de94 (0xb58 bytes) 000883 1788839384.360375 fallocate 0x0131de thru 0x01ef31 (0xbd54 bytes) 000884 1788839384.370075 fallocate 0x020b54 thru 0x02cd94 (0xc241 bytes) 000885 1788839384.376068 mapread 0x00eba3 thru 0x00f010 (0x46e bytes) 000886 1788839384.390437 trunc from 0x038749 to 0x03ffff (0x78b7 bytes) 000887 1788839384.398462 trunc from 0x0362ca to 0x03e303 (0x803a bytes) 000888 1788839384.412478 trunc from 0x040000 to 0x00acd9 (0x35326 bytes) 000889 1788839384.452105 fallocate 0x017362 thru 0x01a270 (0x2f0f bytes) 000890 1788839384.460252 mapread 0x019fbc thru 0x01a270 (0x2b5 bytes) 000891 1788839384.472565 mapread 0x0172cc thru 0x01a270 (0x2fa5 bytes) 000893 1788839384.482621 MIRROR_RESYNC 4 mirrors 000894 1788839385.490847 MIRROR_RESYNC 4 mirrors 000895 1788839386.114796 punch from 0x0072ee to 0x00e92f (0x7642 bytes) 000897 1788839386.196509 punch from 0x015e16 to 0x01a270 (0x445b bytes) 000898 1788839386.202192 write 0x001d2f thru 0x00ded7 (0xc1a9 bytes) 000899 1788839386.222225 read 0x01320b thru 0x01a270 (0x7066 bytes) 000900 1788839386.241857 read 0x014e12 thru 0x01a270 (0x545f bytes) 000901 1788839386.259449 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a271): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a271): Mirror 19: 0xb36df442 (stale) Mirror 20: 0xb36df442 (stale) Mirror 21: 0xb36df442 (stale) CRC-32 checksum value for chunk [0, 0x1a271): Mirror 1: 0xb008231c Mirror 22: 0xb008231c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a271: skipped 000902 1788839387.193630 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a271): Mirror 19: 0xb36df442 (stale) Mirror 20: 0xb36df442 (stale) Mirror 21: 0xb36df442 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a271): Mirror 19: 0xb36df442 (stale) Mirror 20: 0xb36df442 (stale) Mirror 21: 0xb36df442 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a271: skipped 000903 1788839387.527704 mapwrite 0x008b58 thru 0x00f80a (0x6cb3 bytes) 000904 1788839387.586811 MIRROR_RESYNC 4 mirrors 000905 1788839388.829855 trunc from 0x01a271 to 0x010a11 (0x985f bytes) 000906 1788839388.900573 mapread 0x00b773 thru 0x00ef4b (0x37d9 bytes) 000909 1788839388.930717 fallocate 0x00d3cd thru 0x0183f9 (0xb02d bytes) 000910 1788839388.943438 read 0x00ccdc thru 0x010a11 (0x3d36 bytes) 000911 1788839388.944878 mapwrite 0x0215ed thru 0x02f01b (0xda2f bytes) 000912 1788839389.140012 mapwrite 0x03c4f9 thru 0x03ffff (0x3b07 bytes) 000913 1788839389.252561 trunc from 0x00d7f1 to 0x01518d (0x799d bytes) 000914 1788839389.260450 read 0x016a90 thru 0x01d48f (0x6a00 bytes) 000915 1788839389.288307 read 0x03c651 thru 0x03ffff (0x39af bytes) 000916 1788839389.289319 trunc from 0x02e4d8 to 0x034a5f (0x6588 bytes) 000917 1788839389.312204 write 0x008535 thru 0x00b4e4 (0x2fb0 bytes) 000919 1788839389.313668 write 0x0259f0 thru 0x02882e (0x2e3f bytes) 000920 1788839389.320178 write 0x02a89d thru 0x0325b5 (0x7d19 bytes) 000921 1788839389.327252 punch from 0x03df4d to 0x03ffff (0x20b3 bytes) 000922 1788839389.336715 mapread 0x00207b thru 0x005691 (0x3617 bytes) 000923 1788839389.337561 write 0x033f33 thru 0x0375d7 (0x36a5 bytes) 000924 1788839389.377535 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x5b28b974 (stale) Mirror 20: 0x5b28b974 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x5b28b974 (stale) Mirror 20: 0x5b28b974 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000925 1788839389.975198 fallocate 0x006377 thru 0x015e02 (0xfa8c bytes) 000926 1788839389.982877 mapwrite 0x0054b7 thru 0x013c34 (0xe77e bytes) 000927 1788839390.090340 mapread 0x0372d2 thru 0x03ffff (0x8d2e bytes) 000928 1788839390.133601 trunc from 0x040000 to 0x006ee9 (0x39116 bytes) 000929 1788839390.162379 mapread 0x001f12 thru 0x002c70 (0xd5f bytes) 000930 1788839390.163852 trunc from 0x006eea to 0x010a2a (0x9b41 bytes) 000931 1788839390.186707 mapread 0x003de2 thru 0x00f2d2 (0xb4f1 bytes) 000932 1788839390.201189 fallocate 0x02b398 thru 0x036612 (0xb27b bytes) 000933 1788839390.205435 trunc from 0x00da8d to 0x010a2a (0x2f9e bytes) 000934 1788839390.209008 write 0x0098e5 thru 0x01108a (0x77a6 bytes) 000935 1788839390.210328 trunc from 0x01108b to 0x006f8a (0xa100 bytes) 000936 1788839390.222358 fallocate 0x025177 thru 0x034561 (0xf3eb bytes) 000938 1788839390.226804 write 0x034ed7 thru 0x03b137 (0x6261 bytes) 000939 1788839390.228641 read 0x03233d thru 0x0361f9 (0x3ebd bytes) 000941 1788839390.232846 punch from 0x004049 to 0x00c002 (0x7fba bytes) 000942 1788839390.237070 trunc from 0x03b138 to 0x005a32 (0x35705 bytes) 000944 1788839390.264999 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5a33): Mirror 19: 0xa5a6fc20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5a33): Mirror 19: 0xa5a6fc20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5a33: skipped 000945 1788839390.513977 fallocate 0x035f29 thru 0x038bb1 (0x2c89 bytes) 000946 1788839390.518996 trunc from 0x005a33 to 0x00cd8b (0x7359 bytes) 000947 1788839390.540474 mapwrite 0x026ce3 thru 0x02e847 (0x7b65 bytes) 000948 1788839390.594988 trunc from 0x02e848 to 0x005755 (0x290f2 bytes) 000949 1788839390.612713 mapwrite 0x007f15 thru 0x00fb5a (0x7c46 bytes) 000950 1788839390.663152 trunc from 0x008bd3 to 0x00fb5a (0x6f88 bytes) 000951 1788839390.681650 fallocate 0x011a92 thru 0x012639 (0xba8 bytes) 000952 1788839390.688740 mapread 0x006cc0 thru 0x00fb5a (0x8e9b bytes) 000953 1788839390.701045 fallocate 0x013d6d thru 0x01a2ff (0x6593 bytes) 000954 1788839390.706694 mapread 0x00169d thru 0x00eeb9 (0xd81d bytes) 000955 1788839390.708632 mapread 0x00b460 thru 0x00fb5a (0x46fb bytes) 000956 1788839390.710591 trunc from 0x000481 to 0x009213 (0x8d93 bytes) 000957 1788839390.716271 trunc from 0x00d35d to 0x00fb5a (0x27fe bytes) 000958 1788839390.722494 read 0x009f71 thru 0x00a3ae (0x43e bytes) 000959 1788839390.727936 mapread 0x00bd58 thru 0x00ebe7 (0x2e90 bytes) 000960 1788839390.739960 mapread 0x00f515 thru 0x00fb5a (0x646 bytes) 000961 1788839390.740840 read 0x002f52 thru 0x006f69 (0x4018 bytes) 000962 1788839390.751243 write 0x01661b thru 0x0252d6 (0xecbc bytes) 000963 1788839390.753904 mapwrite 0x023b66 thru 0x028833 (0x4cce bytes) 000965 1788839390.791807 mapwrite 0x03b043 thru 0x03ffff (0x4fbd bytes) 000966 1788839390.833312 mapread 0x022174 thru 0x02c2fa (0xa187 bytes) 000968 1788839390.834710 mapwrite 0x00b9a4 thru 0x00c464 (0xac1 bytes) 000969 1788839390.851444 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000970 1788839390.984248 read 0x03dabc thru 0x03ffff (0x2544 bytes) 000971 1788839390.991317 write 0x00c9aa thru 0x019d61 (0xd3b8 bytes) 000972 1788839391.007925 fallocate 0x018552 thru 0x01f3f6 (0x6ea5 bytes) 000973 1788839391.012659 punch from 0x001971 to 0x0081e4 (0x6874 bytes) 000975 1788839391.018559 fallocate 0x03ccab thru 0x03ffff (0x3355 bytes) 000976 1788839391.023492 trunc from 0x040000 to 0x01cf07 (0x230f8 bytes) 000977 1788839391.041550 mapwrite 0x02a023 thru 0x03562e (0xb60c bytes) 000978 1788839391.109279 fallocate 0x006e5e thru 0x0117ab (0xa94e bytes) 000979 1788839391.116952 mapwrite 0x022668 thru 0x030b93 (0xe52c bytes) 000980 1788839391.167235 punch from 0x029036 to 0x02e80d (0x57d8 bytes) 000981 1788839391.180378 read 0x026ef1 thru 0x027475 (0x585 bytes) 000982 1788839391.181245 mapwrite 0x004db3 thru 0x013455 (0xe6a3 bytes) 000983 1788839391.280688 trunc from 0x03562f to 0x032e22 (0x280c bytes) 000985 1788839391.299466 write 0x00f049 thru 0x01a416 (0xb3ce bytes) 000986 1788839391.305455 write 0x021d6d thru 0x02fe91 (0xe125 bytes) 000987 1788839391.308473 punch from 0x011fda to 0x01f727 (0xd74e bytes) 000988 1788839391.331606 fallocate 0x0143b0 thru 0x020d6e (0xc9bf bytes) 000989 1788839391.338356 mapread 0x0108d9 thru 0x01de85 (0xd5ad bytes) 000990 1788839391.406035 fallocate 0x0013f6 thru 0x005209 (0x3e14 bytes) 000991 1788839391.411592 mapread 0x0326e4 thru 0x032e22 (0x73f bytes) 000993 1788839391.413602 mapwrite 0x023b8c thru 0x0314ce (0xd943 bytes) 000994 1788839391.446280 trunc from 0x01cd26 to 0x01e783 (0x1a5e bytes) 000995 1788839391.453349 mapwrite 0x029428 thru 0x038771 (0xf34a bytes) 000996 1788839391.524667 mapread 0x008895 thru 0x009db8 (0x1524 bytes) 000997 1788839391.532433 MIRROR_EXTEND to 2 mirrors 000998 1788839392.414870 fallocate 0x018e31 thru 0x0238a4 (0xaa74 bytes) 000999 1788839392.456462 trunc from 0x0198fb to 0x0277dd (0xdee3 bytes) 001000 1788839392.466474 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38772): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38772): Mirror 2: 0xf4a2d368 (stale) CRC-32 checksum value for chunk [0, 0x38772): Mirror 1: 0x6c5df41d Mirror 3: 0x6c5df41d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38772: skipped All operations completed A-OK! PASS 70a (154s) == sanity-flr test 71: check mirror extend parent fid ==== 23:49:59 (1788839399) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d3:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 23:50:11 (1788839411) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154099 s, 13.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951507 s, 11.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 23:50:20 (1788839420) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1212704 .. Extending file size to 5000544 .. done Extending file size to 7407360 .. Extending file size to 2886272 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8648896 .. Extending file size to 8020480 .. Extending file size to 8574784 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61279 61280 61282 61283 61284 done 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) 192.168.202.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:51:48 (1788839508) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5614656 .. Extending file size to 3589856 .. done Extending file size to 5734624 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6602880 .. Extending file size to 6803936 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7292992 .. Extending file size to 8700800 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8118944 .. Extending file size to 8332416 .. Extending file size to 8503520 .. Extending file size to 8745920 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8560704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8643968 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62724 62725 62728 62729 62730 failed 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) 192.168.202.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:53:11 (1788839591) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2288 1269016 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23880 3583140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7920 3599100 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 31800 7182240 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2288 1269016 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23880 3583140 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 7920 3599100 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 31800 7182240 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0928031 s, 22.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129903 s, 16.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb36e5860 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:53:32 (1788839612) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0828137 s, 25.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:53:40 (1788839620) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x103:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x103:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:53:47 (1788839627) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0642706 s, 32.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:53:55 (1788839635) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0813798 s, 25.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:54:02 (1788839642) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198876 s, 10.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:54:36 (1788839676) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg229-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843317 s, 12.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:55:30 (1788839730) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg229-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156951 s, 13.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:56:22 (1788839782) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg229-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg229-server: Pool lustre.test_204f created Adding targets to pool oleg229-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg229-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147099 s, 14.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg229-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg229-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg229-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:57:35 (1788839855) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:57:42 (1788839862) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:57:50 (1788839870) oleg229-server: Pool lustre.test_206 created Adding targets to pool oleg229-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg229-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg229-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg229-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg229-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:58:18 (1788839898) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567658 s, 18.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228206 s, 45.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4fa3cfe6 Mirror 3: 0x4fa3cfe6 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:58:30 (1788839910) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:58:34 (1788839914) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:58:37 (1788839917) Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0601502 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1bd0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2520000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2520000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2520000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xde0000 Read this (4 bytes): BBBB 192.168.202.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg229-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:59:20 (1788839960) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.045605 s, 180 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:59:29 (1788839969) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609137 s, 17.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611229 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:59:31.000000000 -0400 Modify: 2026-09-07 23:59:31.000000000 -0400 Change: 2026-09-07 23:59:33.000000000 -0400 Birth: 2026-09-07 23:59:30.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:59:40 (1788839980) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238686 s, 43.9 MB/s oleg229-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg229-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg229-server: O/280000401/d21/135: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:00:01 (1788840001) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.237419 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.011305 s, 362 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 00:00 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 00:00 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:00:09 (1788840009) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:00:17 (1788840017) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1908 sec =========== 00:00:26 (1788840026) === sanity-flr: start cleanup 00:00:27 (1788840027) === === sanity-flr: finish cleanup 00:00:37 (1788840037) ===