-----============= acceptance-small: sanity-flr ============----- Wed Sep 9 01:00:52 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg409-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:01:15 (1788930075) === oleg409-client.virtnet: executing check_config_client /mnt/lustre oleg409-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg409-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c12502ea800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c12502ea800.idle_timeout=debug disable quota as required oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:02:03 (1788930123) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8236] == sanity-flr test 0a: lfs mirror create with -N option == 01:02:06 (1788930126) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:02:27 (1788930147) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:02:32 (1788930152) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:02:36 (1788930156) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:03:03 (1788930183) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:03:06 (1788930186) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:03:10 (1788930190) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c12502ea800.stats=clear osc.lustre-OST0001-osc-ffff9c12502ea800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:03:20 (1788930200) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:03:31 (1788930211) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:03:40 (1788930220) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:03:44 (1788930224) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:03:51 (1788930231) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 01:04:02 (1788930242) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:04:09 (1788930249) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:04:17 (1788930257) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:04:26 (1788930266) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:04:40 (1788930280) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:04:48 (1788930288) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.10102 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807083 s, 13.0 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:05:00 (1788930300) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.433713 s, 33.8 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:05:10 (1788930310) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:05:23 (1788930323) Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg409-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:06:47 (1788930407) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.557594 s, 16.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:07:33 (1788930453) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg409-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (96s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:09:09 (1788930549) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.51955 s, 14.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 9 01:09 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.038s user 0m0.000s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c12502ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.210s user 0m0.000s sys 0m0.183s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg409-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg409-server oleg409-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.145s user 0m0.001s sys 0m0.172s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg409-server: oleg409-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg409-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg409-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c12502ea800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:10:47 (1788930647) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:10:51 (1788930651) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:10:54 (1788930654) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:10:57 (1788930657) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 01:11:07 (1788930667) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.520916 s, 24.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:11:21 (1788930681) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386411 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250059 s, 41.9 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:11:29 (1788930689) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:11:39 (1788930699) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00159238 s, 643 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000930539 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00094312 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00342382 s, 299 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 01:11:51 (1788930711) create test files with size 15735625 .. 28c16c5e0c0a20b71d3f4567cf33771c - 810d1f0ba1ea86f19acd86a5ee4da996 - 97f3cd9bfd7d49d651329bf294c878dd - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg409-client.virtnet /mnt/lustre (opts:) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 01:12:19 (1788930739) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:12:35 (1788930755) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:12:44 (1788930764) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804379 s, 13.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543122 s, 19.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 01:12:55 (1788930775) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109459 s, 19.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.507829 s, 8.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227635 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 01:13:14 (1788930794) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:13:17 (1788930797) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:13:21 (1788930801) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053084 s, 19.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 01:13:30 (1788930810) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.229308 s, 13.7 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:13:41 (1788930821) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:13:52 (1788930832) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.371141 s, 28.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg409-client.virtnet /mnt/lustre (opts:) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:14:04 (1788930844) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9c12603a2800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281176 s, 37.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg409-client.virtnet /mnt/lustre (opts:) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9c1249ed8800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:14:21 (1788930861) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:15:03 (1788930903) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:15:43 (1788930943) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:15:46 (1788930946) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.301451 s, 10.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.263183 s, 12.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd16623dd (nosync,stale) Mirror 2: 0xd16623dd (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6f7dd017c77d61a268cc44b8803a4090 - mirror 1 checksum: cd792f7e382827f310ff6746a6065eb0 - mirror 2 checksum: 6f7dd017c77d61a268cc44b8803a4090 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-09T01:15:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-09T01:15:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6f7dd017c77d61a268cc44b8803a4090 - mirror 1 checksum: cd792f7e382827f310ff6746a6065eb0 - mirror 2 checksum: cd792f7e382827f310ff6746a6065eb0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-09T01:15:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-09T01:15:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:15:59 (1788930959) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314765 s, 33.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:16:08 (1788930968) Failing mds1 on oleg409-server Stopping /mnt/lustre-mds1 (opts:) on oleg409-server 01:16:22 (1788930982) shut down facet: mds1 facet_host: oleg409-server facet_failover_host: oleg409-server Failover mds1 to oleg409-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 01:16:59 (1788931019) targets are mounted 01:16:59 (1788931019) facet_failover done oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921855 s, 11.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 50A (72s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:17:21 (1788931041) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0755999 s, 271 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0389301 s, 526 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (15s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:17:36 (1788931056) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0688446 s, 297 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00631087 s, 3.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:17:49 (1788931069) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0213587 s, 767 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:17:58 (1788931078) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0606534 s, 338 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:18:08 (1788931088) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00389936 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0410002 s, 99.9 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:18:18 (1788931098) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:18:45 (1788931125) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:19:17 (1788931157) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:19:43 (1788931183) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:20:10 (1788931210) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 01:20:19 (1788931219) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57613 mirror split pid 57614 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57613 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57614 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 01:21:29 (1788931289) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 1292 truncating to largest ever: 0x2b160 000001 1788931292.706459 trunc from 00000000 to 0x02b15f (0x2b160 bytes) 000003 1788931292.723459 write 0x02edf0 thru 0x03c983 (0xdb94 bytes) 000006 1788931292.728827 read 0x009fa7 thru 0x013433 (0x948d bytes) 000007 1788931292.738987 read 0x0169e7 thru 0x01be3e (0x5458 bytes) 000008 1788931292.752899 write 0x025f6d thru 0x02f7ad (0x9841 bytes) 000009 1788931292.755771 trunc from 0x019921 to 0x020a13 (0x70f3 bytes) 000010 1788931292.761070 fallocate 0x02029d thru 0x027fdd (0x7d41 bytes) 000011 1788931292.766804 mapwrite 0x00b3aa thru 0x01833a (0xcf91 bytes) zero_range to largest ever: 0x2c7f7 000012 1788931292.816382 trunc from 0x021a8d to 0x02c7f6 (0xad6a bytes) 000014 1788931292.824656 trunc from 0x018ccf to 0x01a8e2 (0x1c14 bytes) 000016 1788931292.832777 read 0x01a950 thru 0x02764d (0xccfe bytes) 000018 1788931292.846737 punch from 0x002fe6 to 0x011cb7 (0xecd2 bytes) 000019 1788931292.852772 mapwrite 0x03e805 thru 0x03f49c (0xc98 bytes) 000020 1788931292.907064 mapwrite 0x025cd0 thru 0x02fccf (0xa000 bytes) 000021 1788931292.931864 read 0x0204c7 thru 0x02c6bf (0xc1f9 bytes) 000022 1788931292.933055 mapread 0x015277 thru 0x015527 (0x2b1 bytes) 000023 1788931292.935520 mapwrite 0x039e1a thru 0x03ffff (0x61e6 bytes) 000024 1788931292.991633 punch from 0x007e44 to 0x016caa (0xee67 bytes) 000025 1788931292.997862 mapread 0x0157d9 thru 0x0164f8 (0xd20 bytes) 000026 1788931293.014190 punch from 0x015f0b to 0x01d894 (0x798a bytes) 000027 1788931293.039012 mapread 0x014673 thru 0x01de21 (0x97af bytes) fallocating to largest ever: 0x40000 000029 1788931293.117555 fallocate 0x0353c8 thru 0x03ffff (0xac38 bytes) 000030 1788931293.124850 write 0x01d25d thru 0x027931 (0xa6d5 bytes) 000031 1788931293.127732 punch from 0x0374b7 to 0x038fa3 (0x1aed bytes) 000032 1788931293.133570 fallocate 0x038d3a thru 0x03ffff (0x72c6 bytes) 000033 1788931293.139778 fallocate 0x0108c9 thru 0x01bcf4 (0xb42c bytes) 000034 1788931293.145698 fallocate 0x03b40d thru 0x03ffff (0x4bf3 bytes) 000035 1788931293.151787 fallocate 0x01b30f thru 0x023b82 (0x8874 bytes) 000036 1788931293.158341 punch from 0x028d6d to 0x034032 (0xb2c6 bytes) 000037 1788931293.164688 trunc from 0x040000 to 0x0113ad (0x2ec52 bytes) 000038 1788931293.185441 MIRROR_EXTEND to 2 mirrors 000039 1788931293.942169 fallocate 0x0030e7 thru 0x012c1d (0xfb37 bytes) 000041 1788931293.981002 punch from 0x008a16 to 0x008afe (0xe9 bytes) 000042 1788931293.987115 punch from 0x005488 to 0x00bd79 (0x68f2 bytes) 000043 1788931293.994067 mapread 0x0033af thru 0x00a637 (0x7289 bytes) 000044 1788931294.023768 trunc from 0x00da8e to 0x0113ad (0x3920 bytes) 000045 1788931294.029082 write 0x03b9ee thru 0x03de2a (0x243d bytes) 000046 1788931294.035774 trunc from 0x013158 to 0x0137a7 (0x650 bytes) 000048 1788931294.041896 trunc from 0x037ca1 to 0x03de2a (0x618a bytes) 000049 1788931294.055798 mapwrite 0x02f4e0 thru 0x036e85 (0x79a6 bytes) 000050 1788931294.086295 trunc from 0x0115a7 to 0x01e639 (0xd093 bytes) 000051 1788931294.094359 write 0x00dde1 thru 0x01225f (0x447f bytes) 000053 1788931294.110062 read 0x03581a thru 0x036960 (0x1147 bytes) 000054 1788931294.111547 trunc from 0x00f337 to 0x011fc9 (0x2c93 bytes) 000055 1788931294.127666 fallocate 0x0280f7 thru 0x02f7ea (0x76f4 bytes) 000056 1788931294.134667 trunc from 0x02920d to 0x02bfe9 (0x2ddd bytes) 000057 1788931294.142397 trunc from 0x03de2b to 0x000789 (0x3d6a1 bytes) 000058 1788931294.177002 trunc from 0x00077b to 0x000789 (0xf bytes) 000059 1788931294.184753 punch from 0x000353 to 0x000789 (0x437 bytes) 000060 1788931294.189476 trunc from 0x0005ec to 0x000789 (0x19e bytes) 000061 1788931294.195890 read 0x000047 thru 0x000789 (0x743 bytes) 000062 1788931294.202764 punch from 0x00042f to 0x000789 (0x35b bytes) 000063 1788931294.209758 mapread 0x00075c thru 0x000789 (0x2e bytes) 000064 1788931294.217752 fallocate 0x01777f thru 0x023b74 (0xc3f6 bytes) 000065 1788931294.224054 mapwrite 0x0298e5 thru 0x02fcea (0x6406 bytes) 000066 1788931294.282865 mapwrite 0x006086 thru 0x014be8 (0xeb63 bytes) 000067 1788931294.310138 mapread 0x02ec91 thru 0x02fcea (0x105a bytes) 000068 1788931294.311524 write 0x003ce8 thru 0x009cd5 (0x5fee bytes) 000070 1788931294.314246 mapread 0x0067d7 thru 0x00c554 (0x5d7e bytes) 000071 1788931294.315721 read 0x00b531 thru 0x00e629 (0x30f9 bytes) 000072 1788931294.316908 punch from 0x016b3c to 0x021d07 (0xb1cc bytes) 000074 1788931294.328247 trunc from 0x02fceb to 0x00cedd (0x22e0d bytes) 000075 1788931294.358839 mapwrite 0x001c44 thru 0x00c944 (0xad01 bytes) 000076 1788931294.383109 mapwrite 0x00dd71 thru 0x015f21 (0x81b1 bytes) 000077 1788931294.497327 mapwrite 0x027b8d thru 0x02ec38 (0x70ac bytes) 000078 1788931294.609728 read 0x000c20 thru 0x00a0c6 (0x94a7 bytes) 000080 1788931294.617336 write 0x01ec5b thru 0x01ff97 (0x133d bytes) 000081 1788931294.622207 write 0x03cfa7 thru 0x03ffff (0x3059 bytes) 000082 1788931294.628947 fallocate 0x00eb78 thru 0x01699d (0x7e26 bytes) 000084 1788931294.651224 punch from 0x02b19a to 0x02cc96 (0x1afd bytes) 000085 1788931294.661426 punch from 0x007aa3 to 0x016819 (0xed77 bytes) 000086 1788931294.698060 MIRROR_RESYNC 2 mirrors 000088 1788931295.970010 trunc from 0x022d6f to 0x02d959 (0xabeb bytes) 000089 1788931296.056898 read 0x035ad3 thru 0x037219 (0x1747 bytes) 000090 1788931296.078489 write 0x01bec1 thru 0x022542 (0x6682 bytes) 000091 1788931296.126025 read 0x01ebad thru 0x0291e2 (0xa636 bytes) 000092 1788931296.152637 write 0x0076f5 thru 0x01358a (0xbe96 bytes) 000093 1788931296.154152 trunc from 0x017ace to 0x0263f5 (0xe928 bytes) 000094 1788931296.197230 trunc from 0x007b50 to 0x00acac (0x315d bytes) 000095 1788931296.215824 trunc from 0x02d92e to 0x035901 (0x7fd4 bytes) 000096 1788931296.247640 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc6eede62 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x964de409 Mirror 3: 0x964de409 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000097 1788931298.169362 mapread 0x00169f thru 0x004fc3 (0x3925 bytes) 000098 1788931298.190928 trunc from 0x040000 to 0x018dec (0x27213 bytes) 000099 1788931298.237069 trunc from 0x00e9a4 to 0x00eec4 (0x521 bytes) 000100 1788931298.249564 mapread 0x002410 thru 0x011760 (0xf351 bytes) 000102 1788931298.276742 mapwrite 0x038583 thru 0x038d89 (0x807 bytes) 000103 1788931298.324622 fallocate 0x02b97b thru 0x036f9b (0xb621 bytes) 000104 1788931298.330197 trunc from 0x029e89 to 0x02d1e6 (0x335e bytes) 000106 1788931298.342708 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38d8a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d8a): Mirror 2: 0xf88d7092 (stale) Mirror 3: 0xb2c52f5c (stale) CRC-32 checksum value for chunk [0, 0x38d8a): Mirror 1: 0x4f9c79f9 Mirror 4: 0x4f9c79f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d8a: skipped 000107 1788931299.533223 MIRROR_RESYNC 4 mirrors 000109 1788931300.755311 read 0x031a53 thru 0x03346c (0x1a1a bytes) 000111 1788931300.776578 mapread 0x0023d6 thru 0x011b62 (0xf78d bytes) 000112 1788931300.857335 mapread 0x0135de thru 0x01d25c (0x9c7f bytes) 000113 1788931300.871791 trunc from 0x02432a to 0x0268e2 (0x25b9 bytes) 000114 1788931300.922394 read 0x01eff2 thru 0x023ed5 (0x4ee4 bytes) 000115 1788931300.936711 write 0x0158bb thru 0x018e41 (0x3587 bytes) 000116 1788931300.937977 mapwrite 0x00f378 thru 0x01ca3f (0xd6c8 bytes) 000117 1788931300.978800 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d8a): Mirror 3: 0x4f9c79f9 (stale) Mirror 4: 0x4f9c79f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d8a): Mirror 3: 0x4f9c79f9 (stale) Mirror 4: 0x4f9c79f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d8a: skipped 000118 1788931301.398216 fallocate 0x003206 thru 0x0038a5 (0x6a0 bytes) 000119 1788931301.405050 trunc from 0x00da4d to 0x011c27 (0x41db bytes) 000120 1788931301.410127 write 0x017399 thru 0x01cdd7 (0x5a3f bytes) 000121 1788931301.427817 write 0x011e49 thru 0x014000 (0x21b8 bytes) 000123 1788931301.439568 mapwrite 0x000dcb thru 0x00f822 (0xea58 bytes) 000124 1788931301.504888 trunc from 0x01f22b to 0x0256f9 (0x64cf bytes) 000125 1788931301.510481 mapread 0x01fe7f thru 0x02bf86 (0xc108 bytes) 000126 1788931301.540944 trunc from 0x038d8a to 0x02cfe7 (0xbda2 bytes) 000127 1788931301.589871 trunc from 0x02cfe8 to 0x0384f0 (0xb509 bytes) 000128 1788931301.613877 punch from 0x01e54e to 0x021cb8 (0x376b bytes) 000130 1788931301.634441 trunc from 0x00c463 to 0x016c31 (0xa7cf bytes) 000131 1788931301.643907 mapread 0x0276ac thru 0x035832 (0xe187 bytes) 000132 1788931301.656784 trunc from 0x0384f1 to 0x03ab38 (0x2648 bytes) 000134 1788931301.684513 trunc from 0x03ab39 to 0x037562 (0x35d6 bytes) 000135 1788931301.706619 fallocate 0x0194a0 thru 0x01af9d (0x1afe bytes) 000136 1788931301.717091 trunc from 0x037563 to 0x00891d (0x2ec45 bytes) 000137 1788931301.740307 fallocate 0x02dad6 thru 0x030d28 (0x3253 bytes) 000138 1788931301.747109 write 0x03a07e thru 0x03aa16 (0x999 bytes) 000139 1788931301.749361 punch from 0x0397f2 to 0x03aa16 (0x1225 bytes) 000140 1788931301.761011 write 0x0230de thru 0x028fa6 (0x5ec9 bytes) 000141 1788931301.772146 fallocate 0x018f18 thru 0x021661 (0x874a bytes) 000142 1788931301.777638 trunc from 0x02708d to 0x0327a6 (0xb71a bytes) 000143 1788931301.797005 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3aa17): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa17): Mirror 3: 0x1674cf04 (stale) Mirror 4: 0x1674cf04 (stale) CRC-32 checksum value for chunk [0, 0x3aa17): Mirror 1: 0x39e906ae Mirror 5: 0x39e906ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa17: skipped 000144 1788931303.502984 write 0x020d1f thru 0x02f913 (0xebf5 bytes) 000145 1788931303.582746 trunc from 0x00cce4 to 0x01aac3 (0xdde0 bytes) 000147 1788931303.593054 read 0x037888 thru 0x03aa16 (0x318f bytes) 000148 1788931303.602264 punch from 0x039b59 to 0x03aa16 (0xebe bytes) 000149 1788931303.613936 trunc from 0x03aa17 to 0x03dcd3 (0x32bd bytes) 000150 1788931303.662943 mapwrite 0x012333 thru 0x01ba4b (0x9719 bytes) 000151 1788931303.768954 trunc from 0x0161d5 to 0x01678c (0x5b8 bytes) 000152 1788931303.783946 mapread 0x03aee2 thru 0x03dcd3 (0x2df2 bytes) 000153 1788931303.828682 mapwrite 0x033d49 thru 0x03640c (0x26c4 bytes) 000154 1788931303.884953 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 3: 0xaa8acf8d (stale) Mirror 4: 0xaa8acf8d (stale) Mirror 5: 0x66f94ba8 (stale) CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 1: 0xcbbdacaf Mirror 6: 0xcbbdacaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dcd4: skipped 000155 1788931305.524313 mapwrite 0x008cb8 thru 0x00b512 (0x285b bytes) 000156 1788931305.765284 fallocate 0x004281 thru 0x01363d (0xf3bd bytes) 000157 1788931305.777764 punch from 0x00517e to 0x00f77d (0xa600 bytes) 000159 1788931305.791481 mapread 0x00c8b3 thru 0x014903 (0x8051 bytes) 000160 1788931305.828920 write 0x021e22 thru 0x02efa6 (0xd185 bytes) 000161 1788931305.885851 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 3: 0xaa8acf8d (stale) Mirror 4: 0xaa8acf8d (stale) Mirror 6: 0xcbbdacaf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 3: 0xaa8acf8d (stale) Mirror 4: 0xaa8acf8d (stale) Mirror 6: 0xcbbdacaf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dcd4: skipped 000162 1788931306.406895 mapwrite 0x0048b0 thru 0x006013 (0x1764 bytes) 000163 1788931306.491791 mapread 0x025624 thru 0x034799 (0xf176 bytes) 000165 1788931306.539696 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 4: 0xaa8acf8d (stale) Mirror 6: 0xcbbdacaf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 4: 0xaa8acf8d (stale) Mirror 6: 0xcbbdacaf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dcd4: skipped 000166 1788931307.183749 MIRROR_RESYNC 3 mirrors 000167 1788931309.082881 mapread 0x00fe71 thru 0x01fcfd (0xfe8d bytes) 000169 1788931309.132398 write 0x01edd7 thru 0x024234 (0x545e bytes) 000170 1788931309.236450 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 4: 0x1b37248b (stale) Mirror 6: 0x1b37248b (stale) CRC-32 checksum value for chunk [0, 0x3dcd4): Mirror 1: 0x8aaf806c Mirror 7: 0x8aaf806c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dcd4: skipped 000171 1788931311.049849 mapread 0x01074d thru 0x01fb61 (0xf415 bytes) 000172 1788931311.183040 trunc from 0x014850 to 0x021cc0 (0xd471 bytes) 000173 1788931311.277228 mapwrite 0x01bb7c thru 0x028efe (0xd383 bytes) 000174 1788931311.386720 trunc from 0x03dcd4 to 0x00655a (0x37779 bytes) 000175 1788931311.426311 trunc from 0x003ff1 to 0x00655a (0x256a bytes) 000176 1788931311.434035 trunc from 0x00655b to 0x01ce26 (0x168cc bytes) 000177 1788931311.473578 mapread 0x00f6c8 thru 0x0167d1 (0x710a bytes) 000178 1788931311.508167 read 0x00c7f0 thru 0x011254 (0x4a65 bytes) 000179 1788931311.508994 write 0x01e2c3 thru 0x027003 (0x8d41 bytes) 000180 1788931311.523047 mapread 0x0231ef thru 0x027003 (0x3e15 bytes) 000182 1788931311.537737 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27004): Mirror 4: 0x9b87737a (stale) Mirror 6: 0x9b87737a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27004): Mirror 4: 0x9b87737a (stale) Mirror 6: 0x9b87737a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27004: skipped 000183 1788931312.137285 trunc from 0x013ecb to 0x014198 (0x2ce bytes) 000184 1788931312.147167 read 0x024c8b thru 0x027003 (0x2379 bytes) 000185 1788931312.156363 read 0x025da2 thru 0x027003 (0x1262 bytes) 000186 1788931312.157396 read 0x0252a2 thru 0x027003 (0x1d62 bytes) 000187 1788931312.158958 trunc from 0x027004 to 0x033026 (0xc023 bytes) 000188 1788931312.188483 trunc from 0x033027 to 0x02143a (0x11bec bytes) 000189 1788931312.224003 MIRROR_RESYNC 3 mirrors 000191 1788931313.198997 mapwrite 0x015a43 thru 0x01ea56 (0x9014 bytes) 000192 1788931313.324669 trunc from 0x0086d3 to 0x015670 (0xcf9e bytes) 000193 1788931313.334774 read 0x004fd9 thru 0x00991c (0x4944 bytes) 000194 1788931313.354194 write 0x00e5cd thru 0x01d599 (0xefcd bytes) 000195 1788931313.363406 punch from 0x0135d0 to 0x0213f7 (0xde28 bytes) 000197 1788931313.386032 write 0x033dd6 thru 0x03c212 (0x843d bytes) 000198 1788931313.389033 trunc from 0x03c213 to 0x02f07a (0xd198 bytes) 000199 1788931313.444328 trunc from 0x02f07b to 0x01dcf0 (0x1138a bytes) 000200 1788931313.464916 mapread 0x010da2 thru 0x01dcf0 (0xcf4f bytes) 000201 1788931313.488024 fallocate 0x01b11c thru 0x01c4ea (0x13cf bytes) 000202 1788931313.493758 trunc from 0x0194e5 to 0x01dcf0 (0x480c bytes) 000203 1788931313.499817 mapwrite 0x039f4d thru 0x03ffff (0x60b3 bytes) 000204 1788931313.561424 mapread 0x02125d thru 0x026918 (0x56bc bytes) 000205 1788931313.573204 trunc from 0x026c4d to 0x026f47 (0x2fb bytes) 000207 1788931313.580435 write 0x01844e thru 0x021082 (0x8c35 bytes) 000208 1788931313.588995 fallocate 0x027c43 thru 0x02c965 (0x4d23 bytes) 000209 1788931313.596023 mapread 0x02e27e thru 0x02fbf4 (0x1977 bytes) 000210 1788931313.617011 mapread 0x037dff thru 0x03f00b (0x720d bytes) 000211 1788931313.618164 trunc from 0x040000 to 0x0208c2 (0x1f73d bytes) 000212 1788931313.647967 write 0x00315d thru 0x003c37 (0xadb bytes) 000213 1788931313.662199 read 0x014fc0 thru 0x01517a (0x1bb bytes) 000214 1788931313.662909 trunc from 0x0208c3 to 0x02e152 (0xd890 bytes) 000215 1788931313.696135 mapwrite 0x01eecd thru 0x02e5e3 (0xf717 bytes) 000217 1788931313.835931 punch from 0x016927 to 0x02005d (0x9737 bytes) 000218 1788931313.847084 fallocate 0x031e06 thru 0x03cd04 (0xaeff bytes) 000219 1788931313.856775 write 0x00195b thru 0x0102ad (0xe953 bytes) 000220 1788931313.864682 mapwrite 0x001b73 thru 0x002127 (0x5b5 bytes) 000221 1788931313.895214 trunc from 0x02e5e4 to 0x0182c1 (0x16322 bytes) 000222 1788931313.945712 write 0x0045d1 thru 0x0054fc (0xf2c bytes) 000223 1788931313.948447 write 0x01dbd0 thru 0x022385 (0x47b6 bytes) 000225 1788931313.952091 fallocate 0x00139b thru 0x005d27 (0x498d bytes) 000226 1788931313.962548 mapread 0x0090e8 thru 0x00acf6 (0x1c0f bytes) 000228 1788931313.964934 mapread 0x002151 thru 0x006d1f (0x4bcf bytes) 000229 1788931313.976236 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22386): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22386): Mirror 4: 0x24fda25b (stale) Mirror 6: 0x24fda25b (stale) CRC-32 checksum value for chunk [0, 0x22386): Mirror 1: 0xe82f83c9 Mirror 7: 0xe82f83c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22386: skipped 000230 1788931314.981130 punch from 0x00dcab to 0x01bd6d (0xe0c3 bytes) 000231 1788931315.045409 trunc from 0x016f6e to 0x01fecf (0x8f62 bytes) 000232 1788931315.069892 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22386): Mirror 6: 0x24fda25b (stale) Mirror 7: 0xe82f83c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22386): Mirror 6: 0x24fda25b (stale) Mirror 7: 0xe82f83c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22386: skipped 000233 1788931315.654822 mapread 0x01fe78 thru 0x022385 (0x250e bytes) 000235 1788931315.703290 trunc from 0x022386 to 0x0072eb (0x1b09a bytes) 000236 1788931315.748996 read 0x005007 thru 0x0072eb (0x22e5 bytes) 000237 1788931315.759467 punch from 0x004962 to 0x0072eb (0x298a bytes) 000238 1788931315.773836 fallocate 0x00d074 thru 0x00fd73 (0x2d00 bytes) 000239 1788931315.783408 trunc from 0x0072ec to 0x00373f (0x3bac bytes) 000240 1788931315.828534 punch from 0x002386 to 0x00373f (0x13ba bytes) 000241 1788931315.836576 fallocate 0x02c6f6 thru 0x03677b (0xa086 bytes) 000243 1788931315.852372 mapwrite 0x02fc5a thru 0x03e11e (0xe4c5 bytes) 000244 1788931315.958193 MIRROR_RESYNC 3 mirrors 000245 1788931317.240217 read 0x00fde8 thru 0x014861 (0x4a7a bytes) 000246 1788931317.266796 trunc from 0x03e11f to 0x01a81c (0x23902 bytes) 000247 1788931317.362558 write 0x02b7d8 thru 0x0332a0 (0x7ac9 bytes) 000248 1788931317.367833 punch from 0x00eff5 to 0x013f9e (0x4faa bytes) 000249 1788931317.378189 fallocate 0x000e71 thru 0x00af47 (0xa0d7 bytes) 000250 1788931317.385489 mapwrite 0x03e4dd thru 0x03fd3c (0x1860 bytes) 000252 1788931317.508719 punch from 0x01ac62 to 0x024bcd (0x9f6c bytes) 000253 1788931317.535248 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd3d): Mirror 7: 0xbe4692ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd3d): Mirror 7: 0xbe4692ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd3d: skipped 000254 1788931318.031161 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) 000255 1788931318.274312 punch from 0x007931 to 0x00fe28 (0x84f8 bytes) 000256 1788931318.282400 mapwrite 0x00588e thru 0x00f853 (0x9fc6 bytes) 000257 1788931318.370460 mapwrite 0x02c345 thru 0x036d46 (0xaa02 bytes) 000258 1788931318.436282 fallocate 0x01e964 thru 0x01f795 (0xe32 bytes) 000259 1788931318.444997 mapread 0x036778 thru 0x03fd3c (0x95c5 bytes) 000260 1788931318.460923 trunc from 0x03b08e to 0x03fd3c (0x4caf bytes) 000261 1788931318.467155 write 0x00371d thru 0x0088f5 (0x51d9 bytes) 000262 1788931318.476103 write 0x00a2c1 thru 0x00d5d8 (0x3318 bytes) 000263 1788931318.477484 punch from 0x010e5d to 0x01a6dc (0x9880 bytes) 000264 1788931318.485729 fallocate 0x02d5bf thru 0x039872 (0xc2b4 bytes) 000265 1788931318.499900 mapread 0x022859 thru 0x02c82d (0x9fd5 bytes) 000266 1788931318.525914 mapwrite 0x033b0a thru 0x03a754 (0x6c4b bytes) 000267 1788931318.567506 punch from 0x002a26 to 0x00b10b (0x86e6 bytes) 000268 1788931318.576578 read 0x027ce1 thru 0x02d7a4 (0x5ac4 bytes) 000269 1788931318.578367 read 0x038340 thru 0x039de6 (0x1aa7 bytes) 000270 1788931318.584081 trunc from 0x0273e6 to 0x030653 (0x926e bytes) 000271 1788931318.596470 mapread 0x00fce4 thru 0x018045 (0x8362 bytes) 000272 1788931318.626116 read 0x00987d thru 0x00b68d (0x1e11 bytes) 000273 1788931318.640863 MIRROR_EXTEND to 2 mirrors 000274 1788931319.706669 mapwrite 0x00446f thru 0x0078ef (0x3481 bytes) 000275 1788931319.919757 trunc from 0x03fd3d to 0x039b3c (0x6200 bytes) 000276 1788931319.988599 write 0x022350 thru 0x02c4fb (0xa1ac bytes) 000277 1788931320.017894 trunc from 0x026859 to 0x026cad (0x455 bytes) 000278 1788931320.049435 read 0x00a959 thru 0x00b108 (0x7b0 bytes) 000279 1788931320.056205 mapread 0x00c049 thru 0x00f83e (0x37f6 bytes) 000280 1788931320.058817 read 0x02ef90 thru 0x03117d (0x21ee bytes) 000281 1788931320.087727 trunc from 0x00203d to 0x0083c7 (0x638b bytes) 000282 1788931320.109672 MIRROR_RESYNC 2 mirrors 000284 1788931321.366574 trunc from 0x039b3d to 0x000cea (0x38e52 bytes) 000285 1788931321.447127 write 0x02fd6f thru 0x030186 (0x418 bytes) 000286 1788931321.448826 mapread 0x004e4f thru 0x00f678 (0xa82a bytes) 000287 1788931321.470248 fallocate 0x00c624 thru 0x00cdf0 (0x7cd bytes) 000288 1788931321.477712 mapread 0x0235b1 thru 0x027806 (0x4256 bytes) 000289 1788931321.494386 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) 000291 1788931321.816433 trunc from 0x030187 to 0x012349 (0x1de3d bytes) 000292 1788931321.877289 mapwrite 0x01e23f thru 0x01e75f (0x521 bytes) 000294 1788931322.004003 punch from 0x012dad to 0x018510 (0x5764 bytes) 000295 1788931322.018953 fallocate 0x016c78 thru 0x020be9 (0x9f72 bytes) 000296 1788931322.025473 mapread 0x0092e5 thru 0x0112b6 (0x7fd2 bytes) 000297 1788931322.052242 mapread 0x0017eb thru 0x00fe5c (0xe672 bytes) 000298 1788931322.063176 mapwrite 0x0093d1 thru 0x00c4d0 (0x3100 bytes) 000300 1788931322.114975 trunc from 0x00b8f5 to 0x00cf0e (0x161a bytes) 000301 1788931322.130351 fallocate 0x036169 thru 0x03ffff (0x9e97 bytes) 000303 1788931322.141632 mapwrite 0x01c769 thru 0x028956 (0xc1ee bytes) 000306 1788931322.213284 read 0x00e36e thru 0x016e04 (0x8a97 bytes) 000307 1788931322.222686 punch from 0x0229bd to 0x02d6bb (0xacff bytes) 000309 1788931322.235421 punch from 0x00f894 to 0x01e3c1 (0xeb2e bytes) 000310 1788931322.244446 fallocate 0x03cd4f thru 0x03ffff (0x32b1 bytes) 000312 1788931322.258802 mapwrite 0x019459 thru 0x02632c (0xced4 bytes) 000313 1788931322.339527 fallocate 0x01ac1f thru 0x0271b7 (0xc599 bytes) 000314 1788931322.349498 write 0x027255 thru 0x0328e2 (0xb68e bytes) 000315 1788931322.357862 mapwrite 0x01d127 thru 0x01d679 (0x553 bytes) 000316 1788931322.384575 mapread 0x03d5af thru 0x03ffff (0x2a51 bytes) 000317 1788931322.402453 fallocate 0x011eec thru 0x01bed4 (0x9fe9 bytes) 000318 1788931322.411347 fallocate 0x01d255 thru 0x01f5ca (0x2376 bytes) 000319 1788931322.417539 trunc from 0x040000 to 0x00d3e8 (0x32c17 bytes) 000320 1788931322.454970 write 0x00aa3d thru 0x00e3aa (0x396e bytes) 000321 1788931322.460135 trunc from 0x0066e7 to 0x00a7e7 (0x4101 bytes) 000322 1788931322.485827 trunc from 0x00e3ab to 0x020d80 (0x129d6 bytes) 000323 1788931322.522524 trunc from 0x020d81 to 0x035dc2 (0x15042 bytes) 000324 1788931322.561061 mapwrite 0x00f893 thru 0x0146cb (0x4e39 bytes) 000325 1788931322.615027 mapread 0x029c58 thru 0x02e3fe (0x47a7 bytes) 000327 1788931322.633473 mapread 0x03100a thru 0x035dc2 (0x4db9 bytes) 000329 1788931322.642445 mapread 0x01f53f thru 0x024b0e (0x55d0 bytes) 000331 1788931322.659495 mapread 0x01aa22 thru 0x025774 (0xad53 bytes) 000332 1788931322.661338 read 0x004c21 thru 0x0109a6 (0xbd86 bytes) 000333 1788931322.672203 write 0x02a908 thru 0x02ddaf (0x34a8 bytes) 000334 1788931322.674400 trunc from 0x035dc3 to 0x03f7d3 (0x9a11 bytes) 000335 1788931322.697113 mapread 0x000e1b thru 0x004f29 (0x410f bytes) 000336 1788931322.699562 mapread 0x00dca3 thru 0x017e77 (0xa1d5 bytes) 000337 1788931322.706444 read 0x026498 thru 0x02d5c8 (0x7131 bytes) 000339 1788931322.710084 mapwrite 0x039d73 thru 0x03ffff (0x628d bytes) 000340 1788931322.795939 mapwrite 0x03210e thru 0x0377f2 (0x56e5 bytes) 000341 1788931322.828190 mapwrite 0x028b8f thru 0x036a61 (0xded3 bytes) 000342 1788931322.885079 trunc from 0x02317f to 0x0260c3 (0x2f45 bytes) 000343 1788931322.893524 punch from 0x02e993 to 0x0304ca (0x1b38 bytes) 000344 1788931322.908203 write 0x03550f thru 0x03b86f (0x6361 bytes) 000346 1788931322.912739 mapread 0x033efb thru 0x03f761 (0xb867 bytes) 000347 1788931322.913999 punch from 0x03a640 to 0x03ffff (0x59c0 bytes) 000348 1788931322.938717 mapread 0x03f2a6 thru 0x03ffff (0xd5a bytes) 000350 1788931322.945775 trunc from 0x040000 to 0x037f0f (0x80f0 bytes) 000351 1788931322.975482 trunc from 0x037f10 to 0x01f80d (0x18702 bytes) 000352 1788931323.001554 read 0x0197a6 thru 0x01a62c (0xe87 bytes) 000353 1788931323.006335 fallocate 0x02ae16 thru 0x03a612 (0xf7fd bytes) 000354 1788931323.012463 punch from 0x001a12 to 0x011724 (0xfd13 bytes) 000357 1788931323.020357 punch from 0x006e4f to 0x00e412 (0x75c4 bytes) 000358 1788931323.028408 write 0x023171 thru 0x029d87 (0x6c17 bytes) 000359 1788931323.037414 mapread 0x026d3b thru 0x029d87 (0x304d bytes) 000360 1788931323.041278 mapread 0x0025cb thru 0x0110e0 (0xeb16 bytes) 000362 1788931323.068419 fallocate 0x01cfd1 thru 0x022d70 (0x5da0 bytes) 000363 1788931323.077557 punch from 0x016113 to 0x025e13 (0xfd01 bytes) 000364 1788931323.123237 mapwrite 0x029011 thru 0x038dcf (0xfdbf bytes) 000365 1788931323.204928 read 0x031bc6 thru 0x035283 (0x36be bytes) 000366 1788931323.206182 fallocate 0x021b41 thru 0x02dc78 (0xc138 bytes) 000367 1788931323.214555 write 0x02d5b9 thru 0x032d43 (0x578b bytes) 000369 1788931323.219420 mapread 0x032c2c thru 0x038dcf (0x61a4 bytes) 000370 1788931323.221476 mapwrite 0x031c83 thru 0x034a5a (0x2dd8 bytes) 000373 1788931323.272332 trunc from 0x038dd0 to 0x03b34b (0x257c bytes) 000374 1788931323.321544 punch from 0x007c24 to 0x00e800 (0x6bdd bytes) 000376 1788931323.341792 trunc from 0x026f60 to 0x02d0ea (0x618b bytes) 000378 1788931323.353243 write 0x00dc16 thru 0x018fee (0xb3d9 bytes) 000380 1788931323.376489 read 0x021303 thru 0x0264bb (0x51b9 bytes) 000381 1788931323.379082 fallocate 0x0122f2 thru 0x01515a (0x2e69 bytes) 000382 1788931323.386534 trunc from 0x0057c6 to 0x007c75 (0x24b0 bytes) 000383 1788931323.396921 write 0x039e84 thru 0x03e473 (0x45f0 bytes) 000384 1788931323.398369 write 0x01be5e thru 0x02768f (0xb832 bytes) 000385 1788931323.412464 mapread 0x0275b6 thru 0x02ad5a (0x37a5 bytes) 000386 1788931323.414739 mapread 0x036b81 thru 0x03aa3e (0x3ebe bytes) 000387 1788931323.415685 trunc from 0x005047 to 0x00f823 (0xa7dd bytes) 000388 1788931323.438108 punch from 0x02f38f to 0x037f80 (0x8bf2 bytes) 000389 1788931323.448418 mapwrite 0x005b25 thru 0x009def (0x42cb bytes) 000390 1788931323.508001 punch from 0x0100e2 to 0x01f5a7 (0xf4c6 bytes) 000391 1788931323.517411 fallocate 0x0348fd thru 0x038d40 (0x4444 bytes) 000392 1788931323.526001 read 0x0293d0 thru 0x02be2b (0x2a5c bytes) 000393 1788931323.527171 trunc from 0x02ff90 to 0x03a57a (0xa5eb bytes) 000394 1788931323.533983 trunc from 0x010551 to 0x0137ea (0x329a bytes) 000395 1788931323.541011 mapread 0x021c7f thru 0x02dec0 (0xc242 bytes) 000396 1788931323.542402 fallocate 0x00110a thru 0x00bc64 (0xab5b bytes) 000397 1788931323.549143 write 0x00c463 thru 0x01b012 (0xebb0 bytes) 000398 1788931323.557480 mapwrite 0x02191f thru 0x025cdd (0x43bf bytes) 000399 1788931323.588043 punch from 0x011ccc to 0x0184fa (0x682f bytes) 000400 1788931323.595057 trunc from 0x021c54 to 0x02ad37 (0x90e4 bytes) 000401 1788931323.610577 mapread 0x008aa5 thru 0x012cce (0xa22a bytes) 000402 1788931323.629571 trunc from 0x010563 to 0x01b8d1 (0xb36f bytes) 000403 1788931323.637489 mapwrite 0x024c4d thru 0x033b58 (0xef0c bytes) 000404 1788931323.679024 trunc from 0x01d2cc to 0x027fd0 (0xad05 bytes) 000405 1788931323.689058 punch from 0x0098b5 to 0x00dd63 (0x44af bytes) 000406 1788931323.695861 read 0x0290f2 thru 0x0386f9 (0xf608 bytes) 000407 1788931323.704465 MIRROR_EXTEND to 2 mirrors 000408 1788931324.900216 write 0x01eb8d thru 0x025dbe (0x7232 bytes) 000409 1788931324.956903 trunc from 0x03e474 to 0x02706a (0x17409 bytes) 000410 1788931324.995559 fallocate 0x03a7b7 thru 0x03ffff (0x5849 bytes) 000413 1788931325.018919 write 0x03bb3b thru 0x03ffff (0x44c5 bytes) 000414 1788931325.025308 punch from 0x01ce17 to 0x0214c1 (0x46ab bytes) 000415 1788931325.059328 write 0x03d01b thru 0x03ffff (0x2fe5 bytes) 000417 1788931325.085959 MIRROR_RESYNC 2 mirrors 000418 1788931326.346743 trunc from 0x00566e to 0x007c1a (0x25ad bytes) 000420 1788931326.409894 punch from 0x00bdb7 to 0x00ed02 (0x2f4c bytes) 000423 1788931326.421774 mapread 0x00abcb thru 0x011bdd (0x7013 bytes) 000424 1788931326.451473 write 0x01c075 thru 0x025d99 (0x9d25 bytes) 000425 1788931326.464157 punch from 0x008466 to 0x010344 (0x7edf bytes) 000426 1788931326.470956 punch from 0x03b88b to 0x03ffff (0x4775 bytes) 000427 1788931326.479957 mapwrite 0x0047e1 thru 0x008609 (0x3e29 bytes) 000428 1788931326.522990 fallocate 0x0026e9 thru 0x002940 (0x258 bytes) 000429 1788931326.536161 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) 000430 1788931326.691411 fallocate 0x012151 thru 0x0180c9 (0x5f79 bytes) 000431 1788931326.698281 trunc from 0x02d4c6 to 0x02e1cf (0xd0a bytes) 000432 1788931326.704757 read 0x02eb65 thru 0x03715f (0x85fb bytes) 000433 1788931326.717341 fallocate 0x03535e thru 0x03ffff (0xaca2 bytes) 000436 1788931326.723058 write 0x01e44a thru 0x01e4dd (0x94 bytes) 000437 1788931326.740081 mapwrite 0x000a86 thru 0x00aa88 (0xa003 bytes) 000438 1788931326.797046 write 0x024829 thru 0x025bd7 (0x13af bytes) 000439 1788931326.811204 mapread 0x0127bb thru 0x019539 (0x6d7f bytes) 000440 1788931326.829307 mapread 0x004528 thru 0x01359c (0xf075 bytes) 000441 1788931326.832001 fallocate 0x039249 thru 0x03edf5 (0x5bad bytes) 000442 1788931326.840373 write 0x0099f9 thru 0x016028 (0xc630 bytes) 000443 1788931326.844909 trunc from 0x036d06 to 0x03896b (0x1c66 bytes) 000444 1788931326.851241 mapwrite 0x033e98 thru 0x03ceb8 (0x9021 bytes) 000445 1788931326.899055 mapread 0x028209 thru 0x032b70 (0xa968 bytes) 000447 1788931326.913972 fallocate 0x032b9f thru 0x03603d (0x349f bytes) 000448 1788931326.921970 mapread 0x007147 thru 0x011ebc (0xad76 bytes) 000450 1788931326.925170 trunc from 0x040000 to 0x00072b (0x3f8d4 bytes) 000451 1788931326.958040 MIRROR_EXTEND to 2 mirrors 000452 1788931328.063384 read 0x0000e0 thru 0x00072b (0x64c bytes) 000453 1788931328.102221 punch from 0x000510 to 0x00072b (0x21c bytes) 000454 1788931328.169190 trunc from 0x000139 to 0x00072b (0x5f3 bytes) 000455 1788931328.177963 punch from 0x0002d4 to 0x00072b (0x458 bytes) 000456 1788931328.192396 trunc from 0x000194 to 0x00072b (0x598 bytes) 000457 1788931328.201791 trunc from 0x00072c to 0x011053 (0x10928 bytes) 000458 1788931328.272110 punch from 0x0024fa to 0x004026 (0x1b2d bytes) 000459 1788931328.284354 fallocate 0x026adc thru 0x030c97 (0xa1bc bytes) 000461 1788931328.296719 punch from 0x009774 to 0x00d585 (0x3e12 bytes) 000462 1788931328.310282 mapwrite 0x021146 thru 0x024024 (0x2edf bytes) 000463 1788931328.392127 mapwrite 0x02ceff thru 0x02f6e2 (0x27e4 bytes) 000464 1788931328.493501 mapread 0x01a6d1 thru 0x02559a (0xaeca bytes) 000465 1788931328.547392 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f6e3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6e3): Mirror 2: 0xb8394b0c (stale) CRC-32 checksum value for chunk [0, 0x2f6e3): Mirror 1: 0x635aebf5 Mirror 3: 0x635aebf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6e3: skipped 000466 1788931329.910705 fallocate 0x02732c thru 0x031646 (0xa31b bytes) 000467 1788931329.980360 trunc from 0x02f6e3 to 0x03a72a (0xb048 bytes) 000468 1788931330.019445 read 0x009507 thru 0x016507 (0xd001 bytes) 000469 1788931330.037062 fallocate 0x02164c thru 0x02404a (0x29ff bytes) 000470 1788931330.048717 trunc from 0x01f0af to 0x026acb (0x7a1d bytes) 000472 1788931330.068449 trunc from 0x03a72b to 0x03b9f8 (0x12ce bytes) 000473 1788931330.116256 write 0x019198 thru 0x01e8c1 (0x572a bytes) 000474 1788931330.141627 mapwrite 0x00c3d9 thru 0x0125df (0x6207 bytes) 000475 1788931330.209551 read 0x009c1e thru 0x0182b7 (0xe69a bytes) 000476 1788931330.245838 mapwrite 0x0145b5 thru 0x02364e (0xf09a bytes) 000477 1788931330.315818 write 0x003923 thru 0x009136 (0x5814 bytes) 000478 1788931330.363182 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b9f9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9f9): Mirror 2: 0x7906c6eb (stale) Mirror 3: 0x3c620e8d (stale) CRC-32 checksum value for chunk [0, 0x3b9f9): Mirror 1: 0x3307e251 Mirror 4: 0x3307e251 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9f9: skipped 000479 1788931332.472965 write 0x018718 thru 0x020f3a (0x8823 bytes) 000480 1788931332.620175 mapread 0x02e4fb thru 0x038926 (0xa42c bytes) 000481 1788931332.662360 read 0x0113d7 thru 0x013984 (0x25ae bytes) 000482 1788931332.675224 mapread 0x0265bc thru 0x02eea7 (0x88ec bytes) 000483 1788931332.696192 MIRROR_RESYNC 4 mirrors 000484 1788931335.416004 punch from 0x03a7c2 to 0x03b9f8 (0x1237 bytes) 000486 1788931335.477365 mapread 0x019363 thru 0x01f597 (0x6235 bytes) 000488 1788931335.524710 mapwrite 0x038434 thru 0x03d8b6 (0x5483 bytes) 000489 1788931335.666060 write 0x004f81 thru 0x00a672 (0x56f2 bytes) 000490 1788931335.694867 mapwrite 0x038a28 thru 0x03afdc (0x25b5 bytes) 000491 1788931335.779930 MIRROR_RESYNC 4 mirrors 000492 1788931337.926813 mapread 0x02989a thru 0x03027e (0x69e5 bytes) 000494 1788931337.951791 write 0x005583 thru 0x0130a4 (0xdb22 bytes) 000495 1788931337.995897 read 0x013484 thru 0x01fe92 (0xca0f bytes) 000496 1788931338.011111 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d8b7): Mirror 3: 0x5e2f84f7 (stale) Mirror 4: 0x5e2f84f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8b7): Mirror 3: 0x5e2f84f7 (stale) Mirror 4: 0x5e2f84f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8b7: skipped 000497 1788931338.716164 mapwrite 0x012932 thru 0x01f8b1 (0xcf80 bytes) 000498 1788931338.848091 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d8b7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8b7): Mirror 3: 0x5e2f84f7 (stale) Mirror 4: 0x5e2f84f7 (stale) CRC-32 checksum value for chunk [0, 0x3d8b7): Mirror 1: 0xf3cbaf48 Mirror 5: 0xf3cbaf48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8b7: skipped 000499 1788931340.876888 fallocate 0x020397 thru 0x02910f (0x8d79 bytes) 000500 1788931340.944498 read 0x021da4 thru 0x02f6a3 (0xd900 bytes) 000501 1788931340.998682 MIRROR_RESYNC 4 mirrors 000502 1788931342.806639 trunc from 0x037c6e to 0x03d8b6 (0x5c49 bytes) 000503 1788931342.874918 mapread 0x034c63 thru 0x03d8b6 (0x8c54 bytes) 000505 1788931342.926986 punch from 0x017b9a to 0x01f5f3 (0x7a5a bytes) 000506 1788931342.934197 mapread 0x038c12 thru 0x03d8b6 (0x4ca5 bytes) 000507 1788931342.935187 write 0x02f04d thru 0x035efd (0x6eb1 bytes) 000508 1788931342.938960 mapwrite 0x00d3d3 thru 0x0111c5 (0x3df3 bytes) 000509 1788931343.012156 read 0x005878 thru 0x009ea1 (0x462a bytes) 000510 1788931343.023223 trunc from 0x01a90c to 0x01ad38 (0x42d bytes) 000511 1788931343.037572 trunc from 0x013f9a to 0x0195a5 (0x560c bytes) 000512 1788931343.048469 mapread 0x007e7d thru 0x0167f4 (0xe978 bytes) 000513 1788931343.070364 trunc from 0x03d8b7 to 0x03a025 (0x3891 bytes) 000514 1788931343.135512 MIRROR_RESYNC 4 mirrors 000515 1788931345.403820 read 0x0274ee thru 0x02d9fd (0x6510 bytes) 000516 1788931345.423399 fallocate 0x02c916 thru 0x0370eb (0xa7d6 bytes) 000517 1788931345.461122 read 0x000955 thru 0x00eb05 (0xe1b1 bytes) 000518 1788931345.476102 write 0x0380e0 thru 0x03ffff (0x7f20 bytes) 000519 1788931345.478132 mapread 0x0255cd thru 0x02ba28 (0x645c bytes) 000520 1788931345.479712 read 0x00b62f thru 0x01926f (0xdc41 bytes) 000521 1788931345.483845 trunc from 0x040000 to 0x02e214 (0x11deb bytes) 000522 1788931345.526324 trunc from 0x026d00 to 0x02e214 (0x7515 bytes) 000523 1788931345.533667 read 0x01269f thru 0x014673 (0x1fd5 bytes) 000524 1788931345.534457 read 0x00fdbb thru 0x015f9c (0x61e2 bytes) 000525 1788931345.536375 trunc from 0x02e215 to 0x023112 (0xb102 bytes) 000526 1788931345.560307 punch from 0x000d4e to 0x00a2c8 (0x957b bytes) 000527 1788931345.566605 write 0x017e62 thru 0x0232f3 (0xb492 bytes) 000528 1788931345.578100 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x232f4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232f4): Mirror 3: 0xc7b87bbb (stale) Mirror 4: 0xc7b87bbb (stale) Mirror 5: 0xc7b87bbb (stale) CRC-32 checksum value for chunk [0, 0x232f4): Mirror 1: 0xcd5a62f7 Mirror 6: 0xcd5a62f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232f4: skipped 000529 1788931346.982751 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x232f4): Mirror 3: 0xc7b87bbb (stale) Mirror 4: 0xc7b87bbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232f4): Mirror 3: 0xc7b87bbb (stale) Mirror 4: 0xc7b87bbb (stale) CRC-32 checksum value for chunk [0, 0x232f4): Mirror 1: 0xcd5a62f7 Mirror 6: 0xcd5a62f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232f4: skipped 000530 1788931347.887464 trunc from 0x0014f2 to 0x00328c (0x1d9b bytes) 000531 1788931347.947332 punch from 0x001ab1 to 0x002ac8 (0x1018 bytes) 000532 1788931347.957069 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x232f4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232f4): Mirror 3: 0xc7b87bbb (stale) Mirror 4: 0xc7b87bbb (stale) Mirror 6: 0xcd5a62f7 (stale) CRC-32 checksum value for chunk [0, 0x232f4): Mirror 1: 0xcd5a62f7 Mirror 7: 0xcd5a62f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232f4: skipped 000533 1788931349.153438 trunc from 0x01561f to 0x01a8e3 (0x52c5 bytes) 000534 1788931349.193899 mapwrite 0x02388b thru 0x02b990 (0x8106 bytes) 000535 1788931349.271740 write 0x0306d1 thru 0x03ed37 (0xe667 bytes) 000536 1788931349.274892 trunc from 0x0334f1 to 0x036723 (0x3233 bytes) 000538 1788931349.291871 trunc from 0x03ed38 to 0x018e75 (0x25ec2 bytes) 000539 1788931349.317300 MIRROR_RESYNC 5 mirrors 000540 1788931351.082076 mapwrite 0x00a3a3 thru 0x00a7e9 (0x447 bytes) 000541 1788931351.176839 MIRROR_RESYNC 5 mirrors 000542 1788931352.522796 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) 000543 1788931352.981906 fallocate 0x03f0a5 thru 0x03ffff (0xf5b bytes) 000544 1788931353.028433 mapwrite 0x00d8e6 thru 0x00f8b5 (0x1fd0 bytes) 000546 1788931353.075212 fallocate 0x006e2c thru 0x0075e5 (0x7ba bytes) 000547 1788931353.080960 trunc from 0x00cbe4 to 0x0117f8 (0x4c15 bytes) 000548 1788931353.087509 fallocate 0x01d491 thru 0x01e539 (0x10a9 bytes) 000549 1788931353.093965 mapread 0x00b42b thru 0x00fb26 (0x46fc bytes) 000550 1788931353.103282 trunc from 0x0032d0 to 0x00ee49 (0xbb7a bytes) 000551 1788931353.108999 trunc from 0x00ef63 to 0x00f3a0 (0x43e bytes) 000553 1788931353.114487 write 0x00a718 thru 0x017c5e (0xd547 bytes) 000554 1788931353.126457 write 0x01abe9 thru 0x01d9e9 (0x2e01 bytes) 000555 1788931353.128465 mapread 0x009949 thru 0x0147e6 (0xae9e bytes) 000556 1788931353.138548 trunc from 0x007e16 to 0x016826 (0xea11 bytes) 000557 1788931353.168910 mapwrite 0x03eecb thru 0x03ffff (0x1135 bytes) 000558 1788931353.224888 trunc from 0x022a08 to 0x02d855 (0xae4e bytes) 000559 1788931353.233954 fallocate 0x01d3c6 thru 0x02ba0b (0xe646 bytes) 000561 1788931353.256386 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x923b954f (stale) Mirror 4: 0x923b954f (stale) Mirror 6: 0x923b954f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9102e3d Mirror 7: 0xd9102e3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000562 1788931354.556461 fallocate 0x03e797 thru 0x03f182 (0x9ec bytes) 000563 1788931354.644311 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x923b954f (stale) Mirror 4: 0x923b954f (stale) Mirror 6: 0x923b954f (stale) Mirror 7: 0xd9102e3d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9102e3d Mirror 8: 0xd9102e3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000564 1788931356.232899 mapwrite 0x03ad47 thru 0x03ffff (0x52b9 bytes) 000565 1788931356.445222 read 0x000a9c thru 0x001923 (0xe88 bytes) 000566 1788931356.477559 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x923b954f (stale) Mirror 4: 0x923b954f (stale) Mirror 6: 0x923b954f (stale) Mirror 7: 0xd9102e3d (stale) Mirror 8: 0xd9102e3d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x652c665c Mirror 9: 0x652c665c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000567 1788931358.372927 trunc from 0x040000 to 0x029528 (0x16ad7 bytes) 000568 1788931358.432351 write 0x03eacc thru 0x03ffff (0x1534 bytes) 000569 1788931358.435330 read 0x009ca1 thru 0x012873 (0x8bd3 bytes) 000570 1788931358.450505 mapread 0x02b85a thru 0x032642 (0x6de9 bytes) 000571 1788931358.476490 fallocate 0x00be3c thru 0x00d31b (0x14e0 bytes) 000573 1788931358.487936 fallocate 0x03e61e thru 0x03ffff (0x19e2 bytes) 000574 1788931358.504759 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x923b954f (stale) Mirror 4: 0x923b954f (stale) Mirror 6: 0x923b954f (stale) Mirror 7: 0xd9102e3d (stale) Mirror 8: 0xd9102e3d (stale) Mirror 9: 0x652c665c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4e1138b3 Mirror 10: 0x4e1138b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000575 1788931360.034280 trunc from 0x040000 to 0x01b4ae (0x24b51 bytes) 000576 1788931360.104020 write 0x035d4d thru 0x037707 (0x19bb bytes) 000577 1788931360.105441 mapread 0x034ee1 thru 0x037707 (0x2827 bytes) 000578 1788931360.116139 fallocate 0x02eff5 thru 0x03e3ea (0xf3f6 bytes) 000579 1788931360.123700 trunc from 0x00d241 to 0x012bbc (0x597c bytes) 000580 1788931360.131867 write 0x01e446 thru 0x02879f (0xa35a bytes) 000581 1788931360.146414 trunc from 0x0188fa to 0x026ec5 (0xe5cc bytes) 000582 1788931360.168708 fallocate 0x006662 thru 0x01611e (0xfabd bytes) 000583 1788931360.176532 mapread 0x02f1e4 thru 0x037707 (0x8524 bytes) 000584 1788931360.212105 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37708): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37708): Mirror 3: 0xe89d65d5 (stale) Mirror 4: 0xe89d65d5 (stale) Mirror 6: 0xe89d65d5 (stale) Mirror 7: 0x424c41f2 (stale) Mirror 8: 0x424c41f2 (stale) Mirror 9: 0x424c41f2 (stale) Mirror 10: 0x424c41f2 (stale) CRC-32 checksum value for chunk [0, 0x37708): Mirror 1: 0x42409a7c Mirror 11: 0x42409a7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37708: skipped 000585 1788931361.628376 trunc from 0x037708 to 0x0095f9 (0x2e10e bytes) 000587 1788931361.728283 punch from 0x002ab5 to 0x0095f9 (0x6b45 bytes) 000588 1788931361.735743 trunc from 0x00539f to 0x0095f9 (0x425b bytes) 000589 1788931361.741533 mapread 0x0085b2 thru 0x0095f9 (0x1048 bytes) 000591 1788931361.756775 write 0x039202 thru 0x03ffff (0x6dfe bytes) 000592 1788931361.761829 read 0x0020b4 thru 0x00a91a (0x8867 bytes) 000593 1788931361.780103 MIRROR_RESYNC 9 mirrors 000594 1788931364.595476 trunc from 0x035f72 to 0x03e6fe (0x878d bytes) 000595 1788931364.722543 mapwrite 0x009bce thru 0x00dee8 (0x431b bytes) 000596 1788931364.853907 read 0x03efbf thru 0x03ffff (0x1041 bytes) 000597 1788931364.893704 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x78770e46 (stale) Mirror 4: 0x78770e46 (stale) Mirror 6: 0x78770e46 (stale) Mirror 7: 0x78770e46 (stale) Mirror 9: 0x78770e46 (stale) Mirror 10: 0x78770e46 (stale) Mirror 11: 0x78770e46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x78770e46 (stale) Mirror 4: 0x78770e46 (stale) Mirror 6: 0x78770e46 (stale) Mirror 7: 0x78770e46 (stale) Mirror 9: 0x78770e46 (stale) Mirror 10: 0x78770e46 (stale) Mirror 11: 0x78770e46 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000598 1788931366.053273 MIRROR_RESYNC 8 mirrors 000600 1788931369.033466 mapwrite 0x00e761 thru 0x01ca71 (0xe311 bytes) 000601 1788931369.190618 write 0x023175 thru 0x02e73c (0xb5c8 bytes) 000602 1788931369.249263 MIRROR_RESYNC 8 mirrors 000603 1788931372.430029 punch from 0x019261 to 0x01f4b8 (0x6258 bytes) 000604 1788931372.520497 mapwrite 0x009445 thru 0x013960 (0xa51c bytes) 000605 1788931372.602852 mapwrite 0x01be80 thru 0x023b12 (0x7c93 bytes) 000606 1788931372.688881 fallocate 0x010e30 thru 0x0182bd (0x748e bytes) 000607 1788931372.715376 read 0x00b8a5 thru 0x01864f (0xcdab bytes) 000608 1788931372.716040 read 0x029cc2 thru 0x035ba2 (0xbee1 bytes) 000609 1788931372.735995 trunc from 0x006caa to 0x016c6f (0xffc6 bytes) 000610 1788931372.749756 fallocate 0x015427 thru 0x01b301 (0x5edb bytes) 000611 1788931372.771024 trunc from 0x00f5c2 to 0x0157cd (0x620c bytes) 000613 1788931372.784657 read 0x032c45 thru 0x03ecc9 (0xc085 bytes) 000614 1788931372.786629 mapwrite 0x013036 thru 0x01f1ba (0xc185 bytes) 000615 1788931372.848991 fallocate 0x03f139 thru 0x03ffff (0xec7 bytes) 000616 1788931372.863203 trunc from 0x040000 to 0x038745 (0x78ba bytes) 000617 1788931372.913879 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38746): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38746): Mirror 3: 0x29664045 (stale) Mirror 4: 0x29664045 (stale) Mirror 6: 0x29664045 (stale) Mirror 7: 0x29664045 (stale) Mirror 9: 0x29664045 (stale) Mirror 10: 0x29664045 (stale) Mirror 11: 0x29664045 (stale) CRC-32 checksum value for chunk [0, 0x38746): Mirror 1: 0xb314d478 Mirror 12: 0xb314d478 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38746: skipped 000618 1788931374.640401 mapwrite 0x00d28e thru 0x014be7 (0x795a bytes) 000619 1788931374.773129 punch from 0x021c7f to 0x025e5e (0x41e0 bytes) 000621 1788931374.782269 trunc from 0x0330d5 to 0x038745 (0x5671 bytes) 000622 1788931374.789542 read 0x01e525 thru 0x02151a (0x2ff6 bytes) 000623 1788931374.803249 punch from 0x01d00b to 0x01f09c (0x2092 bytes) 000624 1788931374.810128 write 0x0037dc thru 0x0037e6 (0xb bytes) 000625 1788931374.821520 read 0x0025d9 thru 0x0115b2 (0xefda bytes) 000626 1788931374.838288 mapread 0x00210e thru 0x00e40b (0xc2fe bytes) 000627 1788931374.840139 mapwrite 0x018c7b thru 0x01fb06 (0x6e8c bytes) 000628 1788931374.885812 mapwrite 0x02b898 thru 0x034874 (0x8fdd bytes) 000630 1788931374.930275 punch from 0x02d1d1 to 0x03500d (0x7e3d bytes) 000632 1788931374.938456 read 0x01babf thru 0x02a71a (0xec5c bytes) 000633 1788931374.939244 mapread 0x02ba80 thru 0x02fb50 (0x40d1 bytes) 000634 1788931374.955927 mapread 0x017d3f thru 0x01f692 (0x7954 bytes) 000635 1788931374.957043 read 0x006a3a thru 0x00f633 (0x8bfa bytes) 000636 1788931374.967440 read 0x034876 thru 0x038745 (0x3ed0 bytes) 000637 1788931374.968122 read 0x0373c7 thru 0x038745 (0x137f bytes) 000638 1788931374.968737 trunc from 0x017558 to 0x01fef0 (0x8999 bytes) 000639 1788931374.975207 punch from 0x00a086 to 0x0123de (0x8359 bytes) 000640 1788931374.980210 mapread 0x0070d0 thru 0x00bc66 (0x4b97 bytes) 000641 1788931374.989999 write 0x033d23 thru 0x03ffff (0xc2dd bytes) 000642 1788931374.992477 trunc from 0x00cef2 to 0x01345d (0x656c bytes) 000643 1788931374.998917 trunc from 0x019bda to 0x019ec5 (0x2ec bytes) 000644 1788931375.007675 punch from 0x039beb to 0x03ffff (0x6415 bytes) 000645 1788931375.050400 mapwrite 0x01022e thru 0x01b6c0 (0xb493 bytes) 000646 1788931375.165385 trunc from 0x040000 to 0x039601 (0x69fe bytes) 000647 1788931375.212485 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39602): Mirror 3: 0xd802fbd8 (stale) Mirror 4: 0xd802fbd8 (stale) Mirror 6: 0xd802fbd8 (stale) Mirror 7: 0xd802fbd8 (stale) Mirror 9: 0xd802fbd8 (stale) Mirror 10: 0xd802fbd8 (stale) Mirror 11: 0xd802fbd8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39602): Mirror 3: 0xd802fbd8 (stale) Mirror 4: 0xd802fbd8 (stale) Mirror 6: 0xd802fbd8 (stale) Mirror 7: 0xd802fbd8 (stale) Mirror 9: 0xd802fbd8 (stale) Mirror 10: 0xd802fbd8 (stale) Mirror 11: 0xd802fbd8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39602: skipped 000648 1788931375.834644 trunc from 0x00d4f1 to 0x01206b (0x4b7b bytes) 000649 1788931375.841072 punch from 0x00d249 to 0x00ebdf (0x1997 bytes) 000650 1788931375.846457 mapread 0x020ffc thru 0x027b1b (0x6b20 bytes) 000651 1788931375.877098 fallocate 0x00f4b2 thru 0x01da7d (0xe5cc bytes) 000652 1788931375.882825 write 0x03547d thru 0x03ffff (0xab83 bytes) 000654 1788931375.906080 read 0x03cf67 thru 0x03ffff (0x3099 bytes) 000655 1788931375.909579 trunc from 0x0198de to 0x022bd0 (0x92f3 bytes) 000656 1788931375.935099 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x281d079d (stale) Mirror 4: 0x281d079d (stale) Mirror 6: 0x281d079d (stale) Mirror 7: 0x281d079d (stale) Mirror 9: 0x281d079d (stale) Mirror 10: 0x281d079d (stale) Mirror 11: 0x281d079d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1bbbc7fb Mirror 12: 0x1bbbc7fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000657 1788931377.301940 fallocate 0x017e12 thru 0x01b90a (0x3af9 bytes) 000658 1788931377.343144 trunc from 0x040000 to 0x03d198 (0x2e67 bytes) 000659 1788931377.404174 MIRROR_RESYNC 9 mirrors 000660 1788931381.773393 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) 000661 1788931382.211910 mapread 0x03856a thru 0x03d198 (0x4c2f bytes) 000662 1788931382.236701 punch from 0x01e072 to 0x02d4ab (0xf43a bytes) 000663 1788931382.303296 read 0x0291f1 thru 0x02e542 (0x5352 bytes) 000664 1788931382.315861 fallocate 0x002685 thru 0x0043ec (0x1d68 bytes) 000665 1788931382.321964 punch from 0x037ac1 to 0x03d198 (0x56d8 bytes) 000666 1788931382.327920 punch from 0x02536b to 0x030cf5 (0xb98b bytes) 000667 1788931382.333466 mapread 0x003219 thru 0x00f99c (0xc784 bytes) 000668 1788931382.343798 mapwrite 0x02280b thru 0x025a3c (0x3232 bytes) 000669 1788931382.372166 write 0x0050fa thru 0x00ced6 (0x7ddd bytes) 000670 1788931382.381908 mapread 0x003954 thru 0x0047e2 (0xe8f bytes) 000672 1788931382.384388 mapread 0x01e201 thru 0x02b723 (0xd523 bytes) 000673 1788931382.398821 trunc from 0x037015 to 0x03c386 (0x5372 bytes) 000674 1788931382.404736 mapread 0x01b240 thru 0x025257 (0xa018 bytes) 000675 1788931382.405853 trunc from 0x020c16 to 0x023f62 (0x334d bytes) 000676 1788931382.414312 trunc from 0x0284ea to 0x036813 (0xe32a bytes) 000677 1788931382.422363 write 0x002490 thru 0x0113d9 (0xef4a bytes) 000678 1788931382.431865 write 0x03ef56 thru 0x03ffff (0x10aa bytes) 000679 1788931382.433145 mapread 0x03d9cd thru 0x03ffff (0x2633 bytes) 000680 1788931382.443334 trunc from 0x040000 to 0x024acc (0x1b533 bytes) 000681 1788931382.468043 fallocate 0x0035e1 thru 0x00df8d (0xa9ad bytes) 000682 1788931382.473293 punch from 0x0094f8 to 0x018fa9 (0xfab2 bytes) 000683 1788931382.492347 trunc from 0x00a1a5 to 0x01171f (0x757b bytes) 000684 1788931382.498043 write 0x022357 thru 0x026e61 (0x4b0b bytes) 000688 1788931382.508449 fallocate 0x00ec19 thru 0x015be5 (0x6fcd bytes) 000689 1788931382.514543 trunc from 0x026e62 to 0x01fba0 (0x72c1 bytes) 000690 1788931382.536629 fallocate 0x013a4e thru 0x0155cd (0x1b80 bytes) 000691 1788931382.547183 read 0x004e06 thru 0x00ec96 (0x9e91 bytes) 000692 1788931382.566161 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1fba1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fba1): Mirror 3: 0xf50997f7 (stale) Mirror 4: 0xf50997f7 (stale) Mirror 6: 0xf50997f7 (stale) Mirror 7: 0xf50997f7 (stale) Mirror 10: 0xf50997f7 (stale) Mirror 11: 0xf50997f7 (stale) Mirror 12: 0xf50997f7 (stale) CRC-32 checksum value for chunk [0, 0x1fba1): Mirror 1: 0xea9a196 Mirror 13: 0xea9a196 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fba1: skipped 000693 1788931384.094414 fallocate 0x034887 thru 0x037fa3 (0x371d bytes) 000694 1788931384.152549 write 0x01b4f3 thru 0x020626 (0x5134 bytes) 000695 1788931384.181117 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x37fa4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fa4): Mirror 3: 0xba4c6ea9 (stale) Mirror 4: 0xba4c6ea9 (stale) Mirror 6: 0xba4c6ea9 (stale) Mirror 7: 0xba4c6ea9 (stale) Mirror 10: 0xba4c6ea9 (stale) Mirror 11: 0xba4c6ea9 (stale) Mirror 12: 0xba4c6ea9 (stale) Mirror 13: 0x12f5c10e (stale) CRC-32 checksum value for chunk [0, 0x37fa4): Mirror 1: 0x9b7a4188 Mirror 14: 0x9b7a4188 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fa4: skipped 000696 1788931385.928656 read 0x00c974 thru 0x00f015 (0x26a2 bytes) 000697 1788931385.950301 write 0x02c716 thru 0x03246f (0x5d5a bytes) 000698 1788931386.063581 mapread 0x01c27a thru 0x0263cf (0xa156 bytes) 000699 1788931386.088428 mapwrite 0x006861 thru 0x00ee36 (0x85d6 bytes) 000700 1788931386.136475 punch from 0x037c76 to 0x037fa3 (0x32e bytes) 000701 1788931386.144384 trunc from 0x002e13 to 0x00659f (0x378d bytes) 000702 1788931386.152051 write 0x03b148 thru 0x03de9c (0x2d55 bytes) 000704 1788931386.159333 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3de9d): Mirror 3: 0x75b777fc (stale) Mirror 4: 0x75b777fc (stale) Mirror 6: 0x75b777fc (stale) Mirror 7: 0x75b777fc (stale) Mirror 11: 0x75b777fc (stale) Mirror 12: 0x75b777fc (stale) Mirror 13: 0xe86993db (stale) Mirror 14: 0xbf4544e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de9d): Mirror 3: 0x75b777fc (stale) Mirror 4: 0x75b777fc (stale) Mirror 6: 0x75b777fc (stale) Mirror 7: 0x75b777fc (stale) Mirror 11: 0x75b777fc (stale) Mirror 12: 0x75b777fc (stale) Mirror 13: 0xe86993db (stale) Mirror 14: 0xbf4544e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de9d: skipped 000705 1788931386.769690 fallocate 0x02c1ab thru 0x03c045 (0xfe9b bytes) 000707 1788931386.778277 trunc from 0x0106fe to 0x012eef (0x27f2 bytes) 000708 1788931386.783277 trunc from 0x03de9d to 0x0245d9 (0x198c3 bytes) 000709 1788931386.805120 read 0x00be09 thru 0x00f8d3 (0x3acb bytes) 000710 1788931386.815280 fallocate 0x033846 thru 0x03e4d0 (0xac8b bytes) 000711 1788931386.827822 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3e4d1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d1): Mirror 3: 0x51aef381 (stale) Mirror 4: 0x51aef381 (stale) Mirror 6: 0x51aef381 (stale) Mirror 7: 0x51aef381 (stale) Mirror 11: 0x51aef381 (stale) Mirror 12: 0x51aef381 (stale) Mirror 13: 0xef3156ac (stale) Mirror 14: 0x7431aa07 (stale) CRC-32 checksum value for chunk [0, 0x3e4d1): Mirror 1: 0x61b37127 Mirror 15: 0x61b37127 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d1: skipped 000712 1788931388.570284 read 0x006868 thru 0x00efda (0x8773 bytes) 000713 1788931388.592248 read 0x00b097 thru 0x018708 (0xd672 bytes) 000714 1788931388.613268 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4d1): Mirror 4: 0x51aef381 (stale) Mirror 6: 0x51aef381 (stale) Mirror 7: 0x51aef381 (stale) Mirror 11: 0x51aef381 (stale) Mirror 12: 0x51aef381 (stale) Mirror 13: 0xef3156ac (stale) Mirror 14: 0x7431aa07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d1): Mirror 4: 0x51aef381 (stale) Mirror 6: 0x51aef381 (stale) Mirror 7: 0x51aef381 (stale) Mirror 11: 0x51aef381 (stale) Mirror 12: 0x51aef381 (stale) Mirror 13: 0xef3156ac (stale) Mirror 14: 0x7431aa07 (stale) CRC-32 checksum value for chunk [0, 0x3e4d1): Mirror 1: 0x61b37127 Mirror 15: 0x61b37127 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d1: skipped 000715 1788931389.713521 trunc from 0x03e4d1 to 0x023cee (0x1a7e2 bytes) 000716 1788931389.794759 trunc from 0x023cef to 0x033294 (0xf5a6 bytes) 000717 1788931389.815556 fallocate 0x03e346 thru 0x03ffff (0x1cba bytes) 000718 1788931389.825067 trunc from 0x033295 to 0x0234e8 (0xfdac bytes) 000719 1788931389.844891 write 0x005a00 thru 0x00ee8e (0x948f bytes) 000720 1788931389.866658 MIRROR_RESYNC 9 mirrors 000721 1788931394.270724 write 0x002ace thru 0x00a613 (0x7b46 bytes) 000722 1788931394.355982 trunc from 0x0062ca to 0x008b51 (0x2888 bytes) 000723 1788931394.388125 mapread 0x01c2eb thru 0x021364 (0x507a bytes) 000724 1788931394.446450 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x234e9): Mirror 4: 0xab962581 (stale) Mirror 6: 0xab962581 (stale) Mirror 7: 0xab962581 (stale) Mirror 11: 0xab962581 (stale) Mirror 13: 0xab962581 (stale) Mirror 14: 0xab962581 (stale) Mirror 15: 0xab962581 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x234e9): Mirror 4: 0xab962581 (stale) Mirror 6: 0xab962581 (stale) Mirror 7: 0xab962581 (stale) Mirror 11: 0xab962581 (stale) Mirror 13: 0xab962581 (stale) Mirror 14: 0xab962581 (stale) Mirror 15: 0xab962581 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x234e9: skipped 000725 1788931395.210574 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x234e9): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x234e9): Mirror 4: 0xab962581 (stale) Mirror 6: 0xab962581 (stale) Mirror 7: 0xab962581 (stale) Mirror 11: 0xab962581 (stale) Mirror 13: 0xab962581 (stale) Mirror 14: 0xab962581 (stale) Mirror 15: 0xab962581 (stale) CRC-32 checksum value for chunk [0, 0x234e9): Mirror 1: 0xa8d0bf4e Mirror 16: 0xa8d0bf4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x234e9: skipped 000726 1788931396.421523 read 0x021cc3 thru 0x0234e8 (0x1826 bytes) 000727 1788931396.430657 MIRROR_RESYNC 9 mirrors 000729 1788931400.376207 mapread 0x0002c1 thru 0x0084d7 (0x8217 bytes) 000730 1788931400.396555 read 0x01ea66 thru 0x0234e8 (0x4a83 bytes) 000731 1788931400.406071 punch from 0x00626f to 0x0112d6 (0xb068 bytes) 000732 1788931400.439801 punch from 0x0169ac to 0x01e0a7 (0x76fc bytes) 000734 1788931400.445345 mapread 0x016f52 thru 0x023412 (0xc4c1 bytes) 000735 1788931400.459216 mapwrite 0x033de9 thru 0x03e51d (0xa735 bytes) 000736 1788931400.527817 trunc from 0x037a13 to 0x03c7d7 (0x4dc5 bytes) 000737 1788931400.532249 punch from 0x000f8f to 0x00dd95 (0xce07 bytes) 000738 1788931400.536902 mapwrite 0x01275a thru 0x01fe15 (0xd6bc bytes) 000739 1788931400.556253 mapread 0x035b54 thru 0x038d86 (0x3233 bytes) 000741 1788931400.568477 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3e51e): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e51e): Mirror 4: 0x74f89199 (stale) Mirror 6: 0x74f89199 (stale) Mirror 7: 0x74f89199 (stale) Mirror 11: 0x74f89199 (stale) Mirror 13: 0x74f89199 (stale) Mirror 14: 0x74f89199 (stale) Mirror 15: 0x74f89199 (stale) Mirror 16: 0x74f89199 (stale) CRC-32 checksum value for chunk [0, 0x3e51e): Mirror 1: 0x9f67d94a Mirror 17: 0x9f67d94a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e51e: skipped 000742 1788931402.520814 mapread 0x00cd55 thru 0x013e23 (0x70cf bytes) 000743 1788931402.583862 mapread 0x0277a3 thru 0x02aa35 (0x3293 bytes) 000744 1788931402.624223 mapread 0x01a8c5 thru 0x022a5f (0x819b bytes) 000746 1788931402.666970 write 0x02a7e6 thru 0x03742e (0xcc49 bytes) 000747 1788931402.715106 write 0x03d2df thru 0x03d556 (0x278 bytes) 000748 1788931402.723885 fallocate 0x035afe thru 0x036da1 (0x12a4 bytes) 000749 1788931402.730135 mapread 0x039630 thru 0x03e51d (0x4eee bytes) 000750 1788931402.743421 read 0x013642 thru 0x013b22 (0x4e1 bytes) 000751 1788931402.751471 read 0x024bba thru 0x02db77 (0x8fbe bytes) 000753 1788931402.767907 mapread 0x03297b thru 0x039dd6 (0x745c bytes) 000754 1788931402.769353 mapread 0x017e3c thru 0x01b821 (0x39e6 bytes) 000755 1788931402.772753 punch from 0x028757 to 0x03577c (0xd026 bytes) 000756 1788931402.798243 mapread 0x0195d3 thru 0x025c47 (0xc675 bytes) 000757 1788931402.800646 write 0x00bef1 thru 0x011d19 (0x5e29 bytes) 000758 1788931402.805039 mapwrite 0x02c758 thru 0x033c32 (0x74db bytes) 000760 1788931402.907271 mapwrite 0x018b88 thru 0x024cfd (0xc176 bytes) 000762 1788931402.985421 punch from 0x02fb47 to 0x033c3b (0x40f5 bytes) 000764 1788931402.995417 trunc from 0x039a8e to 0x03e51d (0x4a90 bytes) 000765 1788931403.004861 trunc from 0x03b912 to 0x03e51d (0x2c0c bytes) 000768 1788931403.019270 punch from 0x027884 to 0x02b12c (0x38a9 bytes) 000770 1788931403.024496 write 0x0343f7 thru 0x037569 (0x3173 bytes) 000771 1788931403.030225 write 0x02430b thru 0x0296c1 (0x53b7 bytes) 000772 1788931403.035287 mapwrite 0x009c39 thru 0x00f51c (0x58e4 bytes) 000773 1788931403.054059 punch from 0x0112ed to 0x020a50 (0xf764 bytes) 000774 1788931403.068119 punch from 0x01c331 to 0x028b51 (0xc821 bytes) 000776 1788931403.078236 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e51e): Mirror 4: 0x74f89199 (stale) Mirror 6: 0x74f89199 (stale) Mirror 7: 0x74f89199 (stale) Mirror 11: 0x74f89199 (stale) Mirror 13: 0x74f89199 (stale) Mirror 14: 0x74f89199 (stale) Mirror 15: 0x74f89199 (stale) Mirror 16: 0x74f89199 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e51e): Mirror 4: 0x74f89199 (stale) Mirror 6: 0x74f89199 (stale) Mirror 7: 0x74f89199 (stale) Mirror 11: 0x74f89199 (stale) Mirror 13: 0x74f89199 (stale) Mirror 14: 0x74f89199 (stale) Mirror 15: 0x74f89199 (stale) Mirror 16: 0x74f89199 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e51e: skipped 000777 1788931403.725237 write 0x012b06 thru 0x019f81 (0x747c bytes) 000779 1788931403.745150 trunc from 0x03e51e to 0x022907 (0x1bc16 bytes) 000780 1788931403.779478 punch from 0x01299f to 0x015d69 (0x33cb bytes) 000781 1788931403.798132 write 0x00c8a7 thru 0x01824c (0xb9a6 bytes) 000782 1788931403.812838 mapread 0x007350 thru 0x00a14c (0x2dfd bytes) 000783 1788931403.854794 read 0x01ee09 thru 0x022907 (0x3aff bytes) 000784 1788931403.878000 read 0x007905 thru 0x01560b (0xdd07 bytes) 000785 1788931403.878993 fallocate 0x023040 thru 0x02cdd0 (0x9d91 bytes) 000786 1788931403.892462 trunc from 0x0121ba to 0x0163e1 (0x4228 bytes) 000787 1788931403.946908 fallocate 0x0359ae thru 0x037b28 (0x217b bytes) 000788 1788931403.970772 trunc from 0x037b29 to 0x031890 (0x6298 bytes) 000789 1788931404.011515 punch from 0x018ff1 to 0x02240d (0x941d bytes) 000790 1788931404.022645 trunc from 0x02603e to 0x029fe7 (0x3faa bytes) 000791 1788931404.031226 mapwrite 0x02566d thru 0x031f56 (0xc8ea bytes) 000792 1788931404.145632 MIRROR_RESYNC 9 mirrors 000793 1788931407.527143 mapread 0x02a795 thru 0x031466 (0x6cd2 bytes) 000794 1788931407.551432 mapwrite 0x00ac2e thru 0x0119de (0x6db1 bytes) 000795 1788931407.715959 trunc from 0x031f57 to 0x0190d2 (0x18e84 bytes) 000797 1788931407.747086 write 0x00fa42 thru 0x01b00d (0xb5cc bytes) 000798 1788931407.750208 trunc from 0x01b00e to 0x01c3a3 (0x1396 bytes) 000800 1788931407.803885 fallocate 0x02c239 thru 0x03802b (0xbdf3 bytes) 000801 1788931407.817116 MIRROR_RESYNC 9 mirrors 000802 1788931409.978324 punch from 0x0180d1 to 0x01c3a3 (0x42d3 bytes) 000803 1788931410.045824 read 0x0018e3 thru 0x00e3ed (0xcb0b bytes) 000804 1788931410.070840 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1c3a4): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c3a4): Mirror 4: 0x9cf205de (stale) Mirror 6: 0x9cf205de (stale) Mirror 7: 0x9cf205de (stale) Mirror 11: 0x9cf205de (stale) Mirror 13: 0x9cf205de (stale) Mirror 14: 0x9cf205de (stale) Mirror 15: 0x9cf205de (stale) Mirror 16: 0x9cf205de (stale) CRC-32 checksum value for chunk [0, 0x1c3a4): Mirror 1: 0x5ed5c7c5 Mirror 17: 0x5ed5c7c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c3a4: skipped 000805 1788931411.384013 write 0x03c3ad thru 0x03edf8 (0x2a4c bytes) 000806 1788931411.485783 punch from 0x00aadd to 0x010d3c (0x6260 bytes) 000807 1788931411.501438 write 0x019098 thru 0x022184 (0x90ed bytes) 000809 1788931411.546660 MIRROR_RESYNC 10 mirrors 000810 1788931415.030964 trunc from 0x03edf9 to 0x0239a2 (0x1b456 bytes) 000811 1788931415.150556 trunc from 0x0239a3 to 0x004943 (0x1f05f bytes) 000813 1788931415.189191 trunc from 0x00137d to 0x0042a1 (0x2f25 bytes) 000814 1788931415.207069 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x4944): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4944): Mirror 4: 0xe69e9a0e (stale) Mirror 6: 0xe69e9a0e (stale) Mirror 7: 0xe69e9a0e (stale) Mirror 11: 0xe69e9a0e (stale) Mirror 13: 0xe69e9a0e (stale) Mirror 14: 0xe69e9a0e (stale) Mirror 15: 0xe69e9a0e (stale) Mirror 16: 0xe69e9a0e (stale) Mirror 17: 0xe69e9a0e (stale) CRC-32 checksum value for chunk [0, 0x4944): Mirror 1: 0xe69e9a0e Mirror 18: 0xe69e9a0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4944: skipped 000815 1788931416.804999 MIRROR_RESYNC 11 mirrors 000816 1788931419.014676 read 0x00093d thru 0x003444 (0x2b08 bytes) 000817 1788931419.022801 trunc from 0x004944 to 0x03dd8a (0x39447 bytes) 000818 1788931419.113173 trunc from 0x03dd8b to 0x024373 (0x19a17 bytes) 000819 1788931419.149394 trunc from 0x00f638 to 0x01a941 (0xb30a bytes) 000820 1788931419.159964 trunc from 0x011271 to 0x017842 (0x65d2 bytes) 000821 1788931419.179816 trunc from 0x024374 to 0x01c45b (0x7f18 bytes) 000822 1788931419.214413 punch from 0x016cfd to 0x01c45b (0x575f bytes) 000823 1788931419.221985 fallocate 0x02cc33 thru 0x03a473 (0xd841 bytes) 000824 1788931419.241782 write 0x01929d thru 0x01c9fd (0x3761 bytes) 000825 1788931419.264220 mapread 0x01bc8c thru 0x01c9fd (0xd72 bytes) 000826 1788931419.265916 mapwrite 0x0031d8 thru 0x0034ee (0x317 bytes) 000827 1788931419.306085 mapwrite 0x0365ee thru 0x03cac9 (0x64dc bytes) 000828 1788931419.453936 fallocate 0x00603e thru 0x011204 (0xb1c7 bytes) 000829 1788931419.467711 trunc from 0x02f839 to 0x03bab5 (0xc27d bytes) 000830 1788931419.482847 mapwrite 0x0010a8 thru 0x00a308 (0x9261 bytes) 000831 1788931419.544470 fallocate 0x01f369 thru 0x020cff (0x1997 bytes) 000832 1788931419.563929 MIRROR_RESYNC 11 mirrors 000833 1788931422.602769 punch from 0x007a72 to 0x00f101 (0x7690 bytes) 000835 1788931422.697526 fallocate 0x020a5d thru 0x02e30f (0xd8b3 bytes) 000837 1788931422.703717 mapwrite 0x023dfd thru 0x024854 (0xa58 bytes) 000838 1788931422.726329 read 0x01a84d thru 0x027f80 (0xd734 bytes) 000839 1788931422.737090 trunc from 0x03b29e to 0x03bd15 (0xa78 bytes) 000840 1788931422.742329 trunc from 0x006c24 to 0x00a89d (0x3c7a bytes) 000841 1788931422.750258 trunc from 0x02abf2 to 0x034674 (0x9a83 bytes) 000842 1788931422.755722 mapwrite 0x016bbc thru 0x02016f (0x95b4 bytes) 000844 1788931422.788877 fallocate 0x01bb4c thru 0x026a37 (0xaeec bytes) 000845 1788931422.806952 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3caca): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3caca): Mirror 4: 0x673f07e3 (stale) Mirror 6: 0x673f07e3 (stale) Mirror 7: 0x673f07e3 (stale) Mirror 11: 0x673f07e3 (stale) Mirror 13: 0x673f07e3 (stale) Mirror 14: 0x673f07e3 (stale) Mirror 15: 0x673f07e3 (stale) Mirror 16: 0x673f07e3 (stale) Mirror 17: 0x673f07e3 (stale) Mirror 18: 0x673f07e3 (stale) CRC-32 checksum value for chunk [0, 0x3caca): Mirror 1: 0x1c6b061 Mirror 19: 0x1c6b061 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3caca: skipped 000846 1788931424.279931 trunc from 0x036406 to 0x03968c (0x3287 bytes) 000848 1788931424.351930 punch from 0x02d2a4 to 0x03baf1 (0xe84e bytes) 000849 1788931424.365369 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3caca): Mirror 4: 0x673f07e3 (stale) Mirror 7: 0x673f07e3 (stale) Mirror 11: 0x673f07e3 (stale) Mirror 13: 0x673f07e3 (stale) Mirror 14: 0x673f07e3 (stale) Mirror 15: 0x673f07e3 (stale) Mirror 16: 0x673f07e3 (stale) Mirror 17: 0x673f07e3 (stale) Mirror 18: 0x673f07e3 (stale) Mirror 19: 0x1c6b061 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3caca): Mirror 4: 0x673f07e3 (stale) Mirror 7: 0x673f07e3 (stale) Mirror 11: 0x673f07e3 (stale) Mirror 13: 0x673f07e3 (stale) Mirror 14: 0x673f07e3 (stale) Mirror 15: 0x673f07e3 (stale) Mirror 16: 0x673f07e3 (stale) Mirror 17: 0x673f07e3 (stale) Mirror 18: 0x673f07e3 (stale) Mirror 19: 0x1c6b061 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3caca: skipped 000850 1788931425.233560 write 0x020211 thru 0x02ea7d (0xe86d bytes) 000851 1788931425.252551 punch from 0x031245 to 0x03cac9 (0xb885 bytes) 000852 1788931425.258285 trunc from 0x03caca to 0x00ab2d (0x31f9c bytes) 000853 1788931425.280644 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xab2e): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xab2e): Mirror 4: 0xd81f1603 (stale) Mirror 7: 0xd81f1603 (stale) Mirror 11: 0xd81f1603 (stale) Mirror 13: 0xd81f1603 (stale) Mirror 14: 0xd81f1603 (stale) Mirror 15: 0xd81f1603 (stale) Mirror 16: 0xd81f1603 (stale) Mirror 17: 0xd81f1603 (stale) Mirror 18: 0xd81f1603 (stale) Mirror 19: 0x66ec0534 (stale) CRC-32 checksum value for chunk [0, 0xab2e): Mirror 1: 0x66ec0534 Mirror 20: 0x66ec0534 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xab2e: skipped 000854 1788931426.537966 mapwrite 0x0164d1 thru 0x01966d (0x319d bytes) 000855 1788931426.640073 trunc from 0x011afc to 0x0176e2 (0x5be7 bytes) 000856 1788931426.647207 mapwrite 0x014420 thru 0x0161ab (0x1d8c bytes) 000857 1788931426.669160 fallocate 0x032c8b thru 0x0359e0 (0x2d56 bytes) 000858 1788931426.674046 trunc from 0x0359e1 to 0x00afec (0x2a9f4 bytes) 000859 1788931426.697978 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xafed): Mirror 7: 0xd74a3d91 (stale) Mirror 11: 0xd74a3d91 (stale) Mirror 13: 0xd74a3d91 (stale) Mirror 14: 0xd74a3d91 (stale) Mirror 15: 0xd74a3d91 (stale) Mirror 16: 0xd74a3d91 (stale) Mirror 17: 0xd74a3d91 (stale) Mirror 18: 0xd74a3d91 (stale) Mirror 19: 0x7a57d992 (stale) Mirror 20: 0x7a57d992 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xafed): Mirror 7: 0xd74a3d91 (stale) Mirror 11: 0xd74a3d91 (stale) Mirror 13: 0xd74a3d91 (stale) Mirror 14: 0xd74a3d91 (stale) Mirror 15: 0xd74a3d91 (stale) Mirror 16: 0xd74a3d91 (stale) Mirror 17: 0xd74a3d91 (stale) Mirror 18: 0xd74a3d91 (stale) Mirror 19: 0x7a57d992 (stale) Mirror 20: 0x7a57d992 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xafed: skipped 000860 1788931427.275228 write 0x037679 thru 0x03a55e (0x2ee6 bytes) 000862 1788931427.286637 MIRROR_RESYNC 11 mirrors 000864 1788931429.557171 punch from 0x02ed36 to 0x03a55e (0xb829 bytes) 000865 1788931429.639109 trunc from 0x03a55f to 0x0049f9 (0x35b65 bytes) 000866 1788931429.712109 read 0x003859 thru 0x0049f9 (0x11a1 bytes) 000867 1788931429.719797 trunc from 0x002d20 to 0x0049f9 (0x1cda bytes) 000869 1788931429.734170 fallocate 0x016801 thru 0x01f786 (0x8f86 bytes) 000870 1788931429.748802 trunc from 0x011fd7 to 0x01e9da (0xca04 bytes) 000871 1788931429.762477 write 0x022883 thru 0x023c12 (0x1390 bytes) 000872 1788931429.764329 read 0x01b9bc thru 0x020c43 (0x5288 bytes) 000873 1788931429.790479 fallocate 0x01811c thru 0x020dea (0x8ccf bytes) 000874 1788931429.809157 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x23c13): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23c13): Mirror 7: 0x97cc4885 (stale) Mirror 11: 0x97cc4885 (stale) Mirror 13: 0x97cc4885 (stale) Mirror 14: 0x97cc4885 (stale) Mirror 15: 0x97cc4885 (stale) Mirror 16: 0x97cc4885 (stale) Mirror 17: 0x97cc4885 (stale) Mirror 18: 0x97cc4885 (stale) Mirror 19: 0x97cc4885 (stale) Mirror 20: 0x97cc4885 (stale) CRC-32 checksum value for chunk [0, 0x23c13): Mirror 1: 0x7c605808 Mirror 21: 0x7c605808 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23c13: skipped 000875 1788931431.480021 punch from 0x018bc1 to 0x01ecdf (0x611f bytes) 000876 1788931431.534348 read 0x01f1fb thru 0x023c12 (0x4a18 bytes) 000877 1788931431.560145 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23c13): Mirror 7: 0x97cc4885 (stale) Mirror 11: 0x97cc4885 (stale) Mirror 13: 0x97cc4885 (stale) Mirror 14: 0x97cc4885 (stale) Mirror 15: 0x97cc4885 (stale) Mirror 17: 0x97cc4885 (stale) Mirror 18: 0x97cc4885 (stale) Mirror 19: 0x97cc4885 (stale) Mirror 20: 0x97cc4885 (stale) Mirror 21: 0x7c605808 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23c13): Mirror 7: 0x97cc4885 (stale) Mirror 11: 0x97cc4885 (stale) Mirror 13: 0x97cc4885 (stale) Mirror 14: 0x97cc4885 (stale) Mirror 15: 0x97cc4885 (stale) Mirror 17: 0x97cc4885 (stale) Mirror 18: 0x97cc4885 (stale) Mirror 19: 0x97cc4885 (stale) Mirror 20: 0x97cc4885 (stale) Mirror 21: 0x7c605808 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23c13: skipped 000878 1788931432.323229 fallocate 0x02d748 thru 0x037288 (0x9b41 bytes) 000879 1788931432.332522 mapwrite 0x02aacd thru 0x0316a6 (0x6bda bytes) 000880 1788931432.370440 write 0x03833d thru 0x03ffff (0x7cc3 bytes) 000881 1788931432.377295 fallocate 0x021e3f thru 0x02b34c (0x950e bytes) 000882 1788931432.383077 fallocate 0x00510a thru 0x00f9e9 (0xa8e0 bytes) 000883 1788931432.388049 fallocate 0x029958 thru 0x032488 (0x8b31 bytes) 000884 1788931432.392758 trunc from 0x040000 to 0x02f242 (0x10dbd bytes) 000885 1788931432.413921 punch from 0x02e9b6 to 0x02f242 (0x88d bytes) 000886 1788931432.419524 mapwrite 0x00340e thru 0x00ada4 (0x7997 bytes) 000888 1788931432.464365 read 0x017972 thru 0x019f96 (0x2625 bytes) 000889 1788931432.471250 read 0x016392 thru 0x01efc6 (0x8c35 bytes) 000890 1788931432.488156 write 0x018c3d thru 0x024c94 (0xc058 bytes) 000891 1788931432.489526 write 0x007505 thru 0x00e9a7 (0x74a3 bytes) 000893 1788931432.492541 write 0x030287 thru 0x035e84 (0x5bfe bytes) 000894 1788931432.499287 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e85): Mirror 7: 0x5e8ba67 (stale) Mirror 11: 0x5e8ba67 (stale) Mirror 13: 0x5e8ba67 (stale) Mirror 14: 0x5e8ba67 (stale) Mirror 15: 0x5e8ba67 (stale) Mirror 17: 0x5e8ba67 (stale) Mirror 18: 0x5e8ba67 (stale) Mirror 19: 0x5e8ba67 (stale) Mirror 20: 0x5e8ba67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e85): Mirror 7: 0x5e8ba67 (stale) Mirror 11: 0x5e8ba67 (stale) Mirror 13: 0x5e8ba67 (stale) Mirror 14: 0x5e8ba67 (stale) Mirror 15: 0x5e8ba67 (stale) Mirror 17: 0x5e8ba67 (stale) Mirror 18: 0x5e8ba67 (stale) Mirror 19: 0x5e8ba67 (stale) Mirror 20: 0x5e8ba67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e85: skipped 000895 1788931433.379322 write 0x0059b7 thru 0x00de7a (0x84c4 bytes) 000896 1788931433.400251 mapwrite 0x031259 thru 0x03ffff (0xeda7 bytes) 000897 1788931433.515095 write 0x0150f1 thru 0x01f257 (0xa167 bytes) 000898 1788931433.539361 MIRROR_RESYNC 10 mirrors 000900 1788931435.553532 trunc from 0x01205e to 0x02073f (0xe6e2 bytes) 000901 1788931435.599387 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x991c8681 (stale) Mirror 13: 0x991c8681 (stale) Mirror 14: 0x991c8681 (stale) Mirror 15: 0x991c8681 (stale) Mirror 17: 0x991c8681 (stale) Mirror 18: 0x991c8681 (stale) Mirror 19: 0x991c8681 (stale) Mirror 20: 0x991c8681 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x991c8681 (stale) Mirror 13: 0x991c8681 (stale) Mirror 14: 0x991c8681 (stale) Mirror 15: 0x991c8681 (stale) Mirror 17: 0x991c8681 (stale) Mirror 18: 0x991c8681 (stale) Mirror 19: 0x991c8681 (stale) Mirror 20: 0x991c8681 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000902 1788931436.507398 punch from 0x03d850 to 0x03ffff (0x27b0 bytes) 000903 1788931436.516195 read 0x020caf thru 0x02dddd (0xd12f bytes) 000904 1788931436.531807 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x991c8681 (stale) Mirror 13: 0x991c8681 (stale) Mirror 14: 0x991c8681 (stale) Mirror 15: 0x991c8681 (stale) Mirror 17: 0x991c8681 (stale) Mirror 19: 0x991c8681 (stale) Mirror 20: 0x991c8681 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x991c8681 (stale) Mirror 13: 0x991c8681 (stale) Mirror 14: 0x991c8681 (stale) Mirror 15: 0x991c8681 (stale) Mirror 17: 0x991c8681 (stale) Mirror 19: 0x991c8681 (stale) Mirror 20: 0x991c8681 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000905 1788931437.054678 read 0x039e98 thru 0x03d12f (0x3298 bytes) 000906 1788931437.061510 read 0x02de36 thru 0x034b2f (0x6cfa bytes) 000907 1788931437.078085 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x991c8681 (stale) Mirror 13: 0x991c8681 (stale) Mirror 15: 0x991c8681 (stale) Mirror 17: 0x991c8681 (stale) Mirror 19: 0x991c8681 (stale) Mirror 20: 0x991c8681 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x991c8681 (stale) Mirror 13: 0x991c8681 (stale) Mirror 15: 0x991c8681 (stale) Mirror 17: 0x991c8681 (stale) Mirror 19: 0x991c8681 (stale) Mirror 20: 0x991c8681 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000908 1788931437.610257 mapread 0x01fbd2 thru 0x02931d (0x974c bytes) 000909 1788931437.639667 fallocate 0x01e261 thru 0x0279e3 (0x9783 bytes) 000910 1788931437.651273 punch from 0x02b725 to 0x03418f (0x8a6b bytes) 000911 1788931437.660705 trunc from 0x021e95 to 0x02ee50 (0xcfbc bytes) 000912 1788931437.673828 read 0x01dc24 thru 0x0253b6 (0x7793 bytes) 000913 1788931437.696045 mapread 0x02a6d8 thru 0x03699c (0xc2c5 bytes) 000915 1788931437.755282 read 0x00e150 thru 0x010468 (0x2319 bytes) 000916 1788931437.773840 trunc from 0x03253d to 0x03ffff (0xdac3 bytes) 000918 1788931437.788223 MIRROR_RESYNC 7 mirrors 000919 1788931439.804651 mapwrite 0x028966 thru 0x02ab3d (0x21d8 bytes) 000920 1788931439.901347 trunc from 0x040000 to 0x01e87c (0x21783 bytes) 000921 1788931439.920647 fallocate 0x02163b thru 0x02ce7e (0xb844 bytes) 000922 1788931439.927171 write 0x01ea80 thru 0x02e206 (0xf787 bytes) 000923 1788931439.937083 MIRROR_RESYNC 7 mirrors 000925 1788931441.191283 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) 000926 1788931441.730988 mapread 0x015a02 thru 0x01f6a3 (0x9ca2 bytes) 000927 1788931441.785133 trunc from 0x02e207 to 0x0287ad (0x5a59 bytes) 000929 1788931441.850022 write 0x023d1c thru 0x02dac2 (0x9da7 bytes) 000930 1788931441.861995 trunc from 0x02dac3 to 0x011f24 (0x1bb9e bytes) 000931 1788931441.889572 trunc from 0x011a4b to 0x011f24 (0x4da bytes) 000932 1788931441.894512 punch from 0x00ce53 to 0x011f24 (0x50d2 bytes) 000933 1788931441.901103 mapread 0x0091da thru 0x00d333 (0x415a bytes) 000935 1788931441.912848 read 0x00b0cc thru 0x011f24 (0x6e59 bytes) 000936 1788931441.916246 mapread 0x00aa49 thru 0x011f24 (0x74dc bytes) 000938 1788931441.918507 trunc from 0x011f25 to 0x01b176 (0x9252 bytes) 000939 1788931441.940280 write 0x03a380 thru 0x03ffff (0x5c80 bytes) 000940 1788931441.942057 fallocate 0x00a7c7 thru 0x016d29 (0xc563 bytes) 000941 1788931441.948306 trunc from 0x040000 to 0x026feb (0x19014 bytes) 000942 1788931441.970487 mapread 0x009aac thru 0x0184c3 (0xea18 bytes) 000943 1788931441.980878 mapwrite 0x02abe0 thru 0x039bbe (0xefdf bytes) 000944 1788931442.060055 trunc from 0x039bbf to 0x026382 (0x1383c bytes) 000945 1788931442.092175 trunc from 0x026383 to 0x01aca6 (0xb6dc bytes) 000946 1788931442.114371 punch from 0x00c7a1 to 0x014e7a (0x86da bytes) 000948 1788931442.133933 punch from 0x0164f6 to 0x01aca6 (0x47b1 bytes) 000949 1788931442.142430 mapwrite 0x010a8b thru 0x01f441 (0xe9b7 bytes) 000950 1788931442.273491 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1f442): Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f442): Mirror 7: 0x83e8b42a (stale) Mirror 13: 0x83e8b42a (stale) Mirror 17: 0x83e8b42a (stale) Mirror 19: 0x83e8b42a (stale) Mirror 20: 0x83e8b42a (stale) CRC-32 checksum value for chunk [0, 0x1f442): Mirror 1: 0x288a23ef Mirror 21: 0x288a23ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f442: skipped 000951 1788931443.632212 MIRROR_RESYNC 7 mirrors 000952 1788931445.078905 mapwrite 0x02730f thru 0x031973 (0xa665 bytes) 000953 1788931445.210503 punch from 0x01b4ba to 0x01f31e (0x3e65 bytes) 000954 1788931445.220742 trunc from 0x031974 to 0x002493 (0x2f4e0 bytes) 000955 1788931445.279196 trunc from 0x002494 to 0x005f33 (0x3aa0 bytes) 000956 1788931445.297686 punch from 0x000902 to 0x005f33 (0x5632 bytes) 000957 1788931445.304151 mapread 0x002f6d thru 0x005f33 (0x2fc7 bytes) 000958 1788931445.314509 fallocate 0x02fabf thru 0x03301d (0x355f bytes) 000959 1788931445.320889 write 0x02707b thru 0x0309d6 (0x995c bytes) 000960 1788931445.337970 fallocate 0x019438 thru 0x01d769 (0x4332 bytes) 000961 1788931445.343481 mapread 0x02940f thru 0x03036d (0x6f5f bytes) 000962 1788931445.357695 MIRROR_RESYNC 7 mirrors 000963 1788931446.876053 trunc from 0x003d4e to 0x00d80b (0x9abe bytes) 000964 1788931446.906871 trunc from 0x02270b to 0x024b30 (0x2426 bytes) 000965 1788931446.911919 trunc from 0x03301e to 0x012729 (0x208f4 bytes) 000966 1788931446.931137 mapread 0x00eeca thru 0x012729 (0x3860 bytes) 000967 1788931446.953581 MIRROR_RESYNC 7 mirrors 000968 1788931448.555466 mapwrite 0x03c409 thru 0x03de1e (0x1a16 bytes) 000969 1788931448.638555 trunc from 0x038ef4 to 0x03de1e (0x4f2b bytes) 000970 1788931448.653042 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3de1f): Mirror 7: 0xc1411883 (stale) Mirror 13: 0xc1411883 (stale) Mirror 19: 0xc1411883 (stale) Mirror 20: 0xc1411883 (stale) Mirror 21: 0xc1411883 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de1f): Mirror 7: 0xc1411883 (stale) Mirror 13: 0xc1411883 (stale) Mirror 19: 0xc1411883 (stale) Mirror 20: 0xc1411883 (stale) Mirror 21: 0xc1411883 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de1f: skipped 000971 1788931449.144155 read 0x014285 thru 0x01fcbe (0xba3a bytes) 000972 1788931449.155222 fallocate 0x030f91 thru 0x03cd94 (0xbe04 bytes) 000973 1788931449.160539 mapwrite 0x00772f thru 0x0148da (0xd1ac bytes) 000974 1788931449.206077 mapwrite 0x03fa1b thru 0x03ffff (0x5e5 bytes) 000975 1788931449.249899 write 0x0366a4 thru 0x036ac8 (0x425 bytes) 000976 1788931449.256992 trunc from 0x01840a to 0x01dd84 (0x597b bytes) 000977 1788931449.262913 punch from 0x0103a0 to 0x012c01 (0x2862 bytes) 000978 1788931449.268053 trunc from 0x040000 to 0x0252d7 (0x1ad28 bytes) 000979 1788931449.293386 trunc from 0x001105 to 0x00a530 (0x942c bytes) 000980 1788931449.300168 read 0x0006df thru 0x00a54c (0x9e6e bytes) 000981 1788931449.311636 write 0x01eacc thru 0x02be07 (0xd33c bytes) 000982 1788931449.314694 mapwrite 0x02225d thru 0x027a9a (0x583e bytes) 000983 1788931449.337192 mapread 0x0181b4 thru 0x018de6 (0xc33 bytes) 000984 1788931449.339054 trunc from 0x02be08 to 0x00432a (0x27add bytes) 000985 1788931449.365181 trunc from 0x00432b to 0x0395f4 (0x352ca bytes) 000986 1788931449.381759 trunc from 0x0395f5 to 0x01c7f8 (0x1cdfc bytes) 000987 1788931449.401023 read 0x0100cc thru 0x01c7f8 (0xc72d bytes) 000989 1788931449.421765 write 0x0113cd thru 0x01aabb (0x96ef bytes) 000990 1788931449.426997 write 0x01089b thru 0x0205be (0xfd24 bytes) 000991 1788931449.435275 read 0x0133bd thru 0x01e037 (0xac7b bytes) 000992 1788931449.437003 mapwrite 0x01bec9 thru 0x026917 (0xaa4f bytes) 000994 1788931449.512183 fallocate 0x012ace thru 0x013b8b (0x10be bytes) 000995 1788931449.522780 trunc from 0x0147b2 to 0x02405d (0xf8ac bytes) 000996 1788931449.535003 trunc from 0x026918 to 0x008e2f (0x1dae8 bytes) 000997 1788931449.570122 mapread 0x006a7a thru 0x008e2f (0x23b6 bytes) 000998 1788931449.572703 mapwrite 0x03f000 thru 0x03ffff (0x1000 bytes) 000999 1788931449.626773 read 0x028a79 thru 0x02e689 (0x5c11 bytes) 001000 1788931449.627705 punch from 0x03ecfe to 0x03ffff (0x1302 bytes) All operations completed A-OK! PASS 70a (165s) == sanity-flr test 71: check mirror extend parent fid ==== 01:24:14 (1788931454) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x181:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 01:24:25 (1788931465) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20602 s, 10.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136676 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 01:24:34 (1788931474) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 786912 .. Extending file size to 2081824 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5355680 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5627552 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8800960 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9079200 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9245952 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9118688 .. Waiting 61329 61330 61332 61333 61335 done 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) 192.168.204.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:26:00 (1788931560) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7356288 .. Extending file size to 3037824 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3888512 .. failed Extending file size to 8608864 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8189632 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8365984 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9287168 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8510656 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9302080 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8641216 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63013 63014 63015 63016 63017 done 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) 192.168.204.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:27:23 (1788931643) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17944 3589076 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26060 3580960 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44004 7170036 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17944 3589076 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 26060 3580960 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44004 7170036 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107733 s, 19.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251048 s, 8.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8dd5e258 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:27:46 (1788931666) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156015 s, 13.4 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 01:27:55 (1788931675) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:28:04 (1788931684) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152524 s, 13.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:28:13 (1788931693) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125114 s, 16.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:28:23 (1788931703) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0840576 s, 24.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:28:55 (1788931735) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg409-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849556 s, 12.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:29:41 (1788931781) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg409-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179163 s, 11.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:30:32 (1788931832) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg409-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg409-server: Pool lustre.test_204f created Adding targets to pool oleg409-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg409-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122381 s, 17.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg409-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg409-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg409-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:31:41 (1788931901) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:31:48 (1788931908) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:31:56 (1788931916) oleg409-server: Pool lustre.test_206 created Adding targets to pool oleg409-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg409-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg409-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg409-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg409-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:32:24 (1788931944) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725961 s, 14.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431612 s, 24.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8510d954 Mirror 3: 0x8510d954 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:32:34 (1788931954) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:32:38 (1788931958) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:32:41 (1788931961) Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0660146 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1580000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1580000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1580000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f90000 Read this (4 bytes): BBBB 192.168.204.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg409-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:33:20 (1788932000) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0445319 s, 184 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:33:28 (1788932008) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697049 s, 15.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611262 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 01:33:30.000000000 -0400 Modify: 2026-09-09 01:33:30.000000000 -0400 Change: 2026-09-09 01:33:31.000000000 -0400 Birth: 2026-09-09 01:33:30.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:33:36 (1788932016) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268842 s, 39.0 MB/s oleg409-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg409-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg409-server: O/280000401/d27/13b: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:33:58 (1788932038) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.278648 s, 147 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0306191 s, 134 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 9 01:34 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x83:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 9 01:34 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:34:07 (1788932047) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:34:17 (1788932057) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (16s) == sanity-flr test complete, duration 2019 sec =========== 01:34:33 (1788932073) === sanity-flr: start cleanup 01:34:36 (1788932076) === === sanity-flr: finish cleanup 01:34:49 (1788932089) ===