-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 22:51:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg257-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:51:33 (1789181493) === oleg257-client.virtnet: executing check_config_client /mnt/lustre oleg257-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg257-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff969a4bae4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff969a4bae4000.idle_timeout=debug disable quota as required oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:52:18 (1789181538) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8224] == sanity-flr test 0a: lfs mirror create with -N option == 22:52:21 (1789181541) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:52:42 (1789181562) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:52:46 (1789181566) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:52:50 (1789181570) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:53:18 (1789181598) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:53:23 (1789181603) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:53:27 (1789181607) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff969a4bae4000.stats=clear osc.lustre-OST0001-osc-ffff969a4bae4000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:53:37 (1789181617) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:53:49 (1789181629) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:53:57 (1789181637) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:54:00 (1789181640) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:54:09 (1789181649) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (15s) == sanity-flr test 2: create components from existing files ========================================================== 22:54:24 (1789181664) PASS 2 (11s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:54:35 (1789181675) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:54:45 (1789181685) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:54:56 (1789181696) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:55:08 (1789181708) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:55:16 (1789181716) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.3402 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143858 s, 7.3 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:55:28 (1789181728) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.252185 s, 24.9 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:55:38 (1789181738) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:55:52 (1789181752) Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg257-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:57:14 (1789181834) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.203248 s, 15.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:58:00 (1789181880) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg257-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:59:24 (1789181964) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.61067 s, 14.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 22:59 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.224s user 0m0.001s sys 0m0.213s Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff969a4bae4000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.054s user 0m0.001s sys 0m0.184s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.960s user 0m0.003s sys 0m0.194s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg257-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg257-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff969a4bae4000.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:01:07 (1789182067) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:01:11 (1789182071) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:01:15 (1789182075) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:01:18 (1789182078) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:01:26 (1789182086) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.424007 s, 29.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:01:37 (1789182097) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286246 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256633 s, 40.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:01:44 (1789182104) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:01:51 (1789182111) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165796 s, 618 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00182611 s, 561 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00314287 s, 326 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000982512 s, 1.0 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 23:02:00 (1789182120) create test files with size 15735917 .. 300e38771b2bd49fedb00c2a3277c860 - 274a8528da04c0950b2323b4b5986ab2 - dc93b66f427c29386d44d2032c7e186a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg257-client.virtnet /mnt/lustre (opts:) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 23:02:27 (1789182147) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:02:42 (1789182162) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:02:48 (1789182168) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523338 s, 20.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627203 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 23:02:58 (1789182178) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0552259 s, 38.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.248736 s, 16.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151336 s, 13.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 23:03:12 (1789182192) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 23:03:15 (1789182195) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:03:18 (1789182198) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445858 s, 23.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 23:03:25 (1789182205) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.292177 s, 10.8 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:03:35 (1789182215) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:03:47 (1789182227) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.36462 s, 28.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg257-client.virtnet /mnt/lustre (opts:) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:04:00 (1789182240) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff969a46082000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.207617 s, 50.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg257-client.virtnet /mnt/lustre (opts:) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff969a44948800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:04:18 (1789182258) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:04:59 (1789182299) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:05:35 (1789182335) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 23:05:38 (1789182338) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.27947 s, 11.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.112363 s, 28.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9da8f304 (nosync,stale) Mirror 2: 0x9da8f304 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0706c0bcb47ea8a14b95e64cab1093f8 - mirror 1 checksum: e7bd74499b5734cd10b22c73538a0d6c - mirror 2 checksum: 0706c0bcb47ea8a14b95e64cab1093f8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T23:05:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T23:05:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0706c0bcb47ea8a14b95e64cab1093f8 - mirror 1 checksum: e7bd74499b5734cd10b22c73538a0d6c - mirror 2 checksum: e7bd74499b5734cd10b22c73538a0d6c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T23:05:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T23:05:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:05:49 (1789182349) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391337 s, 26.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:05:57 (1789182357) Failing mds1 on oleg257-server Stopping /mnt/lustre-mds1 (opts:) on oleg257-server 23:06:10 (1789182370) shut down facet: mds1 facet_host: oleg257-server facet_failover_host: oleg257-server Failover mds1 to oleg257-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 23:06:42 (1789182402) targets are mounted 23:06:42 (1789182402) facet_failover done oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325604 s, 32.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:06:59 (1789182419) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.046229 s, 443 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0174061 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:07:08 (1789182428) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0720686 s, 284 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0167237 s, 1.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:07:19 (1789182439) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00603081 s, 2.7 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:07:28 (1789182448) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0409076 s, 501 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:07:38 (1789182458) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00799908 s, 512 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0354378 s, 116 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:07:47 (1789182467) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:08:12 (1789182492) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:08:43 (1789182523) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:09:10 (1789182550) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:09:36 (1789182576) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 23:09:44 (1789182584) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57599 mirror split pid 57600 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57599 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57600 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 23:10:53 (1789182653) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2656 fallocating to largest ever: 0x40000 000001 1789182656.645603 fallocate 0x036121 thru 0x03ffff (0x9edf bytes) 000002 1789182656.666184 write 0x00cfd6 thru 0x00f291 (0x22bc bytes) 000003 1789182656.671138 fallocate 0x000466 thru 0x001804 (0x139f bytes) 000004 1789182656.678152 mapread 0x00ef31 thru 0x00f291 (0x361 bytes) 000005 1789182656.682434 mapread 0x002cd5 thru 0x00d954 (0xac80 bytes) 000006 1789182656.754153 fallocate 0x017285 thru 0x018bae (0x192a bytes) 000007 1789182656.770403 fallocate 0x009fcc thru 0x011a42 (0x7a77 bytes) 000008 1789182656.786115 mapread 0x00ebd3 thru 0x010b27 (0x1f55 bytes) 000009 1789182656.808527 mapread 0x007e34 thru 0x009211 (0x13de bytes) 000010 1789182656.811409 trunc from 0x018baf to 0x025ab6 (0xcf08 bytes) 000012 1789182656.840122 mapread 0x0241d0 thru 0x025ab6 (0x18e7 bytes) 000013 1789182656.858625 fallocate 0x01daf1 thru 0x023207 (0x5717 bytes) 000014 1789182656.877146 mapwrite 0x022668 thru 0x02d087 (0xaa20 bytes) 000016 1789182657.008381 mapread 0x01c9ba thru 0x026c44 (0xa28b bytes) 000017 1789182657.017069 trunc from 0x006377 to 0x011c9a (0xb924 bytes) 000018 1789182657.024201 fallocate 0x038947 thru 0x03ffff (0x76b9 bytes) 000019 1789182657.032075 trunc from 0x0212d2 to 0x021ebe (0xbed bytes) 000020 1789182657.044844 trunc from 0x03c5dc to 0x03ffff (0x3a24 bytes) 000021 1789182657.060366 trunc from 0x040000 to 0x0017b2 (0x3e84d bytes) 000022 1789182657.080029 trunc from 0x0017b3 to 0x004d54 (0x35a2 bytes) 000023 1789182657.100426 trunc from 0x004c99 to 0x004d54 (0xbc bytes) 000024 1789182657.108709 mapwrite 0x01bfb6 thru 0x01fad9 (0x3b24 bytes) 000025 1789182657.199480 fallocate 0x03f15a thru 0x03ffff (0xea6 bytes) 000026 1789182657.223178 punch from 0x02ed2e to 0x031003 (0x22d6 bytes) 000027 1789182657.236388 trunc from 0x040000 to 0x01722c (0x28dd3 bytes) 000030 1789182657.263649 punch from 0x00ed42 to 0x01722c (0x84eb bytes) 000031 1789182657.274888 trunc from 0x016e81 to 0x01722c (0x3ac bytes) 000032 1789182657.285887 read 0x00f2a7 thru 0x012785 (0x34df bytes) 000034 1789182657.323578 punch from 0x00820c to 0x00a77e (0x2573 bytes) 000035 1789182657.344036 mapread 0x00772c thru 0x00931a (0x1bef bytes) 000036 1789182657.370289 trunc from 0x00cdc7 to 0x0114ae (0x46e8 bytes) 000037 1789182657.377522 trunc from 0x01722d to 0x01f7ac (0x8580 bytes) 000038 1789182657.391659 mapread 0x003111 thru 0x00fd28 (0xcc18 bytes) 000039 1789182657.406814 punch from 0x002051 to 0x003323 (0x12d3 bytes) 000040 1789182657.414157 fallocate 0x009065 thru 0x00db94 (0x4b30 bytes) 000041 1789182657.432843 write 0x00860b thru 0x00c374 (0x3d6a bytes) 000042 1789182657.435497 trunc from 0x01f7ad to 0x0076fa (0x180b2 bytes) 000043 1789182657.469419 punch from 0x005f55 to 0x0076fa (0x17a6 bytes) 000044 1789182657.477355 read 0x0032e9 thru 0x0076fa (0x4412 bytes) 000045 1789182657.488094 trunc from 0x000bff to 0x005dd5 (0x51d7 bytes) 000046 1789182657.495731 trunc from 0x0076fb to 0x009a31 (0x2337 bytes) 000047 1789182657.511984 trunc from 0x009a32 to 0x01e94c (0x14f1b bytes) 000048 1789182657.529458 read 0x010469 thru 0x015b58 (0x56f0 bytes) 000049 1789182657.541952 mapwrite 0x024be9 thru 0x03215e (0xd576 bytes) 000050 1789182657.599914 punch from 0x02db22 to 0x03215e (0x463d bytes) 000051 1789182657.620786 trunc from 0x03215f to 0x03c56c (0xa40e bytes) 000052 1789182657.655637 read 0x012d4e thru 0x01d622 (0xa8d5 bytes) 000053 1789182657.656389 mapread 0x00c4d8 thru 0x015846 (0x936f bytes) 000054 1789182657.657355 trunc from 0x03c56d to 0x01f271 (0x1d2fb bytes) 000055 1789182657.702889 trunc from 0x017707 to 0x01c3dc (0x4cd6 bytes) 000057 1789182657.710424 write 0x01ae2b thru 0x025fed (0xb1c3 bytes) 000058 1789182657.730579 trunc from 0x0159b6 to 0x024512 (0xeb5d bytes) 000059 1789182657.755992 read 0x0015c5 thru 0x00ca8c (0xb4c8 bytes) 000060 1789182657.756980 read 0x023d4e thru 0x025fed (0x22a0 bytes) 000061 1789182657.770258 mapwrite 0x00bd6a thru 0x00c008 (0x29f bytes) 000062 1789182657.786325 mapread 0x013952 thru 0x01ea85 (0xb134 bytes) 000064 1789182657.790373 read 0x011ff5 thru 0x01ccef (0xacfb bytes) 000065 1789182657.791822 mapread 0x01ec4e thru 0x025fed (0x73a0 bytes) 000066 1789182657.794125 fallocate 0x001117 thru 0x0076d3 (0x65bd bytes) 000067 1789182657.802671 read 0x01b2f6 thru 0x0230fd (0x7e08 bytes) 000068 1789182657.803700 trunc from 0x025fee to 0x0372aa (0x112bd bytes) 000069 1789182657.827124 fallocate 0x0098b1 thru 0x013a8e (0xa1de bytes) 000070 1789182657.835377 mapwrite 0x01c799 thru 0x020495 (0x3cfd bytes) 000073 1789182657.914933 mapwrite 0x022794 thru 0x0298ba (0x7127 bytes) 000074 1789182658.051874 mapwrite 0x0211bd thru 0x02e779 (0xd5bd bytes) 000075 1789182658.138775 read 0x01f016 thru 0x02a8ce (0xb8b9 bytes) 000076 1789182658.139896 punch from 0x0245dc to 0x025555 (0xf7a bytes) 000077 1789182658.154494 trunc from 0x02dadd to 0x035187 (0x76ab bytes) 000078 1789182658.167234 write 0x02951a thru 0x032335 (0x8e1c bytes) 000079 1789182658.190293 read 0x031bed thru 0x0372aa (0x56be bytes) 000081 1789182658.206632 fallocate 0x021475 thru 0x0237aa (0x2336 bytes) 000083 1789182658.230126 trunc from 0x0372ab to 0x01affa (0x1c2b0 bytes) 000084 1789182658.258396 trunc from 0x01affb to 0x03d724 (0x2272a bytes) 000087 1789182658.276012 punch from 0x03632c to 0x03d724 (0x73f9 bytes) 000088 1789182658.291374 MIRROR_EXTEND to 2 mirrors 000089 1789182659.371029 mapwrite 0x00f4e8 thru 0x01aea7 (0xb9c0 bytes) 000090 1789182659.614064 trunc from 0x000657 to 0x004ace (0x4478 bytes) 000091 1789182659.644048 fallocate 0x016d83 thru 0x017b0b (0xd89 bytes) 000092 1789182659.663380 fallocate 0x0110b8 thru 0x01b34d (0xa296 bytes) 000093 1789182659.676160 read 0x03701f thru 0x03d724 (0x6706 bytes) 000094 1789182659.692420 read 0x02f778 thru 0x03179e (0x2027 bytes) 000095 1789182659.705599 mapwrite 0x03ac1d thru 0x03ffff (0x53e3 bytes) 000096 1789182659.816517 read 0x014315 thru 0x01bc34 (0x7920 bytes) 000097 1789182659.818051 trunc from 0x040000 to 0x023211 (0x1cdee bytes) 000098 1789182659.850033 fallocate 0x0387aa thru 0x03ffff (0x7856 bytes) 000099 1789182659.869488 fallocate 0x01736a thru 0x023d16 (0xc9ad bytes) 000100 1789182659.879484 read 0x0352d6 thru 0x03f012 (0x9d3d bytes) 000101 1789182659.899972 fallocate 0x01ece8 thru 0x022803 (0x3b1c bytes) 000102 1789182659.907976 punch from 0x01815c to 0x01cd73 (0x4c18 bytes) 000103 1789182659.918085 trunc from 0x01a21a to 0x023be1 (0x99c8 bytes) 000104 1789182659.928210 trunc from 0x040000 to 0x02e9f0 (0x1160f bytes) 000105 1789182659.969456 trunc from 0x02e9f1 to 0x016c6a (0x17d86 bytes) 000106 1789182660.019396 read 0x0018e9 thru 0x00fb00 (0xe218 bytes) 000107 1789182660.021093 fallocate 0x014222 thru 0x017a4b (0x382a bytes) 000108 1789182660.038239 MIRROR_RESYNC 2 mirrors 000109 1789182661.130083 fallocate 0x03ee99 thru 0x03ffff (0x1167 bytes) 000110 1789182661.234556 MIRROR_RESYNC 2 mirrors 000111 1789182662.901339 read 0x015362 thru 0x017a4b (0x26ea bytes) 000112 1789182662.910141 write 0x0040b8 thru 0x00d0cd (0x9016 bytes) 000113 1789182663.044473 mapread 0x00ebfc thru 0x00f541 (0x946 bytes) 000114 1789182663.077068 punch from 0x007108 to 0x009627 (0x2520 bytes) 000115 1789182663.134648 read 0x0134be thru 0x013f01 (0xa44 bytes) 000116 1789182663.173054 trunc from 0x017a4c to 0x00d551 (0xa4fa bytes) 000117 1789182663.224727 punch from 0x003d14 to 0x0088b4 (0x4ba1 bytes) 000118 1789182663.233497 fallocate 0x019a17 thru 0x01eb4d (0x5137 bytes) 000119 1789182663.239636 trunc from 0x008974 to 0x00d551 (0x4bde bytes) 000120 1789182663.262973 MIRROR_RESYNC 2 mirrors 000121 1789182664.508573 fallocate 0x002a12 thru 0x01178c (0xed7b bytes) 000122 1789182664.592014 punch from 0x00b98e to 0x01178c (0x5dff bytes) 000123 1789182664.604564 fallocate 0x024591 thru 0x029c7a (0x56ea bytes) 000124 1789182664.613485 trunc from 0x01178d to 0x00aa5c (0x6d30 bytes) 000125 1789182664.678905 write 0x02bb83 thru 0x02ca40 (0xebe bytes) 000126 1789182664.692039 read 0x022662 thru 0x02575e (0x30fd bytes) 000127 1789182664.700773 fallocate 0x008851 thru 0x01743a (0xebea bytes) 000128 1789182664.705724 punch from 0x0236ac to 0x02b8c8 (0x821d bytes) 000129 1789182664.724442 trunc from 0x0179cc to 0x01828f (0x8c4 bytes) 000130 1789182664.747656 MIRROR_RESYNC 2 mirrors 000131 1789182665.129067 trunc from 0x02ca41 to 0x01d123 (0xf91d bytes) 000132 1789182665.252918 mapread 0x005f16 thru 0x010dc9 (0xaeb4 bytes) 000133 1789182665.285143 mapread 0x00bbb7 thru 0x01a43f (0xe889 bytes) 000134 1789182665.298344 fallocate 0x03d47d thru 0x03e8cc (0x1450 bytes) 000135 1789182665.304385 mapwrite 0x002492 thru 0x00bb6e (0x96dd bytes) 000136 1789182665.322785 mapread 0x000806 thru 0x008f18 (0x8713 bytes) 000137 1789182665.327238 punch from 0x02999c to 0x02e13c (0x47a1 bytes) 000138 1789182665.331495 write 0x008c9c thru 0x00eab4 (0x5e19 bytes) 000139 1789182665.339672 MIRROR_RESYNC 2 mirrors 000140 1789182666.578592 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000141 1789182666.888383 write 0x0251dc thru 0x033ad6 (0xe8fb bytes) 000142 1789182666.923387 mapread 0x02971f thru 0x033fe4 (0xa8c6 bytes) 000143 1789182666.931984 trunc from 0x02fa6c to 0x03e190 (0xe725 bytes) 000145 1789182666.963396 MIRROR_EXTEND to 2 mirrors 000146 1789182668.100020 trunc from 0x03e8cd to 0x01ae88 (0x23a44 bytes) 000147 1789182668.262529 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000148 1789182668.600659 trunc from 0x01ae89 to 0x03ff26 (0x2509e bytes) 000149 1789182668.655642 read 0x008952 thru 0x00d13f (0x47ee bytes) 000150 1789182668.677077 trunc from 0x03ff27 to 0x039a8c (0x649a bytes) 000151 1789182668.700053 read 0x021439 thru 0x023bf4 (0x27bc bytes) 000152 1789182668.722044 read 0x01b1ca thru 0x0251b5 (0x9fec bytes) 000153 1789182668.741626 MIRROR_EXTEND to 2 mirrors 000154 1789182669.695624 punch from 0x0367ac to 0x039a8c (0x32e1 bytes) 000155 1789182669.738715 mapwrite 0x00b0bd thru 0x019472 (0xe3b6 bytes) 000156 1789182669.847581 punch from 0x01c126 to 0x0271bd (0xb098 bytes) 000157 1789182669.851384 trunc from 0x007da7 to 0x012dd5 (0xb02f bytes) 000158 1789182669.861613 read 0x02e429 thru 0x02f985 (0x155d bytes) 000160 1789182669.880357 MIRROR_RESYNC 2 mirrors 000161 1789182671.017671 write 0x015614 thru 0x01a98e (0x537b bytes) 000162 1789182671.096306 read 0x03652c thru 0x039a8c (0x3561 bytes) 000163 1789182671.111252 mapwrite 0x01cfe8 thru 0x02c713 (0xf72c bytes) 000164 1789182671.235236 read 0x023502 thru 0x025160 (0x1c5f bytes) 000165 1789182671.245286 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000166 1789182671.553074 trunc from 0x01f0c8 to 0x02ca2d (0xd966 bytes) 000167 1789182671.568053 mapwrite 0x01ff21 thru 0x028cbe (0x8d9e bytes) 000168 1789182671.685879 fallocate 0x0355e3 thru 0x039f8b (0x49a9 bytes) 000169 1789182671.706721 punch from 0x02afa5 to 0x039c3f (0xec9b bytes) 000170 1789182671.716432 read 0x020c94 thru 0x02c0e7 (0xb454 bytes) 000172 1789182671.740250 read 0x0334c8 thru 0x039f8b (0x6ac4 bytes) 000173 1789182671.771321 trunc from 0x039f8c to 0x014c8d (0x252fe bytes) 000174 1789182671.803839 mapwrite 0x020d8b thru 0x0233ff (0x2675 bytes) 000175 1789182671.966952 trunc from 0x023400 to 0x006600 (0x1cdff bytes) 000176 1789182672.000000 mapread 0x000286 thru 0x00185a (0x15d5 bytes) 000178 1789182672.001383 mapwrite 0x01cc3e thru 0x029a03 (0xcdc6 bytes) 000179 1789182672.274794 mapwrite 0x00010d thru 0x00a857 (0xa74b bytes) 000180 1789182672.390616 read 0x013754 thru 0x01b739 (0x7fe6 bytes) 000181 1789182672.404340 mapwrite 0x0122f7 thru 0x017ae3 (0x57ed bytes) 000182 1789182672.464680 trunc from 0x0142b8 to 0x019382 (0x50cb bytes) 000183 1789182672.488485 punch from 0x024db5 to 0x029a03 (0x4c4f bytes) 000185 1789182672.506786 trunc from 0x0181a3 to 0x01d6bf (0x551d bytes) 000186 1789182672.529645 fallocate 0x007344 thru 0x00cd05 (0x59c2 bytes) 000187 1789182672.538886 read 0x000e86 thru 0x00f9e8 (0xeb63 bytes) 000189 1789182672.551901 trunc from 0x029a04 to 0x03cedb (0x134d8 bytes) 000190 1789182672.574016 punch from 0x025449 to 0x03458f (0xf147 bytes) 000191 1789182672.609014 punch from 0x00ca63 to 0x0128cd (0x5e6b bytes) 000192 1789182672.633680 fallocate 0x01ae18 thru 0x0279d8 (0xcbc1 bytes) 000194 1789182672.659286 mapwrite 0x02e32f thru 0x0364e9 (0x81bb bytes) 000195 1789182672.764518 trunc from 0x03cedc to 0x032699 (0xa842 bytes) 000197 1789182672.795956 trunc from 0x03269a to 0x000d09 (0x31990 bytes) 000198 1789182672.893011 punch from 0x000aa5 to 0x000d09 (0x265 bytes) 000199 1789182672.910607 punch from 0x000c76 to 0x000d09 (0x94 bytes) 000202 1789182672.933699 trunc from 0x000d0a to 0x034c21 (0x33f18 bytes) 000203 1789182673.001556 fallocate 0x03fc5c thru 0x03ffff (0x3a4 bytes) 000204 1789182673.028524 trunc from 0x01f836 to 0x022bb6 (0x3381 bytes) 000205 1789182673.047580 mapwrite 0x003223 thru 0x005254 (0x2032 bytes) 000206 1789182673.158751 read 0x030fcd thru 0x03ac06 (0x9c3a bytes) 000207 1789182673.212049 punch from 0x03c744 to 0x03ffff (0x38bc bytes) 000209 1789182673.230603 trunc from 0x040000 to 0x0299f7 (0x16608 bytes) 000210 1789182673.284360 read 0x027a87 thru 0x0299f7 (0x1f71 bytes) 000211 1789182673.291132 trunc from 0x02229b to 0x0299f7 (0x775d bytes) 000212 1789182673.306429 trunc from 0x0299f8 to 0x0187cf (0x11228 bytes) 000213 1789182673.344497 read 0x000eb0 thru 0x00c3a5 (0xb4f6 bytes) 000214 1789182673.345216 trunc from 0x00f2cb to 0x0185af (0x92e5 bytes) 000215 1789182673.351532 trunc from 0x0158ff to 0x017c05 (0x2307 bytes) 000216 1789182673.356950 punch from 0x004173 to 0x010b12 (0xc9a0 bytes) 000217 1789182673.367106 MIRROR_EXTEND to 2 mirrors 000218 1789182674.596328 mapwrite 0x03f76a thru 0x03ffff (0x896 bytes) 000219 1789182674.755253 mapwrite 0x004f87 thru 0x00bf64 (0x6fde bytes) 000220 1789182674.810520 trunc from 0x040000 to 0x026947 (0x196b8 bytes) 000221 1789182674.841043 write 0x012fa4 thru 0x01f820 (0xc87d bytes) 000222 1789182674.853616 trunc from 0x026948 to 0x03a247 (0x13900 bytes) 000223 1789182674.870491 trunc from 0x03457c to 0x037062 (0x2ae7 bytes) 000224 1789182674.880684 mapwrite 0x033126 thru 0x03ffff (0xceda bytes) 000225 1789182675.004439 trunc from 0x0016dc to 0x005f7d (0x48a2 bytes) 000226 1789182675.013337 trunc from 0x040000 to 0x006213 (0x39dec bytes) 000227 1789182675.030458 write 0x029606 thru 0x02e972 (0x536d bytes) 000228 1789182675.038684 write 0x00065d thru 0x00b240 (0xabe4 bytes) 000229 1789182675.064036 write 0x018209 thru 0x027395 (0xf18d bytes) 000230 1789182675.087749 punch from 0x022de3 to 0x02ca62 (0x9c80 bytes) 000231 1789182675.109964 punch from 0x015584 to 0x0235ac (0xe029 bytes) 000232 1789182675.120009 trunc from 0x02443b to 0x02e972 (0xa538 bytes) 000233 1789182675.131409 MIRROR_RESYNC 2 mirrors 000234 1789182675.842412 MIRROR_EXTEND to 3 mirrors 000236 1789182676.782485 mapwrite 0x019f41 thru 0x021d1a (0x7dda bytes) 000237 1789182676.868238 trunc from 0x02e973 to 0x022d79 (0xbbf9 bytes) 000238 1789182676.908365 punch from 0x019161 to 0x01fd52 (0x6bf2 bytes) 000240 1789182676.912689 fallocate 0x0282c1 thru 0x02c43b (0x417b bytes) 000242 1789182676.916290 punch from 0x01d5a5 to 0x025140 (0x7b9c bytes) 000243 1789182676.920463 read 0x002d15 thru 0x00d8d7 (0xabc3 bytes) 000244 1789182676.932295 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c43c): Mirror 3: 0x59356d02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c43c): Mirror 3: 0x59356d02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c43c: skipped 000246 1789182677.185951 mapread 0x004638 thru 0x012720 (0xe0e9 bytes) 000247 1789182677.209184 fallocate 0x021567 thru 0x0281d0 (0x6c6a bytes) 000248 1789182677.212284 fallocate 0x03adc9 thru 0x03ffff (0x5237 bytes) 000249 1789182677.215299 mapread 0x00d407 thru 0x016a29 (0x9623 bytes) 000250 1789182677.218771 trunc from 0x040000 to 0x00117e (0x3ee81 bytes) 000251 1789182677.239221 trunc from 0x00117f to 0x00fd38 (0xebba bytes) 000252 1789182677.255029 read 0x00682f thru 0x00de87 (0x7659 bytes) 000254 1789182677.264953 trunc from 0x00fd39 to 0x02e9d8 (0x1eca0 bytes) 000255 1789182677.278115 write 0x023ab8 thru 0x026950 (0x2e99 bytes) 000257 1789182677.285371 punch from 0x017425 to 0x018b3b (0x1717 bytes) 000258 1789182677.292350 mapwrite 0x004588 thru 0x009a6f (0x54e8 bytes) 000260 1789182677.351438 read 0x000120 thru 0x000a0c (0x8ed bytes) 000261 1789182677.379318 trunc from 0x02bff4 to 0x02e9d8 (0x29e5 bytes) 000262 1789182677.387582 mapread 0x014e49 thru 0x01cd34 (0x7eec bytes) 000263 1789182677.388715 punch from 0x02351d to 0x02e9d8 (0xb4bc bytes) 000264 1789182677.396643 mapread 0x012196 thru 0x012ba4 (0xa0f bytes) 000265 1789182677.399477 write 0x00c959 thru 0x018d35 (0xc3dd bytes) 000266 1789182677.407696 write 0x011749 thru 0x01a026 (0x88de bytes) 000268 1789182677.411506 write 0x03fe8d thru 0x03ffff (0x173 bytes) 000269 1789182677.417622 read 0x018792 thru 0x020c8a (0x84f9 bytes) 000270 1789182677.418454 mapwrite 0x00bdd9 thru 0x0146e4 (0x890c bytes) 000271 1789182677.498120 mapread 0x00e46e thru 0x0147cf (0x6362 bytes) 000272 1789182677.502274 write 0x03b4c7 thru 0x03ffff (0x4b39 bytes) 000273 1789182677.523169 write 0x02f79c thru 0x031dbe (0x2623 bytes) 000274 1789182677.524302 trunc from 0x040000 to 0x00900d (0x36ff2 bytes) 000275 1789182677.559627 punch from 0x005b55 to 0x00900d (0x34b9 bytes) 000277 1789182677.574330 mapread 0x000cc4 thru 0x00900d (0x834a bytes) 000278 1789182677.612583 trunc from 0x00900e to 0x0031a1 (0x5e6c bytes) 000279 1789182677.641603 mapread 0x0030a5 thru 0x0031a1 (0xfd bytes) 000280 1789182677.646530 trunc from 0x0031a2 to 0x01f1df (0x1c03e bytes) 000283 1789182677.700238 write 0x0360ab thru 0x036f77 (0xecd bytes) 000284 1789182677.706936 write 0x015a7c thru 0x01b9bd (0x5f42 bytes) 000285 1789182677.728580 mapread 0x01fc55 thru 0x020f32 (0x12de bytes) 000286 1789182677.751699 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36f78): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f78): Mirror 3: 0x8560a09a (stale) CRC-32 checksum value for chunk [0, 0x36f78): Mirror 1: 0xbb2b82b5 Mirror 4: 0xbb2b82b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f78: skipped 000287 1789182679.692515 mapread 0x00d9ab thru 0x01b6f6 (0xdd4c bytes) 000288 1789182679.792440 MIRROR_RESYNC 3 mirrors 000289 1789182680.957285 trunc from 0x036f78 to 0x011c19 (0x2535e bytes) 000291 1789182681.041380 write 0x00d1db thru 0x0114ca (0x42f0 bytes) 000293 1789182681.066281 punch from 0x004282 to 0x00c136 (0x7eb5 bytes) 000294 1789182681.073404 fallocate 0x01a02d thru 0x01e94a (0x491e bytes) 000295 1789182681.086399 write 0x01f1fa thru 0x02d2dd (0xe0e4 bytes) 000296 1789182681.088367 read 0x00d7b6 thru 0x01c16f (0xe9ba bytes) 000297 1789182681.119327 trunc from 0x02d2de to 0x038599 (0xb2bc bytes) 000298 1789182681.154407 read 0x02c52d thru 0x037c89 (0xb75d bytes) 000300 1789182681.168127 trunc from 0x03859a to 0x017b33 (0x20a66 bytes) 000302 1789182681.188936 fallocate 0x02847a thru 0x03224e (0x9dd5 bytes) 000303 1789182681.195433 trunc from 0x002309 to 0x00f28d (0xcf85 bytes) 000304 1789182681.226179 mapwrite 0x012b4c thru 0x01534b (0x2800 bytes) 000305 1789182681.249466 trunc from 0x002364 to 0x00e7fe (0xc49b bytes) 000306 1789182681.262623 mapwrite 0x01e676 thru 0x02718e (0x8b19 bytes) 000307 1789182681.441518 mapwrite 0x03a13c thru 0x03e22f (0x40f4 bytes) 000308 1789182681.530689 trunc from 0x0092ad to 0x010240 (0x6f94 bytes) 000309 1789182681.542316 mapread 0x034ba3 thru 0x035978 (0xdd6 bytes) 000310 1789182681.560537 mapwrite 0x00aec4 thru 0x00fbc1 (0x4cfe bytes) 000311 1789182681.666234 write 0x03a05d thru 0x03ffff (0x5fa3 bytes) 000312 1789182681.705596 MIRROR_RESYNC 3 mirrors 000313 1789182683.899953 mapread 0x03c86f thru 0x03ffff (0x3791 bytes) 000315 1789182683.955295 mapread 0x0156e2 thru 0x015ef6 (0x815 bytes) 000316 1789182683.968191 write 0x035c4b thru 0x03df55 (0x830b bytes) 000317 1789182684.062546 mapwrite 0x03a0af thru 0x03ffff (0x5f51 bytes) 000318 1789182684.111657 punch from 0x02681d to 0x02983d (0x3021 bytes) 000319 1789182684.131662 punch from 0x00ef52 to 0x01acf6 (0xbda5 bytes) 000320 1789182684.141584 punch from 0x031f4e to 0x03ffff (0xe0b2 bytes) 000321 1789182684.151700 mapwrite 0x017b08 thru 0x01fd03 (0x81fc bytes) 000322 1789182684.194796 trunc from 0x040000 to 0x024591 (0x1ba6e bytes) 000323 1789182684.262513 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24592): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24592): Mirror 3: 0xbe174bbf (stale) Mirror 4: 0xbe174bbf (stale) CRC-32 checksum value for chunk [0, 0x24592): Mirror 1: 0xb0e9115 Mirror 5: 0xb0e9115 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24592: skipped 000324 1789182686.590685 write 0x02289a thru 0x024b5b (0x22c2 bytes) 000325 1789182686.687596 read 0x0158c8 thru 0x01a600 (0x4d39 bytes) 000327 1789182686.713847 write 0x03cb71 thru 0x03ffff (0x348f bytes) 000328 1789182686.717016 trunc from 0x040000 to 0x00f2df (0x30d20 bytes) 000329 1789182686.772251 write 0x00bb9a thru 0x018e03 (0xd26a bytes) 000330 1789182686.786338 punch from 0x0058e3 to 0x008f99 (0x36b7 bytes) 000331 1789182686.798842 fallocate 0x0390fe thru 0x03ffff (0x6f02 bytes) 000332 1789182686.807395 trunc from 0x0168b7 to 0x01eedc (0x8626 bytes) 000335 1789182686.859081 read 0x02d9a5 thru 0x03bf83 (0xe5df bytes) 000336 1789182686.970034 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xfd533b1c (stale) Mirror 5: 0x87d2e1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xfd533b1c (stale) Mirror 5: 0x87d2e1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000337 1789182687.753731 punch from 0x03deba to 0x03ffff (0x2146 bytes) 000338 1789182687.760522 trunc from 0x02a4f9 to 0x036936 (0xc43e bytes) 000339 1789182687.768105 punch from 0x03a461 to 0x03ffff (0x5b9f bytes) 000340 1789182687.788255 mapwrite 0x03ed3f thru 0x03ffff (0x12c1 bytes) 000342 1789182687.837838 trunc from 0x040000 to 0x00e6bf (0x31940 bytes) 000343 1789182687.875600 write 0x0071fe thru 0x0127cb (0xb5ce bytes) 000344 1789182687.887402 punch from 0x002e8c to 0x00f9f9 (0xcb6e bytes) 000345 1789182687.908592 write 0x01c4ea thru 0x027f80 (0xba97 bytes) 000346 1789182687.913888 read 0x00d9af thru 0x01b750 (0xdda2 bytes) 000347 1789182687.924602 punch from 0x002726 to 0x005add (0x33b8 bytes) 000348 1789182687.930491 punch from 0x00bb91 to 0x0155fe (0x9a6e bytes) 000349 1789182687.937543 trunc from 0x01ac9f to 0x027824 (0xcb86 bytes) 000350 1789182687.961414 fallocate 0x031ced thru 0x03b790 (0x9aa4 bytes) 000351 1789182687.970288 mapwrite 0x026a2c thru 0x027bd2 (0x11a7 bytes) 000352 1789182688.008243 write 0x020efc thru 0x0292ca (0x83cf bytes) 000353 1789182688.020252 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b791): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b791): Mirror 4: 0xfa627b7 (stale) Mirror 5: 0xb3cf349c (stale) CRC-32 checksum value for chunk [0, 0x3b791): Mirror 1: 0x62ac76ac Mirror 6: 0x62ac76ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b791: skipped 000354 1789182689.749872 trunc from 0x02b2b0 to 0x031a75 (0x67c6 bytes) 000357 1789182689.802175 mapwrite 0x0069b0 thru 0x014de6 (0xe437 bytes) 000358 1789182689.853232 read 0x007261 thru 0x0092dc (0x207c bytes) 000359 1789182689.854866 read 0x024d7a thru 0x026e3d (0x20c4 bytes) 000360 1789182689.866833 mapwrite 0x014b41 thru 0x0176ee (0x2bae bytes) 000361 1789182689.884857 trunc from 0x023786 to 0x025f31 (0x27ac bytes) 000362 1789182689.891351 trunc from 0x03b791 to 0x03102c (0xa764 bytes) 000363 1789182689.939392 read 0x01047b thru 0x01450b (0x4091 bytes) 000364 1789182689.941351 fallocate 0x012981 thru 0x013d4e (0x13ce bytes) 000365 1789182689.946562 punch from 0x02cb33 to 0x02f837 (0x2d05 bytes) 000367 1789182689.954876 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3102d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3102d): Mirror 4: 0x45fc007c (stale) Mirror 5: 0x6d5bee55 (stale) Mirror 6: 0x4cdc7da0 (stale) CRC-32 checksum value for chunk [0, 0x3102d): Mirror 1: 0xa175a401 Mirror 7: 0xa175a401 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3102d: skipped 000368 1789182691.650442 mapwrite 0x00555a thru 0x013330 (0xddd7 bytes) 000369 1789182691.883403 read 0x009594 thru 0x018512 (0xef7f bytes) 000370 1789182691.884229 punch from 0x00e9fa to 0x01be3d (0xd444 bytes) 000371 1789182691.894298 mapwrite 0x03920d thru 0x03ffff (0x6df3 bytes) 000372 1789182691.990597 trunc from 0x040000 to 0x02d709 (0x128f6 bytes) 000373 1789182692.013453 trunc from 0x007193 to 0x0074e3 (0x351 bytes) 000374 1789182692.039982 trunc from 0x02d70a to 0x022e96 (0xa873 bytes) 000375 1789182692.075592 write 0x021953 thru 0x02ac9d (0x934b bytes) 000376 1789182692.082541 mapread 0x023bfe thru 0x02ac9d (0x70a0 bytes) 000377 1789182692.083463 mapwrite 0x00f921 thru 0x0109b9 (0x1099 bytes) 000378 1789182692.175501 punch from 0x027f22 to 0x02ac9d (0x2d7c bytes) 000379 1789182692.185580 trunc from 0x01ede9 to 0x0238ae (0x4ac6 bytes) 000380 1789182692.203868 mapwrite 0x002bf1 thru 0x011736 (0xeb46 bytes) 000381 1789182692.265473 write 0x032029 thru 0x03ffff (0xdfd7 bytes) 000382 1789182692.271539 punch from 0x01f9c2 to 0x02d63b (0xdc7a bytes) 000383 1789182692.283772 read 0x01cc3b thru 0x02c993 (0xfd59 bytes) 000384 1789182692.322325 fallocate 0x00cc07 thru 0x0106b7 (0x3ab1 bytes) 000385 1789182692.343541 fallocate 0x03e858 thru 0x03ffff (0x17a8 bytes) 000386 1789182692.349810 punch from 0x01c8f6 to 0x025baa (0x92b5 bytes) 000387 1789182692.373607 read 0x0256b8 thru 0x0296d3 (0x401c bytes) 000388 1789182692.389438 trunc from 0x03ae76 to 0x03ffff (0x518a bytes) 000389 1789182692.412793 read 0x02c071 thru 0x03a0d7 (0xe067 bytes) 000390 1789182692.440995 write 0x00edc5 thru 0x01557d (0x67b9 bytes) 000391 1789182692.446543 mapread 0x00284e thru 0x01020b (0xd9be bytes) 000392 1789182692.450874 write 0x00a963 thru 0x013eaa (0x9548 bytes) 000393 1789182692.455984 trunc from 0x034623 to 0x03a957 (0x6335 bytes) 000394 1789182692.463609 trunc from 0x040000 to 0x0031a0 (0x3ce5f bytes) 000395 1789182692.488496 write 0x0115ec thru 0x01d1e5 (0xbbfa bytes) 000396 1789182692.490664 mapwrite 0x02b334 thru 0x03ac82 (0xf94f bytes) 000397 1789182692.615390 fallocate 0x018a12 thru 0x01f1fd (0x67ec bytes) 000398 1789182692.624916 punch from 0x02cf7a to 0x032fa8 (0x602f bytes) 000399 1789182692.635275 MIRROR_RESYNC 5 mirrors 000400 1789182693.796344 fallocate 0x0011dd thru 0x00b7df (0xa603 bytes) 000401 1789182693.835866 write 0x019f3f thru 0x01b77d (0x183f bytes) 000402 1789182693.847078 punch from 0x006c31 to 0x016329 (0xf6f9 bytes) 000403 1789182693.851295 fallocate 0x03b8e1 thru 0x03ffff (0x471f bytes) 000404 1789182693.854783 write 0x038761 thru 0x03e83a (0x60da bytes) 000405 1789182693.864398 mapread 0x0172ff thru 0x01d779 (0x647b bytes) 000406 1789182693.887462 trunc from 0x040000 to 0x02897d (0x17682 bytes) 000407 1789182693.912722 MIRROR_RESYNC 5 mirrors 000408 1789182695.317724 mapwrite 0x0149d8 thru 0x022ff3 (0xe61c bytes) 000409 1789182695.483668 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2897e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2897e): Mirror 4: 0x1c2800a (stale) Mirror 5: 0x1c2800a (stale) Mirror 6: 0x1c2800a (stale) Mirror 7: 0x1c2800a (stale) CRC-32 checksum value for chunk [0, 0x2897e): Mirror 1: 0xc20c2153 Mirror 8: 0xc20c2153 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2897e: skipped 000410 1789182696.730991 trunc from 0x02897e to 0x003c24 (0x24d59 bytes) 000412 1789182696.815661 write 0x017507 thru 0x01fde3 (0x88dd bytes) 000413 1789182696.819133 mapwrite 0x02739a thru 0x029998 (0x25ff bytes) 000414 1789182696.873385 mapwrite 0x025560 thru 0x02e228 (0x8cc9 bytes) 000415 1789182696.910932 punch from 0x00174b to 0x0082f5 (0x6bab bytes) 000416 1789182696.915235 mapread 0x02dc4a thru 0x02e228 (0x5df bytes) 000417 1789182696.916235 punch from 0x009953 to 0x00ade4 (0x1492 bytes) 000418 1789182696.924461 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e229): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e229): Mirror 4: 0xff88a975 (stale) Mirror 5: 0xff88a975 (stale) Mirror 6: 0xff88a975 (stale) Mirror 7: 0xff88a975 (stale) Mirror 8: 0x6653df55 (stale) CRC-32 checksum value for chunk [0, 0x2e229): Mirror 1: 0x755a5986 Mirror 9: 0x755a5986 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e229: skipped 000419 1789182698.014108 trunc from 0x02e229 to 0x01a8dd (0x1394b bytes) 000420 1789182698.148141 fallocate 0x02b397 thru 0x03b0d7 (0xfd41 bytes) 000421 1789182698.155293 trunc from 0x01a8de to 0x002aba (0x17e23 bytes) 000423 1789182698.177546 fallocate 0x0333c9 thru 0x0393bd (0x5ff5 bytes) 000424 1789182698.185219 fallocate 0x011f3d thru 0x01e996 (0xca5a bytes) 000425 1789182698.192731 trunc from 0x002abb to 0x016118 (0x1365e bytes) 000426 1789182698.209566 trunc from 0x016119 to 0x017072 (0xf5a bytes) 000428 1789182698.232062 trunc from 0x017073 to 0x02ccff (0x15c8d bytes) 000429 1789182698.256108 fallocate 0x01a156 thru 0x02647e (0xc329 bytes) 000431 1789182698.262403 trunc from 0x02cd00 to 0x0330e7 (0x63e8 bytes) 000432 1789182698.291387 fallocate 0x025ea5 thru 0x027ded (0x1f49 bytes) 000433 1789182698.304566 write 0x01b213 thru 0x01c234 (0x1022 bytes) 000434 1789182698.328004 fallocate 0x0028c1 thru 0x00aa0f (0x814f bytes) 000435 1789182698.336241 read 0x0217e2 thru 0x030a00 (0xf21f bytes) 000436 1789182698.353076 fallocate 0x0117d0 thru 0x016707 (0x4f38 bytes) 000437 1789182698.359458 read 0x030e13 thru 0x0330e7 (0x22d5 bytes) 000438 1789182698.372551 fallocate 0x01f99d thru 0x025ba9 (0x620d bytes) 000439 1789182698.377972 mapwrite 0x0103c7 thru 0x01df27 (0xdb61 bytes) 000440 1789182698.405161 trunc from 0x0330e8 to 0x039594 (0x64ad bytes) 000441 1789182698.422395 read 0x017d33 thru 0x02290f (0xabdd bytes) 000443 1789182698.426073 write 0x0106fe thru 0x0112ce (0xbd1 bytes) 000444 1789182698.428195 fallocate 0x015cb3 thru 0x01edba (0x9108 bytes) 000446 1789182698.435401 trunc from 0x039595 to 0x019643 (0x1ff51 bytes) 000447 1789182698.458149 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x19644): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19644): Mirror 4: 0x1d090fd8 (stale) Mirror 5: 0x1d090fd8 (stale) Mirror 6: 0x1d090fd8 (stale) Mirror 7: 0x1d090fd8 (stale) Mirror 8: 0x420bc78a (stale) Mirror 9: 0x19be2947 (stale) CRC-32 checksum value for chunk [0, 0x19644): Mirror 1: 0x4f6931f8 Mirror 10: 0x4f6931f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19644: skipped 000448 1789182699.371235 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x19644): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19644): Mirror 4: 0x1d090fd8 (stale) Mirror 5: 0x1d090fd8 (stale) Mirror 6: 0x1d090fd8 (stale) Mirror 7: 0x1d090fd8 (stale) Mirror 8: 0x420bc78a (stale) Mirror 9: 0x19be2947 (stale) CRC-32 checksum value for chunk [0, 0x19644): Mirror 1: 0x4f6931f8 Mirror 10: 0x4f6931f8 Mirror 11: 0x4f6931f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19644: skipped 000449 1789182700.463052 trunc from 0x019644 to 0x02ab95 (0x11552 bytes) 000450 1789182700.540482 trunc from 0x02ab96 to 0x02a4c1 (0x6d4 bytes) 000451 1789182700.569185 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 4: 0xc73b0338 (stale) Mirror 5: 0xc73b0338 (stale) Mirror 6: 0xc73b0338 (stale) Mirror 7: 0xc73b0338 (stale) Mirror 8: 0x16a78972 (stale) Mirror 9: 0xd5731e02 (stale) Mirror 10: 0x8f7a1d4d (stale) Mirror 11: 0x8f7a1d4d (stale) CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 1: 0x8f7a1d4d Mirror 12: 0x8f7a1d4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a4c2: skipped 000452 1789182701.956210 mapread 0x01f330 thru 0x0290b0 (0x9d81 bytes) 000453 1789182701.979491 mapread 0x002407 thru 0x00971e (0x7318 bytes) 000455 1789182701.988801 punch from 0x029740 to 0x02a4c1 (0xd82 bytes) 000456 1789182702.030661 punch from 0x00810e to 0x00ca77 (0x496a bytes) 000457 1789182702.036576 fallocate 0x017d34 thru 0x0187e5 (0xab2 bytes) 000458 1789182702.041478 write 0x00a63c thru 0x00c34c (0x1d11 bytes) 000460 1789182702.055084 trunc from 0x02a4c2 to 0x0223f5 (0x80cc bytes) 000461 1789182702.075047 mapwrite 0x00019f thru 0x004372 (0x41d4 bytes) 000462 1789182702.101288 trunc from 0x00c0d1 to 0x01461e (0x854e bytes) 000463 1789182702.106870 write 0x013f55 thru 0x015db9 (0x1e65 bytes) 000464 1789182702.117081 read 0x00a49a thru 0x015b41 (0xb6a8 bytes) 000465 1789182702.128223 trunc from 0x0223f6 to 0x015512 (0xcee3 bytes) 000466 1789182702.150478 trunc from 0x015513 to 0x0134c2 (0x2050 bytes) 000467 1789182702.168480 write 0x01bfdd thru 0x024456 (0x847a bytes) 000468 1789182702.171325 trunc from 0x024457 to 0x02cc21 (0x87cb bytes) 000469 1789182702.193701 mapwrite 0x01714f thru 0x01e146 (0x6ff8 bytes) 000470 1789182702.219268 fallocate 0x021f2b thru 0x02e4ed (0xc5c3 bytes) 000471 1789182702.224557 mapwrite 0x00910f thru 0x016c0a (0xdafc bytes) 000472 1789182702.243021 trunc from 0x02cc22 to 0x03024b (0x362a bytes) 000473 1789182702.263409 trunc from 0x005481 to 0x00fb6b (0xa6eb bytes) 000474 1789182702.269491 fallocate 0x034b97 thru 0x038fa8 (0x4412 bytes) 000475 1789182702.274599 trunc from 0x03024c to 0x01c695 (0x13bb6 bytes) 000476 1789182702.289433 read 0x0129ca thru 0x013110 (0x747 bytes) 000477 1789182702.290115 write 0x007a27 thru 0x00c296 (0x4870 bytes) 000478 1789182702.301401 read 0x000036 thru 0x006365 (0x6330 bytes) 000479 1789182702.302345 trunc from 0x00e267 to 0x01c695 (0xe42f bytes) 000480 1789182702.308468 read 0x00a52c thru 0x00f19b (0x4c70 bytes) 000481 1789182702.317925 fallocate 0x0328a4 thru 0x036f11 (0x466e bytes) 000482 1789182702.322608 trunc from 0x0097ed to 0x013d24 (0xa538 bytes) 000483 1789182702.335577 punch from 0x00abbc to 0x012004 (0x7449 bytes) 000484 1789182702.341711 mapread 0x01bb5c thru 0x01c695 (0xb3a bytes) 000485 1789182702.346608 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1c696): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c696): Mirror 4: 0x132ec987 (stale) Mirror 5: 0x132ec987 (stale) Mirror 6: 0x132ec987 (stale) Mirror 7: 0x132ec987 (stale) Mirror 8: 0xce19917e (stale) Mirror 9: 0x1b92714d (stale) Mirror 10: 0xf7d2d8ad (stale) Mirror 11: 0xf7d2d8ad (stale) Mirror 12: 0xf7d2d8ad (stale) CRC-32 checksum value for chunk [0, 0x1c696): Mirror 1: 0x43332623 Mirror 13: 0x43332623 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c696: skipped 000486 1789182703.240325 fallocate 0x03ecf1 thru 0x03ffff (0x130f bytes) 000487 1789182703.298091 punch from 0x00bf15 to 0x015de3 (0x9ecf bytes) 000488 1789182703.303399 read 0x034559 thru 0x03ffff (0xbaa7 bytes) 000489 1789182703.314477 write 0x009b38 thru 0x01171a (0x7be3 bytes) 000490 1789182703.331321 mapwrite 0x0310cf thru 0x037792 (0x66c4 bytes) 000491 1789182703.373906 mapwrite 0x0384bc thru 0x03b3d1 (0x2f16 bytes) 000492 1789182703.399334 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5f600c0b (stale) Mirror 5: 0x5f600c0b (stale) Mirror 6: 0x5f600c0b (stale) Mirror 7: 0x5f600c0b (stale) Mirror 8: 0x9c71fdde (stale) Mirror 9: 0x75ab5ad3 (stale) Mirror 10: 0xb2aa5feb (stale) Mirror 11: 0xb2aa5feb (stale) Mirror 12: 0xb2aa5feb (stale) Mirror 13: 0x1845015d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa1251c72 Mirror 14: 0xa1251c72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000493 1789182704.769519 write 0x018b2f thru 0x022b9e (0xa070 bytes) 000494 1789182704.866006 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5f600c0b (stale) Mirror 5: 0x5f600c0b (stale) Mirror 6: 0x5f600c0b (stale) Mirror 7: 0x5f600c0b (stale) Mirror 8: 0x9c71fdde (stale) Mirror 10: 0xb2aa5feb (stale) Mirror 11: 0xb2aa5feb (stale) Mirror 12: 0xb2aa5feb (stale) Mirror 13: 0x1845015d (stale) Mirror 14: 0xa1251c72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5f600c0b (stale) Mirror 5: 0x5f600c0b (stale) Mirror 6: 0x5f600c0b (stale) Mirror 7: 0x5f600c0b (stale) Mirror 8: 0x9c71fdde (stale) Mirror 10: 0xb2aa5feb (stale) Mirror 11: 0xb2aa5feb (stale) Mirror 12: 0xb2aa5feb (stale) Mirror 13: 0x1845015d (stale) Mirror 14: 0xa1251c72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000495 1789182705.692991 write 0x016a1e thru 0x024d7d (0xe360 bytes) 000496 1789182705.712017 fallocate 0x0302b5 thru 0x038afa (0x8846 bytes) 000497 1789182705.717160 punch from 0x018989 to 0x01b23e (0x28b6 bytes) 000498 1789182705.736335 punch from 0x0214a0 to 0x02218e (0xcef bytes) 000499 1789182705.742214 mapread 0x00c441 thru 0x01a71a (0xe2da bytes) 000500 1789182705.767578 write 0x007834 thru 0x0135a3 (0xbd70 bytes) 000501 1789182705.770909 write 0x025bef thru 0x02eda1 (0x91b3 bytes) 000503 1789182705.780961 mapwrite 0x029285 thru 0x0301ac (0x6f28 bytes) 000504 1789182705.811294 trunc from 0x040000 to 0x0022e6 (0x3dd19 bytes) 000505 1789182705.864680 read 0x000823 thru 0x0022e6 (0x1ac4 bytes) 000507 1789182705.876851 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22e7): Mirror 4: 0x76072a9d (stale) Mirror 6: 0x76072a9d (stale) Mirror 7: 0x76072a9d (stale) Mirror 8: 0x76072a9d (stale) Mirror 10: 0x76072a9d (stale) Mirror 11: 0x76072a9d (stale) Mirror 12: 0x76072a9d (stale) Mirror 13: 0xb284e38b (stale) Mirror 14: 0xb284e38b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e7): Mirror 4: 0x76072a9d (stale) Mirror 6: 0x76072a9d (stale) Mirror 7: 0x76072a9d (stale) Mirror 8: 0x76072a9d (stale) Mirror 10: 0x76072a9d (stale) Mirror 11: 0x76072a9d (stale) Mirror 12: 0x76072a9d (stale) Mirror 13: 0xb284e38b (stale) Mirror 14: 0xb284e38b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e7: skipped 000508 1789182706.354068 write 0x00c2be thru 0x00cd86 (0xac9 bytes) 000509 1789182706.358905 mapwrite 0x03d085 thru 0x03ffff (0x2f7b bytes) 000511 1789182706.398549 mapwrite 0x02ec1c thru 0x036236 (0x761b bytes) 000512 1789182706.420109 fallocate 0x02bc76 thru 0x036a88 (0xae13 bytes) 000514 1789182706.424425 mapread 0x018d27 thru 0x0209dd (0x7cb7 bytes) 000515 1789182706.432887 trunc from 0x026284 to 0x02946f (0x31ec bytes) 000516 1789182706.442140 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5f600c0b (stale) Mirror 6: 0x5f600c0b (stale) Mirror 7: 0x5f600c0b (stale) Mirror 8: 0x9c71fdde (stale) Mirror 10: 0xb2aa5feb (stale) Mirror 11: 0xb2aa5feb (stale) Mirror 12: 0xb2aa5feb (stale) Mirror 13: 0x1845015d (stale) Mirror 14: 0xa1251c72 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xacb23f59 Mirror 15: 0xacb23f59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000517 1789182707.882611 trunc from 0x040000 to 0x000037 (0x3ffc8 bytes) 000518 1789182707.986936 trunc from 0x000026 to 0x000037 (0x12 bytes) 000519 1789182707.992540 trunc from 0x000038 to 0x01d669 (0x1d632 bytes) 000520 1789182708.024176 trunc from 0x01d66a to 0x0375af (0x19f46 bytes) 000521 1789182708.053260 MIRROR_RESYNC 11 mirrors 000522 1789182710.479582 trunc from 0x01a54e to 0x01feac (0x595f bytes) 000523 1789182710.522902 read 0x030344 thru 0x0375af (0x726c bytes) 000524 1789182710.534024 fallocate 0x03adeb thru 0x03ffff (0x5215 bytes) 000525 1789182710.538948 write 0x037fed thru 0x0383e1 (0x3f5 bytes) 000526 1789182710.540502 trunc from 0x0109a4 to 0x01e29d (0xd8fa bytes) 000527 1789182710.545878 write 0x01cb18 thru 0x027cf4 (0xb1dd bytes) 000528 1789182710.555925 trunc from 0x002aa0 to 0x00c6a8 (0x9c09 bytes) 000529 1789182710.561021 mapwrite 0x0059b5 thru 0x00764b (0x1c97 bytes) 000530 1789182710.598699 mapread 0x0077e4 thru 0x0131b1 (0xb9ce bytes) 000531 1789182710.610009 mapread 0x00d364 thru 0x00fb19 (0x27b6 bytes) 000532 1789182710.611198 trunc from 0x0084b8 to 0x01151f (0x9068 bytes) 000533 1789182710.618111 mapwrite 0x00446a thru 0x007cd0 (0x3867 bytes) 000534 1789182710.632039 fallocate 0x0211d3 thru 0x0231b8 (0x1fe6 bytes) 000535 1789182710.636648 write 0x00591a thru 0x012854 (0xcf3b bytes) 000536 1789182710.640127 trunc from 0x0383e2 to 0x026160 (0x12281 bytes) 000537 1789182710.667371 fallocate 0x020bc8 thru 0x02c71e (0xbb57 bytes) 000538 1789182710.680935 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x26161): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26161): Mirror 4: 0x5e11c859 (stale) Mirror 6: 0x5e11c859 (stale) Mirror 7: 0x5e11c859 (stale) Mirror 8: 0x5e11c859 (stale) Mirror 10: 0x5e11c859 (stale) Mirror 11: 0x5e11c859 (stale) Mirror 12: 0x5e11c859 (stale) Mirror 13: 0x5e11c859 (stale) Mirror 14: 0x5e11c859 (stale) Mirror 15: 0x5e11c859 (stale) CRC-32 checksum value for chunk [0, 0x26161): Mirror 1: 0x4bc6bd11 Mirror 16: 0x4bc6bd11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26161: skipped 000539 1789182711.774325 mapwrite 0x02f348 thru 0x03ac7d (0xb936 bytes) 000540 1789182711.874588 fallocate 0x025946 thru 0x031418 (0xbad3 bytes) 000541 1789182711.880403 read 0x021e03 thru 0x0295b8 (0x77b6 bytes) 000542 1789182711.894121 fallocate 0x002cfa thru 0x009235 (0x653c bytes) 000543 1789182711.898899 trunc from 0x0248c2 to 0x027237 (0x2976 bytes) 000544 1789182711.905380 mapread 0x00a0c1 thru 0x018d32 (0xec72 bytes) 000545 1789182711.926113 read 0x02646a thru 0x02d480 (0x7017 bytes) 000546 1789182711.935645 fallocate 0x02f0b7 thru 0x032a33 (0x397d bytes) 000547 1789182711.941046 mapread 0x002139 thru 0x0058d5 (0x379d bytes) 000548 1789182711.942560 write 0x008071 thru 0x00ba32 (0x39c2 bytes) 000549 1789182711.944504 fallocate 0x013c0a thru 0x01d3a0 (0x9797 bytes) 000550 1789182711.949165 read 0x011d9e thru 0x01b0dd (0x9340 bytes) 000551 1789182711.949939 mapwrite 0x020190 thru 0x02ebe5 (0xea56 bytes) 000552 1789182711.970924 mapwrite 0x03439c thru 0x03ad42 (0x69a7 bytes) 000553 1789182712.004661 mapread 0x010fc0 thru 0x01c1b8 (0xb1f9 bytes) 000554 1789182712.006339 fallocate 0x03ae1f thru 0x03f72f (0x4911 bytes) 000555 1789182712.012487 read 0x025e78 thru 0x03479f (0xe928 bytes) 000557 1789182712.017925 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3ad43): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad43): Mirror 4: 0xa929218c (stale) Mirror 6: 0xa929218c (stale) Mirror 7: 0xa929218c (stale) Mirror 8: 0xa929218c (stale) Mirror 10: 0xa929218c (stale) Mirror 11: 0xa929218c (stale) Mirror 12: 0xa929218c (stale) Mirror 13: 0xa929218c (stale) Mirror 14: 0xa929218c (stale) Mirror 15: 0xa929218c (stale) Mirror 16: 0xb4485f44 (stale) CRC-32 checksum value for chunk [0, 0x3ad43): Mirror 1: 0xffbc9c4a Mirror 17: 0xffbc9c4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad43: skipped 000558 1789182713.015607 fallocate 0x000435 thru 0x006591 (0x615d bytes) 000559 1789182713.061049 trunc from 0x03ad43 to 0x034d0a (0x6038 bytes) 000560 1789182713.086297 trunc from 0x00c9c4 to 0x014fe2 (0x861f bytes) 000561 1789182713.091963 punch from 0x0153c4 to 0x01c8a8 (0x74e5 bytes) 000562 1789182713.097685 mapwrite 0x023c31 thru 0x02ac20 (0x6ff0 bytes) 000563 1789182713.146882 trunc from 0x034d0b to 0x000da0 (0x33f6a bytes) 000564 1789182713.172163 mapwrite 0x03fe1c thru 0x03ffff (0x1e4 bytes) 000567 1789182713.225074 punch from 0x01de84 to 0x02cc17 (0xed94 bytes) 000568 1789182713.238116 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0x5e53688b (stale) Mirror 17: 0xbc377229 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x952852c1 Mirror 18: 0x952852c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000569 1789182714.162015 fallocate 0x034083 thru 0x03ffff (0xbf7d bytes) 000570 1789182714.209081 fallocate 0x0180f0 thru 0x023127 (0xb038 bytes) 000571 1789182714.215062 mapwrite 0x0218ac thru 0x02bf67 (0xa6bc bytes) 000572 1789182714.259245 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0x5e53688b (stale) Mirror 17: 0xbc377229 (stale) Mirror 18: 0x952852c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0x5e53688b (stale) Mirror 17: 0xbc377229 (stale) Mirror 18: 0x952852c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000573 1789182714.923658 read 0x01f4b2 thru 0x027344 (0x7e93 bytes) 000574 1789182714.934529 write 0x029e47 thru 0x02daea (0x3ca4 bytes) 000575 1789182714.948035 trunc from 0x040000 to 0x03e180 (0x1e7f bytes) 000576 1789182714.986139 mapread 0x016831 thru 0x01bb6b (0x533b bytes) 000578 1789182715.005933 mapread 0x03aabc thru 0x03d567 (0x2aac bytes) 000579 1789182715.020492 MIRROR_RESYNC 13 mirrors 000580 1789182717.359950 punch from 0x029da1 to 0x0311e6 (0x7446 bytes) 000581 1789182717.422088 fallocate 0x019dc4 thru 0x022de5 (0x9022 bytes) 000582 1789182717.428784 fallocate 0x01cd6a thru 0x026996 (0x9c2d bytes) 000583 1789182717.435076 fallocate 0x02f9cd thru 0x03cf53 (0xd587 bytes) 000584 1789182717.448930 write 0x034f38 thru 0x03e521 (0x95ea bytes) 000585 1789182717.466101 mapwrite 0x03c206 thru 0x03ffff (0x3dfa bytes) 000586 1789182717.523956 trunc from 0x00a7ee to 0x011fb8 (0x77cb bytes) 000587 1789182717.530925 trunc from 0x040000 to 0x00f834 (0x307cb bytes) 000588 1789182717.550573 write 0x033db1 thru 0x03fcf0 (0xbf40 bytes) 000589 1789182717.554172 trunc from 0x000d16 to 0x0031fa (0x24e5 bytes) 000590 1789182717.561792 write 0x016e44 thru 0x02022b (0x93e8 bytes) 000591 1789182717.573343 trunc from 0x020940 to 0x0260a7 (0x5768 bytes) 000592 1789182717.596873 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3fcf1): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fcf1): Mirror 4: 0xd51b69b0 (stale) Mirror 6: 0xd51b69b0 (stale) Mirror 7: 0xd51b69b0 (stale) Mirror 8: 0xd51b69b0 (stale) Mirror 10: 0xd51b69b0 (stale) Mirror 11: 0xd51b69b0 (stale) Mirror 12: 0xd51b69b0 (stale) Mirror 13: 0xd51b69b0 (stale) Mirror 15: 0xd51b69b0 (stale) Mirror 16: 0xd51b69b0 (stale) Mirror 17: 0xd51b69b0 (stale) Mirror 18: 0xd51b69b0 (stale) CRC-32 checksum value for chunk [0, 0x3fcf1): Mirror 1: 0x7c36fef1 Mirror 19: 0x7c36fef1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fcf1: skipped 000594 1789182719.027957 mapread 0x03396b thru 0x03fcf0 (0xc386 bytes) 000596 1789182719.067906 read 0x02dde9 thru 0x03b82d (0xda45 bytes) 000597 1789182719.078994 trunc from 0x012de8 to 0x014da4 (0x1fbd bytes) 000598 1789182719.137238 read 0x03c362 thru 0x03fcf0 (0x398f bytes) 000599 1789182719.178206 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3fcf1): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fcf1): Mirror 4: 0xd51b69b0 (stale) Mirror 6: 0xd51b69b0 (stale) Mirror 7: 0xd51b69b0 (stale) Mirror 8: 0xd51b69b0 (stale) Mirror 10: 0xd51b69b0 (stale) Mirror 11: 0xd51b69b0 (stale) Mirror 12: 0xd51b69b0 (stale) Mirror 13: 0xd51b69b0 (stale) Mirror 15: 0xd51b69b0 (stale) Mirror 16: 0xd51b69b0 (stale) Mirror 17: 0xd51b69b0 (stale) Mirror 18: 0xd51b69b0 (stale) Mirror 19: 0x7c36fef1 (stale) CRC-32 checksum value for chunk [0, 0x3fcf1): Mirror 1: 0x7c36fef1 Mirror 20: 0x7c36fef1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fcf1: skipped 000600 1789182720.694109 mapwrite 0x01ad79 thru 0x022658 (0x78e0 bytes) 000601 1789182720.878574 punch from 0x003c3c to 0x00af67 (0x732c bytes) 000602 1789182720.896160 write 0x0088f1 thru 0x013114 (0xa824 bytes) 000603 1789182720.913548 write 0x02f70c thru 0x031aac (0x23a1 bytes) 000606 1789182720.942529 mapwrite 0x02e692 thru 0x030cf7 (0x2666 bytes) 000607 1789182721.001784 trunc from 0x03fcf1 to 0x031ced (0xe003 bytes) 000608 1789182721.073439 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31cee): Mirror 6: 0xcaad23ec (stale) Mirror 7: 0xcaad23ec (stale) Mirror 8: 0xcaad23ec (stale) Mirror 10: 0xcaad23ec (stale) Mirror 11: 0xcaad23ec (stale) Mirror 12: 0xcaad23ec (stale) Mirror 13: 0xcaad23ec (stale) Mirror 15: 0xcaad23ec (stale) Mirror 16: 0xcaad23ec (stale) Mirror 17: 0xcaad23ec (stale) Mirror 18: 0xcaad23ec (stale) Mirror 19: 0x57e34849 (stale) Mirror 20: 0x57e34849 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31cee): Mirror 6: 0xcaad23ec (stale) Mirror 7: 0xcaad23ec (stale) Mirror 8: 0xcaad23ec (stale) Mirror 10: 0xcaad23ec (stale) Mirror 11: 0xcaad23ec (stale) Mirror 12: 0xcaad23ec (stale) Mirror 13: 0xcaad23ec (stale) Mirror 15: 0xcaad23ec (stale) Mirror 16: 0xcaad23ec (stale) Mirror 17: 0xcaad23ec (stale) Mirror 18: 0xcaad23ec (stale) Mirror 19: 0x57e34849 (stale) Mirror 20: 0x57e34849 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31cee: skipped 000609 1789182722.051319 mapwrite 0x02d360 thru 0x0347c7 (0x7468 bytes) 000610 1789182722.161177 trunc from 0x03385b to 0x0347c7 (0xf6d bytes) 000611 1789182722.182015 mapwrite 0x014a90 thru 0x01f953 (0xaec4 bytes) 000612 1789182722.227388 write 0x013ba4 thru 0x01654f (0x29ac bytes) 000613 1789182722.228692 trunc from 0x010d03 to 0x01a970 (0x9c6e bytes) 000614 1789182722.240122 read 0x011306 thru 0x014692 (0x338d bytes) 000615 1789182722.245269 fallocate 0x039adc thru 0x03ffff (0x6524 bytes) 000616 1789182722.252904 mapwrite 0x02ef4b thru 0x030ad4 (0x1b8a bytes) 000617 1789182722.288476 mapwrite 0x00b1c1 thru 0x0198f8 (0xe738 bytes) 000618 1789182722.353118 read 0x03a996 thru 0x03d8f3 (0x2f5e bytes) 000619 1789182722.384639 read 0x01219e thru 0x017bea (0x5a4d bytes) 000620 1789182722.385339 trunc from 0x040000 to 0x02a09a (0x15f65 bytes) 000621 1789182722.436781 mapread 0x00327f thru 0x004f6f (0x1cf1 bytes) 000622 1789182722.439151 fallocate 0x03c9f7 thru 0x03ffff (0x3609 bytes) 000623 1789182722.445407 trunc from 0x02a09b to 0x016217 (0x13e83 bytes) 000624 1789182722.460953 trunc from 0x0140a0 to 0x014b4b (0xaac bytes) 000625 1789182722.471969 fallocate 0x02dc89 thru 0x03a522 (0xc89a bytes) 000626 1789182722.484083 trunc from 0x03a523 to 0x008ad2 (0x31a50 bytes) 000627 1789182722.508487 mapwrite 0x037d85 thru 0x03ffff (0x827b bytes) 000629 1789182722.601845 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x99b761e5 (stale) Mirror 7: 0x99b761e5 (stale) Mirror 8: 0x99b761e5 (stale) Mirror 10: 0x99b761e5 (stale) Mirror 11: 0x99b761e5 (stale) Mirror 12: 0x99b761e5 (stale) Mirror 13: 0x99b761e5 (stale) Mirror 15: 0x99b761e5 (stale) Mirror 16: 0x99b761e5 (stale) Mirror 17: 0x99b761e5 (stale) Mirror 19: 0x9002684b (stale) Mirror 20: 0x9002684b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x99b761e5 (stale) Mirror 7: 0x99b761e5 (stale) Mirror 8: 0x99b761e5 (stale) Mirror 10: 0x99b761e5 (stale) Mirror 11: 0x99b761e5 (stale) Mirror 12: 0x99b761e5 (stale) Mirror 13: 0x99b761e5 (stale) Mirror 15: 0x99b761e5 (stale) Mirror 16: 0x99b761e5 (stale) Mirror 17: 0x99b761e5 (stale) Mirror 19: 0x9002684b (stale) Mirror 20: 0x9002684b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1789182723.480607 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x99b761e5 (stale) Mirror 7: 0x99b761e5 (stale) Mirror 8: 0x99b761e5 (stale) Mirror 10: 0x99b761e5 (stale) Mirror 11: 0x99b761e5 (stale) Mirror 12: 0x99b761e5 (stale) Mirror 13: 0x99b761e5 (stale) Mirror 15: 0x99b761e5 (stale) Mirror 16: 0x99b761e5 (stale) Mirror 17: 0x99b761e5 (stale) Mirror 19: 0x9002684b (stale) Mirror 20: 0x9002684b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84afdcc3 Mirror 21: 0x84afdcc3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000631 1789182724.914490 mapread 0x0207cd thru 0x029d6a (0x959e bytes) 000632 1789182724.956525 read 0x034ea5 thru 0x0381a7 (0x3303 bytes) 000633 1789182724.968569 fallocate 0x02ff22 thru 0x0327b0 (0x288f bytes) 000634 1789182725.039551 read 0x02604f thru 0x03460e (0xe5c0 bytes) 000635 1789182725.079277 write 0x010bc4 thru 0x01ab1b (0x9f58 bytes) 000636 1789182725.087546 mapread 0x03f077 thru 0x03f35f (0x2e9 bytes) 000638 1789182725.089019 trunc from 0x020feb to 0x0288c4 (0x78da bytes) 000639 1789182725.095260 punch from 0x0312a7 to 0x03f46d (0xe1c7 bytes) 000640 1789182725.101677 write 0x023292 thru 0x02d4af (0xa21e bytes) 000642 1789182725.111422 trunc from 0x03acb2 to 0x03c88e (0x1bdd bytes) 000643 1789182725.117453 fallocate 0x0081de thru 0x008be9 (0xa0c bytes) 000644 1789182725.122262 mapwrite 0x02f3f9 thru 0x03519c (0x5da4 bytes) 000645 1789182725.179197 mapread 0x0123b0 thru 0x01764b (0x529c bytes) 000647 1789182725.180485 trunc from 0x040000 to 0x02c63c (0x139c3 bytes) 000648 1789182725.194617 fallocate 0x02c53d thru 0x034fd7 (0x8a9b bytes) 000649 1789182725.200983 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2c63d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c63d): Mirror 6: 0x764c4a67 (stale) Mirror 7: 0x764c4a67 (stale) Mirror 8: 0x764c4a67 (stale) Mirror 10: 0x764c4a67 (stale) Mirror 11: 0x764c4a67 (stale) Mirror 12: 0x764c4a67 (stale) Mirror 13: 0x764c4a67 (stale) Mirror 15: 0x764c4a67 (stale) Mirror 16: 0x764c4a67 (stale) Mirror 17: 0x764c4a67 (stale) Mirror 19: 0x2f13594f (stale) Mirror 20: 0x2f13594f (stale) Mirror 21: 0xd83364e7 (stale) CRC-32 checksum value for chunk [0, 0x2c63d): Mirror 1: 0x70d2dacf Mirror 22: 0x70d2dacf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c63d: skipped 000650 1789182726.566065 mapread 0x023d55 thru 0x02c63c (0x88e8 bytes) 000651 1789182726.631206 read 0x029db7 thru 0x02c24a (0x2494 bytes) 000652 1789182726.653099 punch from 0x002e46 to 0x006c42 (0x3dfd bytes) 000653 1789182726.729144 fallocate 0x03caa9 thru 0x03ffff (0x3557 bytes) 000654 1789182726.734684 trunc from 0x029dc0 to 0x02c63c (0x287d bytes) 000655 1789182726.744704 trunc from 0x0095aa to 0x01161e (0x8075 bytes) 000656 1789182726.754710 trunc from 0x000eb1 to 0x008581 (0x76d1 bytes) 000657 1789182726.768215 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c63d): Mirror 6: 0x764c4a67 (stale) Mirror 7: 0x764c4a67 (stale) Mirror 8: 0x764c4a67 (stale) Mirror 10: 0x764c4a67 (stale) Mirror 11: 0x764c4a67 (stale) Mirror 12: 0x764c4a67 (stale) Mirror 13: 0x764c4a67 (stale) Mirror 15: 0x764c4a67 (stale) Mirror 16: 0x764c4a67 (stale) Mirror 17: 0x764c4a67 (stale) Mirror 20: 0x2f13594f (stale) Mirror 21: 0xd83364e7 (stale) Mirror 22: 0x70d2dacf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c63d): Mirror 6: 0x764c4a67 (stale) Mirror 7: 0x764c4a67 (stale) Mirror 8: 0x764c4a67 (stale) Mirror 10: 0x764c4a67 (stale) Mirror 11: 0x764c4a67 (stale) Mirror 12: 0x764c4a67 (stale) Mirror 13: 0x764c4a67 (stale) Mirror 15: 0x764c4a67 (stale) Mirror 16: 0x764c4a67 (stale) Mirror 17: 0x764c4a67 (stale) Mirror 20: 0x2f13594f (stale) Mirror 21: 0xd83364e7 (stale) Mirror 22: 0x70d2dacf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c63d: skipped 000659 1789182727.629429 fallocate 0x0257c0 thru 0x03050c (0xad4d bytes) 000660 1789182727.635367 mapread 0x029cd5 thru 0x02c63c (0x2968 bytes) 000661 1789182727.657878 write 0x0299d8 thru 0x02d071 (0x369a bytes) 000662 1789182727.663361 mapread 0x0292fe thru 0x02d071 (0x3d74 bytes) 000663 1789182727.664498 write 0x00d3fa thru 0x018b80 (0xb787 bytes) 000664 1789182727.678181 trunc from 0x02d072 to 0x03b4a5 (0xe434 bytes) 000666 1789182727.725450 mapread 0x00c14c thru 0x01c09a (0xff4f bytes) 000667 1789182727.743660 trunc from 0x03b4a6 to 0x023b8e (0x17917 bytes) 000668 1789182727.769099 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x23b8f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b8f): Mirror 6: 0xc89ddad (stale) Mirror 7: 0xc89ddad (stale) Mirror 8: 0xc89ddad (stale) Mirror 10: 0xc89ddad (stale) Mirror 11: 0xc89ddad (stale) Mirror 12: 0xc89ddad (stale) Mirror 13: 0xc89ddad (stale) Mirror 15: 0xc89ddad (stale) Mirror 16: 0xc89ddad (stale) Mirror 17: 0xc89ddad (stale) Mirror 20: 0x408ae4be (stale) Mirror 21: 0x5078f807 (stale) Mirror 22: 0xc426f283 (stale) CRC-32 checksum value for chunk [0, 0x23b8f): Mirror 1: 0x7b1e5636 Mirror 23: 0x7b1e5636 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b8f: skipped 000669 1789182729.092113 punch from 0x019bce to 0x0225b2 (0x89e5 bytes) 000671 1789182729.148070 read 0x001446 thru 0x00b649 (0xa204 bytes) 000673 1789182729.162517 mapwrite 0x00a7b8 thru 0x019bd2 (0xf41b bytes) 000675 1789182729.214701 read 0x020fd7 thru 0x0229ed (0x1a17 bytes) 000676 1789182729.225874 read 0x00b29b thru 0x012ac8 (0x782e bytes) 000677 1789182729.231301 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23b8f): Mirror 6: 0xc89ddad (stale) Mirror 7: 0xc89ddad (stale) Mirror 8: 0xc89ddad (stale) Mirror 10: 0xc89ddad (stale) Mirror 11: 0xc89ddad (stale) Mirror 12: 0xc89ddad (stale) Mirror 13: 0xc89ddad (stale) Mirror 15: 0xc89ddad (stale) Mirror 16: 0xc89ddad (stale) Mirror 20: 0x408ae4be (stale) Mirror 21: 0x5078f807 (stale) Mirror 22: 0xc426f283 (stale) Mirror 23: 0x7b1e5636 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b8f): Mirror 6: 0xc89ddad (stale) Mirror 7: 0xc89ddad (stale) Mirror 8: 0xc89ddad (stale) Mirror 10: 0xc89ddad (stale) Mirror 11: 0xc89ddad (stale) Mirror 12: 0xc89ddad (stale) Mirror 13: 0xc89ddad (stale) Mirror 15: 0xc89ddad (stale) Mirror 16: 0xc89ddad (stale) Mirror 20: 0x408ae4be (stale) Mirror 21: 0x5078f807 (stale) Mirror 22: 0xc426f283 (stale) Mirror 23: 0x7b1e5636 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b8f: skipped 000678 1789182729.959414 fallocate 0x025e4d thru 0x02aed7 (0x508b bytes) 000679 1789182729.966020 trunc from 0x01cc68 to 0x01d22d (0x5c6 bytes) 000681 1789182729.974413 mapwrite 0x01852c thru 0x019098 (0xb6d bytes) 000682 1789182729.999809 mapwrite 0x037e9f thru 0x03ffff (0x8161 bytes) 000683 1789182730.061250 MIRROR_RESYNC 14 mirrors 000684 1789182734.446087 punch from 0x03884b to 0x03ffff (0x77b5 bytes) 000685 1789182734.492662 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa07a824a (stale) Mirror 8: 0xa07a824a (stale) Mirror 10: 0xa07a824a (stale) Mirror 11: 0xa07a824a (stale) Mirror 12: 0xa07a824a (stale) Mirror 13: 0xa07a824a (stale) Mirror 15: 0xa07a824a (stale) Mirror 16: 0xa07a824a (stale) Mirror 20: 0xa07a824a (stale) Mirror 21: 0xa07a824a (stale) Mirror 22: 0xa07a824a (stale) Mirror 23: 0xa07a824a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa07a824a (stale) Mirror 8: 0xa07a824a (stale) Mirror 10: 0xa07a824a (stale) Mirror 11: 0xa07a824a (stale) Mirror 12: 0xa07a824a (stale) Mirror 13: 0xa07a824a (stale) Mirror 15: 0xa07a824a (stale) Mirror 16: 0xa07a824a (stale) Mirror 20: 0xa07a824a (stale) Mirror 21: 0xa07a824a (stale) Mirror 22: 0xa07a824a (stale) Mirror 23: 0xa07a824a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000687 1789182735.124109 mapwrite 0x03f5d3 thru 0x03ffff (0xa2d bytes) 000689 1789182735.145353 trunc from 0x040000 to 0x01e47b (0x21b84 bytes) 000690 1789182735.167373 trunc from 0x017fdd to 0x01e47b (0x649f bytes) 000691 1789182735.176105 mapread 0x000b3d thru 0x00d16d (0xc631 bytes) 000692 1789182735.192000 mapread 0x008138 thru 0x00cab2 (0x497b bytes) 000693 1789182735.193379 mapread 0x01944a thru 0x01dd81 (0x4938 bytes) 000694 1789182735.203763 read 0x001d25 thru 0x0058b8 (0x3b94 bytes) 000695 1789182735.204874 write 0x027f5c thru 0x030d79 (0x8e1e bytes) 000696 1789182735.206704 mapwrite 0x00b53b thru 0x00f6fe (0x41c4 bytes) 000697 1789182735.239112 MIRROR_RESYNC 13 mirrors 000698 1789182738.501540 trunc from 0x01e421 to 0x025829 (0x7409 bytes) 000699 1789182738.558329 mapread 0x020953 thru 0x028845 (0x7ef3 bytes) 000700 1789182738.585375 fallocate 0x00b353 thru 0x01a727 (0xf3d5 bytes) 000701 1789182738.590285 trunc from 0x030d7a to 0x03020a (0xb6f bytes) 000702 1789182738.617778 fallocate 0x010775 thru 0x015030 (0x48bc bytes) 000703 1789182738.624669 mapread 0x00c5d0 thru 0x019887 (0xd2b8 bytes) 000704 1789182738.641993 write 0x013bd8 thru 0x01f1e9 (0xb612 bytes) 000705 1789182738.649283 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3020b): Mirror 7: 0xea946dea (stale) Mirror 8: 0xea946dea (stale) Mirror 10: 0xea946dea (stale) Mirror 11: 0xea946dea (stale) Mirror 12: 0xea946dea (stale) Mirror 13: 0xea946dea (stale) Mirror 16: 0xea946dea (stale) Mirror 20: 0xea946dea (stale) Mirror 21: 0xea946dea (stale) Mirror 22: 0xea946dea (stale) Mirror 23: 0xea946dea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3020b): Mirror 7: 0xea946dea (stale) Mirror 8: 0xea946dea (stale) Mirror 10: 0xea946dea (stale) Mirror 11: 0xea946dea (stale) Mirror 12: 0xea946dea (stale) Mirror 13: 0xea946dea (stale) Mirror 16: 0xea946dea (stale) Mirror 20: 0xea946dea (stale) Mirror 21: 0xea946dea (stale) Mirror 22: 0xea946dea (stale) Mirror 23: 0xea946dea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3020b: skipped 000706 1789182739.345160 fallocate 0x034529 thru 0x03ffff (0xbad7 bytes) 000707 1789182739.351645 mapread 0x010b69 thru 0x01b79f (0xac37 bytes) 000708 1789182739.385624 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xec43cf04 (stale) Mirror 10: 0xec43cf04 (stale) Mirror 11: 0xec43cf04 (stale) Mirror 12: 0xec43cf04 (stale) Mirror 13: 0xec43cf04 (stale) Mirror 16: 0xec43cf04 (stale) Mirror 20: 0xec43cf04 (stale) Mirror 21: 0xec43cf04 (stale) Mirror 22: 0xec43cf04 (stale) Mirror 23: 0xec43cf04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xec43cf04 (stale) Mirror 10: 0xec43cf04 (stale) Mirror 11: 0xec43cf04 (stale) Mirror 12: 0xec43cf04 (stale) Mirror 13: 0xec43cf04 (stale) Mirror 16: 0xec43cf04 (stale) Mirror 20: 0xec43cf04 (stale) Mirror 21: 0xec43cf04 (stale) Mirror 22: 0xec43cf04 (stale) Mirror 23: 0xec43cf04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000709 1789182740.008544 mapwrite 0x012634 thru 0x01b6ca (0x9097 bytes) 000710 1789182740.107644 mapwrite 0x037bad thru 0x03e6ab (0x6aff bytes) 000711 1789182740.161198 trunc from 0x01371b to 0x0182fa (0x4be0 bytes) 000712 1789182740.173934 fallocate 0x002384 thru 0x00ad09 (0x8986 bytes) 000713 1789182740.182114 punch from 0x022ab3 to 0x0307f8 (0xdd46 bytes) 000714 1789182740.191594 write 0x017d66 thru 0x01c47d (0x4718 bytes) 000716 1789182740.209537 trunc from 0x040000 to 0x038851 (0x77ae bytes) 000717 1789182740.238678 trunc from 0x02b046 to 0x034b7a (0x9b35 bytes) 000718 1789182740.248917 punch from 0x03036c to 0x032fd8 (0x2c6d bytes) 000719 1789182740.257242 trunc from 0x038852 to 0x005459 (0x333f8 bytes) 000720 1789182740.278250 mapread 0x003934 thru 0x005459 (0x1b26 bytes) 000721 1789182740.291359 fallocate 0x00dff4 thru 0x01941c (0xb429 bytes) 000722 1789182740.297875 mapread 0x00f49d thru 0x01941c (0x9f80 bytes) 000723 1789182740.331408 trunc from 0x01941d to 0x027fba (0xeb9e bytes) 000724 1789182740.361413 read 0x0087fc thru 0x009707 (0xf0c bytes) 000725 1789182740.362569 write 0x02f14e thru 0x0392a4 (0xa157 bytes) 000726 1789182740.366890 fallocate 0x00df4f thru 0x01a447 (0xc4f9 bytes) 000727 1789182740.379311 MIRROR_RESYNC 11 mirrors 000728 1789182742.425514 read 0x025561 thru 0x026175 (0xc15 bytes) 000729 1789182742.438074 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000730 1789182742.985155 trunc from 0x000bae to 0x002925 (0x1d78 bytes) 000731 1789182743.062696 trunc from 0x01a74b to 0x020717 (0x5fcd bytes) 000732 1789182743.076962 write 0x0179a4 thru 0x020ff4 (0x9651 bytes) 000733 1789182743.094294 trunc from 0x0392a5 to 0x02bade (0xd7c6 bytes) 000734 1789182743.130480 mapwrite 0x006e58 thru 0x00dbd2 (0x6d7b bytes) 000735 1789182743.181637 punch from 0x00b7f6 to 0x010b52 (0x535d bytes) 000736 1789182743.187844 punch from 0x002d63 to 0x00e276 (0xb514 bytes) 000737 1789182743.193863 read 0x00172a thru 0x0114d1 (0xfda8 bytes) 000738 1789182743.208159 mapwrite 0x000a43 thru 0x00af24 (0xa4e2 bytes) 000739 1789182743.241257 mapwrite 0x008691 thru 0x00bdd1 (0x3741 bytes) 000740 1789182743.260584 mapread 0x027bb4 thru 0x02bade (0x3f2b bytes) 000741 1789182743.274013 punch from 0x010d2d to 0x01e0bb (0xd38f bytes) 000742 1789182743.280968 write 0x010350 thru 0x01c12d (0xbdde bytes) 000743 1789182743.298336 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2badf): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2badf): Mirror 7: 0x14946b4c (stale) Mirror 10: 0x14946b4c (stale) Mirror 12: 0x14946b4c (stale) Mirror 13: 0x14946b4c (stale) Mirror 16: 0x14946b4c (stale) Mirror 20: 0x14946b4c (stale) Mirror 21: 0x14946b4c (stale) Mirror 22: 0x14946b4c (stale) Mirror 23: 0x14946b4c (stale) CRC-32 checksum value for chunk [0, 0x2badf): Mirror 1: 0xeeee8bc7 Mirror 24: 0xeeee8bc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2badf: skipped 000744 1789182744.789517 mapread 0x01ba26 thru 0x021c83 (0x625e bytes) 000745 1789182744.843999 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2badf): Mirror 10: 0x14946b4c (stale) Mirror 12: 0x14946b4c (stale) Mirror 13: 0x14946b4c (stale) Mirror 16: 0x14946b4c (stale) Mirror 20: 0x14946b4c (stale) Mirror 21: 0x14946b4c (stale) Mirror 22: 0x14946b4c (stale) Mirror 23: 0x14946b4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2badf): Mirror 10: 0x14946b4c (stale) Mirror 12: 0x14946b4c (stale) Mirror 13: 0x14946b4c (stale) Mirror 16: 0x14946b4c (stale) Mirror 20: 0x14946b4c (stale) Mirror 21: 0x14946b4c (stale) Mirror 22: 0x14946b4c (stale) Mirror 23: 0x14946b4c (stale) CRC-32 checksum value for chunk [0, 0x2badf): Mirror 1: 0xeeee8bc7 Mirror 24: 0xeeee8bc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2badf: skipped 000746 1789182745.937957 trunc from 0x02badf to 0x021312 (0xa7cc bytes) 000747 1789182746.047478 trunc from 0x01ea56 to 0x021312 (0x28bd bytes) 000748 1789182746.054320 mapread 0x013631 thru 0x021312 (0xdce2 bytes) 000749 1789182746.082244 read 0x008bed thru 0x010478 (0x788c bytes) 000750 1789182746.095843 mapwrite 0x03e8a4 thru 0x03ffff (0x175c bytes) 000751 1789182746.150694 write 0x0240bb thru 0x0301bd (0xc103 bytes) 000752 1789182746.169244 fallocate 0x00baea thru 0x014a60 (0x8f77 bytes) 000754 1789182746.176534 punch from 0x03c493 to 0x03ffff (0x3b6d bytes) 000755 1789182746.181989 mapread 0x03a269 thru 0x03ffff (0x5d97 bytes) 000756 1789182746.203666 MIRROR_RESYNC 10 mirrors 000757 1789182749.838502 punch from 0x01845c to 0x026440 (0xdfe5 bytes) 000758 1789182749.883624 trunc from 0x040000 to 0x00d581 (0x32a7e bytes) 000759 1789182749.907625 fallocate 0x004ab0 thru 0x011d6d (0xd2be bytes) 000760 1789182749.913560 trunc from 0x00eed7 to 0x011d6d (0x2e97 bytes) 000761 1789182749.918523 write 0x02b1f8 thru 0x02ff24 (0x4d2d bytes) 000762 1789182749.920185 read 0x01693f thru 0x0188c6 (0x1f88 bytes) 000763 1789182749.925963 mapwrite 0x0064f2 thru 0x00e03a (0x7b49 bytes) 000764 1789182749.967829 trunc from 0x02ff25 to 0x00d4f3 (0x22a31 bytes) 000766 1789182749.995907 read 0x001eeb thru 0x00d0be (0xb1d4 bytes) 000767 1789182750.001307 mapread 0x00aa9f thru 0x00d4f3 (0x2a55 bytes) 000768 1789182750.013439 punch from 0x00cfd5 to 0x00d4f3 (0x51f bytes) 000769 1789182750.026595 mapread 0x00576b thru 0x00d4f3 (0x7d89 bytes) 000770 1789182750.060094 mapread 0x008f50 thru 0x009123 (0x1d4 bytes) 000771 1789182750.062481 punch from 0x000ce1 to 0x00adec (0xa10c bytes) 000772 1789182750.072209 fallocate 0x00b13e thru 0x018eae (0xdd71 bytes) 000773 1789182750.080012 fallocate 0x00cecc thru 0x00e0e7 (0x121c bytes) 000774 1789182750.103645 trunc from 0x0019c4 to 0x00542b (0x3a68 bytes) 000775 1789182750.115586 mapread 0x00632d thru 0x00e0e7 (0x7dbb bytes) 000776 1789182750.168273 read 0x001183 thru 0x00d337 (0xc1b5 bytes) 000777 1789182750.187163 mapwrite 0x03dd1c thru 0x03ffff (0x22e4 bytes) 000778 1789182750.264289 read 0x0076be thru 0x00864c (0xf8f bytes) 000779 1789182750.268514 read 0x016311 thru 0x02630e (0xfffe bytes) 000780 1789182750.269474 fallocate 0x021d08 thru 0x025ca4 (0x3f9d bytes) 000781 1789182750.279216 read 0x03ba94 thru 0x03ffff (0x456c bytes) 000782 1789182750.283156 punch from 0x03b0eb to 0x03ffff (0x4f15 bytes) 000783 1789182750.288531 punch from 0x00fa5c to 0x01bdce (0xc373 bytes) 000784 1789182750.297179 punch from 0x01d6ac to 0x02c7a7 (0xf0fc bytes) 000785 1789182750.314933 punch from 0x008944 to 0x010dd9 (0x8496 bytes) 000787 1789182750.331890 trunc from 0x00e7c1 to 0x00f320 (0xb60 bytes) 000788 1789182750.347335 write 0x00cad4 thru 0x0141a3 (0x76d0 bytes) 000789 1789182750.370624 mapread 0x022b60 thru 0x03230c (0xf7ad bytes) 000790 1789182750.372391 write 0x009588 thru 0x010f74 (0x79ed bytes) 000791 1789182750.385139 fallocate 0x00a7cf thru 0x00f90e (0x5140 bytes) 000792 1789182750.404999 mapwrite 0x039540 thru 0x03b1c0 (0x1c81 bytes) 000793 1789182750.449010 fallocate 0x03c807 thru 0x03f8da (0x30d4 bytes) 000794 1789182750.461328 write 0x020af5 thru 0x02c6f8 (0xbc04 bytes) 000795 1789182750.464294 trunc from 0x040000 to 0x02fdb4 (0x1024b bytes) 000796 1789182750.492252 MIRROR_RESYNC 10 mirrors 000797 1789182753.100158 punch from 0x00a70f to 0x00be04 (0x16f6 bytes) 000798 1789182753.141296 trunc from 0x02fdb5 to 0x0301d7 (0x423 bytes) 000799 1789182753.159068 mapread 0x024b83 thru 0x027e5f (0x32dd bytes) 000800 1789182753.172411 punch from 0x02dbf3 to 0x0301d7 (0x25e5 bytes) 000801 1789182753.182138 MIRROR_RESYNC 10 mirrors 000802 1789182755.701487 read 0x022ac0 thru 0x02ee53 (0xc394 bytes) 000803 1789182755.710920 fallocate 0x02573f thru 0x034859 (0xf11b bytes) 000804 1789182755.793901 trunc from 0x02b0e5 to 0x03334d (0x8269 bytes) 000805 1789182755.805491 write 0x02e8b0 thru 0x038b65 (0xa2b6 bytes) 000808 1789182755.826455 write 0x026b33 thru 0x03675b (0xfc29 bytes) 000809 1789182755.842730 punch from 0x034a05 to 0x03675a (0x1d56 bytes) 000810 1789182755.863385 fallocate 0x000d2d thru 0x003c86 (0x2f5a bytes) 000812 1789182755.869529 trunc from 0x038b66 to 0x007f65 (0x30c00 bytes) 000813 1789182755.892666 trunc from 0x002b42 to 0x007f65 (0x5424 bytes) 000814 1789182755.899203 write 0x01ac39 thru 0x01c69f (0x1a67 bytes) 000816 1789182755.901182 read 0x01915a thru 0x01c69f (0x3546 bytes) 000817 1789182755.914913 write 0x035b00 thru 0x0363e1 (0x8e2 bytes) 000818 1789182755.916828 punch from 0x033e07 to 0x0363e1 (0x25db bytes) 000819 1789182755.938069 trunc from 0x0363e2 to 0x03ef64 (0x8b83 bytes) 000820 1789182755.959597 write 0x009292 thru 0x0135f6 (0xa365 bytes) 000821 1789182755.963171 fallocate 0x0067ab thru 0x00b351 (0x4ba7 bytes) 000822 1789182755.969426 mapread 0x001a32 thru 0x00c914 (0xaee3 bytes) 000823 1789182755.971239 fallocate 0x007cd4 thru 0x01794d (0xfc7a bytes) 000824 1789182755.976976 mapread 0x02228f thru 0x0250ae (0x2e20 bytes) 000825 1789182755.988187 write 0x01d725 thru 0x02196c (0x4248 bytes) 000826 1789182755.990141 trunc from 0x00cce1 to 0x00e8a5 (0x1bc5 bytes) 000827 1789182756.004782 fallocate 0x023acc thru 0x02f6ee (0xbc23 bytes) 000829 1789182756.009939 fallocate 0x007434 thru 0x011d5b (0xa928 bytes) 000830 1789182756.019953 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 10: 0x3421e9b7 (stale) Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 22: 0x3421e9b7 (stale) Mirror 23: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 10: 0x3421e9b7 (stale) Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 22: 0x3421e9b7 (stale) Mirror 23: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef65: skipped 000831 1789182756.730852 mapwrite 0x00d7e1 thru 0x016f5c (0x977c bytes) 000832 1789182756.772503 write 0x01bfd5 thru 0x02206f (0x609b bytes) 000833 1789182756.789590 fallocate 0x024058 thru 0x02f07c (0xb025 bytes) 000834 1789182756.795124 fallocate 0x029ad0 thru 0x037968 (0xde99 bytes) 000836 1789182756.803121 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 22: 0x3421e9b7 (stale) Mirror 23: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 22: 0x3421e9b7 (stale) Mirror 23: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef65: skipped 000837 1789182757.302097 punch from 0x0085b2 to 0x011d53 (0x97a2 bytes) 000838 1789182757.305906 punch from 0x022bfb to 0x02a316 (0x771c bytes) 000839 1789182757.310167 write 0x01c600 thru 0x0236bf (0x70c0 bytes) 000841 1789182757.322341 punch from 0x00dc35 to 0x011d31 (0x40fd bytes) 000842 1789182757.327022 mapread 0x00e8f1 thru 0x01d78d (0xee9d bytes) 000843 1789182757.353966 fallocate 0x00900a thru 0x01631f (0xd316 bytes) 000844 1789182757.359648 fallocate 0x006ab3 thru 0x00961a (0x2b68 bytes) 000845 1789182757.376672 mapread 0x030a99 thru 0x034f44 (0x44ac bytes) 000846 1789182757.391898 read 0x00028f thru 0x008137 (0x7ea9 bytes) 000848 1789182757.410486 punch from 0x031242 to 0x03ef64 (0xdd23 bytes) 000849 1789182757.415987 trunc from 0x030f40 to 0x031fbc (0x107d bytes) 000850 1789182757.425434 read 0x014754 thru 0x01e0de (0x998b bytes) 000852 1789182757.426620 read 0x02d778 thru 0x0367c5 (0x904e bytes) 000853 1789182757.467777 mapwrite 0x034631 thru 0x03876a (0x413a bytes) 000855 1789182757.531838 punch from 0x019e31 to 0x01f7e6 (0x59b6 bytes) 000856 1789182757.541107 punch from 0x020927 to 0x0274a0 (0x6b7a bytes) 000858 1789182757.548672 read 0x0047dc thru 0x010431 (0xbc56 bytes) 000859 1789182757.555983 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 23: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 23: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef65: skipped 000860 1789182758.135850 mapread 0x038f1d thru 0x03ef64 (0x6048 bytes) 000861 1789182758.160841 write 0x01af39 thru 0x028541 (0xd609 bytes) 000862 1789182758.184546 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 16: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef65: skipped 000863 1789182758.650362 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef65: skipped 000864 1789182758.995028 write 0x0213a5 thru 0x0255ab (0x4207 bytes) 000865 1789182759.012247 write 0x01f563 thru 0x0262be (0x6d5c bytes) 000866 1789182759.023499 punch from 0x021f96 to 0x02ffb2 (0xe01d bytes) 000867 1789182759.046103 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 12: 0x3421e9b7 (stale) Mirror 13: 0x3421e9b7 (stale) Mirror 20: 0x3421e9b7 (stale) Mirror 24: 0x3421e9b7 (stale) CRC-32 checksum value for chunk [0, 0x3ef65): Mirror 1: 0x7457e979 Mirror 25: 0x7457e979 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef65: skipped 000868 1789182760.282104 trunc from 0x03ef65 to 0x0328b0 (0xc6b4 bytes) 000869 1789182760.369188 trunc from 0x029529 to 0x0328b0 (0x9388 bytes) 000870 1789182760.377901 read 0x011906 thru 0x0189e0 (0x70db bytes) 000871 1789182760.385403 trunc from 0x0328b1 to 0x007d1f (0x2ab91 bytes) 000872 1789182760.415687 mapwrite 0x010271 thru 0x011687 (0x1417 bytes) 000873 1789182760.459345 fallocate 0x027e51 thru 0x035328 (0xd4d8 bytes) 000874 1789182760.464566 mapwrite 0x00385d thru 0x009b95 (0x6339 bytes) 000875 1789182760.498094 punch from 0x001c53 to 0x00b623 (0x99d1 bytes) 000876 1789182760.505513 punch from 0x016c5f to 0x025bad (0xef4f bytes) 000877 1789182760.512431 write 0x03ab0c thru 0x03ffff (0x54f4 bytes) 000878 1789182760.515906 mapwrite 0x03af3e thru 0x03ffff (0x50c2 bytes) 000879 1789182760.538598 trunc from 0x030d58 to 0x03304d (0x22f6 bytes) 000880 1789182760.544092 trunc from 0x00d803 to 0x016a8e (0x928c bytes) 000881 1789182760.550398 punch from 0x01faa5 to 0x02ded4 (0xe430 bytes) 000882 1789182760.555549 write 0x0150ec thru 0x01f547 (0xa45c bytes) 000883 1789182760.565307 write 0x018457 thru 0x0247e1 (0xc38b bytes) 000884 1789182760.570637 mapwrite 0x01ab41 thru 0x02100e (0x64ce bytes) 000885 1789182760.584567 read 0x008d2d thru 0x01442a (0xb6fe bytes) 000886 1789182760.596163 punch from 0x01325f to 0x01a2e5 (0x7087 bytes) 000887 1789182760.602512 read 0x02048b thru 0x023825 (0x339b bytes) 000888 1789182760.603574 mapread 0x00cc67 thru 0x0141c7 (0x7561 bytes) 000889 1789182760.611694 trunc from 0x040000 to 0x016cc0 (0x2933f bytes) 000890 1789182760.632971 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16cc1): Mirror 12: 0x437b41e5 (stale) Mirror 13: 0x437b41e5 (stale) Mirror 20: 0x437b41e5 (stale) Mirror 24: 0x437b41e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cc1): Mirror 12: 0x437b41e5 (stale) Mirror 13: 0x437b41e5 (stale) Mirror 20: 0x437b41e5 (stale) Mirror 24: 0x437b41e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cc1: skipped 000892 1789182761.006693 fallocate 0x007403 thru 0x009030 (0x1c2e bytes) 000894 1789182761.026408 fallocate 0x01a2f0 thru 0x0229ae (0x86bf bytes) 000895 1789182761.036692 punch from 0x00e162 to 0x00fcef (0x1b8e bytes) 000896 1789182761.051159 write 0x03258c thru 0x03739e (0x4e13 bytes) 000898 1789182761.062525 fallocate 0x01e1df thru 0x02cd23 (0xeb45 bytes) 000899 1789182761.071575 punch from 0x021d45 to 0x02ef9f (0xd25b bytes) 000900 1789182761.082038 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3739f): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3739f): Mirror 12: 0x9670183 (stale) Mirror 13: 0x9670183 (stale) Mirror 20: 0x9670183 (stale) Mirror 24: 0x9670183 (stale) CRC-32 checksum value for chunk [0, 0x3739f): Mirror 1: 0x863b10f1 Mirror 25: 0x863b10f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3739f: skipped 000901 1789182762.351336 mapwrite 0x02f991 thru 0x03e312 (0xe982 bytes) 000902 1789182762.459225 trunc from 0x03e313 to 0x02f269 (0xf0a9 bytes) 000903 1789182762.490421 write 0x024f4f thru 0x02adc9 (0x5e7b bytes) 000904 1789182762.504928 punch from 0x01bb0e to 0x020f81 (0x5474 bytes) 000906 1789182762.512042 trunc from 0x028853 to 0x02f269 (0x6a17 bytes) 000907 1789182762.528721 trunc from 0x0036c8 to 0x006867 (0x31a0 bytes) 000908 1789182762.535220 mapwrite 0x00ef68 thru 0x018742 (0x97db bytes) 000909 1789182762.577532 read 0x004db8 thru 0x006aba (0x1d03 bytes) 000910 1789182762.584360 punch from 0x00115e to 0x0060e9 (0x4f8c bytes) 000911 1789182762.593288 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f26a): Mirror 12: 0x158f1b1 (stale) Mirror 13: 0x158f1b1 (stale) Mirror 20: 0x158f1b1 (stale) Mirror 24: 0x158f1b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f26a): Mirror 12: 0x158f1b1 (stale) Mirror 13: 0x158f1b1 (stale) Mirror 20: 0x158f1b1 (stale) Mirror 24: 0x158f1b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f26a: skipped 000913 1789182763.016497 read 0x00a730 thru 0x01a0e8 (0xf9b9 bytes) 000914 1789182763.034119 write 0x0211c6 thru 0x022098 (0xed3 bytes) 000916 1789182763.070818 read 0x004b16 thru 0x00bbf6 (0x70e1 bytes) 000917 1789182763.084660 trunc from 0x02f26a to 0x0355f8 (0x638f bytes) 000918 1789182763.110910 write 0x00afb8 thru 0x00c7ab (0x17f4 bytes) 000919 1789182763.111720 read 0x0082fd thru 0x00df0c (0x5c10 bytes) 000920 1789182763.112172 trunc from 0x0355f9 to 0x012be1 (0x22a17 bytes) 000921 1789182763.134402 mapread 0x004756 thru 0x00b597 (0x6e42 bytes) 000922 1789182763.141020 mapread 0x0031f9 thru 0x009812 (0x661a bytes) 000923 1789182763.143719 punch from 0x00cced to 0x012be1 (0x5ef5 bytes) 000924 1789182763.156607 mapwrite 0x031fe7 thru 0x039a21 (0x7a3b bytes) 000925 1789182763.216583 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39a22): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a22): Mirror 12: 0x82342544 (stale) Mirror 13: 0x82342544 (stale) Mirror 20: 0x82342544 (stale) Mirror 24: 0x82342544 (stale) CRC-32 checksum value for chunk [0, 0x39a22): Mirror 1: 0x2744b32f Mirror 25: 0x2744b32f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a22: skipped 000926 1789182764.414195 trunc from 0x039a22 to 0x025625 (0x143fc bytes) 000927 1789182764.539404 MIRROR_RESYNC 6 mirrors 000928 1789182766.487731 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000929 1789182766.818476 trunc from 0x025626 to 0x0058c9 (0x1fd5c bytes) 000930 1789182766.926509 read 0x001908 thru 0x0058c9 (0x3fc2 bytes) 000931 1789182766.960520 mapread 0x0050ea thru 0x0058c9 (0x7e0 bytes) 000932 1789182766.961480 mapread 0x00460f thru 0x0058c9 (0x12bb bytes) 000933 1789182766.962303 trunc from 0x003b7c to 0x0058c9 (0x1d4e bytes) 000934 1789182766.979544 write 0x02bda7 thru 0x0356b8 (0x9912 bytes) 000935 1789182766.992519 MIRROR_RESYNC 5 mirrors 000937 1789182767.959366 mapwrite 0x008cb3 thru 0x0165e5 (0xd933 bytes) 000938 1789182768.134584 fallocate 0x0306e3 thru 0x036c3d (0x655b bytes) 000939 1789182768.150902 punch from 0x00d342 to 0x00dfd7 (0xc96 bytes) 000940 1789182768.159635 write 0x02e12a thru 0x035b2f (0x7a06 bytes) 000941 1789182768.185973 fallocate 0x02ce02 thru 0x034326 (0x7525 bytes) 000942 1789182768.192136 trunc from 0x003257 to 0x0078bf (0x4669 bytes) 000943 1789182768.205172 trunc from 0x00d0d2 to 0x00e910 (0x183f bytes) 000944 1789182768.216712 read 0x01ee8e thru 0x01f7e4 (0x957 bytes) 000945 1789182768.230981 trunc from 0x014bbc to 0x015779 (0xbbe bytes) 000946 1789182768.256939 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36c3e): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c3e): Mirror 12: 0xfcd0f4c8 (stale) Mirror 13: 0xfcd0f4c8 (stale) Mirror 24: 0xfcd0f4c8 (stale) Mirror 25: 0xfcd0f4c8 (stale) CRC-32 checksum value for chunk [0, 0x36c3e): Mirror 1: 0x7b50d90b Mirror 26: 0x7b50d90b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c3e: skipped 000947 1789182769.737127 fallocate 0x003733 thru 0x00a5f9 (0x6ec7 bytes) 000948 1789182769.774301 fallocate 0x0343bb thru 0x035b3c (0x1782 bytes) 000949 1789182769.793365 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36c3e): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c3e): Mirror 12: 0xfcd0f4c8 (stale) Mirror 13: 0xfcd0f4c8 (stale) Mirror 24: 0xfcd0f4c8 (stale) Mirror 25: 0xfcd0f4c8 (stale) Mirror 26: 0x7b50d90b (stale) CRC-32 checksum value for chunk [0, 0x36c3e): Mirror 1: 0x7b50d90b Mirror 27: 0x7b50d90b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c3e: skipped 000950 1789182770.912330 punch from 0x01f1a8 to 0x023c78 (0x4ad1 bytes) 000951 1789182770.949776 read 0x011a63 thru 0x02146c (0xfa0a bytes) 000952 1789182770.962547 read 0x02acc9 thru 0x036c3d (0xbf75 bytes) 000953 1789182770.969217 read 0x00a51c thru 0x010607 (0x60ec bytes) 000954 1789182770.969904 write 0x009881 thru 0x010de3 (0x7563 bytes) 000955 1789182770.978628 trunc from 0x036c3e to 0x016fb1 (0x1fc8c bytes) 000956 1789182771.004698 write 0x010ae8 thru 0x018577 (0x7a90 bytes) 000957 1789182771.007634 trunc from 0x016ad7 to 0x018577 (0x1aa1 bytes) 000958 1789182771.033699 punch from 0x012cd3 to 0x018577 (0x58a5 bytes) 000959 1789182771.046076 trunc from 0x00c0fe to 0x00f76c (0x366f bytes) 000960 1789182771.050922 punch from 0x005e3d to 0x00a9cb (0x4b8f bytes) 000962 1789182771.057722 read 0x008657 thru 0x012d14 (0xa6be bytes) 000963 1789182771.076807 trunc from 0x010198 to 0x0182ae (0x8117 bytes) 000964 1789182771.083124 read 0x005145 thru 0x0078fa (0x27b6 bytes) 000965 1789182771.083866 trunc from 0x00bddd to 0x00d3b1 (0x15d5 bytes) 000966 1789182771.088781 mapread 0x00b44f thru 0x016b0c (0xb6be bytes) 000967 1789182771.111492 mapread 0x00a8c0 thru 0x0109f7 (0x6138 bytes) 000968 1789182771.115125 write 0x016513 thru 0x0224e5 (0xbfd3 bytes) 000969 1789182771.118351 mapread 0x01dbaf thru 0x0224e5 (0x4937 bytes) 000970 1789182771.119389 write 0x02c153 thru 0x030295 (0x4143 bytes) 000971 1789182771.126441 write 0x01d263 thru 0x01eb33 (0x18d1 bytes) 000972 1789182771.131004 trunc from 0x026ec5 to 0x030295 (0x93d1 bytes) 000973 1789182771.160148 fallocate 0x03f286 thru 0x03ffff (0xd7a bytes) 000974 1789182771.175472 read 0x0139ca thru 0x01c824 (0x8e5b bytes) 000975 1789182771.187048 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30296): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30296): Mirror 12: 0x50cc76ec (stale) Mirror 13: 0x50cc76ec (stale) Mirror 24: 0x50cc76ec (stale) Mirror 25: 0x50cc76ec (stale) Mirror 26: 0xf7c7a2de (stale) Mirror 27: 0xf7c7a2de (stale) CRC-32 checksum value for chunk [0, 0x30296): Mirror 1: 0x2945a7ae Mirror 28: 0x2945a7ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30296: skipped 000976 1789182772.760249 trunc from 0x02983b to 0x030295 (0x6a5b bytes) 000977 1789182772.856140 punch from 0x0140ec to 0x020973 (0xc888 bytes) 000978 1789182772.868591 punch from 0x02c04f to 0x030295 (0x4247 bytes) 000979 1789182772.877012 trunc from 0x028e82 to 0x030295 (0x7414 bytes) 000980 1789182772.887224 read 0x02046d thru 0x02c993 (0xc527 bytes) 000981 1789182772.908040 read 0x0267c0 thru 0x02a9d0 (0x4211 bytes) 000982 1789182772.910013 trunc from 0x023de1 to 0x02fec5 (0xc0e5 bytes) 000983 1789182772.915264 read 0x0235cb thru 0x030134 (0xcb6a bytes) 000984 1789182772.956974 mapread 0x0220ca thru 0x02f24e (0xd185 bytes) 000985 1789182772.960203 trunc from 0x00a10a to 0x01114b (0x7042 bytes) 000986 1789182772.976809 trunc from 0x00029d to 0x00b36e (0xb0d2 bytes) 000987 1789182772.989435 punch from 0x017bc4 to 0x019f68 (0x23a5 bytes) 000988 1789182772.999123 punch from 0x002857 to 0x00920f (0x69b9 bytes) 000990 1789182773.013552 mapwrite 0x03e524 thru 0x03ffff (0x1adc bytes) 000991 1789182773.131446 mapwrite 0x00a874 thru 0x0175dd (0xcd6a bytes) 000992 1789182773.261672 mapread 0x0381d1 thru 0x03ffff (0x7e2f bytes) 000994 1789182773.266493 read 0x019428 thru 0x01ff89 (0x6b62 bytes) 000995 1789182773.279064 read 0x01c951 thru 0x029e9b (0xd54b bytes) 000996 1789182773.279873 punch from 0x03b03d to 0x03f63c (0x4600 bytes) 000997 1789182773.291321 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7d48dd0a (stale) Mirror 13: 0x7d48dd0a (stale) Mirror 24: 0x7d48dd0a (stale) Mirror 25: 0x7d48dd0a (stale) Mirror 26: 0x5690cc21 (stale) Mirror 27: 0x5690cc21 (stale) Mirror 28: 0x7a9ccb06 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5649eee Mirror 29: 0xf5649eee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000998 1789182774.823039 trunc from 0x033697 to 0x03d479 (0x9de3 bytes) 000999 1789182774.850340 trunc from 0x00170b to 0x01030f (0xec05 bytes) 001000 1789182774.854897 trunc from 0x040000 to 0x00aacf (0x35530 bytes) All operations completed A-OK! PASS 70a (128s) == sanity-flr test 71: check mirror extend parent fid ==== 23:13:00 (1789182780) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x198:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 23:13:10 (1789182790) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.321293 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085703 s, 12.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 23:13:20 (1789182800) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8159840 .. Extending file size to 7717120 .. Extending file size to 7980960 .. Extending file size to 8237216 .. Extending file size to 8364640 .. Extending file size to 8512160 .. Extending file size to 9087840 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8620640 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9039008 .. Extending file size to 9079328 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61347 61348 61351 61352 61353 done 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) 192.168.202.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:14:42 (1789182882) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6450592 .. Extending file size to 1358624 .. Extending file size to 7417120 .. Extending file size to 8380288 .. Extending file size to 8198880 .. Extending file size to 8381376 .. Extending file size to 8257696 .. Extending file size to 8870336 .. Extending file size to 8878816 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9187520 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62971 62973 62974 62976 62979 done 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) 192.168.202.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:16:10 (1789182970) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18908 3588112 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24740 3582280 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43648 7170392 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18908 3588112 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24740 3582280 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43648 7170392 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114358 s, 18.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248911 s, 8.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xaf96dd53 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:16:30 (1789182990) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108926 s, 19.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:16:38 (1789182998) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:16:46 (1789183006) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112442 s, 18.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:16:55 (1789183015) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0780438 s, 26.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:17:05 (1789183025) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117632 s, 17.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:17:40 (1789183060) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg257-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498451 s, 21.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:18:34 (1789183114) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg257-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175664 s, 11.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:19:29 (1789183169) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg257-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg257-server: Pool lustre.test_204f created Adding targets to pool oleg257-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg257-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149766 s, 14.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg257-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg257-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg257-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:20:47 (1789183247) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:20:55 (1789183255) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:21:03 (1789183263) oleg257-server: Pool lustre.test_206 created Adding targets to pool oleg257-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg257-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg257-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg257-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg257-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:21:31 (1789183291) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892343 s, 11.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363423 s, 28.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2ed0b6d6 Mirror 3: 0x2ed0b6d6 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:21:43 (1789183303) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:21:47 (1789183307) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:21:52 (1789183312) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.096439 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1cd0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1550000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1550000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1550000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe50000 Read this (4 bytes): BBBB 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:22:35 (1789183355) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0184871 s, 443 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:22:45 (1789183365) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626271 s, 16.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:22:48.000000000 -0400 Modify: 2026-09-11 23:22:48.000000000 -0400 Change: 2026-09-11 23:22:50.000000000 -0400 Birth: 2026-09-11 23:22:48.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:22:57 (1789183377) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651134 s, 16.1 MB/s oleg257-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg257-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg257-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (30s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:23:27 (1789183407) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.232272 s, 176 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.033833 s, 121 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 23:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 23:23 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:23:36 (1789183416) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:23:45 (1789183425) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1962 sec =========== 23:23:55 (1789183435) === sanity-flr: start cleanup 23:23:56 (1789183436) === === sanity-flr: finish cleanup 23:24:04 (1789183444) ===