-----============= acceptance-small: sanity-flr ============----- Wed Jun 3 20:36:18 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg312-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:36:25 (1780533385) === oleg312-client.virtnet: executing check_config_client /mnt/lustre oleg312-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg312-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8acaa0e7a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8acaa0e7a800.idle_timeout=debug disable quota as required oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:36:39 (1780533399) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8174] == sanity-flr test 0a: lfs mirror create with -N option == 20:36:40 (1780533400) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:36:46 (1780533406) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:36:47 (1780533407) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:36:49 (1780533409) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:36:58 (1780533418) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:36:59 (1780533419) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 20:37:01 (1780533421) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8acaa0e7a800.stats=clear osc.lustre-OST0001-osc-ffff8acaa0e7a800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:37:04 (1780533424) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:37:09 (1780533429) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:37:13 (1780533433) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:37:15 (1780533435) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:37:18 (1780533438) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 20:37:23 (1780533443) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:37:26 (1780533446) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:37:30 (1780533450) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:37:33 (1780533453) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:37:38 (1780533458) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:37:41 (1780533461) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.335704 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404383 s, 25.9 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:37:44 (1780533464) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.14157 s, 88.9 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:37:47 (1780533467) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:37:51 (1780533471) Stopping /mnt/lustre-ost1 (opts:) on oleg312-server oleg312-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg312-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg312-server oleg312-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg312-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (34s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:38:25 (1780533505) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.112406 s, 46.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg312-server oleg312-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg312-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:38:41 (1780533521) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg312-server oleg312-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg312-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg312-server oleg312-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg312-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (36s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:39:17 (1780533557) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2635 s, 41.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 3 20:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.313s user 0m0.001s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg312-server oleg312-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8acaa0e7a800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.377s user 0m0.002s sys 0m0.158s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg312-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg312-server oleg312-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.473s user 0m0.003s sys 0m0.160s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg312-server: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg312-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg312-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8acaa0e7a800.ost_server_uuid in FULL state after 0 sec PASS 33b (40s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:39:57 (1780533597) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:39:58 (1780533598) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:40:00 (1780533600) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 20:40:01 (1780533601) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 20:40:04 (1780533604) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.241223 s, 52.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:40:10 (1780533610) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134022 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022964 s, 45.7 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:40:13 (1780533613) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:40:17 (1780533617) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113216 s, 904 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000966902 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000800314 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00079008 s, 1.3 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 20:40:21 (1780533621) create test files with size 15751517 .. 6ae571b792447303a49756db4b76460d - f54afd925a83644619c9680152a5d643 - eeadc13a2311a4fbb05da0d70412aa86 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg312-client.virtnet /mnt/lustre (opts:) Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 20:40:29 (1780533629) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:40:34 (1780533634) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:40:36 (1780533636) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289121 s, 36.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187619 s, 55.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 20:40:40 (1780533640) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0419991 s, 49.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.143041 s, 29.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0831092 s, 25.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 20:40:46 (1780533646) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 20:40:47 (1780533647) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:40:49 (1780533649) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016108 s, 65.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 20:40:52 (1780533652) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.121121 s, 26.0 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:40:56 (1780533656) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:41:01 (1780533661) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.128319 s, 81.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg312-client.virtnet /mnt/lustre (opts:) Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:41:15 (1780533675) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8aca84a7e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.112892 s, 92.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg312-client.virtnet /mnt/lustre (opts:) Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8aca87fc6000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:41:20 (1780533680) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (16s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:41:36 (1780533696) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (16s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:41:52 (1780533712) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 20:41:54 (1780533714) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.108422 s, 29.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0787761 s, 39.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x365c12ca (nosync,stale) Mirror 2: 0x365c12ca (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4a84f7f2b5bb6edafb7ba401f0ea14df - mirror 1 checksum: 2349bc92ecceb5597879c471e949f7c1 - mirror 2 checksum: 4a84f7f2b5bb6edafb7ba401f0ea14df - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jun 3 20:41:55 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jun 3 20:41:55 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4a84f7f2b5bb6edafb7ba401f0ea14df - mirror 1 checksum: 2349bc92ecceb5597879c471e949f7c1 - mirror 2 checksum: 2349bc92ecceb5597879c471e949f7c1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jun 3 20:41:55 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jun 3 20:41:55 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:41:58 (1780533718) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229833 s, 45.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:42:02 (1780533722) Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server 20:42:12 (1780533732) shut down facet: mds1 facet_host: oleg312-server facet_failover_host: oleg312-server Failover mds1 to oleg312-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 20:42:28 (1780533748) targets are mounted 20:42:28 (1780533748) facet_failover done oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147026 s, 71.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (34s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:42:36 (1780533756) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0202219 s, 1.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00638498 s, 3.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:42:40 (1780533760) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0287737 s, 712 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00508336 s, 4.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:42:44 (1780533764) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00571418 s, 2.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:42:47 (1780533767) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0226487 s, 904 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:42:51 (1780533771) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00405001 s, 1.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0229481 s, 178 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:42:54 (1780533774) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:43:05 (1780533785) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:43:26 (1780533806) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:43:45 (1780533825) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:44:04 (1780533844) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 20:44:06 (1780533846) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57133 mirror split pid 57135 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5003: 57133 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5003: 57135 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 20:45:10 (1780533910) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 3910 fallocating to largest ever: 0x3994e 000003 1780533911.043749 fallocate 0x036833 thru 0x03994d (0x311b bytes) skipping zero size read 000006 1780533911.050132 fallocate 0x02089c thru 0x02bc9f (0xb404 bytes) 000010 1780533911.056068 read 0x01a42d thru 0x01ba70 (0x1644 bytes) 000011 1780533911.059307 fallocate 0x00fe30 thru 0x01c923 (0xcaf4 bytes) 000012 1780533911.063027 fallocate 0x030b84 thru 0x034dc5 (0x4242 bytes) 000013 1780533911.066726 write 0x01d257 thru 0x023edc (0x6c86 bytes) 000015 1780533911.073849 trunc from 0x034dc6 to 0x01570f (0x1f6b6 bytes) fallocating to largest ever: 0x3fe0c 000016 1780533911.083485 fallocate 0x0376d2 thru 0x03fe0b (0x873a bytes) 000017 1780533911.090131 fallocate 0x032a01 thru 0x03e90a (0xbf0a bytes) 000018 1780533911.097137 trunc from 0x03fe0c to 0x011ba6 (0x2e265 bytes) 000019 1780533911.106682 trunc from 0x011ba7 to 0x01f6f5 (0xdb4f bytes) 000020 1780533911.119468 punch from 0x012f33 to 0x019a34 (0x6b02 bytes) 000021 1780533911.123933 trunc from 0x00cf61 to 0x00dae4 (0xb84 bytes) 000022 1780533911.128404 mapwrite 0x010239 thru 0x015eba (0x5c82 bytes) 000023 1780533911.151591 mapread 0x009453 thru 0x0104dd (0x708b bytes) 000024 1780533911.152841 write 0x03a69d thru 0x03ffff (0x5963 bytes) fallocating to largest ever: 0x40000 000025 1780533911.154188 fallocate 0x03c4b8 thru 0x03ffff (0x3b48 bytes) 000026 1780533911.157735 trunc from 0x040000 to 0x007fe5 (0x3801a bytes) 000027 1780533911.170457 trunc from 0x007fe6 to 0x01e6e6 (0x16701 bytes) 000028 1780533911.190049 read 0x01e20d thru 0x01e6e6 (0x4da bytes) 000029 1780533911.201627 write 0x01b796 thru 0x02653b (0xada6 bytes) 000030 1780533911.203702 write 0x0065f2 thru 0x00c3ff (0x5e0e bytes) 000032 1780533911.205954 mapread 0x008fd3 thru 0x010deb (0x7e19 bytes) 000033 1780533911.207293 mapread 0x00ff20 thru 0x01a8a6 (0xa987 bytes) 000034 1780533911.208960 trunc from 0x021dee to 0x02653b (0x474e bytes) 000035 1780533911.221386 fallocate 0x0239a0 thru 0x0243d4 (0xa35 bytes) 000036 1780533911.227085 mapwrite 0x018121 thru 0x019c2a (0x1b0a bytes) 000038 1780533911.244956 MIRROR_EXTEND to 2 mirrors 000039 1780533911.576747 fallocate 0x0156ca thru 0x01fe47 (0xa77e bytes) 000040 1780533911.594299 trunc from 0x0062c6 to 0x006655 (0x390 bytes) 000041 1780533911.598983 write 0x032fa9 thru 0x03bd0e (0x8d66 bytes) 000042 1780533911.602949 trunc from 0x0071d9 to 0x0125e9 (0xb411 bytes) 000043 1780533911.606118 mapread 0x00ecfa thru 0x01780b (0x8b12 bytes) 000044 1780533911.619890 read 0x01c988 thru 0x0238bb (0x6f34 bytes) 000045 1780533911.629893 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000046 1780533911.716175 mapread 0x038ac8 thru 0x03a2c4 (0x17fd bytes) 000048 1780533911.732788 punch from 0x0050f1 to 0x00fa9e (0xa9ae bytes) 000049 1780533911.736714 punch from 0x01b3ab to 0x01dcb1 (0x2907 bytes) 000050 1780533911.743093 mapread 0x034f91 thru 0x03825b (0x32cb bytes) 000051 1780533911.744287 mapread 0x01041c thru 0x01f109 (0xecee bytes) 000052 1780533911.760581 trunc from 0x03bd0f to 0x029386 (0x12988 bytes) 000053 1780533911.788987 mapread 0x025340 thru 0x027704 (0x23c5 bytes) 000054 1780533911.800516 MIRROR_EXTEND to 2 mirrors 000056 1780533912.267303 trunc from 0x029387 to 0x01b21a (0xe16c bytes) 000057 1780533912.290914 fallocate 0x02b740 thru 0x03748a (0xbd4b bytes) 000058 1780533912.295440 trunc from 0x030733 to 0x031693 (0xf61 bytes) 000059 1780533912.300162 write 0x03f93a thru 0x03ffff (0x6c6 bytes) 000060 1780533912.301198 read 0x013480 thru 0x019af5 (0x6676 bytes) 000061 1780533912.306053 write 0x014ea1 thru 0x0240c0 (0xf220 bytes) 000062 1780533912.310126 trunc from 0x040000 to 0x0054ff (0x3ab00 bytes) 000064 1780533912.320872 read 0x00474e thru 0x0054ff (0xdb2 bytes) 000065 1780533912.326743 fallocate 0x00b949 thru 0x00bc91 (0x349 bytes) 000066 1780533912.331765 mapread 0x001195 thru 0x0054ff (0x436b bytes) 000067 1780533912.336850 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x5500): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5500): Mirror 2: 0xad145d62 (stale) CRC-32 checksum value for chunk [0, 0x5500): Mirror 1: 0xad145d62 Mirror 3: 0xad145d62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5500: skipped 000068 1780533912.707204 punch from 0x003834 to 0x0054ff (0x1ccc bytes) 000069 1780533912.728360 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5500): Mirror 3: 0xad145d62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5500): Mirror 3: 0xad145d62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5500: skipped 000070 1780533912.864496 write 0x0159d7 thru 0x023345 (0xd96f bytes) 000071 1780533912.867939 punch from 0x01c487 to 0x021c58 (0x57d2 bytes) 000072 1780533912.878189 mapread 0x00829b thru 0x0106d8 (0x843e bytes) 000073 1780533912.889892 mapread 0x01ad96 thru 0x01e3e0 (0x364b bytes) 000074 1780533912.896411 mapread 0x021ef7 thru 0x023345 (0x144f bytes) 000075 1780533912.897863 mapwrite 0x0184ae thru 0x021e70 (0x99c3 bytes) 000076 1780533912.911846 fallocate 0x016220 thru 0x01d216 (0x6ff7 bytes) 000077 1780533912.914810 write 0x038d1a thru 0x03ffff (0x72e6 bytes) 000078 1780533912.915829 mapwrite 0x0014af thru 0x0032fa (0x1e4c bytes) 000080 1780533912.930293 read 0x028197 thru 0x03659b (0xe405 bytes) 000081 1780533912.937937 mapwrite 0x01244d thru 0x018212 (0x5dc6 bytes) 000082 1780533912.956535 mapwrite 0x005d55 thru 0x012afe (0xcdaa bytes) 000084 1780533912.972912 trunc from 0x026b10 to 0x027379 (0x86a bytes) 000085 1780533912.978306 mapwrite 0x009c49 thru 0x00b03e (0x13f6 bytes) 000086 1780533912.987840 punch from 0x03a4d3 to 0x03ffff (0x5b2d bytes) 000087 1780533912.996027 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000088 1780533913.065813 read 0x022d80 thru 0x0270ba (0x433b bytes) 000089 1780533913.071148 trunc from 0x03e6fb to 0x03ffff (0x1905 bytes) 000090 1780533913.074433 mapread 0x00c539 thru 0x013bdd (0x76a5 bytes) 000091 1780533913.089794 mapwrite 0x007f56 thru 0x01509a (0xd145 bytes) 000092 1780533913.113781 MIRROR_EXTEND to 2 mirrors 000093 1780533913.450731 fallocate 0x03e746 thru 0x03ffff (0x18ba bytes) 000094 1780533913.473692 trunc from 0x029108 to 0x033a4e (0xa947 bytes) 000097 1780533913.479621 mapread 0x01374e thru 0x016279 (0x2b2c bytes) 000098 1780533913.500803 trunc from 0x040000 to 0x00487e (0x3b781 bytes) 000099 1780533913.525838 read 0x003f60 thru 0x00487e (0x91f bytes) 000100 1780533913.530319 fallocate 0x028aaf thru 0x034314 (0xb866 bytes) 000101 1780533913.535597 mapwrite 0x03a6bc thru 0x03e631 (0x3f76 bytes) 000103 1780533913.562280 punch from 0x0117a0 to 0x01839c (0x6bfd bytes) 000104 1780533913.566116 punch from 0x014fab to 0x01bcdf (0x6d35 bytes) 000105 1780533913.570202 mapread 0x030f91 thru 0x037f3b (0x6fab bytes) 000106 1780533913.579921 mapwrite 0x01b687 thru 0x028fad (0xd927 bytes) 000107 1780533913.607103 punch from 0x021dd0 to 0x031ce5 (0xff16 bytes) 000108 1780533913.611170 fallocate 0x008a1a thru 0x01654d (0xdb34 bytes) 000109 1780533913.616157 trunc from 0x000c70 to 0x005eae (0x523f bytes) 000111 1780533913.624362 read 0x0030a6 thru 0x010034 (0xcf8f bytes) 000112 1780533913.632294 trunc from 0x023fe1 to 0x025e51 (0x1e71 bytes) 000113 1780533913.636294 mapread 0x038a48 thru 0x03e631 (0x5bea bytes) 000114 1780533913.637543 punch from 0x009f12 to 0x015c41 (0xbd30 bytes) 000116 1780533913.641353 fallocate 0x004f48 thru 0x00730c (0x23c5 bytes) 000118 1780533913.645203 mapwrite 0x0082cd thru 0x017f98 (0xfccc bytes) 000119 1780533913.727451 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e632): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e632): Mirror 2: 0xc96ffdbc (stale) CRC-32 checksum value for chunk [0, 0x3e632): Mirror 1: 0x316ecf76 Mirror 3: 0x316ecf76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e632: skipped 000120 1780533914.119578 read 0x014bbb thru 0x018e23 (0x4269 bytes) 000122 1780533914.130026 read 0x01a055 thru 0x0260e7 (0xc093 bytes) 000124 1780533914.138840 trunc from 0x03e632 to 0x00713c (0x374f5 bytes) 000125 1780533914.184768 trunc from 0x00713d to 0x02b0c1 (0x23f85 bytes) 000127 1780533914.203223 write 0x009e9a thru 0x0111d2 (0x7339 bytes) 000128 1780533914.210801 write 0x01d7b7 thru 0x0211cb (0x3a15 bytes) 000129 1780533914.216935 mapread 0x01118a thru 0x012785 (0x15fc bytes) 000130 1780533914.221174 trunc from 0x02b0c2 to 0x0332e4 (0x8223 bytes) 000131 1780533914.235393 punch from 0x01bd3e to 0x01fd68 (0x402b bytes) 000132 1780533914.245615 trunc from 0x004fb4 to 0x010354 (0xb3a1 bytes) 000133 1780533914.253859 mapread 0x0006ea thru 0x006cb9 (0x65d0 bytes) 000135 1780533914.278980 mapwrite 0x030ab3 thru 0x033caf (0x31fd bytes) 000136 1780533914.316249 mapwrite 0x012027 thru 0x0173d2 (0x53ac bytes) 000137 1780533914.342166 mapread 0x0212a0 thru 0x0219f5 (0x756 bytes) 000138 1780533914.343613 trunc from 0x033cb0 to 0x025c70 (0xe03f bytes) 000139 1780533914.357835 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25c71): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c71): Mirror 2: 0xf0d781c5 (stale) Mirror 3: 0xae1c3340 (stale) CRC-32 checksum value for chunk [0, 0x25c71): Mirror 1: 0x2839be3d Mirror 4: 0x2839be3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c71: skipped 000140 1780533914.795950 read 0x00f007 thru 0x017718 (0x8712 bytes) 000141 1780533914.804593 trunc from 0x025041 to 0x025c70 (0xc30 bytes) 000142 1780533914.829065 mapwrite 0x016fb5 thru 0x0242cf (0xd31b bytes) 000143 1780533914.864623 punch from 0x00e2f9 to 0x01524c (0x6f54 bytes) 000144 1780533914.868500 trunc from 0x025c71 to 0x020c44 (0x502c bytes) 000145 1780533914.886780 mapread 0x000396 thru 0x006d3d (0x69a8 bytes) 000146 1780533914.895906 mapwrite 0x00f5ab thru 0x014302 (0x4d58 bytes) 000147 1780533914.912769 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20c45): Mirror 3: 0x86bf7425 (stale) Mirror 4: 0xce6894e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c45): Mirror 3: 0x86bf7425 (stale) Mirror 4: 0xce6894e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c45: skipped 000148 1780533915.096477 mapread 0x000fee thru 0x00e2d8 (0xd2eb bytes) 000149 1780533915.116306 mapwrite 0x01532c thru 0x017bc0 (0x2895 bytes) 000150 1780533915.131906 mapread 0x00cc83 thru 0x00f36d (0x26eb bytes) 000151 1780533915.132721 mapread 0x020743 thru 0x020c44 (0x502 bytes) 000152 1780533915.137113 read 0x00ed85 thru 0x019439 (0xa6b5 bytes) 000153 1780533915.137859 mapread 0x00ddcc thru 0x0164ad (0x86e2 bytes) 000154 1780533915.138842 write 0x024d54 thru 0x03003e (0xb2eb bytes) 000155 1780533915.143211 read 0x00b0fb thru 0x0166bf (0xb5c5 bytes) 000157 1780533915.145177 punch from 0x010cb8 to 0x017025 (0x636e bytes) 000159 1780533915.148926 trunc from 0x03003f to 0x0022c4 (0x2dd7a bytes) 000160 1780533915.160941 trunc from 0x0022c5 to 0x018467 (0x161a3 bytes) 000161 1780533915.171873 punch from 0x00e9af to 0x018467 (0x9ab9 bytes) 000162 1780533915.175406 fallocate 0x005400 thru 0x014f13 (0xfb14 bytes) 000163 1780533915.178590 trunc from 0x00cbcb to 0x00f0f1 (0x2527 bytes) 000165 1780533915.182253 trunc from 0x018468 to 0x00f11e (0x9349 bytes) 000166 1780533915.190754 read 0x001dad thru 0x003dcf (0x2023 bytes) 000167 1780533915.195078 fallocate 0x007a34 thru 0x00ea21 (0x6fee bytes) 000169 1780533915.200366 fallocate 0x00f6ed thru 0x013603 (0x3f17 bytes) 000170 1780533915.204264 read 0x000a73 thru 0x004305 (0x3893 bytes) 000172 1780533915.205309 mapwrite 0x02386b thru 0x0256d2 (0x1e68 bytes) 000173 1780533915.231843 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x256d3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256d3): Mirror 3: 0x3e77847c (stale) Mirror 4: 0xe7e2dffc (stale) CRC-32 checksum value for chunk [0, 0x256d3): Mirror 1: 0x307ba1aa Mirror 5: 0x307ba1aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256d3: skipped 000175 1780533915.651109 punch from 0x005841 to 0x0074ea (0x1caa bytes) 000176 1780533915.683212 mapwrite 0x003e2f thru 0x005f15 (0x20e7 bytes) 000177 1780533915.717532 write 0x032a4b thru 0x039cb9 (0x726f bytes) 000178 1780533915.725594 trunc from 0x01af9e to 0x01c0db (0x113e bytes) 000179 1780533915.731016 fallocate 0x03ebed thru 0x03ffff (0x1413 bytes) 000181 1780533915.735355 write 0x03de00 thru 0x03ffff (0x2200 bytes) 000182 1780533915.746370 trunc from 0x02c712 to 0x03c5af (0xfe9e bytes) 000183 1780533915.761481 punch from 0x028bb9 to 0x033859 (0xaca1 bytes) 000184 1780533915.765563 write 0x02d31f thru 0x032626 (0x5308 bytes) 000185 1780533915.782239 MIRROR_RESYNC 4 mirrors 000186 1780533916.534615 MIRROR_EXTEND to 5 mirrors 000187 1780533916.909166 trunc from 0x040000 to 0x031b76 (0xe489 bytes) 000188 1780533916.943414 trunc from 0x0046a8 to 0x00c739 (0x8092 bytes) 000189 1780533916.947898 write 0x022b8d thru 0x029ea4 (0x7318 bytes) 000190 1780533916.955451 mapwrite 0x02d07c thru 0x035e87 (0x8e0c bytes) 000191 1780533916.996299 mapwrite 0x024c56 thru 0x02ea5e (0x9e09 bytes) 000192 1780533917.016461 trunc from 0x0042e0 to 0x00ed25 (0xaa46 bytes) 000193 1780533917.021298 mapwrite 0x00d13f thru 0x011c6b (0x4b2d bytes) 000194 1780533917.052974 mapwrite 0x02e584 thru 0x03024f (0x1ccc bytes) 000195 1780533917.065220 trunc from 0x035e88 to 0x01a034 (0x1be53 bytes) 000197 1780533917.080724 read 0x00355e thru 0x00fd7f (0xc822 bytes) 000198 1780533917.085425 fallocate 0x02429d thru 0x02bf19 (0x7c7d bytes) 000199 1780533917.090237 mapread 0x021047 thru 0x02bf19 (0xaed3 bytes) 000200 1780533917.107137 read 0x02ae71 thru 0x02bf19 (0x10a9 bytes) 000202 1780533917.108564 trunc from 0x02bf1a to 0x021f2f (0x9fea bytes) 000203 1780533917.122198 trunc from 0x01ee5f to 0x01f468 (0x60a bytes) 000205 1780533917.127425 trunc from 0x021f30 to 0x017bb9 (0xa376 bytes) 000208 1780533917.141619 mapwrite 0x02acca thru 0x02d4fd (0x2834 bytes) 000209 1780533917.176576 MIRROR_RESYNC 5 mirrors 000210 1780533917.877522 trunc from 0x02c30f to 0x02d4fd (0x11ef bytes) 000212 1780533917.911220 punch from 0x00dbf4 to 0x00f466 (0x1873 bytes) 000213 1780533917.915393 punch from 0x0007d1 to 0x00e2b1 (0xdae1 bytes) 000214 1780533917.927037 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d4fe): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d4fe): Mirror 3: 0x54668cc4 (stale) Mirror 4: 0x54668cc4 (stale) Mirror 5: 0x54668cc4 (stale) Mirror 6: 0x54668cc4 (stale) CRC-32 checksum value for chunk [0, 0x2d4fe): Mirror 1: 0x46a2256f Mirror 7: 0x46a2256f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d4fe: skipped 000215 1780533918.495407 write 0x00a76f thru 0x0180ce (0xd960 bytes) 000216 1780533918.529900 trunc from 0x000665 to 0x00831c (0x7cb8 bytes) 000217 1780533918.534961 fallocate 0x02b905 thru 0x02eb26 (0x3222 bytes) 000218 1780533918.547293 MIRROR_RESYNC 6 mirrors 000219 1780533919.415741 MIRROR_EXTEND to 7 mirrors 000221 1780533919.855303 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000222 1780533920.114180 mapread 0x01a1bd thru 0x024b61 (0xa9a5 bytes) 000223 1780533920.138040 punch from 0x000d30 to 0x00bc52 (0xaf23 bytes) 000224 1780533920.177794 mapwrite 0x03316a thru 0x037297 (0x412e bytes) 000225 1780533920.213584 write 0x00e5c3 thru 0x017ed6 (0x9914 bytes) 000226 1780533920.223974 mapread 0x017c69 thru 0x0254fd (0xd895 bytes) 000227 1780533920.237762 trunc from 0x037298 to 0x01a12e (0x1d169 bytes) 000228 1780533920.248268 fallocate 0x029ee0 thru 0x02c94b (0x2a6c bytes) 000229 1780533920.253477 read 0x019f1e thru 0x020f50 (0x7033 bytes) 000230 1780533920.262995 MIRROR_RESYNC 6 mirrors 000231 1780533920.904859 read 0x009b56 thru 0x017b2e (0xdfd9 bytes) 000232 1780533920.910761 write 0x003860 thru 0x01059f (0xcd40 bytes) 000234 1780533920.941320 punch from 0x008123 to 0x012883 (0xa761 bytes) 000235 1780533920.952054 mapwrite 0x00715a thru 0x0145df (0xd486 bytes) 000236 1780533920.979813 trunc from 0x00494a to 0x00b8a7 (0x6f5e bytes) 000237 1780533920.988155 MIRROR_RESYNC 6 mirrors 000238 1780533921.745287 punch from 0x0059b9 to 0x00cfc5 (0x760d bytes) 000239 1780533921.768862 trunc from 0x01d871 to 0x025c36 (0x83c6 bytes) 000240 1780533921.780516 MIRROR_RESYNC 6 mirrors 000241 1780533922.710332 fallocate 0x02b43d thru 0x03573b (0xa2ff bytes) 000242 1780533922.733381 trunc from 0x0136d4 to 0x01e027 (0xa954 bytes) 000243 1780533922.737724 mapwrite 0x03a79d thru 0x03e401 (0x3c65 bytes) 000244 1780533922.796005 fallocate 0x009c26 thru 0x015cc3 (0xc09e bytes) 000246 1780533922.804340 MIRROR_RESYNC 6 mirrors 000247 1780533923.864911 mapwrite 0x032835 thru 0x0399b9 (0x7185 bytes) 000249 1780533923.938766 write 0x0361d2 thru 0x03ffff (0x9e2e bytes) 000250 1780533923.940782 fallocate 0x00fbbc thru 0x0127d2 (0x2c17 bytes) 000251 1780533923.948023 write 0x01d1cb thru 0x021684 (0x44ba bytes) 000252 1780533923.957743 trunc from 0x040000 to 0x029593 (0x16a6c bytes) 000253 1780533923.976833 trunc from 0x00b2f3 to 0x014362 (0x9070 bytes) 000254 1780533923.980329 punch from 0x025123 to 0x026392 (0x1270 bytes) 000255 1780533923.986014 trunc from 0x029594 to 0x01e82d (0xad66 bytes) 000256 1780533924.007176 MIRROR_RESYNC 6 mirrors 000257 1780533924.779513 MIRROR_RESYNC 6 mirrors 000258 1780533925.034562 write 0x013cbb thru 0x020810 (0xcb56 bytes) 000259 1780533925.055815 trunc from 0x020811 to 0x038dea (0x185da bytes) 000260 1780533925.071114 MIRROR_RESYNC 6 mirrors 000261 1780533925.981302 read 0x01074b thru 0x01b925 (0xb1db bytes) 000262 1780533925.991130 mapwrite 0x015fd6 thru 0x01b946 (0x5971 bytes) 000264 1780533926.056165 fallocate 0x012f7a thru 0x015afa (0x2b81 bytes) 000266 1780533926.061471 read 0x031697 thru 0x038dea (0x7754 bytes) 000267 1780533926.072855 trunc from 0x01651d to 0x0237ac (0xd290 bytes) 000268 1780533926.079184 punch from 0x02250b to 0x02baa6 (0x959c bytes) 000269 1780533926.084810 write 0x02b67c thru 0x02c6ca (0x104f bytes) 000270 1780533926.090686 punch from 0x0255ae to 0x031b13 (0xc566 bytes) 000272 1780533926.105701 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38deb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38deb): Mirror 3: 0xad8c86b6 (stale) Mirror 4: 0xad8c86b6 (stale) Mirror 5: 0xad8c86b6 (stale) Mirror 7: 0xad8c86b6 (stale) Mirror 8: 0xad8c86b6 (stale) CRC-32 checksum value for chunk [0, 0x38deb): Mirror 1: 0x3df59ee1 Mirror 9: 0x3df59ee1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38deb: skipped 000273 1780533926.616591 fallocate 0x01c960 thru 0x01e26e (0x190f bytes) 000274 1780533926.646769 read 0x016d9f thru 0x01e9ed (0x7c4f bytes) 000275 1780533926.654458 fallocate 0x00f6f6 thru 0x01b4e7 (0xbdf2 bytes) 000276 1780533926.660029 trunc from 0x038deb to 0x0268f9 (0x124f1 bytes) 000277 1780533926.700886 trunc from 0x012cb2 to 0x01a8cc (0x7c1b bytes) 000278 1780533926.705099 mapwrite 0x02ad3e thru 0x034940 (0x9c03 bytes) 000280 1780533926.750986 read 0x0104b1 thru 0x014669 (0x41b9 bytes) 000281 1780533926.758209 write 0x0355d1 thru 0x03ffff (0xaa2f bytes) 000282 1780533926.760474 trunc from 0x040000 to 0x035e19 (0xa1e6 bytes) 000283 1780533926.773216 trunc from 0x035e1a to 0x008687 (0x2d792 bytes) 000284 1780533926.786138 mapread 0x002672 thru 0x008687 (0x6016 bytes) 000285 1780533926.787132 mapwrite 0x035103 thru 0x038004 (0x2f02 bytes) 000286 1780533926.817594 trunc from 0x038005 to 0x03bec6 (0x3ec2 bytes) 000287 1780533926.830212 fallocate 0x0042f4 thru 0x012e39 (0xeb46 bytes) 000288 1780533926.834747 mapread 0x0065aa thru 0x00aa4d (0x44a4 bytes) 000289 1780533926.848020 MIRROR_RESYNC 7 mirrors 000291 1780533927.751975 mapread 0x00119c thru 0x005b43 (0x49a8 bytes) 000292 1780533927.766137 write 0x014126 thru 0x015378 (0x1253 bytes) 000293 1780533927.798689 fallocate 0x008624 thru 0x00a9c0 (0x239d bytes) 000294 1780533927.803580 trunc from 0x032525 to 0x033f1e (0x19fa bytes) 000295 1780533927.807317 read 0x02364d thru 0x033622 (0xffd6 bytes) 000297 1780533927.814996 fallocate 0x01b0b2 thru 0x029248 (0xe197 bytes) 000298 1780533927.819883 mapwrite 0x00866b thru 0x013f97 (0xb92d bytes) 000299 1780533927.841278 trunc from 0x03bec7 to 0x00804b (0x33e7b bytes) 000300 1780533927.860985 read 0x0064f5 thru 0x007941 (0x144d bytes) 000301 1780533927.861710 write 0x016d31 thru 0x017d57 (0x1027 bytes) 000302 1780533927.862817 punch from 0x00bfd2 to 0x014c77 (0x8ca6 bytes) 000303 1780533927.866805 mapread 0x00f78c thru 0x013687 (0x3efc bytes) 000304 1780533927.874111 punch from 0x00b3bf to 0x00de42 (0x2a84 bytes) 000305 1780533927.883684 MIRROR_RESYNC 7 mirrors 000306 1780533928.939397 write 0x01b531 thru 0x024edc (0x99ac bytes) 000307 1780533928.968521 MIRROR_RESYNC 7 mirrors 000308 1780533930.134510 trunc from 0x024edd to 0x015f03 (0xefd9 bytes) 000309 1780533930.158806 fallocate 0x016e84 thru 0x01fd35 (0x8eb2 bytes) 000310 1780533930.162392 trunc from 0x01b287 to 0x01e147 (0x2ec1 bytes) 000312 1780533930.167710 mapwrite 0x026899 thru 0x0303c9 (0x9b31 bytes) 000313 1780533930.216562 fallocate 0x03e1a1 thru 0x03ffff (0x1e5f bytes) 000314 1780533930.221933 mapread 0x02fecc thru 0x03b708 (0xb83d bytes) 000315 1780533930.239409 write 0x00f776 thru 0x0132e7 (0x3b72 bytes) 000317 1780533930.252411 fallocate 0x0396ca thru 0x03ffff (0x6936 bytes) 000318 1780533930.264006 MIRROR_RESYNC 7 mirrors 000319 1780533931.580403 mapread 0x02f5f7 thru 0x0303b3 (0xdbd bytes) 000320 1780533931.589067 write 0x031787 thru 0x03ffff (0xe879 bytes) 000321 1780533931.622274 punch from 0x02d259 to 0x033e07 (0x6baf bytes) 000323 1780533931.632300 write 0x00ac4a thru 0x01a0ff (0xf4b6 bytes) 000324 1780533931.645411 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1053454b (stale) Mirror 5: 0x1053454b (stale) Mirror 7: 0x1053454b (stale) Mirror 8: 0x1053454b (stale) Mirror 9: 0x1053454b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1053454b (stale) Mirror 5: 0x1053454b (stale) Mirror 7: 0x1053454b (stale) Mirror 8: 0x1053454b (stale) Mirror 9: 0x1053454b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000325 1780533931.894468 punch from 0x017ef9 to 0x023c29 (0xbd31 bytes) 000326 1780533931.907443 fallocate 0x01a39f thru 0x021fc8 (0x7c2a bytes) 000327 1780533931.914352 trunc from 0x017a9a to 0x020d49 (0x92b0 bytes) 000328 1780533931.919540 mapread 0x01d4ae thru 0x02bbfe (0xe751 bytes) 000330 1780533931.948057 write 0x002899 thru 0x008303 (0x5a6b bytes) 000331 1780533931.972456 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1053454b (stale) Mirror 5: 0x1053454b (stale) Mirror 7: 0x1053454b (stale) Mirror 8: 0x1053454b (stale) Mirror 9: 0x1053454b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5fa65d5c Mirror 10: 0x5fa65d5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000332 1780533932.562688 trunc from 0x004fc5 to 0x00734f (0x238b bytes) 000333 1780533932.587485 trunc from 0x013c7c to 0x021170 (0xd4f5 bytes) 000334 1780533932.591271 punch from 0x007d05 to 0x00bf05 (0x4201 bytes) 000335 1780533932.598182 trunc from 0x040000 to 0x0197ac (0x26853 bytes) 000336 1780533932.616428 read 0x00bc19 thru 0x01586a (0x9c52 bytes) 000337 1780533932.627209 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x197ad): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x197ad): Mirror 4: 0x461b84ef (stale) Mirror 5: 0x461b84ef (stale) Mirror 7: 0x461b84ef (stale) Mirror 8: 0x461b84ef (stale) Mirror 9: 0x461b84ef (stale) Mirror 10: 0x4a87060a (stale) CRC-32 checksum value for chunk [0, 0x197ad): Mirror 1: 0x13069cee Mirror 11: 0x13069cee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x197ad: skipped 000338 1780533933.172511 trunc from 0x0197ad to 0x027313 (0xdb67 bytes) 000339 1780533933.220115 write 0x03c6d2 thru 0x03ffff (0x392e bytes) 000341 1780533933.225739 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1053454b (stale) Mirror 5: 0x1053454b (stale) Mirror 7: 0x1053454b (stale) Mirror 8: 0x1053454b (stale) Mirror 9: 0x1053454b (stale) Mirror 10: 0x5fa65d5c (stale) Mirror 11: 0x652c6e9a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3a368121 Mirror 12: 0x3a368121 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000342 1780533933.886440 read 0x0201a8 thru 0x02c789 (0xc5e2 bytes) 000343 1780533933.894682 read 0x03d86f thru 0x03ffff (0x2791 bytes) 000344 1780533933.912512 MIRROR_RESYNC 9 mirrors 000345 1780533935.328520 trunc from 0x038e82 to 0x03f443 (0x65c2 bytes) 000346 1780533935.356895 mapwrite 0x0247f7 thru 0x0280af (0x38b9 bytes) 000347 1780533935.384496 punch from 0x01632f to 0x022ff5 (0xccc7 bytes) 000348 1780533935.395744 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3a368121 (stale) Mirror 5: 0x3a368121 (stale) Mirror 7: 0x3a368121 (stale) Mirror 8: 0x3a368121 (stale) Mirror 9: 0x3a368121 (stale) Mirror 10: 0x3a368121 (stale) Mirror 12: 0x3a368121 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3a368121 (stale) Mirror 5: 0x3a368121 (stale) Mirror 7: 0x3a368121 (stale) Mirror 8: 0x3a368121 (stale) Mirror 9: 0x3a368121 (stale) Mirror 10: 0x3a368121 (stale) Mirror 12: 0x3a368121 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000349 1780533935.685608 mapread 0x029d55 thru 0x02b3ae (0x165a bytes) 000350 1780533935.698997 mapwrite 0x02e4fc thru 0x03e1e5 (0xfcea bytes) 000351 1780533935.728861 trunc from 0x040000 to 0x03cab7 (0x3548 bytes) 000352 1780533935.740908 trunc from 0x03cab8 to 0x02cea0 (0xfc17 bytes) 000353 1780533935.751690 mapread 0x00ccb2 thru 0x00e038 (0x1387 bytes) 000354 1780533935.757744 write 0x01cf2b thru 0x01e361 (0x1437 bytes) 000355 1780533935.763936 read 0x00263d thru 0x0044e8 (0x1eac bytes) 000356 1780533935.767220 fallocate 0x03cc9c thru 0x03def6 (0x125b bytes) 000357 1780533935.772996 read 0x02598b thru 0x026b79 (0x11ef bytes) 000358 1780533935.773779 write 0x02e603 thru 0x031267 (0x2c65 bytes) 000359 1780533935.780750 punch from 0x03bc0c to 0x03def6 (0x22eb bytes) 000360 1780533935.787548 punch from 0x000db5 to 0x007ae9 (0x6d35 bytes) 000361 1780533935.792168 mapread 0x030cc5 thru 0x03d94d (0xcc89 bytes) 000362 1780533935.804304 mapwrite 0x03b82a thru 0x03ffff (0x47d6 bytes) 000363 1780533935.833228 trunc from 0x040000 to 0x01a822 (0x257dd bytes) 000364 1780533935.849842 write 0x03ae46 thru 0x03ffff (0x51ba bytes) 000365 1780533935.851314 trunc from 0x040000 to 0x027d5c (0x182a3 bytes) 000366 1780533935.863933 trunc from 0x027d5d to 0x00a90e (0x1d44e bytes) 000367 1780533935.877802 mapread 0x00664c thru 0x00a90e (0x42c3 bytes) 000368 1780533935.891514 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa90f): Mirror 4: 0xf5f6c24a (stale) Mirror 5: 0xf5f6c24a (stale) Mirror 7: 0xf5f6c24a (stale) Mirror 8: 0xf5f6c24a (stale) Mirror 10: 0xf5f6c24a (stale) Mirror 12: 0xf5f6c24a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa90f): Mirror 4: 0xf5f6c24a (stale) Mirror 5: 0xf5f6c24a (stale) Mirror 7: 0xf5f6c24a (stale) Mirror 8: 0xf5f6c24a (stale) Mirror 10: 0xf5f6c24a (stale) Mirror 12: 0xf5f6c24a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa90f: skipped 000369 1780533936.135273 read 0x00266d thru 0x00a1c2 (0x7b56 bytes) 000371 1780533936.141710 trunc from 0x00a90f to 0x0055b4 (0x535a bytes) 000372 1780533936.155233 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x55b5): Mirror 4: 0xe22723d6 (stale) Mirror 5: 0xe22723d6 (stale) Mirror 7: 0xe22723d6 (stale) Mirror 10: 0xe22723d6 (stale) Mirror 12: 0xe22723d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x55b5): Mirror 4: 0xe22723d6 (stale) Mirror 5: 0xe22723d6 (stale) Mirror 7: 0xe22723d6 (stale) Mirror 10: 0xe22723d6 (stale) Mirror 12: 0xe22723d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x55b5: skipped 000373 1780533936.391222 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x55b5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x55b5): Mirror 4: 0xe22723d6 (stale) Mirror 5: 0xe22723d6 (stale) Mirror 7: 0xe22723d6 (stale) Mirror 10: 0xe22723d6 (stale) Mirror 12: 0xe22723d6 (stale) CRC-32 checksum value for chunk [0, 0x55b5): Mirror 1: 0x53ef9efd Mirror 13: 0x53ef9efd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x55b5: skipped 000376 1780533936.844960 read 0x000851 thru 0x0055b4 (0x4d64 bytes) 000378 1780533936.859279 mapread 0x00170a thru 0x0038d5 (0x21cc bytes) 000379 1780533936.873763 punch from 0x001987 to 0x0055b4 (0x3c2e bytes) 000380 1780533936.898116 fallocate 0x0378ac thru 0x03a456 (0x2bab bytes) 000381 1780533936.905461 fallocate 0x015acd thru 0x015d65 (0x299 bytes) 000382 1780533936.912033 punch from 0x0031f5 to 0x008557 (0x5363 bytes) 000383 1780533936.917493 mapwrite 0x031506 thru 0x033a0b (0x2506 bytes) 000384 1780533936.970593 write 0x012962 thru 0x022392 (0xfa31 bytes) 000386 1780533936.988905 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33a0c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a0c): Mirror 4: 0x99dd8de5 (stale) Mirror 5: 0x99dd8de5 (stale) Mirror 7: 0x99dd8de5 (stale) Mirror 10: 0x99dd8de5 (stale) Mirror 12: 0x99dd8de5 (stale) Mirror 13: 0x77356fb8 (stale) CRC-32 checksum value for chunk [0, 0x33a0c): Mirror 1: 0x39890c42 Mirror 14: 0x39890c42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a0c: skipped 000387 1780533937.506240 mapwrite 0x037320 thru 0x03f99e (0x867f bytes) 000388 1780533937.561257 trunc from 0x03f99f to 0x013967 (0x2c037 bytes) 000389 1780533937.576003 mapwrite 0x0222b3 thru 0x02a289 (0x7fd7 bytes) 000390 1780533937.607266 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2a28a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a28a): Mirror 4: 0x5ed76ec0 (stale) Mirror 5: 0x5ed76ec0 (stale) Mirror 7: 0x5ed76ec0 (stale) Mirror 10: 0x5ed76ec0 (stale) Mirror 12: 0x5ed76ec0 (stale) Mirror 13: 0xa9f82218 (stale) Mirror 14: 0xf4613982 (stale) CRC-32 checksum value for chunk [0, 0x2a28a): Mirror 1: 0x54c25ea0 Mirror 15: 0x54c25ea0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a28a: skipped 000391 1780533938.121419 write 0x004f46 thru 0x013a8c (0xeb47 bytes) 000392 1780533938.158374 fallocate 0x00d3f6 thru 0x01b360 (0xdf6b bytes) 000393 1780533938.162921 mapread 0x01c404 thru 0x024410 (0x800d bytes) 000394 1780533938.178261 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a28a): Mirror 4: 0x5ed76ec0 (stale) Mirror 5: 0x5ed76ec0 (stale) Mirror 7: 0x5ed76ec0 (stale) Mirror 10: 0x5ed76ec0 (stale) Mirror 13: 0xa9f82218 (stale) Mirror 14: 0xf4613982 (stale) Mirror 15: 0x54c25ea0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a28a): Mirror 4: 0x5ed76ec0 (stale) Mirror 5: 0x5ed76ec0 (stale) Mirror 7: 0x5ed76ec0 (stale) Mirror 10: 0x5ed76ec0 (stale) Mirror 13: 0xa9f82218 (stale) Mirror 14: 0xf4613982 (stale) Mirror 15: 0x54c25ea0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a28a: skipped 000395 1780533938.451917 read 0x0053de thru 0x013f2b (0xeb4e bytes) 000396 1780533938.456865 write 0x008985 thru 0x00a7d4 (0x1e50 bytes) 000397 1780533938.460436 fallocate 0x01bd4c thru 0x022d18 (0x6fcd bytes) 000399 1780533938.467428 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a28a): Mirror 5: 0x5ed76ec0 (stale) Mirror 7: 0x5ed76ec0 (stale) Mirror 10: 0x5ed76ec0 (stale) Mirror 13: 0xa9f82218 (stale) Mirror 14: 0xf4613982 (stale) Mirror 15: 0x54c25ea0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a28a): Mirror 5: 0x5ed76ec0 (stale) Mirror 7: 0x5ed76ec0 (stale) Mirror 10: 0x5ed76ec0 (stale) Mirror 13: 0xa9f82218 (stale) Mirror 14: 0xf4613982 (stale) Mirror 15: 0x54c25ea0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a28a: skipped 000401 1780533938.679285 trunc from 0x02a28a to 0x028111 (0x2178 bytes) 000402 1780533938.701107 fallocate 0x02ef4c thru 0x035097 (0x614c bytes) 000403 1780533938.705166 read 0x01f69e thru 0x0235eb (0x3f4e bytes) 000404 1780533938.711595 trunc from 0x0335a9 to 0x035097 (0x1aef bytes) 000405 1780533938.714348 read 0x01892e thru 0x020652 (0x7d25 bytes) 000406 1780533938.724854 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35098): Mirror 5: 0x76afe5bd (stale) Mirror 7: 0x76afe5bd (stale) Mirror 10: 0x76afe5bd (stale) Mirror 14: 0x81dd4cb7 (stale) Mirror 15: 0x32f2f2d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35098): Mirror 5: 0x76afe5bd (stale) Mirror 7: 0x76afe5bd (stale) Mirror 10: 0x76afe5bd (stale) Mirror 14: 0x81dd4cb7 (stale) Mirror 15: 0x32f2f2d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35098: skipped 000407 1780533938.933480 read 0x01490f thru 0x01fe6a (0xb55c bytes) 000409 1780533938.940407 trunc from 0x03485d to 0x035097 (0x83b bytes) 000411 1780533938.944783 write 0x00e958 thru 0x01de86 (0xf52f bytes) 000413 1780533938.951709 fallocate 0x02fa1b thru 0x03b615 (0xbbfb bytes) 000414 1780533938.954814 punch from 0x00272e to 0x00901c (0x68ef bytes) 000415 1780533938.960049 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35098): Mirror 7: 0x76afe5bd (stale) Mirror 10: 0x76afe5bd (stale) Mirror 14: 0x81dd4cb7 (stale) Mirror 15: 0x32f2f2d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35098): Mirror 7: 0x76afe5bd (stale) Mirror 10: 0x76afe5bd (stale) Mirror 14: 0x81dd4cb7 (stale) Mirror 15: 0x32f2f2d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35098: skipped 000416 1780533939.165713 mapread 0x00be79 thru 0x0142cf (0x8457 bytes) 000417 1780533939.179340 fallocate 0x025691 thru 0x02a306 (0x4c76 bytes) 000418 1780533939.184070 punch from 0x00bf8f to 0x01b8d1 (0xf943 bytes) 000419 1780533939.190886 mapwrite 0x038a44 thru 0x03ffff (0x75bc bytes) 000420 1780533939.228613 fallocate 0x00a517 thru 0x00fb78 (0x5662 bytes) 000422 1780533939.234142 trunc from 0x036e6b to 0x03d8e0 (0x6a76 bytes) 000423 1780533939.238415 trunc from 0x02c7b2 to 0x03390d (0x715c bytes) 000424 1780533939.241743 mapread 0x01b86d thru 0x01e8e8 (0x307c bytes) 000425 1780533939.252212 fallocate 0x03ff29 thru 0x03ffff (0xd7 bytes) 000426 1780533939.256703 trunc from 0x00889a to 0x012080 (0x97e7 bytes) 000427 1780533939.261211 fallocate 0x021af0 thru 0x026ef8 (0x5409 bytes) 000428 1780533939.265902 trunc from 0x040000 to 0x0239ee (0x1c611 bytes) 000429 1780533939.276291 write 0x000af8 thru 0x004c53 (0x415c bytes) 000430 1780533939.283771 read 0x009de1 thru 0x00dc1a (0x3e3a bytes) 000432 1780533939.285065 punch from 0x00ac6f to 0x019099 (0xe42b bytes) 000433 1780533939.289784 fallocate 0x02d4d8 thru 0x031caf (0x47d8 bytes) 000434 1780533939.296577 write 0x02b62b thru 0x02fa8d (0x4463 bytes) 000435 1780533939.298404 punch from 0x00f37e to 0x014593 (0x5216 bytes) 000436 1780533939.303740 write 0x034fdf thru 0x03ef7b (0x9f9d bytes) 000437 1780533939.305359 mapwrite 0x036608 thru 0x03ffff (0x99f8 bytes) 000438 1780533939.335768 fallocate 0x02b16e thru 0x02cc08 (0x1a9b bytes) 000439 1780533939.341300 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3a368121 (stale) Mirror 10: 0x3a368121 (stale) Mirror 14: 0xbfb8fa5f (stale) Mirror 15: 0xb4043a6e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeeab80b5 Mirror 16: 0xeeab80b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000440 1780533939.790556 trunc from 0x02b68e to 0x02fafe (0x4471 bytes) 000441 1780533939.811274 trunc from 0x005cf1 to 0x012e19 (0xd129 bytes) 000442 1780533939.814895 trunc from 0x03d6e2 to 0x03ffff (0x291e bytes) 000443 1780533939.819432 read 0x0373cf thru 0x03ffff (0x8c31 bytes) 000444 1780533939.825835 fallocate 0x03a63b thru 0x03ffff (0x59c5 bytes) 000447 1780533939.830603 fallocate 0x02a517 thru 0x033fbe (0x9aa8 bytes) 000448 1780533939.834162 read 0x0358d0 thru 0x03ffff (0xa730 bytes) 000449 1780533939.841763 fallocate 0x021bf6 thru 0x02d65b (0xba66 bytes) 000450 1780533939.845357 trunc from 0x022cfb to 0x02f69e (0xc9a4 bytes) 000451 1780533939.849204 write 0x00d980 thru 0x01b7fc (0xde7d bytes) 000452 1780533939.855846 MIRROR_RESYNC 6 mirrors 000453 1780533940.877571 mapread 0x0031a7 thru 0x01068d (0xd4e7 bytes) 000454 1780533940.887152 mapread 0x03699e thru 0x03b237 (0x489a bytes) 000455 1780533940.895339 trunc from 0x040000 to 0x0256eb (0x1a914 bytes) 000456 1780533940.920854 punch from 0x01dede to 0x0215dc (0x36ff bytes) 000457 1780533940.924291 read 0x0179fb thru 0x01f389 (0x798f bytes) 000458 1780533940.930385 mapwrite 0x03f0bd thru 0x03ffff (0xf43 bytes) 000459 1780533940.960874 trunc from 0x02488e to 0x02a7aa (0x5f1d bytes) 000460 1780533940.965392 write 0x00bd5e thru 0x00e546 (0x27e9 bytes) 000461 1780533940.971599 fallocate 0x013636 thru 0x014411 (0xddc bytes) 000462 1780533940.978453 trunc from 0x03d0fc to 0x03ffff (0x2f04 bytes) 000463 1780533940.982405 trunc from 0x040000 to 0x0050cc (0x3af33 bytes) 000464 1780533940.991491 fallocate 0x03a062 thru 0x03ffff (0x5f9e bytes) 000465 1780533940.995221 mapwrite 0x02a662 thru 0x02ddd0 (0x376f bytes) 000466 1780533941.015189 mapread 0x033b82 thru 0x037086 (0x3505 bytes) 000467 1780533941.022709 trunc from 0x01ea63 to 0x029e69 (0xb407 bytes) 000468 1780533941.030846 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x7c95820f (stale) Mirror 10: 0x7c95820f (stale) Mirror 14: 0x7c95820f (stale) Mirror 15: 0x7c95820f (stale) Mirror 16: 0x7c95820f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb0cf8a61 Mirror 17: 0xb0cf8a61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1780533941.446999 trunc from 0x040000 to 0x03954d (0x6ab2 bytes) 000470 1780533941.484510 write 0x027f58 thru 0x02fe5f (0x7f08 bytes) 000471 1780533941.490574 read 0x018cc6 thru 0x0268bc (0xdbf7 bytes) 000472 1780533941.495352 mapread 0x02dbf9 thru 0x03954d (0xb955 bytes) 000473 1780533941.506116 MIRROR_RESYNC 7 mirrors 000474 1780533942.332248 mapwrite 0x03c1f3 thru 0x03ffff (0x3e0d bytes) 000475 1780533942.384439 punch from 0x01cedd to 0x022393 (0x54b7 bytes) 000476 1780533942.389315 mapwrite 0x00ebd7 thru 0x014072 (0x549c bytes) 000477 1780533942.408708 trunc from 0x01632f to 0x0229f1 (0xc6c3 bytes) 000478 1780533942.416299 MIRROR_RESYNC 7 mirrors 000479 1780533943.365753 trunc from 0x002688 to 0x00d443 (0xadbc bytes) 000480 1780533943.397300 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6d0106ea (stale) Mirror 10: 0x6d0106ea (stale) Mirror 14: 0x6d0106ea (stale) Mirror 15: 0x6d0106ea (stale) Mirror 16: 0x6d0106ea (stale) Mirror 17: 0x6d0106ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9072fbc Mirror 18: 0xd9072fbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000481 1780533943.950724 mapread 0x038235 thru 0x03ffff (0x7dcb bytes) 000482 1780533943.979386 trunc from 0x000e22 to 0x008d78 (0x7f57 bytes) 000483 1780533944.005370 write 0x0260c3 thru 0x0306b6 (0xa5f4 bytes) 000484 1780533944.015749 mapread 0x010b82 thru 0x020925 (0xfda4 bytes) 000485 1780533944.026237 trunc from 0x00fb0a to 0x01008c (0x583 bytes) 000486 1780533944.032100 mapread 0x024050 thru 0x02b4bd (0x746e bytes) 000487 1780533944.033318 punch from 0x00fe05 to 0x01de4e (0xe04a bytes) 000488 1780533944.039596 MIRROR_RESYNC 8 mirrors 000489 1780533945.156302 mapwrite 0x0177aa thru 0x024c23 (0xd47a bytes) 000491 1780533945.242489 mapwrite 0x038c5e thru 0x03ffff (0x73a2 bytes) 000492 1780533945.264991 MIRROR_RESYNC 8 mirrors 000493 1780533946.455724 trunc from 0x040000 to 0x0049e1 (0x3b61e bytes) 000494 1780533946.488072 mapwrite 0x0230d2 thru 0x02baa2 (0x89d1 bytes) 000495 1780533946.540078 trunc from 0x02baa3 to 0x01b2d7 (0x107cb bytes) 000496 1780533946.551800 trunc from 0x008cf8 to 0x00d503 (0x480c bytes) 000497 1780533946.554517 mapread 0x000dfe thru 0x004fe3 (0x41e6 bytes) 000498 1780533946.560097 punch from 0x00f9cc to 0x01b2d7 (0xb90c bytes) 000499 1780533946.563490 trunc from 0x011164 to 0x018a2a (0x78c7 bytes) 000500 1780533946.566677 trunc from 0x00b50b to 0x012842 (0x7338 bytes) 000501 1780533946.570467 write 0x01e852 thru 0x022e42 (0x45f1 bytes) 000502 1780533946.571786 mapwrite 0x000663 thru 0x00b577 (0xaf15 bytes) 000503 1780533946.592877 mapwrite 0x0364c6 thru 0x03bad5 (0x5610 bytes) 000504 1780533946.622346 trunc from 0x00b80b to 0x01089f (0x5095 bytes) 000505 1780533946.626965 fallocate 0x02bd81 thru 0x03720f (0xb48f bytes) 000506 1780533946.632244 fallocate 0x0255fd thru 0x030961 (0xb365 bytes) 000508 1780533946.639131 punch from 0x011d2b to 0x015f80 (0x4256 bytes) 000509 1780533946.643406 read 0x03b63f thru 0x03bad5 (0x497 bytes) 000510 1780533946.644109 write 0x031d8a thru 0x03ffff (0xe276 bytes) 000511 1780533946.646025 trunc from 0x001739 to 0x00689a (0x5162 bytes) 000513 1780533946.651095 fallocate 0x0364e7 thru 0x036d53 (0x86d bytes) 000514 1780533946.654781 punch from 0x00324f to 0x0034ec (0x29e bytes) 000515 1780533946.658656 punch from 0x023777 to 0x027c42 (0x44cc bytes) 000516 1780533946.664196 fallocate 0x011d2c thru 0x0154cb (0x37a0 bytes) 000517 1780533946.668529 trunc from 0x040000 to 0x02d0f5 (0x12f0a bytes) 000518 1780533946.685443 trunc from 0x028fbb to 0x02b036 (0x207c bytes) 000519 1780533946.690992 write 0x023b76 thru 0x025da6 (0x2231 bytes) 000520 1780533946.700574 mapread 0x02cb8b thru 0x02d0f5 (0x56b bytes) 000523 1780533946.706820 mapread 0x000c82 thru 0x00ba8c (0xae0b bytes) 000524 1780533946.715173 mapwrite 0x017c15 thru 0x026be5 (0xefd1 bytes) 000525 1780533946.742613 write 0x024be6 thru 0x024f82 (0x39d bytes) 000526 1780533946.743780 punch from 0x01f57c to 0x02649e (0x6f23 bytes) 000527 1780533946.754898 punch from 0x0024a5 to 0x00c853 (0xa3af bytes) 000528 1780533946.759702 mapread 0x00cfd0 thru 0x00d1c7 (0x1f8 bytes) 000529 1780533946.768711 write 0x0356d1 thru 0x039619 (0x3f49 bytes) 000530 1780533946.770525 mapwrite 0x01cfc4 thru 0x025014 (0x8051 bytes) 000532 1780533946.789016 write 0x03b9f1 thru 0x03f077 (0x3687 bytes) 000533 1780533946.790441 read 0x02e4bf thru 0x037b4f (0x9691 bytes) 000534 1780533946.796968 trunc from 0x019aa2 to 0x0200f2 (0x6651 bytes) 000535 1780533946.800892 write 0x03c1ba thru 0x03ffff (0x3e46 bytes) 000536 1780533946.803323 punch from 0x03dee7 to 0x03fd08 (0x1e22 bytes) 000537 1780533946.813184 read 0x01a2ed thru 0x020635 (0x6349 bytes) 000538 1780533946.823882 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xaad52fc5 (stale) Mirror 14: 0xaad52fc5 (stale) Mirror 15: 0xaad52fc5 (stale) Mirror 16: 0xaad52fc5 (stale) Mirror 17: 0xaad52fc5 (stale) Mirror 18: 0xaad52fc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xaad52fc5 (stale) Mirror 14: 0xaad52fc5 (stale) Mirror 15: 0xaad52fc5 (stale) Mirror 16: 0xaad52fc5 (stale) Mirror 17: 0xaad52fc5 (stale) Mirror 18: 0xaad52fc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000539 1780533947.084993 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xaad52fc5 (stale) Mirror 14: 0xaad52fc5 (stale) Mirror 15: 0xaad52fc5 (stale) Mirror 16: 0xaad52fc5 (stale) Mirror 18: 0xaad52fc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xaad52fc5 (stale) Mirror 14: 0xaad52fc5 (stale) Mirror 15: 0xaad52fc5 (stale) Mirror 16: 0xaad52fc5 (stale) Mirror 18: 0xaad52fc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000540 1780533947.332719 trunc from 0x00937b to 0x00c3eb (0x3071 bytes) 000541 1780533947.336326 fallocate 0x013463 thru 0x017e9f (0x4a3d bytes) 000542 1780533947.340279 write 0x03a7b0 thru 0x03be51 (0x16a2 bytes) 000543 1780533947.349887 mapwrite 0x011c13 thru 0x01e502 (0xc8f0 bytes) 000544 1780533947.382340 fallocate 0x0275ff thru 0x027e24 (0x826 bytes) 000545 1780533947.387834 trunc from 0x040000 to 0x00a22f (0x35dd0 bytes) 000546 1780533947.400045 punch from 0x0026df to 0x003b58 (0x147a bytes) 000547 1780533947.403886 trunc from 0x00a230 to 0x03205c (0x27e2d bytes) 000548 1780533947.418773 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3205d): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3205d): Mirror 10: 0x75e9c710 (stale) Mirror 14: 0x75e9c710 (stale) Mirror 15: 0x75e9c710 (stale) Mirror 16: 0x75e9c710 (stale) Mirror 18: 0x75e9c710 (stale) CRC-32 checksum value for chunk [0, 0x3205d): Mirror 1: 0x4bfd4565 Mirror 19: 0x4bfd4565 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3205d: skipped 000549 1780533947.978845 fallocate 0x03f8d1 thru 0x03ffff (0x72f bytes) 000550 1780533948.004513 write 0x013ba6 thru 0x0152ff (0x175a bytes) 000551 1780533948.015786 fallocate 0x02b6b6 thru 0x036f36 (0xb881 bytes) 000552 1780533948.022919 read 0x029672 thru 0x03435b (0xacea bytes) 000553 1780533948.031296 punch from 0x003faf to 0x007428 (0x347a bytes) 000554 1780533948.035246 fallocate 0x03a6da thru 0x03ffff (0x5926 bytes) 000555 1780533948.038825 fallocate 0x003802 thru 0x00e22d (0xaa2c bytes) 000557 1780533948.042682 trunc from 0x040000 to 0x03c853 (0x37ac bytes) 000558 1780533948.058578 fallocate 0x00830e thru 0x00ad6d (0x2a60 bytes) 000559 1780533948.063005 fallocate 0x03a6a9 thru 0x03c280 (0x1bd8 bytes) 000560 1780533948.066322 trunc from 0x03c854 to 0x026da0 (0x15ab3 bytes) 000561 1780533948.075542 punch from 0x00d174 to 0x014e2c (0x7cb9 bytes) 000562 1780533948.084582 read 0x000752 thru 0x003a9d (0x334c bytes) 000563 1780533948.091074 read 0x01edff thru 0x026da0 (0x7fa2 bytes) 000564 1780533948.091829 mapread 0x006dd8 thru 0x01458e (0xd7b7 bytes) 000565 1780533948.094666 trunc from 0x01b057 to 0x023bb5 (0x8b5f bytes) 000566 1780533948.102914 fallocate 0x012e27 thru 0x01dc64 (0xae3e bytes) 000567 1780533948.108812 mapwrite 0x03d4a5 thru 0x03ffff (0x2b5b bytes) 000568 1780533948.156205 trunc from 0x021ef0 to 0x0251a7 (0x32b8 bytes) 000569 1780533948.164276 trunc from 0x016cd1 to 0x01822f (0x155f bytes) 000570 1780533948.169484 mapread 0x03fd46 thru 0x03ffff (0x2ba bytes) 000571 1780533948.170538 fallocate 0x01b856 thru 0x020481 (0x4c2c bytes) 000572 1780533948.176747 fallocate 0x01efb6 thru 0x026a43 (0x7a8e bytes) 000573 1780533948.188424 MIRROR_RESYNC 7 mirrors 000574 1780533949.115280 write 0x039f6d thru 0x03ffff (0x6093 bytes) 000575 1780533949.153242 punch from 0x00f982 to 0x01f005 (0xf684 bytes) 000576 1780533949.158817 mapread 0x0256ee thru 0x03190f (0xc222 bytes) 000578 1780533949.184349 fallocate 0x025760 thru 0x02cf6a (0x780b bytes) 000579 1780533949.189992 mapwrite 0x002bbb thru 0x012485 (0xf8cb bytes) 000580 1780533949.223928 punch from 0x0163ed to 0x016e20 (0xa34 bytes) 000582 1780533949.230663 trunc from 0x02e46b to 0x033434 (0x4fca bytes) 000583 1780533949.234800 punch from 0x01da9f to 0x02769b (0x9bfd bytes) 000585 1780533949.240070 trunc from 0x023fcf to 0x028db6 (0x4de8 bytes) 000586 1780533949.244346 punch from 0x020d1b to 0x02f1a6 (0xe48c bytes) 000587 1780533949.247853 write 0x02fe9f thru 0x032a7a (0x2bdc bytes) 000588 1780533949.256508 trunc from 0x040000 to 0x038e4f (0x71b0 bytes) 000589 1780533949.285422 trunc from 0x01e3bb to 0x0250b9 (0x6cff bytes) 000590 1780533949.293116 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38e50): Mirror 10: 0x7ae260e1 (stale) Mirror 14: 0x7ae260e1 (stale) Mirror 15: 0x7ae260e1 (stale) Mirror 16: 0x7ae260e1 (stale) Mirror 18: 0x7ae260e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e50): Mirror 10: 0x7ae260e1 (stale) Mirror 14: 0x7ae260e1 (stale) Mirror 15: 0x7ae260e1 (stale) Mirror 16: 0x7ae260e1 (stale) Mirror 18: 0x7ae260e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e50: skipped 000591 1780533949.602129 write 0x03d442 thru 0x03ffff (0x2bbe bytes) 000593 1780533949.608850 write 0x03b2e2 thru 0x03ffff (0x4d1e bytes) 000594 1780533949.613002 read 0x03428e thru 0x034785 (0x4f8 bytes) 000595 1780533949.618002 mapwrite 0x037bbd thru 0x039155 (0x1599 bytes) 000596 1780533949.643414 write 0x0259ee thru 0x027eab (0x24be bytes) 000597 1780533949.654102 trunc from 0x004463 to 0x0132ac (0xee4a bytes) 000598 1780533949.662042 trunc from 0x033029 to 0x033d5c (0xd34 bytes) 000600 1780533949.667860 fallocate 0x031334 thru 0x03ec22 (0xd8ef bytes) 000602 1780533949.675714 mapread 0x028afc thru 0x02e56c (0x5a71 bytes) 000603 1780533949.684216 trunc from 0x040000 to 0x02a21f (0x15de0 bytes) 000604 1780533949.697797 write 0x020249 thru 0x029716 (0x94ce bytes) 000605 1780533949.702896 punch from 0x02a04c to 0x02a21f (0x1d4 bytes) 000606 1780533949.708302 write 0x01a2cd thru 0x01dc9b (0x39cf bytes) 000607 1780533949.723626 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2a220): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a220): Mirror 10: 0xc66091a4 (stale) Mirror 14: 0xc66091a4 (stale) Mirror 15: 0xc66091a4 (stale) Mirror 16: 0xc66091a4 (stale) Mirror 18: 0xc66091a4 (stale) CRC-32 checksum value for chunk [0, 0x2a220): Mirror 1: 0x9d59b7dc Mirror 19: 0x9d59b7dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a220: skipped 000608 1780533950.318276 mapwrite 0x024699 thru 0x02bc4a (0x75b2 bytes) 000609 1780533950.384969 trunc from 0x00d47c to 0x017d05 (0xa88a bytes) 000610 1780533950.389461 read 0x015d4f thru 0x017632 (0x18e4 bytes) 000611 1780533950.393122 mapread 0x00d808 thru 0x0190f4 (0xb8ed bytes) 000612 1780533950.404587 write 0x017433 thru 0x026408 (0xefd6 bytes) 000613 1780533950.406131 write 0x037864 thru 0x03b74c (0x3ee9 bytes) 000614 1780533950.407959 read 0x03ae3d thru 0x03b74c (0x910 bytes) 000615 1780533950.408607 mapread 0x029f4d thru 0x03951c (0xf5d0 bytes) 000616 1780533950.418358 fallocate 0x0125ca thru 0x0212e4 (0xed1b bytes) 000617 1780533950.424176 mapread 0x00c424 thru 0x018fbe (0xcb9b bytes) 000618 1780533950.425484 fallocate 0x007e32 thru 0x01461e (0xc7ed bytes) 000619 1780533950.430426 read 0x032db1 thru 0x03b74c (0x899c bytes) 000621 1780533950.432058 trunc from 0x032ffa to 0x033627 (0x62e bytes) 000622 1780533950.437057 mapwrite 0x01a5b0 thru 0x021dfc (0x784d bytes) 000623 1780533950.457327 read 0x03729a thru 0x03b74c (0x44b3 bytes) 000624 1780533950.458007 read 0x0266b4 thru 0x035c34 (0xf581 bytes) 000625 1780533950.465547 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b74d): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b74d): Mirror 10: 0xece9186 (stale) Mirror 14: 0xece9186 (stale) Mirror 15: 0xece9186 (stale) Mirror 16: 0xece9186 (stale) Mirror 18: 0xece9186 (stale) Mirror 19: 0x40b1f8ec (stale) CRC-32 checksum value for chunk [0, 0x3b74d): Mirror 1: 0x19d7f149 Mirror 20: 0x19d7f149 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b74d: skipped 000626 1780533950.968349 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b74d): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b74d): Mirror 10: 0xece9186 (stale) Mirror 14: 0xece9186 (stale) Mirror 15: 0xece9186 (stale) Mirror 16: 0xece9186 (stale) Mirror 18: 0xece9186 (stale) Mirror 19: 0x40b1f8ec (stale) CRC-32 checksum value for chunk [0, 0x3b74d): Mirror 1: 0x19d7f149 Mirror 20: 0x19d7f149 Mirror 21: 0x19d7f149 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b74d: skipped 000628 1780533951.546451 read 0x020bc3 thru 0x02ca64 (0xbea2 bytes) 000629 1780533951.556261 fallocate 0x00288c thru 0x00d3e4 (0xab59 bytes) 000630 1780533951.590603 read 0x01f097 thru 0x026f9e (0x7f08 bytes) 000631 1780533951.596879 punch from 0x02588a to 0x0285e6 (0x2d5d bytes) 000632 1780533951.600005 write 0x0307c5 thru 0x03479b (0x3fd7 bytes) 000633 1780533951.603540 mapwrite 0x036f88 thru 0x03e9f3 (0x7a6c bytes) 000635 1780533951.652924 punch from 0x02a2e6 to 0x033bc4 (0x98df bytes) 000636 1780533951.656115 trunc from 0x02ef90 to 0x02f552 (0x5c3 bytes) 000637 1780533951.660407 mapread 0x012b2c thru 0x021049 (0xe51e bytes) 000638 1780533951.661909 trunc from 0x00896b to 0x017bb8 (0xf24e bytes) 000639 1780533951.666950 mapwrite 0x008db8 thru 0x00f994 (0x6bdd bytes) 000640 1780533951.709901 mapwrite 0x03b25c thru 0x03ffff (0x4da4 bytes) 000641 1780533951.739478 fallocate 0x002a70 thru 0x00ee56 (0xc3e7 bytes) 000642 1780533951.745770 trunc from 0x040000 to 0x007493 (0x38b6c bytes) 000643 1780533951.759356 trunc from 0x003c99 to 0x004d49 (0x10b1 bytes) 000644 1780533951.763537 mapread 0x00069c thru 0x007493 (0x6df8 bytes) 000645 1780533951.775486 write 0x00371a thru 0x008d57 (0x563e bytes) 000646 1780533951.782261 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x8d58): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8d58): Mirror 10: 0xc9f1c61c (stale) Mirror 14: 0xc9f1c61c (stale) Mirror 15: 0xc9f1c61c (stale) Mirror 16: 0xc9f1c61c (stale) Mirror 18: 0xc9f1c61c (stale) Mirror 19: 0xa6aeaf30 (stale) Mirror 20: 0xa6aeaf30 (stale) Mirror 21: 0xa6aeaf30 (stale) CRC-32 checksum value for chunk [0, 0x8d58): Mirror 1: 0x7074c2b6 Mirror 22: 0x7074c2b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8d58: skipped 000647 1780533952.205809 trunc from 0x003f11 to 0x008d57 (0x4e47 bytes) 000648 1780533952.228916 write 0x03bee4 thru 0x03ffff (0x411c bytes) 000649 1780533952.231969 trunc from 0x031906 to 0x03616f (0x486a bytes) 000650 1780533952.235597 write 0x029053 thru 0x030b8d (0x7b3b bytes) 000651 1780533952.241013 mapwrite 0x011222 thru 0x011bbb (0x99a bytes) 000653 1780533952.264884 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x1ebaea3d (stale) Mirror 14: 0x1ebaea3d (stale) Mirror 15: 0x1ebaea3d (stale) Mirror 16: 0x1ebaea3d (stale) Mirror 18: 0x1ebaea3d (stale) Mirror 19: 0x30d8c81f (stale) Mirror 20: 0xebd25b73 (stale) Mirror 21: 0xebd25b73 (stale) Mirror 22: 0xd3125fe9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86efc5a4 Mirror 23: 0x86efc5a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000654 1780533952.819625 fallocate 0x035615 thru 0x0388c5 (0x32b1 bytes) 000655 1780533952.846611 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x1ebaea3d (stale) Mirror 14: 0x1ebaea3d (stale) Mirror 15: 0x1ebaea3d (stale) Mirror 16: 0x1ebaea3d (stale) Mirror 18: 0x1ebaea3d (stale) Mirror 19: 0x30d8c81f (stale) Mirror 20: 0xebd25b73 (stale) Mirror 21: 0xebd25b73 (stale) Mirror 22: 0xd3125fe9 (stale) Mirror 23: 0x86efc5a4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86efc5a4 Mirror 24: 0x86efc5a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000656 1780533953.477577 read 0x018dca thru 0x01a676 (0x18ad bytes) 000657 1780533953.489557 mapread 0x03d10e thru 0x03ffff (0x2ef2 bytes) 000658 1780533953.517721 punch from 0x0352ef to 0x036771 (0x1483 bytes) 000659 1780533953.538799 trunc from 0x040000 to 0x02534b (0x1acb4 bytes) 000660 1780533953.552876 punch from 0x01ea21 to 0x02534b (0x692b bytes) 000661 1780533953.556834 mapread 0x000f13 thru 0x010866 (0xf954 bytes) 000662 1780533953.571459 trunc from 0x02534c to 0x012676 (0x12cd5 bytes) 000663 1780533953.579486 fallocate 0x008b3a thru 0x00ee7d (0x6344 bytes) 000664 1780533953.583512 trunc from 0x00d775 to 0x012676 (0x4f02 bytes) 000665 1780533953.588703 trunc from 0x012677 to 0x00aacf (0x7ba7 bytes) 000666 1780533953.598858 trunc from 0x00aad0 to 0x008215 (0x28ba bytes) 000667 1780533953.612705 punch from 0x007f0d to 0x008215 (0x309 bytes) 000668 1780533953.616259 trunc from 0x008216 to 0x0254d5 (0x1d2c0 bytes) 000669 1780533953.624355 read 0x014733 thru 0x020e30 (0xc6fe bytes) 000670 1780533953.634246 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x254d6): Mirror 10: 0xb9301405 (stale) Mirror 14: 0xb9301405 (stale) Mirror 15: 0xb9301405 (stale) Mirror 16: 0xb9301405 (stale) Mirror 18: 0xb9301405 (stale) Mirror 19: 0x3f95ac2c (stale) Mirror 20: 0xa9f3bd1a (stale) Mirror 21: 0xa9f3bd1a (stale) Mirror 22: 0x63f46973 (stale) Mirror 23: 0xd91c7f1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x254d6): Mirror 10: 0xb9301405 (stale) Mirror 14: 0xb9301405 (stale) Mirror 15: 0xb9301405 (stale) Mirror 16: 0xb9301405 (stale) Mirror 18: 0xb9301405 (stale) Mirror 19: 0x3f95ac2c (stale) Mirror 20: 0xa9f3bd1a (stale) Mirror 21: 0xa9f3bd1a (stale) Mirror 22: 0x63f46973 (stale) Mirror 23: 0xd91c7f1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x254d6: skipped 000671 1780533953.949408 trunc from 0x0254d6 to 0x011b6c (0x13969 bytes) 000672 1780533953.969094 mapread 0x00d35f thru 0x011b6c (0x480e bytes) 000673 1780533953.981233 mapwrite 0x025e5a thru 0x02e916 (0x8abd bytes) 000674 1780533954.012204 mapwrite 0x02b8a8 thru 0x035a7c (0xa1d5 bytes) 000675 1780533954.037559 mapread 0x01e479 thru 0x028b29 (0xa6b1 bytes) 000676 1780533954.038907 write 0x0358ba thru 0x03ffff (0xa746 bytes) 000677 1780533954.040592 read 0x036534 thru 0x03a2ba (0x3d87 bytes) 000678 1780533954.041308 trunc from 0x040000 to 0x035629 (0xa9d6 bytes) 000680 1780533954.050487 fallocate 0x02215d thru 0x02f0c2 (0xcf66 bytes) 000681 1780533954.054170 read 0x00946e thru 0x00fe84 (0x6a17 bytes) 000682 1780533954.054771 mapwrite 0x01fbf3 thru 0x02d3db (0xd7e9 bytes) 000683 1780533954.070325 mapwrite 0x0237b7 thru 0x02c85b (0x90a5 bytes) 000684 1780533954.089504 mapwrite 0x01c735 thru 0x0250ec (0x89b8 bytes) 000685 1780533954.110453 mapwrite 0x008e4e thru 0x012b9d (0x9d50 bytes) 000686 1780533954.124083 read 0x0281a3 thru 0x02e192 (0x5ff0 bytes) 000688 1780533954.125309 mapwrite 0x000695 thru 0x00ef11 (0xe87d bytes) 000689 1780533954.145520 MIRROR_RESYNC 11 mirrors 000690 1780533955.052229 mapread 0x01fbc1 thru 0x027e4a (0x828a bytes) 000691 1780533955.063795 trunc from 0x02c0d5 to 0x03547a (0x93a6 bytes) 000692 1780533955.092438 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3562a): Mirror 10: 0xbeabb5e (stale) Mirror 14: 0xbeabb5e (stale) Mirror 15: 0xbeabb5e (stale) Mirror 16: 0xbeabb5e (stale) Mirror 18: 0xbeabb5e (stale) Mirror 19: 0xbeabb5e (stale) Mirror 20: 0xbeabb5e (stale) Mirror 21: 0xbeabb5e (stale) Mirror 23: 0xbeabb5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3562a): Mirror 10: 0xbeabb5e (stale) Mirror 14: 0xbeabb5e (stale) Mirror 15: 0xbeabb5e (stale) Mirror 16: 0xbeabb5e (stale) Mirror 18: 0xbeabb5e (stale) Mirror 19: 0xbeabb5e (stale) Mirror 20: 0xbeabb5e (stale) Mirror 21: 0xbeabb5e (stale) Mirror 23: 0xbeabb5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3562a: skipped 000693 1780533955.370490 trunc from 0x03562a to 0x006b7a (0x2eaaf bytes) 000694 1780533955.391910 fallocate 0x009c42 thru 0x01017f (0x653e bytes) 000695 1780533955.395758 read 0x009877 thru 0x01017f (0x6909 bytes) 000696 1780533955.400842 read 0x005f74 thru 0x00ff20 (0x9fad bytes) 000698 1780533955.406453 mapread 0x00a4ab thru 0x01017f (0x5cd5 bytes) 000699 1780533955.407491 trunc from 0x006d50 to 0x01017f (0x9430 bytes) 000700 1780533955.410864 trunc from 0x010180 to 0x0078fd (0x8882 bytes) 000701 1780533955.418990 trunc from 0x001320 to 0x0078fd (0x65de bytes) 000702 1780533955.427747 punch from 0x00154f to 0x0078fd (0x63af bytes) 000703 1780533955.431981 punch from 0x00443e to 0x0078fd (0x34c0 bytes) 000704 1780533955.435813 write 0x027868 thru 0x02bfc0 (0x4759 bytes) 000705 1780533955.437403 punch from 0x013ac2 to 0x016dbb (0x32fa bytes) 000706 1780533955.446984 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfc1): Mirror 10: 0xad581348 (stale) Mirror 14: 0xad581348 (stale) Mirror 16: 0xad581348 (stale) Mirror 18: 0xad581348 (stale) Mirror 19: 0xad581348 (stale) Mirror 20: 0xad581348 (stale) Mirror 21: 0xad581348 (stale) Mirror 23: 0xad581348 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfc1): Mirror 10: 0xad581348 (stale) Mirror 14: 0xad581348 (stale) Mirror 16: 0xad581348 (stale) Mirror 18: 0xad581348 (stale) Mirror 19: 0xad581348 (stale) Mirror 20: 0xad581348 (stale) Mirror 21: 0xad581348 (stale) Mirror 23: 0xad581348 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfc1: skipped 000707 1780533955.747031 write 0x029e2b thru 0x030ac8 (0x6c9e bytes) 000708 1780533955.752433 trunc from 0x030ac9 to 0x001253 (0x2f875 bytes) 000709 1780533955.765966 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1254): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1254): Mirror 10: 0x2cfe9d55 (stale) Mirror 14: 0x2cfe9d55 (stale) Mirror 16: 0x2cfe9d55 (stale) Mirror 18: 0x2cfe9d55 (stale) Mirror 19: 0x2cfe9d55 (stale) Mirror 20: 0x2cfe9d55 (stale) Mirror 21: 0x2cfe9d55 (stale) Mirror 23: 0x2cfe9d55 (stale) CRC-32 checksum value for chunk [0, 0x1254): Mirror 1: 0x2cfe9d55 Mirror 24: 0x2cfe9d55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1254: skipped 000710 1780533956.091834 fallocate 0x03eabf thru 0x03ffa6 (0x14e8 bytes) 000711 1780533956.119293 trunc from 0x001254 to 0x03b2dd (0x3a08a bytes) 000712 1780533956.153001 punch from 0x025fef to 0x035839 (0xf84b bytes) 000713 1780533956.157296 trunc from 0x03b2de to 0x02e233 (0xd0aa bytes) 000714 1780533956.171968 fallocate 0x02155f thru 0x029b87 (0x8629 bytes) 000716 1780533956.181230 MIRROR_RESYNC 10 mirrors 000717 1780533956.925454 fallocate 0x0364d1 thru 0x03ffff (0x9b2f bytes) 000718 1780533956.946234 punch from 0x01cbd8 to 0x029fe0 (0xd409 bytes) 000719 1780533956.949226 punch from 0x00452b to 0x0097ad (0x5283 bytes) 000720 1780533956.952489 write 0x02d434 thru 0x03a1fc (0xcdc9 bytes) 000721 1780533956.959527 trunc from 0x00c5a9 to 0x00f650 (0x30a8 bytes) 000722 1780533956.963530 fallocate 0x03242a thru 0x03ffff (0xdbd6 bytes) 000723 1780533956.966765 mapread 0x00601b thru 0x009805 (0x37eb bytes) 000724 1780533956.972991 trunc from 0x040000 to 0x032e41 (0xd1be bytes) 000725 1780533956.992270 fallocate 0x03182a thru 0x0356d4 (0x3eab bytes) 000726 1780533956.998872 mapwrite 0x029c4a thru 0x0379c8 (0xdd7f bytes) 000727 1780533957.038384 MIRROR_RESYNC 10 mirrors 000728 1780533957.728715 mapwrite 0x0332bb thru 0x03ffff (0xcd45 bytes) 000729 1780533957.788463 trunc from 0x040000 to 0x00bd2b (0x342d4 bytes) 000730 1780533957.803938 trunc from 0x00bd2c to 0x013681 (0x7956 bytes) 000731 1780533957.814267 fallocate 0x01e93f thru 0x02d031 (0xe6f3 bytes) 000733 1780533957.817919 write 0x03d912 thru 0x03ffff (0x26ee bytes) 000734 1780533957.819061 trunc from 0x0106c4 to 0x019db6 (0x96f3 bytes) 000735 1780533957.822543 trunc from 0x03a31a to 0x03ffff (0x5ce6 bytes) 000736 1780533957.827911 mapwrite 0x0159c2 thru 0x023402 (0xda41 bytes) 000737 1780533957.853195 read 0x0345a9 thru 0x035920 (0x1378 bytes) 000738 1780533957.857565 trunc from 0x040000 to 0x03c6e2 (0x391d bytes) 000740 1780533957.869235 punch from 0x033b6b to 0x03672c (0x2bc2 bytes) 000741 1780533957.874008 trunc from 0x03c6e3 to 0x033848 (0x8e9a bytes) 000742 1780533957.883795 trunc from 0x0004ef to 0x001697 (0x11a9 bytes) 000743 1780533957.893221 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33849): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33849): Mirror 10: 0x2ebc18a6 (stale) Mirror 14: 0x2ebc18a6 (stale) Mirror 16: 0x2ebc18a6 (stale) Mirror 18: 0x2ebc18a6 (stale) Mirror 19: 0x2ebc18a6 (stale) Mirror 20: 0x2ebc18a6 (stale) Mirror 21: 0x2ebc18a6 (stale) Mirror 23: 0x2ebc18a6 (stale) Mirror 24: 0x2ebc18a6 (stale) CRC-32 checksum value for chunk [0, 0x33849): Mirror 1: 0xa1db849a Mirror 25: 0xa1db849a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33849: skipped 000744 1780533958.439318 read 0x00e334 thru 0x011517 (0x31e4 bytes) 000745 1780533958.451896 mapread 0x01c569 thru 0x027474 (0xaf0c bytes) 000746 1780533958.484242 trunc from 0x00eb8e to 0x01b719 (0xcb8c bytes) 000747 1780533958.516802 write 0x03102c thru 0x037644 (0x6619 bytes) 000748 1780533958.526664 read 0x01badd thru 0x0241cc (0x86f0 bytes) 000749 1780533958.533975 read 0x0257fd thru 0x02b71b (0x5f1f bytes) 000750 1780533958.534995 mapwrite 0x003765 thru 0x00cdae (0x964a bytes) 000751 1780533958.556483 read 0x02aa0c thru 0x0331e8 (0x87dd bytes) 000752 1780533958.557293 mapread 0x013f77 thru 0x01b6cf (0x7759 bytes) 000753 1780533958.558483 trunc from 0x037645 to 0x001a64 (0x35be0 bytes) 000754 1780533958.572816 trunc from 0x001785 to 0x001a64 (0x2e0 bytes) 000755 1780533958.577277 mapwrite 0x02ff20 thru 0x0387f8 (0x88d9 bytes) 000756 1780533958.616393 MIRROR_RESYNC 11 mirrors 000757 1780533959.511382 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000758 1780533959.764149 write 0x0189cb thru 0x0266ae (0xdce4 bytes) 000759 1780533959.792323 mapread 0x01b0e4 thru 0x02b0e3 (0x10000 bytes) 000760 1780533959.800103 read 0x03519d thru 0x0387f8 (0x365c bytes) 000762 1780533959.804737 fallocate 0x02d245 thru 0x032c7d (0x5a39 bytes) 000763 1780533959.809230 read 0x002bae thru 0x008061 (0x54b4 bytes) 000764 1780533959.815486 fallocate 0x000bc5 thru 0x008283 (0x76bf bytes) 000765 1780533959.821998 mapwrite 0x00f0ea thru 0x0120ff (0x3016 bytes) 000766 1780533959.835125 trunc from 0x018cf0 to 0x0230e9 (0xa3fa bytes) 000768 1780533959.842038 trunc from 0x0013d0 to 0x008e90 (0x7ac1 bytes) 000769 1780533959.847872 read 0x007b81 thru 0x011cba (0xa13a bytes) 000770 1780533959.859218 fallocate 0x009ed6 thru 0x019699 (0xf7c4 bytes) 000771 1780533959.863160 trunc from 0x0387f9 to 0x00c951 (0x2bea7 bytes) 000772 1780533959.879182 trunc from 0x00b905 to 0x00c64b (0xd47 bytes) 000773 1780533959.882466 punch from 0x004488 to 0x006e77 (0x29f0 bytes) 000774 1780533959.886149 write 0x028ada thru 0x0337e3 (0xad0a bytes) 000775 1780533959.887537 fallocate 0x03cf48 thru 0x03ffff (0x30b8 bytes) 000776 1780533959.894530 fallocate 0x02ab68 thru 0x02de62 (0x32fb bytes) 000777 1780533959.899731 read 0x0019f0 thru 0x00c3ff (0xaa10 bytes) 000778 1780533959.909385 read 0x007acb thru 0x00a1ba (0x26f0 bytes) 000781 1780533959.915291 mapread 0x028478 thru 0x0337e3 (0xb36c bytes) 000782 1780533959.916609 trunc from 0x0337e4 to 0x009ae7 (0x29cfc bytes) 000784 1780533959.935423 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9ae8): Mirror 10: 0xcbc548c4 (stale) Mirror 14: 0xcbc548c4 (stale) Mirror 16: 0xcbc548c4 (stale) Mirror 18: 0xcbc548c4 (stale) Mirror 20: 0xcbc548c4 (stale) Mirror 21: 0xcbc548c4 (stale) Mirror 23: 0xcbc548c4 (stale) Mirror 24: 0xcbc548c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9ae8): Mirror 10: 0xcbc548c4 (stale) Mirror 14: 0xcbc548c4 (stale) Mirror 16: 0xcbc548c4 (stale) Mirror 18: 0xcbc548c4 (stale) Mirror 20: 0xcbc548c4 (stale) Mirror 21: 0xcbc548c4 (stale) Mirror 23: 0xcbc548c4 (stale) Mirror 24: 0xcbc548c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9ae8: skipped 000785 1780533960.206963 mapwrite 0x03a747 thru 0x03cb06 (0x23c0 bytes) 000787 1780533960.246128 trunc from 0x03cb07 to 0x02eddd (0xdd29 bytes) 000788 1780533960.257394 trunc from 0x028c8c to 0x028cd3 (0x48 bytes) 000789 1780533960.263591 punch from 0x0199a2 to 0x0203ff (0x6a5e bytes) 000790 1780533960.267184 read 0x00740b thru 0x016885 (0xf47b bytes) 000791 1780533960.274717 punch from 0x0224e0 to 0x02aa45 (0x8566 bytes) 000793 1780533960.278594 mapwrite 0x015c55 thru 0x017f8b (0x2337 bytes) 000795 1780533960.292826 fallocate 0x00fabc thru 0x01e00a (0xe54f bytes) 000796 1780533960.296613 read 0x004f2a thru 0x00f71a (0xa7f1 bytes) 000797 1780533960.306317 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2edde): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edde): Mirror 10: 0x79e6ae47 (stale) Mirror 14: 0x79e6ae47 (stale) Mirror 16: 0x79e6ae47 (stale) Mirror 18: 0x79e6ae47 (stale) Mirror 20: 0x79e6ae47 (stale) Mirror 21: 0x79e6ae47 (stale) Mirror 23: 0x79e6ae47 (stale) Mirror 24: 0x79e6ae47 (stale) CRC-32 checksum value for chunk [0, 0x2edde): Mirror 1: 0x8de672a Mirror 25: 0x8de672a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edde: skipped 000798 1780533960.831521 write 0x02b2ea thru 0x039412 (0xe129 bytes) 000799 1780533960.855294 mapwrite 0x00004b thru 0x00cf4e (0xcf04 bytes) 000800 1780533960.876524 fallocate 0x00737b thru 0x015911 (0xe597 bytes) 000801 1780533960.882793 trunc from 0x0009f8 to 0x003be7 (0x31f0 bytes) 000802 1780533960.889225 fallocate 0x039266 thru 0x03ffff (0x6d9a bytes) 000803 1780533960.892706 read 0x00a913 thru 0x019289 (0xe977 bytes) 000804 1780533960.899259 trunc from 0x0148ef to 0x017cc4 (0x33d6 bytes) 000805 1780533960.903251 mapread 0x015d4d thru 0x01eb35 (0x8de9 bytes) 000806 1780533960.919947 read 0x01d94f thru 0x027c57 (0xa309 bytes) 000807 1780533960.928829 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39413): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39413): Mirror 10: 0x14a7d5b9 (stale) Mirror 14: 0x14a7d5b9 (stale) Mirror 16: 0x14a7d5b9 (stale) Mirror 18: 0x14a7d5b9 (stale) Mirror 20: 0x14a7d5b9 (stale) Mirror 21: 0x14a7d5b9 (stale) Mirror 23: 0x14a7d5b9 (stale) Mirror 24: 0x14a7d5b9 (stale) Mirror 25: 0xcb7fd4f5 (stale) CRC-32 checksum value for chunk [0, 0x39413): Mirror 1: 0x17896bb6 Mirror 26: 0x17896bb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39413: skipped 000808 1780533961.475197 read 0x0028f3 thru 0x00a237 (0x7945 bytes) 000809 1780533961.490740 mapread 0x0064a8 thru 0x0084c6 (0x201f bytes) 000810 1780533961.509058 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x39413): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39413): Mirror 10: 0x14a7d5b9 (stale) Mirror 14: 0x14a7d5b9 (stale) Mirror 16: 0x14a7d5b9 (stale) Mirror 18: 0x14a7d5b9 (stale) Mirror 20: 0x14a7d5b9 (stale) Mirror 21: 0x14a7d5b9 (stale) Mirror 23: 0x14a7d5b9 (stale) Mirror 24: 0x14a7d5b9 (stale) Mirror 25: 0xcb7fd4f5 (stale) CRC-32 checksum value for chunk [0, 0x39413): Mirror 1: 0x17896bb6 Mirror 26: 0x17896bb6 Mirror 27: 0x17896bb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39413: skipped 000811 1780533962.051176 trunc from 0x039413 to 0x01663f (0x22dd3 bytes) 000812 1780533962.084580 trunc from 0x016640 to 0x01482c (0x1e13 bytes) 000813 1780533962.099439 mapwrite 0x0372d8 thru 0x03ffff (0x8d28 bytes) 000814 1780533962.134729 punch from 0x006508 to 0x00c08b (0x5b84 bytes) 000815 1780533962.140667 write 0x026448 thru 0x02e995 (0x854e bytes) 000816 1780533962.148689 fallocate 0x0267a2 thru 0x031322 (0xab81 bytes) 000817 1780533962.153482 write 0x03958a thru 0x03ffff (0x6a76 bytes) 000818 1780533962.154603 read 0x00c128 thru 0x0181ad (0xc086 bytes) 000819 1780533962.164983 trunc from 0x040000 to 0x01b297 (0x24d68 bytes) 000820 1780533962.176455 mapread 0x000c2e thru 0x009a0f (0x8de2 bytes) 000821 1780533962.183489 mapwrite 0x017bd1 thru 0x024300 (0xc730 bytes) 000822 1780533962.215170 mapwrite 0x00e1bd thru 0x013933 (0x5777 bytes) 000823 1780533962.229148 mapwrite 0x03871f thru 0x03c9c0 (0x42a2 bytes) 000824 1780533962.264124 fallocate 0x0106bd thru 0x01b13e (0xaa82 bytes) 000825 1780533962.273031 mapwrite 0x0362a0 thru 0x037a72 (0x17d3 bytes) 000827 1780533962.288700 write 0x01ed48 thru 0x025dc3 (0x707c bytes) 000828 1780533962.294253 mapread 0x03a821 thru 0x03c9c0 (0x21a0 bytes) 000829 1780533962.295418 write 0x02a1ab thru 0x03769d (0xd4f3 bytes) 000830 1780533962.303797 punch from 0x0085b2 to 0x011b28 (0x9577 bytes) 000831 1780533962.307841 read 0x009a7a thru 0x009ec0 (0x447 bytes) 000832 1780533962.313608 trunc from 0x008358 to 0x00caeb (0x4794 bytes) 000833 1780533962.320882 MIRROR_RESYNC 12 mirrors 000834 1780533964.271596 MIRROR_EXTEND to 13 mirrors 000836 1780533964.753340 fallocate 0x025316 thru 0x02d0ac (0x7d97 bytes) 000837 1780533964.780764 trunc from 0x03c9c1 to 0x001f25 (0x3aa9b bytes) 000838 1780533964.796752 read 0x000822 thru 0x001f25 (0x1704 bytes) 000839 1780533964.803847 trunc from 0x000c5d to 0x001f25 (0x12c9 bytes) 000840 1780533964.808141 punch from 0x000fd3 to 0x001f25 (0xf53 bytes) 000841 1780533964.811594 punch from 0x00043e to 0x001f25 (0x1ae8 bytes) 000842 1780533964.816983 read 0x001663 thru 0x001f25 (0x8c3 bytes) 000843 1780533964.821028 punch from 0x0006ad to 0x001f25 (0x1879 bytes) 000844 1780533964.829839 MIRROR_RESYNC 13 mirrors 000845 1780533965.671608 mapwrite 0x00721a thru 0x00e51d (0x7304 bytes) 000846 1780533965.727997 mapread 0x005ca8 thru 0x00e51d (0x8876 bytes) 000847 1780533965.729299 write 0x0322ef thru 0x03613a (0x3e4c bytes) 000849 1780533965.731951 write 0x038c67 thru 0x03ffff (0x7399 bytes) 000850 1780533965.733336 mapwrite 0x036cee thru 0x03ffff (0x9312 bytes) 000851 1780533965.750764 trunc from 0x040000 to 0x01fe38 (0x201c7 bytes) 000852 1780533965.764359 mapread 0x01c8ec thru 0x01fe38 (0x354d bytes) 000853 1780533965.773514 MIRROR_RESYNC 13 mirrors 000854 1780533966.957088 trunc from 0x010f0e to 0x017c9d (0x6d90 bytes) 000855 1780533966.982424 punch from 0x005948 to 0x008cd6 (0x338f bytes) 000857 1780533966.987823 mapread 0x00da4f thru 0x01d653 (0xfc05 bytes) 000858 1780533967.012822 trunc from 0x01fe39 to 0x020805 (0x9cd bytes) 000859 1780533967.028130 mapwrite 0x03c105 thru 0x03dbf1 (0x1aed bytes) 000860 1780533967.067322 write 0x019449 thru 0x02209c (0x8c54 bytes) 000861 1780533967.072062 write 0x017783 thru 0x0218fb (0xa179 bytes) 000862 1780533967.073438 trunc from 0x0357df to 0x03dbf1 (0x8413 bytes) 000863 1780533967.079140 fallocate 0x008233 thru 0x012964 (0xa732 bytes) 000864 1780533967.083187 read 0x03a28e thru 0x03d2c0 (0x3033 bytes) 000865 1780533967.089259 trunc from 0x03dbf2 to 0x026bd5 (0x1701c bytes) 000866 1780533967.099349 mapwrite 0x0313b2 thru 0x034ac5 (0x3714 bytes) 000867 1780533967.139310 fallocate 0x01b200 thru 0x026c0d (0xba0e bytes) 000868 1780533967.143150 write 0x01e300 thru 0x02c346 (0xe047 bytes) 000871 1780533967.145759 read 0x01b864 thru 0x01dad6 (0x2273 bytes) 000872 1780533967.146466 trunc from 0x034ac6 to 0x0121e1 (0x228e4 bytes) 000873 1780533967.157559 trunc from 0x0121e2 to 0x03bfe8 (0x29e07 bytes) 000874 1780533967.167372 trunc from 0x03bfe9 to 0x01fa7f (0x1c569 bytes) 000875 1780533967.177002 trunc from 0x01fa80 to 0x00982f (0x16250 bytes) 000876 1780533967.193795 punch from 0x0001ce to 0x00982f (0x9662 bytes) 000878 1780533967.198847 fallocate 0x033b8e thru 0x03d70d (0x9b80 bytes) 000879 1780533967.202817 trunc from 0x009830 to 0x002dc3 (0x6a6c bytes) 000880 1780533967.213917 read 0x00270a thru 0x002dc3 (0x6ba bytes) 000881 1780533967.218481 trunc from 0x002dc4 to 0x0305ef (0x2d82c bytes) 000883 1780533967.232854 write 0x0157c3 thru 0x0244b3 (0xecf1 bytes) 000884 1780533967.239457 write 0x01464d thru 0x0146f7 (0xab bytes) 000886 1780533967.241051 read 0x02ba88 thru 0x0305ef (0x4b68 bytes) 000887 1780533967.241598 trunc from 0x0305f0 to 0x03b875 (0xb286 bytes) 000888 1780533967.249880 punch from 0x007714 to 0x013360 (0xbc4d bytes) 000889 1780533967.252859 punch from 0x025c40 to 0x02e751 (0x8b12 bytes) 000890 1780533967.256031 write 0x0126ab thru 0x020ab9 (0xe40f bytes) 000891 1780533967.266080 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b876): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b876): Mirror 10: 0x2abd9444 (stale) Mirror 14: 0x2abd9444 (stale) Mirror 16: 0x2abd9444 (stale) Mirror 18: 0x2abd9444 (stale) Mirror 20: 0x2abd9444 (stale) Mirror 21: 0x2abd9444 (stale) Mirror 23: 0x2abd9444 (stale) Mirror 24: 0x2abd9444 (stale) Mirror 25: 0x2abd9444 (stale) Mirror 26: 0x2abd9444 (stale) Mirror 27: 0x2abd9444 (stale) Mirror 28: 0x2abd9444 (stale) CRC-32 checksum value for chunk [0, 0x3b876): Mirror 1: 0x5f633c8e Mirror 29: 0x5f633c8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b876: skipped 000892 1780533967.818449 punch from 0x017a2b to 0x01fcf8 (0x82ce bytes) 000893 1780533967.843900 trunc from 0x0068d0 to 0x006d1a (0x44b bytes) 000894 1780533967.848021 mapread 0x0209e4 thru 0x023665 (0x2c82 bytes) 000895 1780533967.858958 mapwrite 0x02cbdd thru 0x032dd0 (0x61f4 bytes) 000896 1780533967.874752 mapread 0x0239ab thru 0x033277 (0xf8cd bytes) 000897 1780533967.876224 fallocate 0x019149 thru 0x027c30 (0xeae8 bytes) 000898 1780533967.886774 MIRROR_RESYNC 14 mirrors 000899 1780533969.815365 write 0x0186dd thru 0x02828b (0xfbaf bytes) 000900 1780533969.844151 trunc from 0x02279c to 0x02eb60 (0xc3c5 bytes) 000901 1780533969.855856 fallocate 0x03b1d4 thru 0x03ffff (0x4e2c bytes) 000902 1780533969.860128 mapwrite 0x01c91f thru 0x01e4af (0x1b91 bytes) 000903 1780533969.871143 fallocate 0x0320f3 thru 0x039899 (0x77a7 bytes) 000904 1780533969.875000 mapread 0x024d6a thru 0x02fe73 (0xb10a bytes) 000905 1780533969.884917 read 0x012762 thru 0x01a80b (0x80aa bytes) 000906 1780533969.890929 trunc from 0x040000 to 0x03dcb9 (0x2346 bytes) 000907 1780533969.904355 mapwrite 0x03ee81 thru 0x03f89d (0xa1d bytes) 000908 1780533969.923732 write 0x0274f9 thru 0x02c1b1 (0x4cb9 bytes) 000909 1780533969.924808 write 0x021681 thru 0x02a648 (0x8fc8 bytes) 000910 1780533969.926039 trunc from 0x03f89e to 0x003a68 (0x3be35 bytes) 000911 1780533969.939872 write 0x004e28 thru 0x0095cc (0x47a5 bytes) 000912 1780533969.941071 fallocate 0x0293ef thru 0x03542b (0xc03d bytes) 000913 1780533969.949556 MIRROR_RESYNC 14 mirrors 000915 1780533971.069521 MIRROR_EXTEND to 15 mirrors 000916 1780533971.465597 mapread 0x0073e4 thru 0x00bba0 (0x47bd bytes) 000917 1780533971.479817 trunc from 0x0067d4 to 0x011c21 (0xb44e bytes) 000918 1780533971.506788 write 0x02753c thru 0x02c165 (0x4c2a bytes) 000919 1780533971.520529 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3542c): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3542c): Mirror 10: 0x244a7111 (stale) Mirror 14: 0x244a7111 (stale) Mirror 16: 0x244a7111 (stale) Mirror 18: 0x244a7111 (stale) Mirror 20: 0x244a7111 (stale) Mirror 21: 0x244a7111 (stale) Mirror 23: 0x244a7111 (stale) Mirror 24: 0x244a7111 (stale) Mirror 25: 0x244a7111 (stale) Mirror 26: 0x244a7111 (stale) Mirror 27: 0x244a7111 (stale) Mirror 28: 0x244a7111 (stale) Mirror 29: 0x244a7111 (stale) Mirror 30: 0x244a7111 (stale) CRC-32 checksum value for chunk [0, 0x3542c): Mirror 1: 0xa4bddde8 Mirror 31: 0xa4bddde8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3542c: skipped 000920 1780533972.181904 write 0x01ab7b thru 0x01def9 (0x337f bytes) 000921 1780533972.210587 read 0x01157c thru 0x01e3b0 (0xce35 bytes) 000922 1780533972.215894 trunc from 0x03542c to 0x03fcf1 (0xa8c6 bytes) 000923 1780533972.227009 mapwrite 0x010bbf thru 0x0122c9 (0x170b bytes) 000924 1780533972.245659 trunc from 0x03fcf2 to 0x02d546 (0x127ab bytes) 000925 1780533972.252792 mapread 0x003054 thru 0x006237 (0x31e4 bytes) 000926 1780533972.259563 read 0x00bba2 thru 0x014f83 (0x93e2 bytes) 000927 1780533972.260194 mapwrite 0x021c87 thru 0x02e30f (0xc689 bytes) 000929 1780533972.284777 punch from 0x011c7d to 0x018c23 (0x6fa7 bytes) 000930 1780533972.288769 read 0x0046f4 thru 0x0047ca (0xd7 bytes) 000931 1780533972.289482 fallocate 0x008149 thru 0x017100 (0xefb8 bytes) 000932 1780533972.292998 mapread 0x027d0b thru 0x02e30f (0x6605 bytes) 000933 1780533972.293983 read 0x005d8a thru 0x015cb7 (0xff2e bytes) 000935 1780533972.303384 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e310): Mirror 14: 0xd46af641 (stale) Mirror 16: 0xd46af641 (stale) Mirror 18: 0xd46af641 (stale) Mirror 20: 0xd46af641 (stale) Mirror 21: 0xd46af641 (stale) Mirror 23: 0xd46af641 (stale) Mirror 24: 0xd46af641 (stale) Mirror 25: 0xd46af641 (stale) Mirror 26: 0xd46af641 (stale) Mirror 27: 0xd46af641 (stale) Mirror 28: 0xd46af641 (stale) Mirror 29: 0xd46af641 (stale) Mirror 30: 0xd46af641 (stale) Mirror 31: 0x41c6c15c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e310): Mirror 14: 0xd46af641 (stale) Mirror 16: 0xd46af641 (stale) Mirror 18: 0xd46af641 (stale) Mirror 20: 0xd46af641 (stale) Mirror 21: 0xd46af641 (stale) Mirror 23: 0xd46af641 (stale) Mirror 24: 0xd46af641 (stale) Mirror 25: 0xd46af641 (stale) Mirror 26: 0xd46af641 (stale) Mirror 27: 0xd46af641 (stale) Mirror 28: 0xd46af641 (stale) Mirror 29: 0xd46af641 (stale) Mirror 30: 0xd46af641 (stale) Mirror 31: 0x41c6c15c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e310: skipped 000937 1780533972.669307 mapread 0x0104c2 thru 0x01c528 (0xc067 bytes) 000938 1780533972.687195 write 0x03101e thru 0x03dc8e (0xcc71 bytes) 000939 1780533972.691053 fallocate 0x03f20a thru 0x03ffff (0xdf6 bytes) 000940 1780533972.694932 mapread 0x02f53f thru 0x0376ba (0x817c bytes) 000941 1780533972.700314 trunc from 0x040000 to 0x026bfc (0x19403 bytes) 000942 1780533972.711810 mapread 0x01ac33 thru 0x01da4e (0x2e1c bytes) 000943 1780533972.713080 punch from 0x0090fd to 0x00b299 (0x219d bytes) 000944 1780533972.717013 mapread 0x012475 thru 0x02218f (0xfd1b bytes) 000945 1780533972.718993 mapwrite 0x00c255 thru 0x01607e (0x9e2a bytes) 000946 1780533972.734256 mapread 0x01bcfa thru 0x0213a6 (0x56ad bytes) 000947 1780533972.739946 MIRROR_RESYNC 15 mirrors 000950 1780533974.890516 read 0x0116ea thru 0x01c1ce (0xaae5 bytes) 000952 1780533974.896923 punch from 0x0095f9 to 0x01202f (0x8a37 bytes) 000953 1780533974.921405 fallocate 0x00ac37 thru 0x01798f (0xcd59 bytes) 000954 1780533974.925017 mapwrite 0x00ac09 thru 0x018b10 (0xdf08 bytes) 000955 1780533974.960780 trunc from 0x026bfd to 0x00458c (0x22670 bytes) 000956 1780533974.972576 write 0x02cf91 thru 0x0309d0 (0x3a40 bytes) 000957 1780533974.973934 fallocate 0x0281c2 thru 0x02c9b9 (0x47f8 bytes) 000958 1780533974.981299 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x309d1): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x309d1): Mirror 14: 0x83f59d2d (stale) Mirror 16: 0x83f59d2d (stale) Mirror 18: 0x83f59d2d (stale) Mirror 20: 0x83f59d2d (stale) Mirror 21: 0x83f59d2d (stale) Mirror 23: 0x83f59d2d (stale) Mirror 24: 0x83f59d2d (stale) Mirror 25: 0x83f59d2d (stale) Mirror 26: 0x83f59d2d (stale) Mirror 27: 0x83f59d2d (stale) Mirror 28: 0x83f59d2d (stale) Mirror 29: 0x83f59d2d (stale) Mirror 30: 0x83f59d2d (stale) Mirror 31: 0x83f59d2d (stale) CRC-32 checksum value for chunk [0, 0x309d1): Mirror 1: 0xed5db9ff Mirror 32: 0xed5db9ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x309d1: skipped 000959 1780533975.590567 punch from 0x0159fc to 0x02245c (0xca61 bytes) 000960 1780533975.614952 punch from 0x02dbc7 to 0x0309d0 (0x2e0a bytes) 000961 1780533975.618742 punch from 0x014c6e to 0x0164e7 (0x187a bytes) 000962 1780533975.622044 mapwrite 0x030661 thru 0x0345e7 (0x3f87 bytes) 000963 1780533975.661771 MIRROR_SPLIT mirror 14 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: 0x66e4254a (stale) Mirror 16: 0x66e4254a (stale) Mirror 18: 0x66e4254a (stale) Mirror 20: 0x66e4254a (stale) Mirror 21: 0x66e4254a (stale) Mirror 23: 0x66e4254a (stale) Mirror 24: 0x66e4254a (stale) Mirror 25: 0x66e4254a (stale) Mirror 26: 0x66e4254a (stale) Mirror 27: 0x66e4254a (stale) Mirror 28: 0x66e4254a (stale) Mirror 29: 0x66e4254a (stale) Mirror 30: 0x66e4254a (stale) Mirror 32: 0xb5ec2a2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: 0x66e4254a (stale) Mirror 16: 0x66e4254a (stale) Mirror 18: 0x66e4254a (stale) Mirror 20: 0x66e4254a (stale) Mirror 21: 0x66e4254a (stale) Mirror 23: 0x66e4254a (stale) Mirror 24: 0x66e4254a (stale) Mirror 25: 0x66e4254a (stale) Mirror 26: 0x66e4254a (stale) Mirror 27: 0x66e4254a (stale) Mirror 28: 0x66e4254a (stale) Mirror 29: 0x66e4254a (stale) Mirror 30: 0x66e4254a (stale) Mirror 32: 0xb5ec2a2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345e8: skipped 000964 1780533976.022017 trunc from 0x00f830 to 0x015bb1 (0x6382 bytes) 000965 1780533976.029431 read 0x028295 thru 0x02b9d8 (0x3744 bytes) 000966 1780533976.038355 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: 0x66e4254a (stale) Mirror 16: 0x66e4254a (stale) Mirror 18: 0x66e4254a (stale) Mirror 21: 0x66e4254a (stale) Mirror 23: 0x66e4254a (stale) Mirror 24: 0x66e4254a (stale) Mirror 25: 0x66e4254a (stale) Mirror 26: 0x66e4254a (stale) Mirror 27: 0x66e4254a (stale) Mirror 28: 0x66e4254a (stale) Mirror 29: 0x66e4254a (stale) Mirror 30: 0x66e4254a (stale) Mirror 32: 0xb5ec2a2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: 0x66e4254a (stale) Mirror 16: 0x66e4254a (stale) Mirror 18: 0x66e4254a (stale) Mirror 21: 0x66e4254a (stale) Mirror 23: 0x66e4254a (stale) Mirror 24: 0x66e4254a (stale) Mirror 25: 0x66e4254a (stale) Mirror 26: 0x66e4254a (stale) Mirror 27: 0x66e4254a (stale) Mirror 28: 0x66e4254a (stale) Mirror 29: 0x66e4254a (stale) Mirror 30: 0x66e4254a (stale) Mirror 32: 0xb5ec2a2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345e8: skipped 000967 1780533976.401549 mapread 0x025b56 thru 0x0317a9 (0xbc54 bytes) 000968 1780533976.421197 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: 0x66e4254a (stale) Mirror 16: 0x66e4254a (stale) Mirror 18: 0x66e4254a (stale) Mirror 21: 0x66e4254a (stale) Mirror 23: 0x66e4254a (stale) Mirror 24: 0x66e4254a (stale) Mirror 25: 0x66e4254a (stale) Mirror 26: 0x66e4254a (stale) Mirror 27: 0x66e4254a (stale) Mirror 28: 0x66e4254a (stale) Mirror 29: 0x66e4254a (stale) Mirror 30: 0x66e4254a (stale) Mirror 32: 0xb5ec2a2a (stale) CRC-32 checksum value for chunk [0, 0x345e8): Mirror 1: 0x7032d02a Mirror 33: 0x7032d02a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345e8: skipped 000969 1780533977.016416 trunc from 0x017815 to 0x020f0c (0x96f8 bytes) 000970 1780533977.046248 read 0x0196d1 thru 0x026cb6 (0xd5e6 bytes) 000971 1780533977.052862 fallocate 0x010dd9 thru 0x01717c (0x63a4 bytes) 000972 1780533977.056467 trunc from 0x01c669 to 0x02a174 (0xdb0c bytes) 000973 1780533977.065916 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345e8): Mirror 14: 0x66e4254a (stale) Mirror 16: 0x66e4254a (stale) Mirror 18: 0x66e4254a (stale) Mirror 21: 0x66e4254a (stale) Mirror 23: 0x66e4254a (stale) Mirror 24: 0x66e4254a (stale) Mirror 25: 0x66e4254a (stale) Mirror 26: 0x66e4254a (stale) Mirror 27: 0x66e4254a (stale) Mirror 28: 0x66e4254a (stale) Mirror 29: 0x66e4254a (stale) Mirror 30: 0x66e4254a (stale) Mirror 32: 0xb5ec2a2a (stale) Mirror 33: 0x7032d02a (stale) CRC-32 checksum value for chunk [0, 0x345e8): Mirror 1: 0x7032d02a Mirror 34: 0x7032d02a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345e8: skipped 000974 1780533977.702560 punch from 0x002b55 to 0x00be1e (0x92ca bytes) 000975 1780533977.730797 mapread 0x00b4ba thru 0x00e0a5 (0x2bec bytes) 000976 1780533977.741337 mapwrite 0x022c61 thru 0x02c61f (0x99bf bytes) 000977 1780533977.768904 fallocate 0x02d759 thru 0x03ab91 (0xd439 bytes) 000978 1780533977.773072 trunc from 0x0345e8 to 0x005c9b (0x2e94c bytes) 000979 1780533977.786906 trunc from 0x005c9c to 0x019b60 (0x13ec5 bytes) 000980 1780533977.797132 read 0x0007bc thru 0x00f594 (0xedd9 bytes) 000981 1780533977.806786 trunc from 0x019b61 to 0x03ef09 (0x253a9 bytes) 000982 1780533977.815504 mapwrite 0x03ff8f thru 0x03ffff (0x71 bytes) 000983 1780533977.842600 mapread 0x00cfd9 thru 0x00d6af (0x6d7 bytes) 000984 1780533977.847302 MIRROR_RESYNC 16 mirrors 000985 1780533979.075260 read 0x00bd52 thru 0x00d49a (0x1749 bytes) 000988 1780533979.079307 write 0x016057 thru 0x020de0 (0xad8a bytes) 000989 1780533979.117845 MIRROR_RESYNC 16 mirrors 000990 1780533980.552943 MIRROR_RESYNC 16 mirrors 000991 1780533980.894965 punch from 0x0138e4 to 0x017850 (0x3f6d bytes) 000992 1780533980.913761 read 0x03cc9d thru 0x03e577 (0x18db bytes) 000993 1780533980.916960 trunc from 0x040000 to 0x002774 (0x3d88b bytes) 000994 1780533980.926735 punch from 0x0018b2 to 0x002774 (0xec3 bytes) 000995 1780533980.930230 mapread 0x000c7b thru 0x002774 (0x1afa bytes) 000996 1780533980.940957 mapread 0x0020e8 thru 0x002774 (0x68d bytes) 000997 1780533980.941783 punch from 0x00061d to 0x002060 (0x1a44 bytes) 000998 1780533980.945325 fallocate 0x02dfd4 thru 0x037bb0 (0x9bdd bytes) 000999 1780533980.948575 mapread 0x0021aa thru 0x002774 (0x5cb bytes) All operations completed A-OK! PASS 70a (73s) == sanity-flr test 71: check mirror extend parent fid ==== 20:46:23 (1780533983) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x416:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 20:46:27 (1780533987) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x675:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x677:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0767266 s, 27.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x675:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x676:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x677:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049215 s, 21.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x675:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x676:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x678:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x677:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x675:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x676:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x678:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x677:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x677:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 20:46:30 (1780533990) Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre2 Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2579456 .. Extending file size to 7488544 .. failed Extending file size to 3781888 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8346944 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9145056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8622016 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8902272 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9092928 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9182304 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62538 62539 62541 62543 62545 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) 192.168.203.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg312-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:47:37 (1780534057) Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre2 Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8252352 .. Extending file size to 4275456 .. done Extending file size to 7082912 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7154592 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8907712 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7877760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8126656 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8624384 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8997120 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9306464 .. Extending file size to 9042912 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9123456 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 65160 65161 65163 65165 65166 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) 192.168.203.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg312-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:48:45 (1780534125) Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2956 1268348 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24792 3582228 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18940 3588080 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43732 7170308 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2956 1268348 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24792 3582228 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18940 3588080 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43732 7170308 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0446074 s, 47.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0796954 s, 26.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x305106fb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:48:52 (1780534132) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0540722 s, 38.8 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 20:48:55 (1780534135) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x425:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x68c:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:48:58 (1780534138) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0316125 s, 66.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:49:01 (1780534141) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0277027 s, 75.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x690:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x691:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x68d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x690:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x691:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x68d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x68e:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:49:04 (1780534144) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0336825 s, 62.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x692:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x68f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x692:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x68f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x690:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:49:28 (1780534168) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg312-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260206 s, 40.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x693:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x692:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x691:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x693:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x692:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x691:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x694:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:50:04 (1780534204) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg312-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x693:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x694:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0268125 s, 78.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x693:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x694:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x695:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x693:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x694:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x695:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:50:39 (1780534239) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg312-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg312-server: Pool lustre.test_204f created Adding targets to pool oleg312-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg312-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0317934 s, 66.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x695:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x696:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x695:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x696:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg312-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg312-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg312-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:51:29 (1780534289) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:51:32 (1780534292) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:51:35 (1780534295) oleg312-server: Pool lustre.test_206 created Adding targets to pool oleg312-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg312-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg312-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg312-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg312-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:51:52 (1780534312) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288743 s, 36.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129854 s, 80.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7b22cedf Mirror 3: 0x7b22cedf PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:51:55 (1780534315) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:51:57 (1780534317) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:51:58 (1780534318) Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0260671 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2070000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1e20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1e20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1e20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19c0000 Read this (4 bytes): BBBB 192.168.203.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg312-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:52:16 (1780534336) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00507561 s, 1.6 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:52:19 (1780534339) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172443 s, 60.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x448 lmm_fid: [0x200000405:0x448:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x448 lmm_fid: [0x200000405:0x448:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612232 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 20:52:19.000000000 -0400 Modify: 2026-06-03 20:52:19.000000000 -0400 Change: 2026-06-03 20:52:20.000000000 -0400 Birth: 2026-06-03 20:52:19.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:52:21 (1780534341) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167694 s, 62.5 MB/s oleg312-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg312-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg312-server: O/280000401/d0/6a0: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:52:33 (1780534353) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0729582 s, 561 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00763688 s, 536 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 3 20:52 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x44d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 3 20:52 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:52:36 (1780534356) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:52:38 (1780534358) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 984 sec ============ 20:52:42 (1780534362) === sanity-flr: start cleanup 20:52:42 (1780534362) === === sanity-flr: finish cleanup 20:52:45 (1780534365) ===