-----============= acceptance-small: sanity-flr ============----- Sun Sep 13 08:29:10 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg249-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:29:34 (1789302574) === oleg249-client.virtnet: executing check_config_client /mnt/lustre oleg249-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg249-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff903382cef000.idle_timeout=debug osc.lustre-OST0001-osc-ffff903382cef000.idle_timeout=debug disable quota as required oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:30:14 (1789302614) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8211] == sanity-flr test 0a: lfs mirror create with -N option == 08:30:16 (1789302616) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:30:30 (1789302630) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:30:33 (1789302633) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:30:36 (1789302636) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:31:02 (1789302662) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:31:06 (1789302666) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 08:31:09 (1789302669) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff903382cef000.stats=clear osc.lustre-OST0001-osc-ffff903382cef000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:31:17 (1789302677) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:31:27 (1789302687) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:31:35 (1789302695) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:31:38 (1789302698) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:31:48 (1789302708) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 08:31:59 (1789302719) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:32:06 (1789302726) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:32:14 (1789302734) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:32:22 (1789302742) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:32:32 (1789302752) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:32:38 (1789302758) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.614212 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622276 s, 16.9 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:32:44 (1789302764) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.344583 s, 39.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:32:52 (1789302772) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:33:02 (1789302782) Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg249-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:34:21 (1789302861) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.544 s, 13.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:35:00 (1789302900) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg249-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:36:15 (1789302975) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.80288 s, 18.7 MB/s -rw-r--r-- 1 root root 52428800 Sep 13 08:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.888s user 0m0.000s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff903382cef000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.974s user 0m0.001s sys 0m0.168s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.511s user 0m0.001s sys 0m0.175s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg249-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg249-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff903382cef000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:37:49 (1789303069) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:37:53 (1789303073) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:37:56 (1789303076) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:37:59 (1789303079) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:38:06 (1789303086) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.469618 s, 26.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:38:18 (1789303098) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223009 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517214 s, 20.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:38:26 (1789303106) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:38:34 (1789303114) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00207056 s, 495 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00230047 s, 445 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129565 s, 790 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00143872 s, 712 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 08:38:44 (1789303124) create test files with size 15758367 .. bf7ca9729a88397e1ddc422d485d4e96 - 35d4b4b6929af2b5eafd6b07c78b9211 - f92f0bd73bdc82e7cd85b66b28ced5a5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg249-client.virtnet /mnt/lustre (opts:) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 08:39:05 (1789303145) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:39:18 (1789303158) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:39:24 (1789303164) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800608 s, 13.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260164 s, 40.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 08:39:32 (1789303172) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126297 s, 16.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.345896 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152575 s, 13.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 08:39:50 (1789303190) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 08:39:54 (1789303194) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:39:58 (1789303198) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346318 s, 30.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:40:07 (1789303207) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.289512 s, 10.9 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:40:17 (1789303217) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:40:26 (1789303226) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.341878 s, 30.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg249-client.virtnet /mnt/lustre (opts:) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:40:38 (1789303238) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff903398a3d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.392252 s, 26.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg249-client.virtnet /mnt/lustre (opts:) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff90338544a000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:40:52 (1789303252) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:41:30 (1789303290) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:42:05 (1789303325) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 08:42:09 (1789303329) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.225955 s, 13.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.108061 s, 29.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x91845f1b (nosync,stale) Mirror 2: 0x91845f1b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9af617189b6be7490c2b3ce52fddafa2 - mirror 1 checksum: 7035905f897d47ecc316d7975b47f18e - mirror 2 checksum: 9af617189b6be7490c2b3ce52fddafa2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-13T08:42:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-13T08:42:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9af617189b6be7490c2b3ce52fddafa2 - mirror 1 checksum: 7035905f897d47ecc316d7975b47f18e - mirror 2 checksum: 7035905f897d47ecc316d7975b47f18e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-13T08:42:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-13T08:42:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:42:19 (1789303339) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394493 s, 26.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:42:26 (1789303346) Failing mds1 on oleg249-server Stopping /mnt/lustre-mds1 (opts:) on oleg249-server 08:42:39 (1789303359) shut down facet: mds1 facet_host: oleg249-server facet_failover_host: oleg249-server Failover mds1 to oleg249-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 08:43:13 (1789303393) targets are mounted 08:43:13 (1789303393) facet_failover done oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160725 s, 65.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:43:29 (1789303409) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0317153 s, 646 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.015547 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:43:39 (1789303419) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0631415 s, 324 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00974775 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:43:50 (1789303430) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0113534 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:43:57 (1789303437) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0388951 s, 527 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:44:04 (1789303444) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00553761 s, 740 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0230595 s, 178 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:44:11 (1789303451) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:44:34 (1789303474) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:45:04 (1789303504) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:45:28 (1789303528) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:45:53 (1789303553) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 08:46:03 (1789303563) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57560 mirror split pid 57561 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57560 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57561 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 08:47:12 (1789303632) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3634 skipping zero size read skipping zero size read 000003 1789303635.306051 write 0x03c17c thru 0x03ffff (0x3e84 bytes) 000004 1789303635.310289 mapwrite 0x02b08a thru 0x02b102 (0x79 bytes) 000005 1789303635.342985 mapwrite 0x02a1f6 thru 0x034fd2 (0xaddd bytes) 000006 1789303635.418266 write 0x027883 thru 0x036c34 (0xf3b2 bytes) zero_range to largest ever: 0x21052 000007 1789303635.426803 trunc from 0x019ab8 to 0x021051 (0x759a bytes) 000008 1789303635.433506 write 0x02151f thru 0x02e6c2 (0xd1a4 bytes) 000009 1789303635.443970 trunc from 0x040000 to 0x0023c5 (0x3dc3a bytes) 000010 1789303635.472075 mapwrite 0x01da4b thru 0x027303 (0x98b9 bytes) 000011 1789303635.551207 write 0x0189f1 thru 0x01d626 (0x4c36 bytes) 000014 1789303635.558108 mapread 0x00981e thru 0x0158a4 (0xc087 bytes) 000015 1789303635.582361 mapwrite 0x019398 thru 0x025ded (0xca56 bytes) 000016 1789303635.608230 mapwrite 0x024068 thru 0x02d9d9 (0x9972 bytes) 000017 1789303635.667252 trunc from 0x02d9da to 0x00fab9 (0x1df20 bytes) truncating to largest ever: 0x3e4db 000018 1789303635.683527 trunc from 0x00faba to 0x03e4da (0x2ea21 bytes) 000020 1789303635.694877 trunc from 0x01f2e0 to 0x0201d9 (0xefa bytes) 000021 1789303635.708721 MIRROR_EXTEND to 2 mirrors 000022 1789303636.629088 trunc from 0x03e4db to 0x03a4db (0x3fff bytes) 000023 1789303636.660308 read 0x033183 thru 0x03a4db (0x7359 bytes) 000024 1789303636.671501 mapwrite 0x032967 thru 0x03ca3f (0xa0d9 bytes) 000025 1789303636.740145 trunc from 0x00079f to 0x009d11 (0x9573 bytes) 000026 1789303636.745764 write 0x00a49a thru 0x015614 (0xb17b bytes) 000027 1789303636.777878 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ca40): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca40): Mirror 2: 0xc1853f82 (stale) CRC-32 checksum value for chunk [0, 0x3ca40): Mirror 1: 0xaa4323c7 Mirror 3: 0xaa4323c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca40: skipped 000028 1789303637.865763 mapwrite 0x03ca1d thru 0x03ffff (0x35e3 bytes) 000029 1789303637.932850 mapread 0x03e453 thru 0x03ffff (0x1bad bytes) 000030 1789303637.933592 write 0x021048 thru 0x024ec9 (0x3e82 bytes) zero_range to largest ever: 0x40000 000031 1789303637.944379 trunc from 0x039681 to 0x03ffff (0x697f bytes) 000032 1789303637.951025 punch from 0x00ceed to 0x00e569 (0x167d bytes) 000033 1789303637.956370 read 0x03b7a1 thru 0x03e0fd (0x295d bytes) 000034 1789303637.969028 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x5ab2aa4f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a5e599f Mirror 4: 0x8a5e599f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000035 1789303639.126681 fallocate 0x008851 thru 0x00dca0 (0x5450 bytes) 000036 1789303639.197752 write 0x026d96 thru 0x02a0c1 (0x332c bytes) 000038 1789303639.224495 mapread 0x0272c5 thru 0x03425e (0xcf9a bytes) 000039 1789303639.261667 fallocate 0x0056b2 thru 0x00efeb (0x993a bytes) 000040 1789303639.276107 trunc from 0x040000 to 0x035ba9 (0xa456 bytes) 000041 1789303639.312997 trunc from 0x0353c7 to 0x035ba9 (0x7e3 bytes) 000042 1789303639.327741 trunc from 0x023237 to 0x02b712 (0x84dc bytes) 000043 1789303639.363220 fallocate 0x034d04 thru 0x03ffff (0xb2fc bytes) 000044 1789303639.374146 write 0x011d1f thru 0x018927 (0x6c09 bytes) 000045 1789303639.404036 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x5ab2aa4f (stale) Mirror 4: 0x8a5e599f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad53e286 Mirror 5: 0xad53e286 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000047 1789303640.421668 write 0x01b383 thru 0x026403 (0xb081 bytes) 000048 1789303640.478780 trunc from 0x040000 to 0x0121fd (0x2de02 bytes) 000049 1789303640.502694 mapwrite 0x028c98 thru 0x036c36 (0xdf9f bytes) 000050 1789303640.573009 mapwrite 0x004ed2 thru 0x0072fc (0x242b bytes) 000051 1789303640.613309 fallocate 0x03826e thru 0x03ffff (0x7d92 bytes) 000052 1789303640.625111 trunc from 0x00654e to 0x011022 (0xaad5 bytes) 000054 1789303640.638691 mapread 0x02ee0a thru 0x03b16d (0xc364 bytes) 000055 1789303640.655742 read 0x002028 thru 0x00eefd (0xced6 bytes) 000056 1789303640.669063 trunc from 0x024bc0 to 0x02539f (0x7e0 bytes) 000057 1789303640.675619 trunc from 0x040000 to 0x03324f (0xcdb0 bytes) 000058 1789303640.698870 fallocate 0x00f790 thru 0x012b51 (0x33c2 bytes) 000059 1789303640.713212 fallocate 0x0075f8 thru 0x00aa7e (0x3487 bytes) 000060 1789303640.726579 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33250): Mirror 3: 0xa565130e (stale) Mirror 4: 0x78226e02 (stale) Mirror 5: 0x931be4aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33250): Mirror 3: 0xa565130e (stale) Mirror 4: 0x78226e02 (stale) Mirror 5: 0x931be4aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33250: skipped 000061 1789303641.200580 MIRROR_RESYNC 4 mirrors 000062 1789303642.301217 mapwrite 0x028c9f thru 0x033416 (0xa778 bytes) 000063 1789303642.387841 trunc from 0x013439 to 0x01bf11 (0x8ad9 bytes) 000064 1789303642.395339 fallocate 0x03e135 thru 0x03f863 (0x172f bytes) 000065 1789303642.400635 trunc from 0x033417 to 0x021ba6 (0x11870 bytes) 000066 1789303642.421630 trunc from 0x021ba7 to 0x0130e1 (0xeac5 bytes) 000068 1789303642.439723 read 0x002c17 thru 0x006991 (0x3d7b bytes) 000069 1789303642.448773 read 0x010f1e thru 0x0130e1 (0x21c4 bytes) 000070 1789303642.458720 mapread 0x006998 thru 0x0130e1 (0xc74a bytes) 000071 1789303642.460078 read 0x0084dd thru 0x00a4d1 (0x1ff5 bytes) 000072 1789303642.461266 mapwrite 0x039d47 thru 0x03ffff (0x62b9 bytes) 000073 1789303642.517064 write 0x0166d6 thru 0x01ec35 (0x8560 bytes) 000074 1789303642.520811 punch from 0x0084b6 to 0x00abfc (0x2747 bytes) 000075 1789303642.526332 mapread 0x0153d7 thru 0x01d15d (0x7d87 bytes) 000077 1789303642.528092 trunc from 0x040000 to 0x00a50a (0x35af5 bytes) 000078 1789303642.548229 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa50b): Mirror 3: 0x9bb419e9 (stale) Mirror 4: 0x9bb419e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa50b): Mirror 3: 0x9bb419e9 (stale) Mirror 4: 0x9bb419e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa50b: skipped 000079 1789303642.906113 punch from 0x005186 to 0x00a50a (0x5385 bytes) 000080 1789303642.913232 trunc from 0x00a50b to 0x02d4f1 (0x22fe7 bytes) 000081 1789303642.952147 punch from 0x024b23 to 0x02d4f1 (0x89cf bytes) 000082 1789303642.959400 punch from 0x00b8f0 to 0x019785 (0xde96 bytes) 000083 1789303642.981933 MIRROR_RESYNC 3 mirrors 000084 1789303643.913957 mapwrite 0x018130 thru 0x01b177 (0x3048 bytes) 000085 1789303644.032305 mapwrite 0x03d1bb thru 0x03d501 (0x347 bytes) 000087 1789303644.094960 write 0x035efe thru 0x03d67d (0x7780 bytes) 000088 1789303644.102975 mapread 0x01f346 thru 0x024964 (0x561f bytes) 000090 1789303644.121871 trunc from 0x03d67e to 0x023adc (0x19ba1 bytes) 000091 1789303644.150207 punch from 0x007711 to 0x013624 (0xbf14 bytes) 000092 1789303644.172634 trunc from 0x023add to 0x037fc6 (0x144ea bytes) 000093 1789303644.198887 mapread 0x024b8a thru 0x02ad26 (0x619d bytes) 000094 1789303644.212013 punch from 0x01714b to 0x01b75f (0x4615 bytes) 000095 1789303644.222923 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fc7): Mirror 4: 0x19fa80ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fc7): Mirror 4: 0x19fa80ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fc7: skipped 000096 1789303644.500023 fallocate 0x03c982 thru 0x03d70f (0xd8e bytes) 000098 1789303644.514484 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d710): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d710): Mirror 4: 0xfb91a5a (stale) CRC-32 checksum value for chunk [0, 0x3d710): Mirror 1: 0xfb91a5a Mirror 5: 0xfb91a5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d710: skipped 000099 1789303646.034638 punch from 0x013218 to 0x01bc4b (0x8a34 bytes) 000100 1789303646.098021 trunc from 0x028802 to 0x02e80f (0x600e bytes) 000101 1789303646.103315 punch from 0x03c1bb to 0x03d70f (0x1555 bytes) 000102 1789303646.132217 MIRROR_RESYNC 3 mirrors 000103 1789303647.089244 write 0x03d662 thru 0x03ffff (0x299e bytes) 000105 1789303647.119723 write 0x0056d0 thru 0x00e552 (0x8e83 bytes) 000106 1789303647.132687 trunc from 0x011ec1 to 0x01a4ca (0x860a bytes) 000107 1789303647.138051 mapread 0x03d82b thru 0x03ffff (0x27d5 bytes) 000108 1789303647.144486 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf37f48f6 (stale) Mirror 5: 0xf37f48f6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xedd72576 Mirror 6: 0xedd72576 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000110 1789303647.866879 write 0x0290e7 thru 0x0299d0 (0x8ea bytes) 000111 1789303647.913546 mapread 0x00b220 thru 0x00c163 (0xf44 bytes) 000112 1789303647.927848 MIRROR_RESYNC 4 mirrors 000113 1789303648.986743 fallocate 0x009549 thru 0x012f78 (0x9a30 bytes) 000114 1789303649.019673 trunc from 0x016bf8 to 0x01926d (0x2676 bytes) 000115 1789303649.026299 read 0x012792 thru 0x0139d8 (0x1247 bytes) 000116 1789303649.032485 fallocate 0x01d198 thru 0x029af8 (0xc961 bytes) 000117 1789303649.038514 trunc from 0x00a2d5 to 0x00da83 (0x37af bytes) 000118 1789303649.043647 read 0x017c4a thru 0x026c44 (0xeffb bytes) 000119 1789303649.064913 mapread 0x03472f thru 0x03badf (0x73b1 bytes) 000120 1789303649.075760 mapwrite 0x02f522 thru 0x036e50 (0x792f bytes) 000121 1789303649.096297 mapread 0x00c4e6 thru 0x01becc (0xf9e7 bytes) 000122 1789303649.098002 trunc from 0x040000 to 0x02b213 (0x14dec bytes) 000123 1789303649.118533 mapwrite 0x030f6e thru 0x0370ad (0x6140 bytes) 000124 1789303649.201258 fallocate 0x02524b thru 0x02830b (0x30c1 bytes) 000125 1789303649.206987 trunc from 0x0370ae to 0x011606 (0x25aa7 bytes) 000126 1789303649.223825 read 0x00f8a4 thru 0x011606 (0x1d63 bytes) 000127 1789303649.224499 trunc from 0x00d497 to 0x011606 (0x4170 bytes) 000128 1789303649.231174 trunc from 0x011607 to 0x017b7e (0x6578 bytes) 000129 1789303649.247577 write 0x036189 thru 0x03aea7 (0x4d1f bytes) 000130 1789303649.248835 fallocate 0x0040e3 thru 0x00a993 (0x68b1 bytes) 000131 1789303649.254400 fallocate 0x038f04 thru 0x039313 (0x410 bytes) 000132 1789303649.260906 punch from 0x01ff15 to 0x0200f2 (0x1de bytes) 000134 1789303649.271830 trunc from 0x0276d5 to 0x02f5db (0x7f07 bytes) 000135 1789303649.281931 MIRROR_RESYNC 4 mirrors 000136 1789303650.383191 trunc from 0x00b50f to 0x00f472 (0x3f64 bytes) 000137 1789303650.415182 mapread 0x02139b thru 0x02ee12 (0xda78 bytes) 000138 1789303650.455040 read 0x007d7f thru 0x00f218 (0x749a bytes) 000139 1789303650.464788 punch from 0x010ba2 to 0x014073 (0x34d2 bytes) 000140 1789303650.469805 trunc from 0x032d88 to 0x03aea7 (0x8120 bytes) 000141 1789303650.475291 read 0x003e4d thru 0x013d05 (0xfeb9 bytes) 000142 1789303650.500271 MIRROR_RESYNC 4 mirrors 000143 1789303651.354001 read 0x030238 thru 0x0368f5 (0x66be bytes) 000145 1789303651.365377 mapread 0x008bd0 thru 0x013da5 (0xb1d6 bytes) 000146 1789303651.389937 write 0x037953 thru 0x03ffff (0x86ad bytes) 000147 1789303651.447348 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x25a3b042 (stale) Mirror 5: 0x25a3b042 (stale) Mirror 6: 0x25a3b042 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2aa53ab4 Mirror 7: 0x2aa53ab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000148 1789303652.446907 trunc from 0x040000 to 0x00d635 (0x329ca bytes) 000149 1789303652.506342 punch from 0x00869d to 0x00d635 (0x4f99 bytes) 000150 1789303652.512232 mapwrite 0x01a373 thru 0x0275fd (0xd28b bytes) 000151 1789303652.578327 mapread 0x008599 thru 0x00d455 (0x4ebd bytes) 000152 1789303652.589587 punch from 0x016054 to 0x01f55f (0x950c bytes) 000153 1789303652.601473 read 0x0036ee thru 0x003c47 (0x55a bytes) 000154 1789303652.602583 write 0x00d388 thru 0x015bca (0x8843 bytes) 000155 1789303652.606727 fallocate 0x0336b4 thru 0x035e82 (0x27cf bytes) 000156 1789303652.611375 read 0x0065ab thru 0x011bbd (0xb613 bytes) 000157 1789303652.611896 mapwrite 0x021335 thru 0x025d59 (0x4a25 bytes) 000158 1789303652.637807 mapread 0x0236f4 thru 0x027d9e (0x46ab bytes) 000159 1789303652.638853 fallocate 0x006182 thru 0x007a0c (0x188b bytes) 000160 1789303652.645733 mapwrite 0x017413 thru 0x01e492 (0x7080 bytes) 000161 1789303652.716402 mapwrite 0x030d28 thru 0x03961b (0x88f4 bytes) 000163 1789303652.791953 trunc from 0x010077 to 0x01bbf4 (0xbb7e bytes) 000164 1789303652.803538 trunc from 0x03961c to 0x02b5d9 (0xe042 bytes) 000165 1789303652.826958 mapwrite 0x009966 thru 0x00fa89 (0x6124 bytes) 000166 1789303652.865332 punch from 0x01cf22 to 0x0235a0 (0x667f bytes) 000168 1789303652.896088 write 0x00890d thru 0x00b0c3 (0x27b7 bytes) 000169 1789303652.897159 trunc from 0x02b5da to 0x0288a1 (0x2d38 bytes) 000170 1789303652.922580 write 0x039511 thru 0x03af05 (0x19f5 bytes) 000171 1789303652.924595 trunc from 0x03af06 to 0x019708 (0x217fd bytes) 000172 1789303652.944830 write 0x03ba26 thru 0x03c965 (0xf40 bytes) 000173 1789303652.945948 trunc from 0x03c966 to 0x03887f (0x40e6 bytes) 000174 1789303652.961282 read 0x01c5c8 thru 0x02b983 (0xf3bc bytes) 000175 1789303652.973963 trunc from 0x038880 to 0x02b589 (0xd2f6 bytes) 000176 1789303653.005861 fallocate 0x018c9c thru 0x024de5 (0xc14a bytes) 000177 1789303653.019121 punch from 0x01b667 to 0x0258eb (0xa285 bytes) 000178 1789303653.034746 mapwrite 0x039d8f thru 0x03c019 (0x228b bytes) 000179 1789303653.096718 read 0x01f72c thru 0x02daf9 (0xe3ce bytes) 000180 1789303653.122163 punch from 0x02e2ae to 0x039821 (0xb574 bytes) 000181 1789303653.127990 trunc from 0x031482 to 0x03592a (0x44a9 bytes) 000182 1789303653.144665 trunc from 0x03c01a to 0x01f4f2 (0x1cb27 bytes) 000183 1789303653.170952 trunc from 0x014c0a to 0x018177 (0x356e bytes) 000184 1789303653.178100 fallocate 0x03651f thru 0x03b0b5 (0x4b97 bytes) 000186 1789303653.185682 fallocate 0x021bdc thru 0x024ef5 (0x331a bytes) 000187 1789303653.192565 punch from 0x026c94 to 0x02d708 (0x6a75 bytes) 000189 1789303653.199192 trunc from 0x01a22b to 0x020e23 (0x6bf9 bytes) 000190 1789303653.205577 fallocate 0x02c517 thru 0x037fdf (0xbac9 bytes) 000191 1789303653.211909 mapwrite 0x007736 thru 0x014e94 (0xd75f bytes) 000192 1789303653.245870 punch from 0x003318 to 0x006f73 (0x3c5c bytes) 000193 1789303653.252158 fallocate 0x01cf96 thru 0x01d540 (0x5ab bytes) 000194 1789303653.256630 read 0x037e69 thru 0x0387f5 (0x98d bytes) 000195 1789303653.266393 mapwrite 0x02b563 thru 0x039ce2 (0xe780 bytes) 000196 1789303653.289941 MIRROR_RESYNC 5 mirrors 000197 1789303654.854124 mapread 0x020014 thru 0x02b2e0 (0xb2cd bytes) 000199 1789303654.896217 mapwrite 0x01d017 thru 0x01e5e8 (0x15d2 bytes) 000200 1789303655.000250 punch from 0x030022 to 0x036aa8 (0x6a87 bytes) 000201 1789303655.008219 fallocate 0x01a7aa thru 0x027b5e (0xd3b5 bytes) 000202 1789303655.015637 fallocate 0x034755 thru 0x03ec2e (0xa4da bytes) 000203 1789303655.028950 read 0x033878 thru 0x033bee (0x377 bytes) 000204 1789303655.035905 trunc from 0x03b0b6 to 0x02028c (0x1ae29 bytes) 000205 1789303655.078078 trunc from 0x01badf to 0x02028c (0x47ae bytes) 000207 1789303655.087299 punch from 0x004434 to 0x009a2f (0x55fc bytes) 000208 1789303655.095081 mapwrite 0x008a0c thru 0x00cf15 (0x450a bytes) 000209 1789303655.179112 read 0x008dc4 thru 0x00a591 (0x17ce bytes) 000210 1789303655.179662 write 0x002889 thru 0x003a5b (0x11d3 bytes) 000211 1789303655.189810 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2028d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2028d): Mirror 4: 0x2243328a (stale) Mirror 5: 0x2243328a (stale) Mirror 6: 0x2243328a (stale) Mirror 7: 0x2243328a (stale) CRC-32 checksum value for chunk [0, 0x2028d): Mirror 1: 0x9c435d63 Mirror 8: 0x9c435d63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2028d: skipped 000212 1789303656.044478 write 0x00b4e4 thru 0x0176d5 (0xc1f2 bytes) 000213 1789303656.096552 write 0x033c68 thru 0x0395e1 (0x597a bytes) 000214 1789303656.097825 punch from 0x001c71 to 0x008c4e (0x6fde bytes) 000215 1789303656.104709 write 0x015f98 thru 0x025254 (0xf2bd bytes) 000216 1789303656.120763 MIRROR_RESYNC 6 mirrors 000217 1789303657.611216 mapread 0x01d39e thru 0x023e31 (0x6a94 bytes) 000218 1789303657.636407 MIRROR_RESYNC 6 mirrors 000219 1789303658.263346 write 0x00a213 thru 0x00dfd8 (0x3dc6 bytes) 000220 1789303658.343466 punch from 0x006a26 to 0x012284 (0xb85f bytes) 000222 1789303658.368360 punch from 0x0334f2 to 0x037a1d (0x452c bytes) 000224 1789303658.374461 trunc from 0x0395e2 to 0x027a60 (0x11b81 bytes) 000225 1789303658.396294 trunc from 0x01e02c to 0x02302d (0x5002 bytes) 000226 1789303658.404550 mapread 0x020f1d thru 0x027456 (0x653a bytes) 000227 1789303658.428638 fallocate 0x03a944 thru 0x03c9d4 (0x2091 bytes) 000228 1789303658.434259 mapwrite 0x0152f9 thru 0x01ec4e (0x9956 bytes) 000229 1789303658.467623 mapwrite 0x008952 thru 0x00de5a (0x5509 bytes) 000230 1789303658.503128 fallocate 0x008197 thru 0x010dfc (0x8c66 bytes) 000231 1789303658.509537 mapwrite 0x022a9f thru 0x02fcf1 (0xd253 bytes) 000232 1789303658.545306 mapread 0x0061ce thru 0x00e531 (0x8364 bytes) 000233 1789303658.546620 trunc from 0x006139 to 0x0086e7 (0x25af bytes) 000234 1789303658.552554 mapread 0x019f73 thru 0x0239c2 (0x9a50 bytes) 000235 1789303658.554291 mapread 0x006283 thru 0x00b80a (0x5588 bytes) 000236 1789303658.561888 mapread 0x02bea3 thru 0x0388ad (0xca0b bytes) 000237 1789303658.574535 mapread 0x00a0f8 thru 0x017fd1 (0xdeda bytes) 000238 1789303658.582782 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c9d5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9d5): Mirror 4: 0xffe4d813 (stale) Mirror 5: 0xffe4d813 (stale) Mirror 6: 0xffe4d813 (stale) Mirror 7: 0xffe4d813 (stale) Mirror 8: 0xffe4d813 (stale) CRC-32 checksum value for chunk [0, 0x3c9d5): Mirror 1: 0xffeee993 Mirror 9: 0xffeee993 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9d5: skipped 000239 1789303659.785062 mapread 0x00718b thru 0x012f22 (0xbd98 bytes) 000240 1789303659.835890 mapread 0x00fabe thru 0x010095 (0x5d8 bytes) 000241 1789303659.866369 punch from 0x006d75 to 0x011382 (0xa60e bytes) 000242 1789303659.925765 write 0x01d289 thru 0x01dd7f (0xaf7 bytes) 000243 1789303659.937970 write 0x0176be thru 0x01b7d5 (0x4118 bytes) 000244 1789303659.948062 write 0x02fec3 thru 0x03569b (0x57d9 bytes) 000245 1789303659.957407 read 0x01fdc7 thru 0x02838b (0x85c5 bytes) 000246 1789303659.968842 trunc from 0x03c9d5 to 0x004616 (0x383be bytes) 000247 1789303659.989922 punch from 0x0018b9 to 0x003333 (0x1a7b bytes) 000249 1789303660.000546 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x4617): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4617): Mirror 4: 0x156afebf (stale) Mirror 5: 0x156afebf (stale) Mirror 6: 0x156afebf (stale) Mirror 7: 0x156afebf (stale) Mirror 8: 0x156afebf (stale) Mirror 9: 0x156afebf (stale) CRC-32 checksum value for chunk [0, 0x4617): Mirror 1: 0x156afebf Mirror 10: 0x156afebf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4617: skipped 000250 1789303660.739122 trunc from 0x004617 to 0x03a20f (0x35bf9 bytes) 000251 1789303660.804418 read 0x024a66 thru 0x0278be (0x2e59 bytes) 000252 1789303660.811911 read 0x010f45 thru 0x012446 (0x1502 bytes) 000253 1789303660.842380 mapread 0x030fc2 thru 0x03a20f (0x924e bytes) 000254 1789303660.843504 trunc from 0x0119a8 to 0x01fbc2 (0xe21b bytes) 000255 1789303660.849765 read 0x016eb6 thru 0x01ab2c (0x3c77 bytes) 000257 1789303660.862854 MIRROR_RESYNC 8 mirrors 000258 1789303662.033538 trunc from 0x03a210 to 0x016ca4 (0x2356b bytes) 000259 1789303662.072035 mapwrite 0x0173b5 thru 0x021ed9 (0xab25 bytes) 000260 1789303662.142514 trunc from 0x0199bd to 0x021ed9 (0x851d bytes) 000261 1789303662.148349 trunc from 0x021eda to 0x00f7dd (0x126fc bytes) 000262 1789303662.172349 read 0x000af9 thru 0x004a43 (0x3f4b bytes) 000263 1789303662.179169 write 0x00ac07 thru 0x01645d (0xb857 bytes) 000264 1789303662.184741 punch from 0x003978 to 0x011de3 (0xe46c bytes) 000265 1789303662.204590 punch from 0x0080cf to 0x00c30a (0x423c bytes) 000266 1789303662.213685 MIRROR_RESYNC 8 mirrors 000267 1789303663.629861 write 0x028884 thru 0x0310a5 (0x8822 bytes) 000268 1789303663.661532 mapread 0x004cd4 thru 0x01403c (0xf369 bytes) 000269 1789303663.700489 read 0x028ad2 thru 0x0310a5 (0x85d4 bytes) 000270 1789303663.701345 fallocate 0x0140ab thru 0x019d29 (0x5c7f bytes) 000271 1789303663.708462 mapwrite 0x001171 thru 0x00ce8f (0xbd1f bytes) 000272 1789303663.758289 write 0x00f8cf thru 0x01d2f2 (0xda24 bytes) 000273 1789303663.770155 fallocate 0x026cb0 thru 0x02df99 (0x72ea bytes) 000274 1789303663.778799 trunc from 0x0310a6 to 0x00c499 (0x24c0c bytes) 000275 1789303663.809821 write 0x000775 thru 0x00d656 (0xcee2 bytes) 000276 1789303663.825614 punch from 0x008c7c to 0x00d656 (0x49db bytes) 000277 1789303663.849348 punch from 0x001cc8 to 0x00c3de (0xa717 bytes) 000278 1789303663.861291 punch from 0x003bb7 to 0x00d656 (0x9aa0 bytes) 000279 1789303663.870783 trunc from 0x00b7f9 to 0x00d656 (0x1e5e bytes) 000280 1789303663.879195 mapread 0x00c1ba thru 0x00c944 (0x78b bytes) 000281 1789303663.894986 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xd657): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd657): Mirror 4: 0xe4c63a15 (stale) Mirror 5: 0xe4c63a15 (stale) Mirror 6: 0xe4c63a15 (stale) Mirror 7: 0xe4c63a15 (stale) Mirror 8: 0xe4c63a15 (stale) Mirror 9: 0xe4c63a15 (stale) Mirror 10: 0xe4c63a15 (stale) CRC-32 checksum value for chunk [0, 0xd657): Mirror 1: 0xadf59d34 Mirror 11: 0xadf59d34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd657: skipped 000282 1789303665.158044 punch from 0x0092e7 to 0x00d656 (0x4370 bytes) 000283 1789303665.206764 read 0x001ac4 thru 0x00d656 (0xbb93 bytes) 000286 1789303665.220875 MIRROR_RESYNC 9 mirrors 000287 1789303666.785876 mapread 0x001345 thru 0x00ac57 (0x9913 bytes) 000288 1789303666.841313 punch from 0x007aa0 to 0x00997f (0x1ee0 bytes) 000291 1789303666.893409 trunc from 0x00d657 to 0x03cf93 (0x2f93d bytes) 000292 1789303666.944126 fallocate 0x036405 thru 0x038600 (0x21fc bytes) 000293 1789303666.966639 trunc from 0x03cf94 to 0x02ef85 (0xe00e bytes) 000294 1789303666.995240 mapread 0x0152a7 thru 0x023bdb (0xe935 bytes) 000295 1789303667.020149 trunc from 0x02ef86 to 0x03038a (0x1405 bytes) 000296 1789303667.046396 read 0x02fcc2 thru 0x03038a (0x6c9 bytes) 000298 1789303667.054013 trunc from 0x0100c7 to 0x01a4cc (0xa406 bytes) 000299 1789303667.059724 trunc from 0x03038b to 0x017ccd (0x186bd bytes) 000300 1789303667.076276 trunc from 0x016ea9 to 0x017ccd (0xe25 bytes) 000303 1789303667.083323 read 0x00748c thru 0x007519 (0x8e bytes) 000304 1789303667.094146 read 0x011ab5 thru 0x017ccd (0x6219 bytes) 000305 1789303667.095171 fallocate 0x03c117 thru 0x03d9f4 (0x18de bytes) 000306 1789303667.101326 write 0x01330f thru 0x01a2bb (0x6fad bytes) 000307 1789303667.104127 read 0x01500e thru 0x01a2bb (0x52ae bytes) 000308 1789303667.105277 mapread 0x00aab8 thru 0x017c2b (0xd174 bytes) 000309 1789303667.111530 MIRROR_RESYNC 9 mirrors 000310 1789303669.056045 trunc from 0x002034 to 0x00bac1 (0x9a8e bytes) 000311 1789303669.079026 fallocate 0x00556e thru 0x006c33 (0x16c6 bytes) 000312 1789303669.083249 trunc from 0x01a2bc to 0x031773 (0x174b8 bytes) 000314 1789303669.097704 trunc from 0x031774 to 0x00e7d2 (0x22fa1 bytes) 000315 1789303669.108728 write 0x02134a thru 0x027696 (0x634d bytes) 000316 1789303669.109852 mapread 0x0091c9 thru 0x00f906 (0x673e bytes) 000317 1789303669.122926 punch from 0x018c31 to 0x018d83 (0x153 bytes) 000318 1789303669.126035 fallocate 0x01a69d thru 0x01ead9 (0x443d bytes) 000319 1789303669.129407 fallocate 0x038d7d thru 0x03ffff (0x7283 bytes) 000320 1789303669.133013 mapwrite 0x01c335 thru 0x026160 (0x9e2c bytes) 000321 1789303669.151032 mapwrite 0x0263e8 thru 0x026807 (0x420 bytes) 000322 1789303669.159024 mapread 0x00ebac thru 0x011d93 (0x31e8 bytes) 000324 1789303669.160251 trunc from 0x027697 to 0x00eb72 (0x18b24 bytes) 000325 1789303669.171115 trunc from 0x00ceba to 0x00eb72 (0x1cb9 bytes) 000327 1789303669.175751 punch from 0x009242 to 0x00eb72 (0x5931 bytes) 000328 1789303669.179412 write 0x01da52 thru 0x02d241 (0xf7f0 bytes) 000329 1789303669.181277 mapread 0x01f7fb thru 0x028053 (0x8859 bytes) 000330 1789303669.182271 trunc from 0x00c30b to 0x018a7a (0xc770 bytes) 000331 1789303669.186598 read 0x00fa75 thru 0x019765 (0x9cf1 bytes) 000333 1789303669.194036 mapwrite 0x00253a thru 0x010417 (0xdede bytes) 000334 1789303669.215994 fallocate 0x02f4a4 thru 0x03b2d2 (0xbe2f bytes) 000336 1789303669.220341 mapwrite 0x03e199 thru 0x03ffff (0x1e67 bytes) 000337 1789303669.260237 trunc from 0x040000 to 0x016994 (0x2966b bytes) 000338 1789303669.280228 trunc from 0x016995 to 0x03e903 (0x27f6f bytes) 000339 1789303669.299251 trunc from 0x036b6e to 0x03dc89 (0x711c bytes) 000340 1789303669.305373 write 0x01c6b1 thru 0x01ff0a (0x385a bytes) 000341 1789303669.322526 fallocate 0x034e00 thru 0x035229 (0x42a bytes) 000343 1789303669.332163 mapread 0x0288b8 thru 0x03144d (0x8b96 bytes) 000345 1789303669.363372 read 0x027987 thru 0x02db7d (0x61f7 bytes) 000347 1789303669.365208 punch from 0x01c472 to 0x02a2f0 (0xde7f bytes) 000348 1789303669.383661 write 0x017cde thru 0x024537 (0xc85a bytes) 000350 1789303669.418007 trunc from 0x03e904 to 0x028e6a (0x15a99 bytes) 000351 1789303669.452484 write 0x02b218 thru 0x033f82 (0x8d6b bytes) 000352 1789303669.454138 write 0x028927 thru 0x02dc9c (0x5376 bytes) 000353 1789303669.457138 read 0x018bce thru 0x01a526 (0x1959 bytes) 000354 1789303669.458533 read 0x0052a2 thru 0x0071d9 (0x1f38 bytes) 000355 1789303669.464713 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f83): Mirror 5: 0x23a2ecbe (stale) Mirror 6: 0x23a2ecbe (stale) Mirror 7: 0x23a2ecbe (stale) Mirror 8: 0x23a2ecbe (stale) Mirror 9: 0x23a2ecbe (stale) Mirror 10: 0x23a2ecbe (stale) Mirror 11: 0x23a2ecbe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f83): Mirror 5: 0x23a2ecbe (stale) Mirror 6: 0x23a2ecbe (stale) Mirror 7: 0x23a2ecbe (stale) Mirror 8: 0x23a2ecbe (stale) Mirror 9: 0x23a2ecbe (stale) Mirror 10: 0x23a2ecbe (stale) Mirror 11: 0x23a2ecbe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f83: skipped 000358 1789303670.075930 read 0x02d644 thru 0x033f82 (0x693f bytes) 000359 1789303670.105348 mapwrite 0x0055ee thru 0x011003 (0xba16 bytes) 000360 1789303670.169670 read 0x009a23 thru 0x019113 (0xf6f1 bytes) 000361 1789303670.170346 read 0x033807 thru 0x033f82 (0x77c bytes) 000362 1789303670.171401 trunc from 0x033f83 to 0x02e7d8 (0x57aa bytes) 000363 1789303670.197702 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e7d9): Mirror 5: 0xc64fda07 (stale) Mirror 6: 0xc64fda07 (stale) Mirror 8: 0xc64fda07 (stale) Mirror 9: 0xc64fda07 (stale) Mirror 10: 0xc64fda07 (stale) Mirror 11: 0xc64fda07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7d9): Mirror 5: 0xc64fda07 (stale) Mirror 6: 0xc64fda07 (stale) Mirror 8: 0xc64fda07 (stale) Mirror 9: 0xc64fda07 (stale) Mirror 10: 0xc64fda07 (stale) Mirror 11: 0xc64fda07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7d9: skipped 000364 1789303670.601940 mapread 0x022f0b thru 0x024883 (0x1979 bytes) 000365 1789303670.624371 trunc from 0x004194 to 0x010a17 (0xc884 bytes) 000366 1789303670.633937 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e7d9): Mirror 5: 0xc64fda07 (stale) Mirror 8: 0xc64fda07 (stale) Mirror 9: 0xc64fda07 (stale) Mirror 10: 0xc64fda07 (stale) Mirror 11: 0xc64fda07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7d9): Mirror 5: 0xc64fda07 (stale) Mirror 8: 0xc64fda07 (stale) Mirror 9: 0xc64fda07 (stale) Mirror 10: 0xc64fda07 (stale) Mirror 11: 0xc64fda07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7d9: skipped 000367 1789303671.001307 fallocate 0x00e35c thru 0x0161ae (0x7e53 bytes) 000368 1789303671.020594 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e7d9): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7d9): Mirror 5: 0xc64fda07 (stale) Mirror 8: 0xc64fda07 (stale) Mirror 9: 0xc64fda07 (stale) Mirror 10: 0xc64fda07 (stale) Mirror 11: 0xc64fda07 (stale) CRC-32 checksum value for chunk [0, 0x2e7d9): Mirror 1: 0xd9aef296 Mirror 12: 0xd9aef296 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7d9: skipped 000370 1789303672.148161 trunc from 0x02e7d9 to 0x018dc4 (0x15a14 bytes) 000371 1789303672.215032 trunc from 0x005b6f to 0x0083e2 (0x2874 bytes) 000372 1789303672.220285 mapread 0x002936 thru 0x006d4a (0x4415 bytes) 000373 1789303672.240300 punch from 0x00ed1f to 0x011f5d (0x323f bytes) 000374 1789303672.245462 mapwrite 0x0358f1 thru 0x03d229 (0x7939 bytes) 000375 1789303672.306506 mapwrite 0x034cfc thru 0x039dbe (0x50c3 bytes) 000376 1789303672.326118 punch from 0x00dd51 to 0x019742 (0xb9f2 bytes) 000378 1789303672.334796 punch from 0x016eef to 0x0171f8 (0x30a bytes) 000380 1789303672.340039 read 0x012f73 thru 0x0207d9 (0xd867 bytes) 000381 1789303672.355850 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d22a): Mirror 5: 0x4b04a6fe (stale) Mirror 8: 0x4b04a6fe (stale) Mirror 10: 0x4b04a6fe (stale) Mirror 11: 0x4b04a6fe (stale) Mirror 12: 0x4eab4bee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d22a): Mirror 5: 0x4b04a6fe (stale) Mirror 8: 0x4b04a6fe (stale) Mirror 10: 0x4b04a6fe (stale) Mirror 11: 0x4b04a6fe (stale) Mirror 12: 0x4eab4bee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d22a: skipped 000382 1789303672.808868 trunc from 0x03d22a to 0x02d24e (0xffdb bytes) 000383 1789303672.827621 trunc from 0x023d56 to 0x02d24e (0x94f9 bytes) 000384 1789303672.832936 trunc from 0x02d24f to 0x025193 (0x80bb bytes) 000385 1789303672.848302 mapwrite 0x01ea5b thru 0x028166 (0x970c bytes) 000386 1789303672.912205 write 0x034e9c thru 0x038aeb (0x3c50 bytes) 000387 1789303672.913527 read 0x016c59 thru 0x01e0c7 (0x746f bytes) 000388 1789303672.928100 punch from 0x018e51 to 0x01ab83 (0x1d33 bytes) 000389 1789303672.933301 mapread 0x031d90 thru 0x038aeb (0x6d5c bytes) 000390 1789303672.944277 mapread 0x005f8f thru 0x00e4e8 (0x855a bytes) 000391 1789303672.962655 trunc from 0x011d23 to 0x020c5c (0xef3a bytes) 000392 1789303672.970085 trunc from 0x038aec to 0x0217ca (0x17321 bytes) 000394 1789303672.989691 fallocate 0x001a3a thru 0x00feaa (0xe471 bytes) 000395 1789303672.994778 punch from 0x011c4e to 0x0182e5 (0x6698 bytes) 000396 1789303673.000397 mapread 0x01d611 thru 0x0217ca (0x41ba bytes) 000397 1789303673.020359 write 0x00c23f thru 0x00f095 (0x2e57 bytes) 000398 1789303673.021983 write 0x03ff3b thru 0x03ffff (0xc5 bytes) 000399 1789303673.023263 trunc from 0x040000 to 0x01072b (0x2f8d4 bytes) 000401 1789303673.049977 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1072c): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1072c): Mirror 5: 0x44747b2b (stale) Mirror 8: 0x44747b2b (stale) Mirror 10: 0x44747b2b (stale) Mirror 11: 0x44747b2b (stale) Mirror 12: 0x91ba0f29 (stale) CRC-32 checksum value for chunk [0, 0x1072c): Mirror 1: 0x743c79c3 Mirror 13: 0x743c79c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1072c: skipped 000402 1789303673.975150 write 0x03f8e4 thru 0x03ffff (0x71c bytes) 000403 1789303674.056390 trunc from 0x034809 to 0x03cdf8 (0x85f0 bytes) 000404 1789303674.071942 read 0x030f53 thru 0x03805f (0x710d bytes) 000405 1789303674.086637 trunc from 0x004280 to 0x0066ef (0x2470 bytes) 000406 1789303674.097227 punch from 0x02ead0 to 0x039c65 (0xb196 bytes) 000407 1789303674.108210 trunc from 0x016afa to 0x01c79d (0x5ca4 bytes) 000408 1789303674.121168 fallocate 0x03fcdb thru 0x03ffff (0x325 bytes) 000409 1789303674.131174 trunc from 0x002bbf to 0x004812 (0x1c54 bytes) 000410 1789303674.146893 punch from 0x03ab32 to 0x03ffff (0x54ce bytes) 000411 1789303674.166811 punch from 0x028537 to 0x0306df (0x81a9 bytes) 000412 1789303674.178063 punch from 0x025c0c to 0x02b62b (0x5a20 bytes) 000413 1789303674.186256 mapread 0x001275 thru 0x005e29 (0x4bb5 bytes) 000414 1789303674.218126 trunc from 0x040000 to 0x03ff93 (0x6c bytes) 000415 1789303674.247111 write 0x02b87c thru 0x02de17 (0x259c bytes) 000416 1789303674.260202 trunc from 0x03ff94 to 0x01bf5c (0x24037 bytes) 000418 1789303674.294365 punch from 0x00b3b1 to 0x014ae1 (0x9731 bytes) 000419 1789303674.303913 fallocate 0x003793 thru 0x00c8b9 (0x9127 bytes) 000420 1789303674.314376 punch from 0x0131b6 to 0x01bf5c (0x8da7 bytes) 000422 1789303674.320843 read 0x0039be thru 0x0112d5 (0xd918 bytes) 000425 1789303674.346162 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1bf5d): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bf5d): Mirror 5: 0x416dbca0 (stale) Mirror 8: 0x416dbca0 (stale) Mirror 10: 0x416dbca0 (stale) Mirror 11: 0x416dbca0 (stale) Mirror 12: 0x194bb9e (stale) Mirror 13: 0xd2618a00 (stale) CRC-32 checksum value for chunk [0, 0x1bf5d): Mirror 1: 0x1e1c401b Mirror 14: 0x1e1c401b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bf5d: skipped 000426 1789303675.618220 fallocate 0x013c08 thru 0x016469 (0x2862 bytes) 000427 1789303675.714276 trunc from 0x01bf5d to 0x0086e8 (0x13874 bytes) 000428 1789303675.787087 punch from 0x005503 to 0x0086e8 (0x31e6 bytes) 000429 1789303675.800611 trunc from 0x0086e9 to 0x03d213 (0x34b2b bytes) 000430 1789303675.818254 fallocate 0x0326c4 thru 0x033ac3 (0x1400 bytes) 000431 1789303675.832068 mapwrite 0x0058d5 thru 0x0064da (0xc06 bytes) 000432 1789303675.883028 mapread 0x014d84 thru 0x01f34d (0xa5ca bytes) 000433 1789303675.918249 read 0x029e9c thru 0x02c571 (0x26d6 bytes) 000434 1789303675.925706 mapread 0x00ecb7 thru 0x01c019 (0xd363 bytes) 000435 1789303675.926877 write 0x00e550 thru 0x0186bf (0xa170 bytes) 000436 1789303675.929298 punch from 0x02069f to 0x0281ff (0x7b61 bytes) 000437 1789303675.944614 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d214): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d214): Mirror 5: 0xfa0571c7 (stale) Mirror 8: 0xfa0571c7 (stale) Mirror 10: 0xfa0571c7 (stale) Mirror 11: 0xfa0571c7 (stale) Mirror 12: 0x97d19c6d (stale) Mirror 13: 0xf462a90d (stale) Mirror 14: 0xf9b74e82 (stale) CRC-32 checksum value for chunk [0, 0x3d214): Mirror 1: 0x62d0311d Mirror 15: 0x62d0311d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d214: skipped 000438 1789303677.307207 MIRROR_RESYNC 9 mirrors 000439 1789303679.991009 mapread 0x020196 thru 0x02f5d8 (0xf443 bytes) 000440 1789303680.038030 trunc from 0x03d214 to 0x03fd1f (0x2b0c bytes) 000441 1789303680.101687 fallocate 0x02f76d thru 0x035aeb (0x637f bytes) 000443 1789303680.111983 trunc from 0x03fd20 to 0x00d9ab (0x32374 bytes) 000444 1789303680.141196 MIRROR_RESYNC 9 mirrors 000445 1789303682.330172 write 0x032e8f thru 0x034b93 (0x1d05 bytes) 000446 1789303682.376214 mapwrite 0x008361 thru 0x00bb10 (0x37b0 bytes) 000447 1789303682.424888 trunc from 0x028c77 to 0x02fd52 (0x70dc bytes) 000448 1789303682.430269 mapwrite 0x036388 thru 0x03777e (0x13f7 bytes) 000449 1789303682.475010 punch from 0x0245f0 to 0x02de61 (0x9872 bytes) 000450 1789303682.481182 trunc from 0x021522 to 0x027530 (0x600f bytes) 000451 1789303682.490710 trunc from 0x02d567 to 0x035f4d (0x89e7 bytes) 000452 1789303682.498194 trunc from 0x00d2d7 to 0x01b039 (0xdd63 bytes) 000453 1789303682.504926 read 0x02938a thru 0x031a26 (0x869d bytes) 000454 1789303682.518081 read 0x001d9c thru 0x008caa (0x6f0f bytes) 000455 1789303682.530050 trunc from 0x01e43e to 0x02573d (0x7300 bytes) 000456 1789303682.534936 trunc from 0x03777f to 0x01a070 (0x1d70e bytes) 000457 1789303682.553690 trunc from 0x013688 to 0x01a070 (0x69e9 bytes) 000458 1789303682.559676 read 0x015166 thru 0x01a070 (0x4f0b bytes) 000459 1789303682.569880 read 0x0187d4 thru 0x01a070 (0x189d bytes) 000460 1789303682.571203 trunc from 0x01a071 to 0x035a20 (0x1b9b0 bytes) 000461 1789303682.593894 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35a21): Mirror 5: 0xfbb71a91 (stale) Mirror 8: 0xfbb71a91 (stale) Mirror 10: 0xfbb71a91 (stale) Mirror 11: 0xfbb71a91 (stale) Mirror 12: 0xfbb71a91 (stale) Mirror 13: 0xfbb71a91 (stale) Mirror 14: 0xfbb71a91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a21): Mirror 5: 0xfbb71a91 (stale) Mirror 8: 0xfbb71a91 (stale) Mirror 10: 0xfbb71a91 (stale) Mirror 11: 0xfbb71a91 (stale) Mirror 12: 0xfbb71a91 (stale) Mirror 13: 0xfbb71a91 (stale) Mirror 14: 0xfbb71a91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a21: skipped 000462 1789303683.186892 fallocate 0x00f336 thru 0x01778c (0x8457 bytes) 000463 1789303683.198238 MIRROR_RESYNC 8 mirrors 000464 1789303685.614966 trunc from 0x035a21 to 0x0225b7 (0x13469 bytes) 000465 1789303685.682934 trunc from 0x014ece to 0x01effc (0xa12f bytes) 000466 1789303685.699973 trunc from 0x0225b8 to 0x03dfd1 (0x1ba1a bytes) 000467 1789303685.758137 trunc from 0x03dfd2 to 0x01236b (0x2bc66 bytes) 000468 1789303685.795383 mapread 0x0108bf thru 0x01236b (0x1aad bytes) 000469 1789303685.834035 punch from 0x00e477 to 0x01236b (0x3ef5 bytes) 000470 1789303685.841654 mapwrite 0x034c64 thru 0x0357f8 (0xb95 bytes) 000471 1789303685.923157 mapread 0x018c4f thru 0x024a9c (0xbe4e bytes) 000472 1789303685.967277 trunc from 0x0357f9 to 0x03f135 (0x993d bytes) 000476 1789303686.012579 write 0x01e0b4 thru 0x01f153 (0x10a0 bytes) 000477 1789303686.019618 mapwrite 0x011c69 thru 0x01f3b8 (0xd750 bytes) 000478 1789303686.072231 mapwrite 0x0269cc thru 0x032088 (0xb6bd bytes) 000479 1789303686.130038 write 0x023c47 thru 0x029601 (0x59bb bytes) 000480 1789303686.137337 read 0x03bb94 thru 0x03f135 (0x35a2 bytes) 000482 1789303686.167710 punch from 0x00a06b to 0x019080 (0xf016 bytes) 000483 1789303686.185653 mapwrite 0x01b487 thru 0x01c166 (0xce0 bytes) 000484 1789303686.226126 trunc from 0x03f136 to 0x00fc43 (0x2f4f2 bytes) 000485 1789303686.260053 mapwrite 0x02dc30 thru 0x035515 (0x78e6 bytes) 000486 1789303686.368611 read 0x00fdf5 thru 0x0175f5 (0x7801 bytes) 000488 1789303686.382130 trunc from 0x01e187 to 0x02c255 (0xe0cf bytes) 000489 1789303686.388007 trunc from 0x035516 to 0x037fc5 (0x2ab0 bytes) 000491 1789303686.407040 write 0x021f70 thru 0x02d146 (0xb1d7 bytes) 000492 1789303686.420089 trunc from 0x037fc6 to 0x00473a (0x3388b bytes) 000493 1789303686.437672 trunc from 0x0007c5 to 0x00459c (0x3dd8 bytes) 000494 1789303686.443732 trunc from 0x000115 to 0x00473a (0x4626 bytes) 000495 1789303686.453042 mapread 0x002558 thru 0x00473a (0x21e3 bytes) 000496 1789303686.477933 write 0x03e0a3 thru 0x03ffff (0x1f5d bytes) 000497 1789303686.479211 read 0x01f8f3 thru 0x024f34 (0x5642 bytes) 000498 1789303686.490798 read 0x00a3dd thru 0x00a883 (0x4a7 bytes) 000499 1789303686.491275 mapread 0x00e64c thru 0x0104c8 (0x1e7d bytes) 000501 1789303686.495789 write 0x037fbb thru 0x03c64b (0x4691 bytes) 000502 1789303686.496880 trunc from 0x040000 to 0x015bc1 (0x2a43e bytes) 000503 1789303686.543413 fallocate 0x02f759 thru 0x03d7cb (0xe073 bytes) 000504 1789303686.555219 mapread 0x011614 thru 0x015bc1 (0x45ae bytes) 000505 1789303686.555930 trunc from 0x015bc2 to 0x018473 (0x28b2 bytes) 000507 1789303686.577371 punch from 0x01058c to 0x01460a (0x407f bytes) 000508 1789303686.582989 punch from 0x016d7b to 0x018473 (0x16f9 bytes) 000509 1789303686.587324 write 0x020c6f thru 0x02b4aa (0xa83c bytes) 000510 1789303686.590016 mapwrite 0x038d81 thru 0x03f568 (0x67e8 bytes) 000511 1789303686.652029 fallocate 0x035a3a thru 0x03dc60 (0x8227 bytes) 000513 1789303686.661972 mapread 0x02eebb thru 0x03326a (0x43b0 bytes) 000515 1789303686.669906 mapwrite 0x03484e thru 0x034dce (0x581 bytes) 000516 1789303686.705504 punch from 0x0200c8 to 0x0239c5 (0x38fe bytes) 000517 1789303686.712258 fallocate 0x022f66 thru 0x025461 (0x24fc bytes) 000518 1789303686.722683 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f569): Mirror 5: 0x150e2c04 (stale) Mirror 10: 0x150e2c04 (stale) Mirror 11: 0x150e2c04 (stale) Mirror 12: 0x150e2c04 (stale) Mirror 13: 0x150e2c04 (stale) Mirror 14: 0x150e2c04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f569): Mirror 5: 0x150e2c04 (stale) Mirror 10: 0x150e2c04 (stale) Mirror 11: 0x150e2c04 (stale) Mirror 12: 0x150e2c04 (stale) Mirror 13: 0x150e2c04 (stale) Mirror 14: 0x150e2c04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f569: skipped 000520 1789303687.330936 trunc from 0x01ffc4 to 0x02fd95 (0xfdd2 bytes) 000521 1789303687.343755 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f569): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f569): Mirror 5: 0x150e2c04 (stale) Mirror 10: 0x150e2c04 (stale) Mirror 11: 0x150e2c04 (stale) Mirror 12: 0x150e2c04 (stale) Mirror 13: 0x150e2c04 (stale) Mirror 14: 0x150e2c04 (stale) CRC-32 checksum value for chunk [0, 0x3f569): Mirror 1: 0x8e414c0d Mirror 15: 0x8e414c0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f569: skipped 000522 1789303688.589655 mapwrite 0x00d4f7 thru 0x018e72 (0xb97c bytes) 000523 1789303688.798321 MIRROR_RESYNC 8 mirrors 000524 1789303691.709924 trunc from 0x03f569 to 0x037d6b (0x77fd bytes) 000526 1789303691.769263 trunc from 0x037d6c to 0x00ef2e (0x28e3d bytes) 000527 1789303691.792175 trunc from 0x00ef2f to 0x0381ce (0x292a0 bytes) 000528 1789303691.813494 mapwrite 0x00b372 thru 0x00f431 (0x40c0 bytes) 000530 1789303691.862033 mapwrite 0x01c13c thru 0x02b137 (0xeffc bytes) 000531 1789303691.921895 mapread 0x0235d3 thru 0x025003 (0x1a31 bytes) 000532 1789303691.922941 trunc from 0x0381cf to 0x00c4be (0x2bd10 bytes) 000533 1789303691.958047 write 0x021994 thru 0x0305cf (0xec3c bytes) 000535 1789303691.972231 fallocate 0x02a3a3 thru 0x030b23 (0x6781 bytes) 000536 1789303691.998032 mapwrite 0x03fad3 thru 0x03ffff (0x52d bytes) 000537 1789303692.084537 mapwrite 0x03ff8d thru 0x03ffff (0x73 bytes) 000538 1789303692.117044 mapread 0x039e40 thru 0x03ffff (0x61c0 bytes) 000539 1789303692.132791 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5c983be (stale) Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5c983be (stale) Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000541 1789303692.838137 write 0x00559d thru 0x00c9a9 (0x740d bytes) 000542 1789303692.878785 punch from 0x0143a2 to 0x01e139 (0x9d98 bytes) 000543 1789303692.887491 read 0x004104 thru 0x0081c0 (0x40bd bytes) 000544 1789303692.896415 trunc from 0x002b80 to 0x004388 (0x1809 bytes) 000545 1789303692.931658 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5c983be (stale) Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfa8b0cb2 Mirror 16: 0xfa8b0cb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000546 1789303694.301800 write 0x03c84f thru 0x03ffff (0x37b1 bytes) 000547 1789303694.393230 punch from 0x030ce2 to 0x038234 (0x7553 bytes) 000548 1789303694.399731 read 0x0298a7 thru 0x03873e (0xee98 bytes) 000549 1789303694.411942 mapread 0x00a022 thru 0x015a1f (0xb9fe bytes) 000550 1789303694.448812 punch from 0x02d1c3 to 0x039e04 (0xcc42 bytes) 000551 1789303694.455598 punch from 0x01dfe8 to 0x01fb53 (0x1b6c bytes) 000552 1789303694.461134 punch from 0x0202f1 to 0x02d0ef (0xcdff bytes) 000553 1789303694.472169 write 0x01146f thru 0x016c58 (0x57ea bytes) 000554 1789303694.473714 fallocate 0x006915 thru 0x010f0e (0xa5fa bytes) 000555 1789303694.479579 read 0x030c8b thru 0x039e92 (0x9208 bytes) 000556 1789303694.491005 write 0x0172de thru 0x01775b (0x47e bytes) 000557 1789303694.492177 fallocate 0x007e30 thru 0x014cbc (0xce8d bytes) 000558 1789303694.498150 write 0x0093b2 thru 0x00cc6e (0x38bd bytes) 000559 1789303694.499575 read 0x01fd94 thru 0x0285dc (0x8849 bytes) 000560 1789303694.500945 punch from 0x013a66 to 0x016c45 (0x31e0 bytes) 000561 1789303694.517494 mapwrite 0x0151e6 thru 0x01559f (0x3ba bytes) 000562 1789303694.536010 trunc from 0x01770b to 0x01c900 (0x51f6 bytes) 000563 1789303694.542291 mapread 0x008f7f thru 0x0112c1 (0x8343 bytes) 000564 1789303694.543973 read 0x001d4c thru 0x003555 (0x180a bytes) 000565 1789303694.546086 read 0x0200c6 thru 0x027a2d (0x7968 bytes) 000567 1789303694.547055 punch from 0x00ed6d to 0x0192e3 (0xa577 bytes) 000568 1789303694.551044 mapread 0x013761 thru 0x017ba3 (0x4443 bytes) 000569 1789303694.560199 write 0x02e3a7 thru 0x031c10 (0x386a bytes) 000570 1789303694.562020 fallocate 0x02bff9 thru 0x035244 (0x924c bytes) 000571 1789303694.566136 mapread 0x028302 thru 0x02dba7 (0x58a6 bytes) 000572 1789303694.567022 punch from 0x02985e to 0x031526 (0x7cc9 bytes) 000573 1789303694.576909 punch from 0x024f3e to 0x02ac45 (0x5d08 bytes) 000574 1789303694.582217 mapwrite 0x002179 thru 0x00c054 (0x9edc bytes) 000575 1789303694.599967 trunc from 0x016926 to 0x016bbe (0x299 bytes) 000576 1789303694.604945 fallocate 0x021f78 thru 0x02744e (0x54d7 bytes) 000577 1789303694.610196 read 0x00e878 thru 0x016d28 (0x84b1 bytes) 000578 1789303694.624594 read 0x00918b thru 0x00ddce (0x4c44 bytes) 000579 1789303694.625247 read 0x015bf4 thru 0x019618 (0x3a25 bytes) 000580 1789303694.630173 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5c983be (stale) Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88606b6c Mirror 17: 0x88606b6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000581 1789303695.996122 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5c983be (stale) Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88606b6c Mirror 17: 0x88606b6c Mirror 18: 0x88606b6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000583 1789303697.231215 trunc from 0x040000 to 0x00b851 (0x347ae bytes) 000584 1789303697.304516 trunc from 0x00380f to 0x00b851 (0x8043 bytes) 000585 1789303697.311266 trunc from 0x00882e to 0x00b851 (0x3024 bytes) 000586 1789303697.317758 trunc from 0x004d67 to 0x0067e1 (0x1a7b bytes) 000587 1789303697.325381 trunc from 0x00b852 to 0x0351e3 (0x29992 bytes) 000588 1789303697.355816 fallocate 0x00dd04 thru 0x0124cb (0x47c8 bytes) 000589 1789303697.371029 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x351e4): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351e4): Mirror 5: 0x7916252a (stale) Mirror 10: 0x7916252a (stale) Mirror 11: 0x7916252a (stale) Mirror 13: 0x7916252a (stale) Mirror 14: 0x7916252a (stale) Mirror 15: 0x7916252a (stale) Mirror 16: 0x63c62ec4 (stale) Mirror 17: 0xa4c44607 (stale) Mirror 18: 0xa4c44607 (stale) CRC-32 checksum value for chunk [0, 0x351e4): Mirror 1: 0x827fb46f Mirror 19: 0x827fb46f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351e4: skipped 000590 1789303698.871801 read 0x015a29 thru 0x019049 (0x3621 bytes) 000591 1789303698.896956 fallocate 0x004c54 thru 0x005c45 (0xff2 bytes) 000592 1789303698.936115 fallocate 0x03403e thru 0x03ca99 (0x8a5c bytes) 000593 1789303698.941339 read 0x020027 thru 0x022406 (0x23e0 bytes) 000594 1789303698.954808 trunc from 0x00b7bf to 0x00ddc5 (0x2607 bytes) 000595 1789303698.959390 mapwrite 0x010e45 thru 0x016a28 (0x5be4 bytes) 000597 1789303698.981080 trunc from 0x0351e4 to 0x02a9df (0xa804 bytes) 000598 1789303699.009155 trunc from 0x0255f6 to 0x02a9df (0x53ea bytes) 000600 1789303699.018126 trunc from 0x005b11 to 0x00a693 (0x4b83 bytes) 000601 1789303699.023196 trunc from 0x02a9e0 to 0x03a80c (0xfe2d bytes) 000602 1789303699.052408 write 0x03d95f thru 0x03ffff (0x26a1 bytes) 000603 1789303699.058701 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5c983be (stale) Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) Mirror 17: 0x88606b6c (stale) Mirror 18: 0x88606b6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5c983be (stale) Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) Mirror 17: 0x88606b6c (stale) Mirror 18: 0x88606b6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000604 1789303699.690883 mapread 0x00258c thru 0x006653 (0x40c8 bytes) 000605 1789303699.725150 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) Mirror 17: 0x88606b6c (stale) Mirror 18: 0x88606b6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 13: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) Mirror 17: 0x88606b6c (stale) Mirror 18: 0x88606b6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000606 1789303700.331084 write 0x007a63 thru 0x00f39b (0x7939 bytes) 000607 1789303700.358742 punch from 0x01b082 to 0x0238d3 (0x8852 bytes) 000608 1789303700.363857 mapwrite 0x03b0a6 thru 0x03ffff (0x4f5a bytes) 000609 1789303700.415023 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) Mirror 17: 0x88606b6c (stale) Mirror 18: 0x88606b6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5c983be (stale) Mirror 11: 0x5c983be (stale) Mirror 14: 0x5c983be (stale) Mirror 15: 0x5c983be (stale) Mirror 16: 0xfa8b0cb2 (stale) Mirror 17: 0x88606b6c (stale) Mirror 18: 0x88606b6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000610 1789303701.009500 trunc from 0x03304e to 0x03f8a6 (0xc859 bytes) 000611 1789303701.015837 write 0x01028a thru 0x013a31 (0x37a8 bytes) 000612 1789303701.033230 MIRROR_RESYNC 8 mirrors 000613 1789303703.450877 mapread 0x0267a1 thru 0x02dce2 (0x7542 bytes) 000614 1789303703.471079 fallocate 0x023b17 thru 0x026704 (0x2bee bytes) 000615 1789303703.511977 trunc from 0x040000 to 0x03af88 (0x5077 bytes) 000616 1789303703.535150 fallocate 0x003385 thru 0x00844e (0x50ca bytes) 000617 1789303703.541696 fallocate 0x008f99 thru 0x010f2a (0x7f92 bytes) 000619 1789303703.548124 trunc from 0x03af89 to 0x0090ae (0x31eda bytes) 000620 1789303703.564245 trunc from 0x008389 to 0x0090ae (0xd26 bytes) 000621 1789303703.569835 mapwrite 0x02be83 thru 0x039d4f (0xdecd bytes) 000622 1789303703.633396 mapwrite 0x020fda thru 0x02c93d (0xb964 bytes) 000623 1789303703.671605 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39d50): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d50): Mirror 10: 0xc91d36d6 (stale) Mirror 11: 0xc91d36d6 (stale) Mirror 14: 0xc91d36d6 (stale) Mirror 15: 0xc91d36d6 (stale) Mirror 16: 0xc91d36d6 (stale) Mirror 17: 0xc91d36d6 (stale) Mirror 18: 0xc91d36d6 (stale) CRC-32 checksum value for chunk [0, 0x39d50): Mirror 1: 0xebf97031 Mirror 19: 0xebf97031 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d50: skipped 000624 1789303704.689859 mapread 0x012610 thru 0x018df6 (0x67e7 bytes) 000626 1789303704.731866 write 0x030abc thru 0x03ca32 (0xbf77 bytes) 000627 1789303704.783998 read 0x033b4f thru 0x0386f7 (0x4ba9 bytes) 000628 1789303704.784889 fallocate 0x011ef5 thru 0x01f709 (0xd815 bytes) 000629 1789303704.791535 write 0x00bba3 thru 0x00de58 (0x22b6 bytes) 000630 1789303704.806758 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ca33): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca33): Mirror 10: 0x482af3fc (stale) Mirror 11: 0x482af3fc (stale) Mirror 14: 0x482af3fc (stale) Mirror 15: 0x482af3fc (stale) Mirror 16: 0x482af3fc (stale) Mirror 17: 0x482af3fc (stale) Mirror 18: 0x482af3fc (stale) Mirror 19: 0xfbefeb99 (stale) CRC-32 checksum value for chunk [0, 0x3ca33): Mirror 1: 0xbdbf2a77 Mirror 20: 0xbdbf2a77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca33: skipped 000631 1789303705.912788 read 0x011777 thru 0x014323 (0x2bad bytes) 000632 1789303705.931243 trunc from 0x03ca33 to 0x02533b (0x176f7 bytes) 000633 1789303706.001492 trunc from 0x02533c to 0x013f8a (0x113b1 bytes) 000634 1789303706.021593 read 0x00cc76 thru 0x013f8a (0x7315 bytes) 000635 1789303706.038086 read 0x00def7 thru 0x00e5aa (0x6b4 bytes) 000636 1789303706.038728 mapwrite 0x02ba71 thru 0x03ae12 (0xf3a2 bytes) 000637 1789303706.094048 write 0x02859a thru 0x028b00 (0x567 bytes) 000638 1789303706.095336 mapwrite 0x007123 thru 0x013da5 (0xcc83 bytes) 000640 1789303706.116870 fallocate 0x029dbe thru 0x03822a (0xe46d bytes) 000641 1789303706.127716 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae13): Mirror 11: 0x56fad0c4 (stale) Mirror 14: 0x56fad0c4 (stale) Mirror 15: 0x56fad0c4 (stale) Mirror 16: 0x56fad0c4 (stale) Mirror 17: 0x56fad0c4 (stale) Mirror 18: 0x56fad0c4 (stale) Mirror 19: 0x25a3d0a0 (stale) Mirror 20: 0xc4d9b114 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae13): Mirror 11: 0x56fad0c4 (stale) Mirror 14: 0x56fad0c4 (stale) Mirror 15: 0x56fad0c4 (stale) Mirror 16: 0x56fad0c4 (stale) Mirror 17: 0x56fad0c4 (stale) Mirror 18: 0x56fad0c4 (stale) Mirror 19: 0x25a3d0a0 (stale) Mirror 20: 0xc4d9b114 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae13: skipped 000642 1789303706.925118 punch from 0x039f38 to 0x03ae12 (0xedb bytes) 000643 1789303706.929934 mapwrite 0x02de48 thru 0x03d64e (0xf807 bytes) 000644 1789303707.010889 trunc from 0x03d64f to 0x00977e (0x33ed0 bytes) 000645 1789303707.037646 MIRROR_RESYNC 9 mirrors 000646 1789303709.307298 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000648 1789303710.013354 trunc from 0x00977f to 0x009894 (0x116 bytes) 000649 1789303710.084227 write 0x03b2c9 thru 0x03ffff (0x4d37 bytes) 000650 1789303710.085819 trunc from 0x040000 to 0x0161e1 (0x29e1e bytes) 000651 1789303710.113136 punch from 0x00b37d to 0x00b6f8 (0x37c bytes) 000652 1789303710.121330 write 0x020447 thru 0x02fc33 (0xf7ed bytes) 000653 1789303710.124424 write 0x03b93c thru 0x03ffff (0x46c4 bytes) 000654 1789303710.127252 mapread 0x03b3dd thru 0x03ffff (0x4c23 bytes) 000656 1789303710.129123 fallocate 0x0239ea thru 0x032844 (0xee5b bytes) 000657 1789303710.135693 fallocate 0x014f32 thru 0x019761 (0x4830 bytes) 000658 1789303710.140857 write 0x00f1f9 thru 0x0168f9 (0x7701 bytes) 000659 1789303710.152058 trunc from 0x034020 to 0x03a764 (0x6745 bytes) 000660 1789303710.157734 trunc from 0x040000 to 0x01b5c2 (0x24a3d bytes) 000662 1789303710.184375 mapread 0x005466 thru 0x0143fa (0xef95 bytes) 000663 1789303710.212367 mapread 0x017164 thru 0x01b5c2 (0x445f bytes) 000664 1789303710.221421 fallocate 0x0333c0 thru 0x0378f6 (0x4537 bytes) 000665 1789303710.227932 read 0x01404e thru 0x019995 (0x5948 bytes) 000666 1789303710.229131 fallocate 0x0247a9 thru 0x0314cb (0xcd23 bytes) 000667 1789303710.235959 write 0x003b29 thru 0x00471a (0xbf2 bytes) 000668 1789303710.237975 trunc from 0x01b737 to 0x024bcb (0x9495 bytes) 000669 1789303710.243741 write 0x00f5b1 thru 0x011e9b (0x28eb bytes) 000670 1789303710.244810 write 0x02381b thru 0x032754 (0xef3a bytes) 000671 1789303710.252129 mapwrite 0x02ccc9 thru 0x0328c4 (0x5bfc bytes) 000672 1789303710.312246 fallocate 0x028818 thru 0x02c60c (0x3df5 bytes) 000673 1789303710.318096 read 0x019a08 thru 0x028cb1 (0xf2aa bytes) 000674 1789303710.328002 punch from 0x0067a0 to 0x01441f (0xdc80 bytes) 000675 1789303710.333736 write 0x02a57f thru 0x039aff (0xf581 bytes) 000676 1789303710.339359 read 0x01defe thru 0x02544b (0x754e bytes) 000677 1789303710.341043 fallocate 0x02fda8 thru 0x039335 (0x958e bytes) 000678 1789303710.347291 trunc from 0x039b00 to 0x00ab30 (0x2efcf bytes) 000679 1789303710.368888 mapread 0x001f60 thru 0x00ab30 (0x8bd1 bytes) 000680 1789303710.380710 punch from 0x004015 to 0x00a020 (0x600c bytes) 000681 1789303710.387289 trunc from 0x00ab31 to 0x004c30 (0x5f00 bytes) 000682 1789303710.402929 read 0x0037df thru 0x004c30 (0x1452 bytes) 000683 1789303710.410007 punch from 0x001430 to 0x004c30 (0x3801 bytes) 000684 1789303710.415804 mapwrite 0x004b78 thru 0x00e817 (0x9ca0 bytes) 000685 1789303710.462759 write 0x016ca3 thru 0x018f76 (0x22d4 bytes) 000686 1789303710.464757 write 0x014460 thru 0x014a3f (0x5e0 bytes) 000687 1789303710.477690 trunc from 0x0062f2 to 0x00640e (0x11d bytes) 000688 1789303710.483296 mapwrite 0x017212 thru 0x0181b3 (0xfa2 bytes) 000689 1789303710.499669 mapwrite 0x02c604 thru 0x03c057 (0xfa54 bytes) 000690 1789303710.554917 read 0x038b18 thru 0x03bebc (0x33a5 bytes) 000691 1789303710.556640 write 0x01db1a thru 0x01f7af (0x1c96 bytes) 000693 1789303710.558821 mapwrite 0x02c1cd thru 0x033ded (0x7c21 bytes) 000694 1789303710.583063 trunc from 0x0389f3 to 0x03c057 (0x3665 bytes) 000695 1789303710.590072 fallocate 0x03e5bd thru 0x03ffff (0x1a43 bytes) 000696 1789303710.601891 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c058): Mirror 14: 0x10963912 (stale) Mirror 16: 0x10963912 (stale) Mirror 17: 0x10963912 (stale) Mirror 18: 0x10963912 (stale) Mirror 19: 0x10963912 (stale) Mirror 20: 0x10963912 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c058): Mirror 14: 0x10963912 (stale) Mirror 16: 0x10963912 (stale) Mirror 17: 0x10963912 (stale) Mirror 18: 0x10963912 (stale) Mirror 19: 0x10963912 (stale) Mirror 20: 0x10963912 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c058: skipped 000697 1789303711.083825 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c058): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c058): Mirror 14: 0x10963912 (stale) Mirror 16: 0x10963912 (stale) Mirror 17: 0x10963912 (stale) Mirror 18: 0x10963912 (stale) Mirror 19: 0x10963912 (stale) Mirror 20: 0x10963912 (stale) CRC-32 checksum value for chunk [0, 0x3c058): Mirror 1: 0xd01b26dc Mirror 21: 0xd01b26dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c058: skipped 000698 1789303712.710946 trunc from 0x02dcf7 to 0x02effc (0x1306 bytes) 000699 1789303712.761195 read 0x008303 thru 0x00d50a (0x5208 bytes) 000701 1789303712.772274 mapread 0x035e22 thru 0x03c057 (0x6236 bytes) 000702 1789303712.790353 trunc from 0x03c058 to 0x00385a (0x387fd bytes) 000703 1789303712.817700 trunc from 0x00001f to 0x000ffd (0xfdf bytes) 000704 1789303712.824891 trunc from 0x00385b to 0x032eec (0x2f692 bytes) 000705 1789303712.849368 punch from 0x00e5b0 to 0x01277d (0x41ce bytes) 000706 1789303712.855402 fallocate 0x01cba1 thru 0x02a9be (0xde1e bytes) 000707 1789303712.861113 write 0x00f5f3 thru 0x01c82e (0xd23c bytes) 000709 1789303712.872827 mapwrite 0x0110e9 thru 0x01f5e0 (0xe4f8 bytes) 000710 1789303712.900714 punch from 0x01fecc to 0x02c09b (0xc1d0 bytes) 000711 1789303712.906960 write 0x0066af thru 0x01021c (0x9b6e bytes) 000712 1789303712.913926 punch from 0x012e1f to 0x0142ad (0x148f bytes) 000713 1789303712.919332 trunc from 0x032eed to 0x00a018 (0x28ed4 bytes) 000714 1789303712.938265 punch from 0x0037f9 to 0x00a018 (0x6820 bytes) 000715 1789303712.958837 fallocate 0x033420 thru 0x035c20 (0x2801 bytes) 000717 1789303712.967864 trunc from 0x035c21 to 0x013531 (0x226ef bytes) 000718 1789303712.982006 write 0x021dfa thru 0x023c9c (0x1ea3 bytes) 000719 1789303712.983166 mapwrite 0x032fb6 thru 0x03e203 (0xb24e bytes) 000720 1789303713.052595 MIRROR_RESYNC 8 mirrors 000721 1789303715.338032 fallocate 0x004380 thru 0x0123a2 (0xe023 bytes) 000722 1789303715.406644 read 0x00be7f thru 0x015c3c (0x9dbe bytes) 000724 1789303715.417496 punch from 0x018798 to 0x0262ce (0xdb37 bytes) 000725 1789303715.427877 fallocate 0x007635 thru 0x00edea (0x77b6 bytes) 000726 1789303715.438942 trunc from 0x013e68 to 0x01f60c (0xb7a5 bytes) 000727 1789303715.445013 write 0x016aa6 thru 0x0175a0 (0xafb bytes) 000729 1789303715.464003 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e204): Mirror 14: 0xb0e5ac26 (stale) Mirror 16: 0xb0e5ac26 (stale) Mirror 17: 0xb0e5ac26 (stale) Mirror 18: 0xb0e5ac26 (stale) Mirror 19: 0xb0e5ac26 (stale) Mirror 20: 0xb0e5ac26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e204): Mirror 14: 0xb0e5ac26 (stale) Mirror 16: 0xb0e5ac26 (stale) Mirror 17: 0xb0e5ac26 (stale) Mirror 18: 0xb0e5ac26 (stale) Mirror 19: 0xb0e5ac26 (stale) Mirror 20: 0xb0e5ac26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e204: skipped 000730 1789303716.057817 mapwrite 0x00b57e thru 0x012722 (0x71a5 bytes) 000731 1789303716.098166 punch from 0x0070d2 to 0x0151d4 (0xe103 bytes) 000732 1789303716.104711 read 0x035a20 thru 0x037149 (0x172a bytes) 000733 1789303716.111386 write 0x0274f7 thru 0x0301e0 (0x8cea bytes) 000734 1789303716.127811 mapread 0x008194 thru 0x008c1f (0xa8c bytes) 000735 1789303716.137753 mapwrite 0x0287d4 thru 0x02b66f (0x2e9c bytes) 000736 1789303716.155977 mapread 0x00c5ce thru 0x0181d6 (0xbc09 bytes) 000737 1789303716.169948 trunc from 0x01218f to 0x017472 (0x52e4 bytes) 000738 1789303716.175698 mapread 0x017faf thru 0x019764 (0x17b6 bytes) 000739 1789303716.193261 MIRROR_RESYNC 7 mirrors 000740 1789303718.138056 punch from 0x029f1f to 0x0361ce (0xc2b0 bytes) 000742 1789303718.192312 mapwrite 0x02ef13 thru 0x036135 (0x7223 bytes) 000743 1789303718.229745 write 0x03c89f thru 0x03ffff (0x3761 bytes) 000744 1789303718.236817 fallocate 0x013a6c thru 0x01eae5 (0xb07a bytes) 000745 1789303718.243721 write 0x005bde thru 0x00b47f (0x58a2 bytes) 000746 1789303718.269985 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x4b8c65a1 (stale) Mirror 16: 0x4b8c65a1 (stale) Mirror 17: 0x4b8c65a1 (stale) Mirror 18: 0x4b8c65a1 (stale) Mirror 19: 0x4b8c65a1 (stale) Mirror 20: 0x4b8c65a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x880b74d3 Mirror 21: 0x880b74d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000747 1789303719.333190 trunc from 0x040000 to 0x03c245 (0x3dba bytes) 000748 1789303719.435223 read 0x0364da thru 0x03c245 (0x5d6c bytes) 000749 1789303719.446645 trunc from 0x023d38 to 0x026f3e (0x3207 bytes) 000750 1789303719.452244 read 0x019c44 thru 0x025560 (0xb91d bytes) 000751 1789303719.468982 punch from 0x037874 to 0x03c245 (0x49d2 bytes) 000752 1789303719.476339 trunc from 0x03c246 to 0x0399b3 (0x2892 bytes) 000753 1789303719.499716 mapwrite 0x029fb9 thru 0x039978 (0xf9c0 bytes) 000754 1789303719.541329 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x399b4): Mirror 16: 0x7af17ca8 (stale) Mirror 17: 0x7af17ca8 (stale) Mirror 18: 0x7af17ca8 (stale) Mirror 19: 0x7af17ca8 (stale) Mirror 20: 0x7af17ca8 (stale) Mirror 21: 0xeed9fc17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399b4): Mirror 16: 0x7af17ca8 (stale) Mirror 17: 0x7af17ca8 (stale) Mirror 18: 0x7af17ca8 (stale) Mirror 19: 0x7af17ca8 (stale) Mirror 20: 0x7af17ca8 (stale) Mirror 21: 0xeed9fc17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399b4: skipped 000755 1789303720.050000 write 0x03fda4 thru 0x03ffff (0x25c bytes) 000756 1789303720.056934 write 0x0369ae thru 0x03e88a (0x7edd bytes) 000757 1789303720.068673 mapread 0x029e62 thru 0x035fbf (0xc15e bytes) 000758 1789303720.097966 write 0x039cbf thru 0x03b3a7 (0x16e9 bytes) 000759 1789303720.099224 mapwrite 0x011d29 thru 0x018bfd (0x6ed5 bytes) 000760 1789303720.151322 fallocate 0x00c39d thru 0x016b36 (0xa79a bytes) 000761 1789303720.169520 mapwrite 0x00b2ac thru 0x00d482 (0x21d7 bytes) 000762 1789303720.208066 mapwrite 0x035927 thru 0x03a6f9 (0x4dd3 bytes) 000763 1789303720.250694 mapread 0x02370b thru 0x02afc3 (0x78b9 bytes) 000764 1789303720.263826 mapread 0x009775 thru 0x013ef1 (0xa77d bytes) 000765 1789303720.274142 punch from 0x01cbdb to 0x01f70d (0x2b33 bytes) 000767 1789303720.283833 mapread 0x005d1a thru 0x00f88c (0x9b73 bytes) 000768 1789303720.296550 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4b8c65a1 (stale) Mirror 18: 0x4b8c65a1 (stale) Mirror 19: 0x4b8c65a1 (stale) Mirror 20: 0x4b8c65a1 (stale) Mirror 21: 0x880b74d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4b8c65a1 (stale) Mirror 18: 0x4b8c65a1 (stale) Mirror 19: 0x4b8c65a1 (stale) Mirror 20: 0x4b8c65a1 (stale) Mirror 21: 0x880b74d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000769 1789303720.734167 mapwrite 0x02a77d thru 0x030db8 (0x663c bytes) 000770 1789303720.777483 trunc from 0x029f21 to 0x03038d (0x646d bytes) 000771 1789303720.783968 mapwrite 0x01dab7 thru 0x024754 (0x6c9e bytes) 000773 1789303720.821853 write 0x00eb9d thru 0x011c91 (0x30f5 bytes) 000774 1789303720.840956 mapread 0x017ba9 thru 0x0246c4 (0xcb1c bytes) 000775 1789303720.848835 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4b8c65a1 (stale) Mirror 18: 0x4b8c65a1 (stale) Mirror 19: 0x4b8c65a1 (stale) Mirror 20: 0x4b8c65a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4b8c65a1 (stale) Mirror 18: 0x4b8c65a1 (stale) Mirror 19: 0x4b8c65a1 (stale) Mirror 20: 0x4b8c65a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000776 1789303721.305753 read 0x01f0a8 thru 0x024103 (0x505c bytes) 000778 1789303721.316349 mapread 0x011b95 thru 0x01f9b5 (0xde21 bytes) 000779 1789303721.345996 read 0x008a73 thru 0x00e8ca (0x5e58 bytes) 000780 1789303721.365595 trunc from 0x026bcc to 0x031421 (0xa856 bytes) 000781 1789303721.372664 trunc from 0x0187da to 0x022ace (0xa2f5 bytes) 000782 1789303721.381174 trunc from 0x0293cc to 0x03049f (0x70d4 bytes) 000783 1789303721.394279 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4b8c65a1 (stale) Mirror 18: 0x4b8c65a1 (stale) Mirror 19: 0x4b8c65a1 (stale) Mirror 20: 0x4b8c65a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe47bea8 Mirror 21: 0xe47bea8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000784 1789303722.835939 mapread 0x005143 thru 0x0079d3 (0x2891 bytes) 000785 1789303722.885894 trunc from 0x00f5b3 to 0x017b14 (0x8562 bytes) 000786 1789303722.957766 MIRROR_RESYNC 6 mirrors 000787 1789303725.562523 write 0x010ab6 thru 0x018aa6 (0x7ff1 bytes) 000788 1789303725.614146 punch from 0x0141a0 to 0x01c8d3 (0x8734 bytes) 000789 1789303725.629074 read 0x023651 thru 0x02e851 (0xb201 bytes) 000790 1789303725.639029 mapwrite 0x0171d4 thru 0x024756 (0xd583 bytes) 000791 1789303725.683550 read 0x03a25c thru 0x03ffff (0x5da4 bytes) 000792 1789303725.694166 fallocate 0x006682 thru 0x00e797 (0x8116 bytes) 000794 1789303725.699569 write 0x01d8a0 thru 0x02b7c5 (0xdf26 bytes) 000795 1789303725.701110 punch from 0x03e77f to 0x03ffff (0x1881 bytes) 000796 1789303725.707293 read 0x036eb1 thru 0x03a826 (0x3976 bytes) 000797 1789303725.708536 trunc from 0x040000 to 0x00bfdf (0x34020 bytes) 000798 1789303725.731273 write 0x020f9b thru 0x027761 (0x67c7 bytes) 000799 1789303725.733716 punch from 0x025f70 to 0x027761 (0x17f2 bytes) 000801 1789303725.747404 trunc from 0x027762 to 0x0100c1 (0x176a0 bytes) 000802 1789303725.768409 trunc from 0x009411 to 0x0100c1 (0x6cb1 bytes) 000803 1789303725.774970 mapread 0x0003c3 thru 0x00819a (0x7dd8 bytes) 000804 1789303725.780737 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x100c2): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x100c2): Mirror 16: 0xb0688eaf (stale) Mirror 18: 0xb0688eaf (stale) Mirror 19: 0xb0688eaf (stale) Mirror 20: 0xb0688eaf (stale) Mirror 21: 0xb0688eaf (stale) CRC-32 checksum value for chunk [0, 0x100c2): Mirror 1: 0x1f6103ab Mirror 22: 0x1f6103ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x100c2: skipped 000805 1789303727.131922 mapwrite 0x03add9 thru 0x03ffff (0x5227 bytes) 000806 1789303727.256418 write 0x019526 thru 0x021feb (0x8ac6 bytes) 000807 1789303727.282931 mapread 0x03c98e thru 0x03ffff (0x3672 bytes) 000808 1789303727.284713 mapread 0x039bb4 thru 0x03ffff (0x644c bytes) 000809 1789303727.286788 trunc from 0x036881 to 0x03ffdd (0x975d bytes) 000810 1789303727.300988 fallocate 0x005b73 thru 0x01418f (0xe61d bytes) 000811 1789303727.306613 read 0x031d33 thru 0x03bd8c (0xa05a bytes) 000812 1789303727.320503 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xa5a88883 (stale) Mirror 18: 0xa5a88883 (stale) Mirror 19: 0xa5a88883 (stale) Mirror 20: 0xa5a88883 (stale) Mirror 21: 0xa5a88883 (stale) Mirror 22: 0x30b9d5f3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x99aff790 Mirror 23: 0x99aff790 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000813 1789303728.443534 fallocate 0x00e4e7 thru 0x01b93d (0xd457 bytes) 000814 1789303728.487636 read 0x035ad8 thru 0x03dedb (0x8404 bytes) 000815 1789303728.494258 mapwrite 0x02bf5a thru 0x03ac78 (0xed1f bytes) 000816 1789303728.518039 mapread 0x00233e thru 0x0034ee (0x11b1 bytes) 000817 1789303728.523912 read 0x0311d4 thru 0x0383d9 (0x7206 bytes) 000818 1789303728.524393 trunc from 0x040000 to 0x033180 (0xce7f bytes) 000819 1789303728.583259 punch from 0x013244 to 0x0149c9 (0x1786 bytes) 000820 1789303728.589841 punch from 0x02abe7 to 0x033180 (0x859a bytes) 000821 1789303728.606487 MIRROR_RESYNC 8 mirrors 000822 1789303730.657808 trunc from 0x02062f to 0x021d89 (0x175b bytes) 000823 1789303730.690314 write 0x01ba52 thru 0x01bdfa (0x3a9 bytes) 000824 1789303730.701416 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33181): Mirror 16: 0x6e2e075d (stale) Mirror 18: 0x6e2e075d (stale) Mirror 19: 0x6e2e075d (stale) Mirror 21: 0x6e2e075d (stale) Mirror 22: 0x6e2e075d (stale) Mirror 23: 0x6e2e075d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33181): Mirror 16: 0x6e2e075d (stale) Mirror 18: 0x6e2e075d (stale) Mirror 19: 0x6e2e075d (stale) Mirror 21: 0x6e2e075d (stale) Mirror 22: 0x6e2e075d (stale) Mirror 23: 0x6e2e075d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33181: skipped 000825 1789303731.112612 mapread 0x024111 thru 0x024842 (0x732 bytes) 000826 1789303731.122409 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33181): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33181): Mirror 16: 0x6e2e075d (stale) Mirror 18: 0x6e2e075d (stale) Mirror 19: 0x6e2e075d (stale) Mirror 21: 0x6e2e075d (stale) Mirror 22: 0x6e2e075d (stale) Mirror 23: 0x6e2e075d (stale) CRC-32 checksum value for chunk [0, 0x33181): Mirror 1: 0xeb8929e3 Mirror 24: 0xeb8929e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33181: skipped 000827 1789303732.168755 trunc from 0x033181 to 0x024642 (0xeb3e bytes) 000828 1789303732.226117 trunc from 0x0161b1 to 0x01b332 (0x5182 bytes) 000829 1789303732.231930 punch from 0x022e79 to 0x024642 (0x17ca bytes) 000830 1789303732.241157 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24643): Mirror 16: 0x109f9689 (stale) Mirror 19: 0x109f9689 (stale) Mirror 21: 0x109f9689 (stale) Mirror 22: 0x109f9689 (stale) Mirror 23: 0x109f9689 (stale) Mirror 24: 0x66233c55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24643): Mirror 16: 0x109f9689 (stale) Mirror 19: 0x109f9689 (stale) Mirror 21: 0x109f9689 (stale) Mirror 22: 0x109f9689 (stale) Mirror 23: 0x109f9689 (stale) Mirror 24: 0x66233c55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24643: skipped 000831 1789303732.751180 punch from 0x021efa to 0x023861 (0x1968 bytes) 000832 1789303732.757792 trunc from 0x00bc85 to 0x014f4a (0x92c6 bytes) 000833 1789303732.763838 read 0x0062fc thru 0x0132d3 (0xcfd8 bytes) 000834 1789303732.778731 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24643): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24643): Mirror 16: 0x109f9689 (stale) Mirror 19: 0x109f9689 (stale) Mirror 21: 0x109f9689 (stale) Mirror 22: 0x109f9689 (stale) Mirror 23: 0x109f9689 (stale) Mirror 24: 0x66233c55 (stale) CRC-32 checksum value for chunk [0, 0x24643): Mirror 1: 0x5f9b3ca1 Mirror 25: 0x5f9b3ca1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24643: skipped 000835 1789303733.878890 trunc from 0x024643 to 0x008da3 (0x1b89f bytes) 000836 1789303733.979321 mapread 0x004f28 thru 0x008da3 (0x3e7c bytes) 000837 1789303734.030556 write 0x034f95 thru 0x0369cf (0x1a3b bytes) 000838 1789303734.049418 MIRROR_RESYNC 8 mirrors 000840 1789303735.674639 fallocate 0x0007ae thru 0x00bcbd (0xb510 bytes) 000841 1789303735.715772 write 0x03ff8e thru 0x03ffff (0x72 bytes) 000842 1789303735.716916 trunc from 0x00e0b3 to 0x00e8b6 (0x804 bytes) 000843 1789303735.724160 mapread 0x01cb7e thru 0x023839 (0x6cbc bytes) 000844 1789303735.752156 MIRROR_RESYNC 8 mirrors 000845 1789303738.397214 trunc from 0x01ec83 to 0x02775c (0x8ada bytes) 000846 1789303738.429886 read 0x0213bb thru 0x0268a2 (0x54e8 bytes) 000847 1789303738.442634 mapwrite 0x03b372 thru 0x03ffff (0x4c8e bytes) 000848 1789303738.492288 mapwrite 0x0229c6 thru 0x02b646 (0x8c81 bytes) 000849 1789303738.525735 MIRROR_RESYNC 8 mirrors 000851 1789303740.685719 punch from 0x0378d8 to 0x03ace1 (0x340a bytes) 000852 1789303740.722324 trunc from 0x040000 to 0x032efe (0xd101 bytes) 000853 1789303740.742300 trunc from 0x0312a0 to 0x031f50 (0xcb1 bytes) 000854 1789303740.748160 trunc from 0x032eff to 0x00cb79 (0x26385 bytes) 000855 1789303740.768081 trunc from 0x007294 to 0x00cb79 (0x58e6 bytes) 000856 1789303740.772865 punch from 0x0098bd to 0x00cb79 (0x32bd bytes) 000857 1789303740.777524 mapread 0x0062b7 thru 0x0068d3 (0x61d bytes) 000858 1789303740.795356 mapwrite 0x01201e thru 0x01c276 (0xa259 bytes) 000859 1789303740.908804 trunc from 0x00ecf7 to 0x01bb15 (0xce1f bytes) 000860 1789303740.915327 write 0x02b66c thru 0x035681 (0xa016 bytes) 000861 1789303740.921597 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35682): Mirror 16: 0x21992e80 (stale) Mirror 19: 0x21992e80 (stale) Mirror 21: 0x21992e80 (stale) Mirror 22: 0x21992e80 (stale) Mirror 23: 0x21992e80 (stale) Mirror 24: 0x21992e80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35682): Mirror 16: 0x21992e80 (stale) Mirror 19: 0x21992e80 (stale) Mirror 21: 0x21992e80 (stale) Mirror 22: 0x21992e80 (stale) Mirror 23: 0x21992e80 (stale) Mirror 24: 0x21992e80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35682: skipped 000862 1789303741.339903 MIRROR_RESYNC 7 mirrors 000863 1789303742.895625 mapread 0x02529a thru 0x0315ff (0xc366 bytes) 000864 1789303742.922274 trunc from 0x035682 to 0x018a60 (0x1cc21 bytes) 000865 1789303742.957949 trunc from 0x018a61 to 0x00777a (0x112e6 bytes) 000866 1789303742.974044 read 0x004431 thru 0x006348 (0x1f18 bytes) 000867 1789303742.980153 mapread 0x003bb0 thru 0x00777a (0x3bcb bytes) 000868 1789303742.987726 mapread 0x00618e thru 0x00777a (0x15ed bytes) 000869 1789303742.988916 trunc from 0x003e3c to 0x00777a (0x393f bytes) 000870 1789303742.993477 read 0x00381f thru 0x00705c (0x383e bytes) 000871 1789303743.001784 punch from 0x005320 to 0x00777a (0x245b bytes) 000872 1789303743.005893 punch from 0x000640 to 0x005589 (0x4f4a bytes) 000873 1789303743.009925 punch from 0x002329 to 0x003a1d (0x16f5 bytes) 000874 1789303743.013726 mapread 0x005099 thru 0x00777a (0x26e2 bytes) 000875 1789303743.025330 trunc from 0x00777b to 0x029d9a (0x22620 bytes) 000876 1789303743.047340 trunc from 0x023d67 to 0x025ab2 (0x1d4c bytes) 000877 1789303743.055734 fallocate 0x035d34 thru 0x03e0b6 (0x8383 bytes) 000878 1789303743.061115 write 0x034587 thru 0x03ffff (0xba79 bytes) 000879 1789303743.062735 trunc from 0x040000 to 0x0310c7 (0xef38 bytes) 000880 1789303743.081487 MIRROR_RESYNC 7 mirrors 000881 1789303744.279684 MIRROR_RESYNC 7 mirrors 000882 1789303744.782292 read 0x015148 thru 0x020e6b (0xbd24 bytes) 000883 1789303744.794296 trunc from 0x02b591 to 0x0310c7 (0x5b37 bytes) 000884 1789303744.829354 read 0x013f76 thru 0x01f06f (0xb0fa bytes) 000887 1789303744.847803 mapread 0x01bc07 thru 0x029be2 (0xdfdc bytes) 000888 1789303744.852593 MIRROR_RESYNC 7 mirrors 000889 1789303746.005611 mapread 0x018d35 thru 0x01bb4a (0x2e16 bytes) 000890 1789303746.023816 mapread 0x01ed7c thru 0x02aa62 (0xbce7 bytes) 000891 1789303746.032654 trunc from 0x02fd76 to 0x0310c7 (0x1352 bytes) 000892 1789303746.063346 write 0x01f356 thru 0x029609 (0xa2b4 bytes) 000893 1789303746.072677 mapwrite 0x022d73 thru 0x029efc (0x718a bytes) 000894 1789303746.093022 mapread 0x00b429 thru 0x00ff7b (0x4b53 bytes) 000895 1789303746.103110 punch from 0x002c10 to 0x00ef08 (0xc2f9 bytes) 000896 1789303746.109580 trunc from 0x012afa to 0x01a42f (0x7936 bytes) 000897 1789303746.113852 mapread 0x007ce1 thru 0x01091a (0x8c3a bytes) 000899 1789303746.137971 fallocate 0x0246aa thru 0x027976 (0x32cd bytes) 000900 1789303746.155276 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x310c8): Mirror 16: 0x6d8360a6 (stale) Mirror 19: 0x6d8360a6 (stale) Mirror 21: 0x6d8360a6 (stale) Mirror 22: 0x6d8360a6 (stale) Mirror 24: 0x6d8360a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x310c8): Mirror 16: 0x6d8360a6 (stale) Mirror 19: 0x6d8360a6 (stale) Mirror 21: 0x6d8360a6 (stale) Mirror 22: 0x6d8360a6 (stale) Mirror 24: 0x6d8360a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x310c8: skipped 000902 1789303746.626961 mapwrite 0x031248 thru 0x032002 (0xdbb bytes) 000903 1789303746.692595 write 0x0127b7 thru 0x014535 (0x1d7f bytes) 000904 1789303746.702973 mapread 0x013197 thru 0x01fb5a (0xc9c4 bytes) 000905 1789303746.724372 mapread 0x02989c thru 0x02d8de (0x4043 bytes) 000906 1789303746.737472 read 0x00f5af thru 0x01e684 (0xf0d6 bytes) 000907 1789303746.738082 punch from 0x024820 to 0x032002 (0xd7e3 bytes) 000908 1789303746.754658 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32003): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32003): Mirror 16: 0x1f2a8daf (stale) Mirror 19: 0x1f2a8daf (stale) Mirror 21: 0x1f2a8daf (stale) Mirror 22: 0x1f2a8daf (stale) Mirror 24: 0x1f2a8daf (stale) CRC-32 checksum value for chunk [0, 0x32003): Mirror 1: 0xcd2bd725 Mirror 25: 0xcd2bd725 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32003: skipped 000909 1789303747.713972 trunc from 0x028eae to 0x032002 (0x9155 bytes) 000910 1789303747.755671 read 0x0200f5 thru 0x02b0f1 (0xaffd bytes) 000911 1789303747.765603 trunc from 0x032003 to 0x01b93f (0x166c3 bytes) 000912 1789303747.797774 write 0x018b0b thru 0x01df82 (0x5478 bytes) 000913 1789303747.808205 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1df83): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1df83): Mirror 16: 0x1182f711 (stale) Mirror 19: 0x1182f711 (stale) Mirror 21: 0x1182f711 (stale) Mirror 22: 0x1182f711 (stale) Mirror 24: 0x1182f711 (stale) Mirror 25: 0x2449c190 (stale) CRC-32 checksum value for chunk [0, 0x1df83): Mirror 1: 0x82a9ae33 Mirror 26: 0x82a9ae33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1df83: skipped 000915 1789303748.647017 fallocate 0x01b9e9 thru 0x01e578 (0x2b90 bytes) 000916 1789303748.694930 fallocate 0x024bda thru 0x02738f (0x27b6 bytes) 000917 1789303748.702635 mapwrite 0x0052d5 thru 0x00a263 (0x4f8f bytes) 000918 1789303748.740977 mapwrite 0x014b40 thru 0x021639 (0xcafa bytes) 000919 1789303748.800295 trunc from 0x02163a to 0x016581 (0xb0b8 bytes) 000920 1789303748.822023 mapread 0x001191 thru 0x00c1fb (0xb06b bytes) 000921 1789303748.827900 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16582): Mirror 16: 0x8c852be8 (stale) Mirror 19: 0x8c852be8 (stale) Mirror 21: 0x8c852be8 (stale) Mirror 24: 0x8c852be8 (stale) Mirror 25: 0x89285b72 (stale) Mirror 26: 0x89285b72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16582): Mirror 16: 0x8c852be8 (stale) Mirror 19: 0x8c852be8 (stale) Mirror 21: 0x8c852be8 (stale) Mirror 24: 0x8c852be8 (stale) Mirror 25: 0x89285b72 (stale) Mirror 26: 0x89285b72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16582: skipped 000922 1789303749.294041 punch from 0x00d3c7 to 0x0138b7 (0x64f1 bytes) 000924 1789303749.300030 mapwrite 0x00bae1 thru 0x0148d7 (0x8df7 bytes) 000925 1789303749.358696 mapread 0x004b46 thru 0x013273 (0xe72e bytes) 000926 1789303749.392150 mapwrite 0x01834d thru 0x01ed10 (0x69c4 bytes) 000927 1789303749.465207 trunc from 0x01ed11 to 0x00e4aa (0x10866 bytes) 000928 1789303749.491958 fallocate 0x002865 thru 0x004188 (0x1924 bytes) 000929 1789303749.498046 punch from 0x0004d2 to 0x00b700 (0xb22f bytes) 000930 1789303749.505144 punch from 0x001571 to 0x00d95e (0xc3ee bytes) 000932 1789303749.511498 fallocate 0x02c49b thru 0x030685 (0x41eb bytes) 000935 1789303749.518202 punch from 0x0044cf to 0x00e4aa (0x9fdc bytes) 000936 1789303749.525058 punch from 0x0094f9 to 0x00e4aa (0x4fb2 bytes) 000937 1789303749.531670 trunc from 0x00e4ab to 0x03eaa3 (0x305f9 bytes) 000940 1789303749.554366 mapwrite 0x00e8f3 thru 0x01a01b (0xb729 bytes) 000941 1789303749.599881 punch from 0x000580 to 0x004b5a (0x45db bytes) 000942 1789303749.612043 MIRROR_RESYNC 7 mirrors 000944 1789303751.075276 fallocate 0x005c0f thru 0x013a5e (0xde50 bytes) 000945 1789303751.124904 trunc from 0x035b48 to 0x03bfcd (0x6486 bytes) 000946 1789303751.132605 read 0x01c667 thru 0x025ac5 (0x945f bytes) 000947 1789303751.144899 mapread 0x03419a thru 0x03eaa3 (0xa90a bytes) 000948 1789303751.168886 fallocate 0x03bd91 thru 0x03f531 (0x37a1 bytes) 000949 1789303751.181257 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3eaa4): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eaa4): Mirror 16: 0x7c828fbf (stale) Mirror 19: 0x7c828fbf (stale) Mirror 21: 0x7c828fbf (stale) Mirror 24: 0x7c828fbf (stale) Mirror 25: 0x7c828fbf (stale) Mirror 26: 0x7c828fbf (stale) CRC-32 checksum value for chunk [0, 0x3eaa4): Mirror 1: 0x7c828fbf Mirror 27: 0x7c828fbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eaa4: skipped 000950 1789303752.459916 punch from 0x00f556 to 0x0173fd (0x7ea8 bytes) 000951 1789303752.493262 mapread 0x015862 thru 0x021527 (0xbcc6 bytes) 000952 1789303752.522473 trunc from 0x000e69 to 0x002b01 (0x1c99 bytes) 000953 1789303752.532579 MIRROR_RESYNC 8 mirrors 000954 1789303754.978209 mapread 0x01faec thru 0x021392 (0x18a7 bytes) 000955 1789303755.008230 write 0x02fb3b thru 0x03b91e (0xbde4 bytes) 000956 1789303755.049870 mapread 0x03dfd2 thru 0x03eaa3 (0xad2 bytes) 000957 1789303755.051270 read 0x010886 thru 0x01ee2c (0xe5a7 bytes) 000958 1789303755.058739 write 0x022dff thru 0x02e48f (0xb691 bytes) 000959 1789303755.066283 mapwrite 0x02f1d9 thru 0x033a67 (0x488f bytes) 000961 1789303755.079544 read 0x005919 thru 0x013035 (0xd71d bytes) 000962 1789303755.090685 mapwrite 0x016dab thru 0x020c53 (0x9ea9 bytes) 000963 1789303755.107379 punch from 0x029233 to 0x0377ef (0xe5bd bytes) 000964 1789303755.112973 mapread 0x037a4d thru 0x03eaa3 (0x7057 bytes) 000965 1789303755.121142 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eaa4): Mirror 16: 0xe05749b8 (stale) Mirror 19: 0xe05749b8 (stale) Mirror 24: 0xe05749b8 (stale) Mirror 25: 0xe05749b8 (stale) Mirror 26: 0xe05749b8 (stale) Mirror 27: 0xe05749b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eaa4): Mirror 16: 0xe05749b8 (stale) Mirror 19: 0xe05749b8 (stale) Mirror 24: 0xe05749b8 (stale) Mirror 25: 0xe05749b8 (stale) Mirror 26: 0xe05749b8 (stale) Mirror 27: 0xe05749b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eaa4: skipped 000966 1789303755.661770 write 0x023f51 thru 0x02aa2f (0x6adf bytes) 000967 1789303755.680382 write 0x0162b1 thru 0x0256d1 (0xf421 bytes) 000968 1789303755.687415 trunc from 0x02ff02 to 0x0349e9 (0x4ae8 bytes) 000969 1789303755.692485 mapread 0x01323e thru 0x016942 (0x3705 bytes) 000970 1789303755.714674 read 0x024572 thru 0x02a049 (0x5ad8 bytes) 000971 1789303755.715342 write 0x029fc1 thru 0x031284 (0x72c4 bytes) 000973 1789303755.723177 punch from 0x031a94 to 0x03720a (0x5777 bytes) 000974 1789303755.742248 trunc from 0x02d1bd to 0x039b3c (0xc980 bytes) 000975 1789303755.746851 fallocate 0x01153d thru 0x019d24 (0x87e8 bytes) 000976 1789303755.752765 fallocate 0x000149 thru 0x0036af (0x3567 bytes) 000977 1789303755.758940 punch from 0x035c9a to 0x03bb22 (0x5e89 bytes) 000978 1789303755.764269 punch from 0x0181f3 to 0x025097 (0xcea5 bytes) 000979 1789303755.774991 trunc from 0x03eaa4 to 0x035393 (0x9710 bytes) 000980 1789303755.795028 write 0x003d93 thru 0x00ea65 (0xacd3 bytes) 000981 1789303755.806931 trunc from 0x00737e to 0x010a40 (0x96c3 bytes) 000982 1789303755.840386 mapread 0x0294b5 thru 0x029f23 (0xa6f bytes) 000983 1789303755.842931 mapread 0x033105 thru 0x035393 (0x228f bytes) 000985 1789303755.863811 punch from 0x017c86 to 0x019c08 (0x1f83 bytes) 000986 1789303755.872888 mapread 0x02d47b thru 0x02d4c3 (0x49 bytes) 000987 1789303755.875117 read 0x002310 thru 0x00272f (0x420 bytes) 000989 1789303755.897720 fallocate 0x0272fd thru 0x033429 (0xc12d bytes) 000990 1789303755.903925 punch from 0x02a3cb to 0x035393 (0xafc9 bytes) 000991 1789303755.911636 mapwrite 0x001a33 thru 0x0057ac (0x3d7a bytes) 000992 1789303755.940420 MIRROR_RESYNC 7 mirrors 000993 1789303757.759763 trunc from 0x00841d to 0x00b3d1 (0x2fb5 bytes) 000994 1789303757.794303 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35394): Mirror 16: 0xe94a28db (stale) Mirror 19: 0xe94a28db (stale) Mirror 25: 0xe94a28db (stale) Mirror 26: 0xe94a28db (stale) Mirror 27: 0xe94a28db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35394): Mirror 16: 0xe94a28db (stale) Mirror 19: 0xe94a28db (stale) Mirror 25: 0xe94a28db (stale) Mirror 26: 0xe94a28db (stale) Mirror 27: 0xe94a28db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35394: skipped 000995 1789303758.107068 punch from 0x02160b to 0x026bcd (0x55c3 bytes) 000996 1789303758.111026 write 0x028d7e thru 0x03807d (0xf300 bytes) 000997 1789303758.119085 trunc from 0x0368bf to 0x03807d (0x17bf bytes) 000998 1789303758.130796 fallocate 0x039639 thru 0x03ffff (0x69c7 bytes) 000999 1789303758.134369 read 0x02b382 thru 0x0385c6 (0xd245 bytes) 001000 1789303758.139121 punch from 0x02c54b to 0x03506a (0x8b20 bytes) All operations completed A-OK! PASS 70a (131s) == sanity-flr test 71: check mirror extend parent fid ==== 08:49:23 (1789303763) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x17e:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 08:49:33 (1789303773) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.238682 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074787 s, 14.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 08:49:41 (1789303781) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6863520 .. Extending file size to 5015808 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8372960 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5870624 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9041248 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6634144 .. Extending file size to 7563712 .. Extending file size to 8020640 .. Extending file size to 8465440 .. Extending file size to 8991136 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61363 61364 61365 61368 61370 done 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) 192.168.202.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:51:04 (1789303864) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8230048 .. Extending file size to 6004192 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9234976 .. Extending file size to 6607424 .. Extending file size to 7497504 .. Extending file size to 7595328 .. Extending file size to 7599488 .. Extending file size to 7912416 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7924768 .. Extending file size to 9320064 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8359552 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8510464 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 62881 62882 62885 62887 62888 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) 192.168.202.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:52:27 (1789303947) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18900 3588120 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24736 3582284 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43636 7170404 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18900 3588120 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24736 3582284 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43636 7170404 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164865 s, 12.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21098 s, 9.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb956ad32 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:52:45 (1789303965) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209081 s, 10.0 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 08:52:54 (1789303974) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:53:02 (1789303982) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0960107 s, 21.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:53:10 (1789303990) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116687 s, 18.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:53:18 (1789303998) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141328 s, 14.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:53:52 (1789304032) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg249-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817774 s, 12.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x203:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:54:52 (1789304092) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg249-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167624 s, 12.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:55:48 (1789304148) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg249-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg249-server: Pool lustre.test_204f created Adding targets to pool oleg249-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg249-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0791424 s, 26.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg249-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg249-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg249-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:57:04 (1789304224) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:57:12 (1789304232) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:57:20 (1789304240) oleg249-server: Pool lustre.test_206 created Adding targets to pool oleg249-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg249-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg249-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg249-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg249-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:57:46 (1789304266) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838826 s, 12.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045146 s, 23.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfc4c6a52 Mirror 3: 0xfc4c6a52 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:57:57 (1789304277) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:58:00 (1789304280) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:58:04 (1789304284) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0822172 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:58:44 (1789304324) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0167562 s, 489 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:58:52 (1789304332) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450401 s, 23.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbf lmm_fid: [0x200000405:0xbf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbf lmm_fid: [0x200000405:0xbf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611327 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-13 08:58:54.000000000 -0400 Modify: 2026-09-13 08:58:54.000000000 -0400 Change: 2026-09-13 08:58:55.000000000 -0400 Birth: 2026-09-13 08:58:54.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:59:02 (1789304342) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266548 s, 39.3 MB/s oleg249-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg249-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg249-server: O/280000401/d30/15e: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:59:23 (1789304363) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.244745 s, 167 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0260718 s, 157 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 13 08:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 13 08:59 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:59:31 (1789304371) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:59:40 (1789304380) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1837 sec =========== 08:59:50 (1789304390) === sanity-flr: start cleanup 08:59:52 (1789304392) === === sanity-flr: finish cleanup 09:00:00 (1789304400) ===