-----============= acceptance-small: sanity-flr ============----- Sun Jun 7 13:00:54 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg302-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:01:14 (1780851674) === oleg302-client.virtnet: executing check_config_client /mnt/lustre oleg302-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg302-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9490096d9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9490096d9800.idle_timeout=debug disable quota as required oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:01:45 (1780851705) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7576] == sanity-flr test 0a: lfs mirror create with -N option == 13:01:47 (1780851707) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:02:04 (1780851724) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:02:07 (1780851727) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:02:10 (1780851730) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:02:34 (1780851754) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:02:38 (1780851758) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:02:41 (1780851761) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9490096d9800.stats=clear osc.lustre-OST0001-osc-ffff9490096d9800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:02:48 (1780851768) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:02:59 (1780851779) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:03:07 (1780851787) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:03:10 (1780851790) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:03:18 (1780851798) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 13:03:28 (1780851808) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:03:35 (1780851815) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:03:38 (1780851818) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:03:44 (1780851824) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:03:54 (1780851834) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:04:01 (1780851841) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.79253 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121454 s, 8.6 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:04:12 (1780851852) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.190648 s, 38.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:04:20 (1780851860) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:04:29 (1780851869) Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (64s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:05:34 (1780851934) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.171103 s, 18.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:06:20 (1780851980) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:07:40 (1780852060) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.65515 s, 9.3 MB/s -rw-r--r-- 1 root root 52428800 Jun 7 13:07 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.289s user 0m0.001s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490096d9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.515s user 0m0.000s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.938s user 0m0.000s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg302-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490096d9800.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:09:25 (1780852165) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:09:28 (1780852168) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:09:32 (1780852172) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 13:09:36 (1780852176) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 13:09:43 (1780852183) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.742525 s, 16.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:09:56 (1780852196) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265883 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247196 s, 42.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:10:03 (1780852203) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:10:10 (1780852210) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00184452 s, 555 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00166471 s, 615 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000959381 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00181274 s, 565 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 13:10:21 (1780852221) create test files with size 15734953 .. 230dbe3ecc722bd3f0b4694e448cf7df - 7c34b7723f9856b721e3b9ea08c922d9 - 6462111db84c7cbd3881d97df6022a98 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 13:10:46 (1780852246) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:11:00 (1780852260) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:11:08 (1780852268) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10679 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038 s, 27.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 13:11:17 (1780852277) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163506 s, 12.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.443306 s, 9.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.380526 s, 5.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 13:11:36 (1780852296) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 13:11:39 (1780852299) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:11:43 (1780852303) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405567 s, 25.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 13:11:50 (1780852310) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:11:54 (1780852314) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:12:03 (1780852323) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.424792 s, 24.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:12:15 (1780852335) llite.lustre-ffff9490030c3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.30715 s, 34.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff949004eae000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:12:29 (1780852349) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:13:07 (1780852387) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:13:46 (1780852426) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 13:13:50 (1780852430) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.234123 s, 13.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.154847 s, 20.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5cd7e1e8 (nosync,stale) Mirror 2: 0x5cd7e1e8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 893a4398266de83071229ada09f85b45 - mirror 1 checksum: 16c9df8e7fb292415a4adc779b069477 - mirror 2 checksum: 893a4398266de83071229ada09f85b45 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jun 7 13:13:51 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jun 7 13:13:51 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 893a4398266de83071229ada09f85b45 - mirror 1 checksum: 16c9df8e7fb292415a4adc779b069477 - mirror 2 checksum: 16c9df8e7fb292415a4adc779b069477 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jun 7 13:13:51 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jun 7 13:13:51 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:14:02 (1780852442) Failing mds1 on oleg302-server Stopping /mnt/lustre-mds1 (opts:) on oleg302-server 13:14:15 (1780852455) shut down facet: mds1 facet_host: oleg302-server facet_failover_host: oleg302-server Failover mds1 to oleg302-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 13:14:40 (1780852480) targets are mounted 13:14:40 (1780852480) facet_failover done oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303241 s, 34.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:15:02 (1780852502) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:15:05 (1780852505) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:15:09 (1780852509) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00709121 s, 2.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:15:17 (1780852517) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:15:20 (1780852520) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:15:24 (1780852524) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:15:49 (1780852549) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:16:19 (1780852579) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:16:43 (1780852603) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:17:07 (1780852627) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 13:17:15 (1780852635) mirror create pid 53545 mirror split pid 53546 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53545 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53546 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 13:18:23 (1780852703) Seed set to 2705 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000006 1780852705.811996 MIRROR_EXTEND to 2 mirrors skipping zero size read 000008 1780852706.337372 write 0x03e632 thru 0x03ffff (0x19ce bytes) 000009 1780852706.419296 mapread 0x007793 thru 0x00dad1 (0x633f bytes) 000010 1780852706.487691 mapwrite 0x015f83 thru 0x01c821 (0x689f bytes) 000013 1780852706.634701 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) truncating to largest ever: 0xf8cb 000014 1780852706.900803 trunc from 0x040000 to 0x00f8ca (0x30735 bytes) 000015 1780852706.935065 mapread 0x009109 thru 0x00ee03 (0x5cfb bytes) 000019 1780852706.982609 read 0x00298d thru 0x00c48c (0x9b00 bytes) 000021 1780852706.993625 mapwrite 0x03cdc6 thru 0x03ffff (0x323a bytes) 000026 1780852707.197855 read 0x0089a5 thru 0x01115e (0x87ba bytes) truncating to largest ever: 0x3929a 000027 1780852707.227873 trunc from 0x040000 to 0x039299 (0x6d66 bytes) 000028 1780852707.267802 mapread 0x0303a4 thru 0x037981 (0x75de bytes) 000032 1780852707.283910 mapwrite 0x00e629 thru 0x017976 (0x934e bytes) 000034 1780852707.393489 read 0x00dca0 thru 0x0161c8 (0x8529 bytes) 000037 1780852707.396837 write 0x01daea thru 0x02c612 (0xeb29 bytes) 000039 1780852707.399858 read 0x020a49 thru 0x023258 (0x2810 bytes) 000040 1780852707.401318 read 0x0004e2 thru 0x006b3b (0x665a bytes) 000042 1780852707.402909 read 0x008a69 thru 0x00c35d (0x38f5 bytes) 000044 1780852707.404852 write 0x02fc61 thru 0x032367 (0x2707 bytes) 000048 1780852707.409433 write 0x027145 thru 0x0331ed (0xc0a9 bytes) 000051 1780852707.421092 mapwrite 0x018744 thru 0x01d0f5 (0x49b2 bytes) 000052 1780852707.507513 trunc from 0x03929a to 0x02729f (0x11ffa bytes) 000055 1780852707.539529 mapread 0x01a523 thru 0x01a75c (0x23a bytes) 000056 1780852707.540572 write 0x000afb thru 0x009885 (0x8d8b bytes) 000057 1780852707.544436 write 0x0063d1 thru 0x0120e2 (0xbd12 bytes) 000058 1780852707.547435 trunc from 0x0272a0 to 0x037a43 (0x107a4 bytes) 000059 1780852707.578371 write 0x00341f thru 0x008311 (0x4ef3 bytes) 000060 1780852707.581492 mapread 0x02d45c thru 0x037a43 (0xa5e8 bytes) 000064 1780852707.597612 trunc from 0x037a44 to 0x013370 (0x246d3 bytes) 000069 1780852707.627127 read 0x00aa0d thru 0x013370 (0x8964 bytes) 000079 1780852707.644600 MIRROR_EXTEND to 2 mirrors 000080 1780852708.572541 mapread 0x0053be thru 0x00b6d9 (0x631c bytes) 000083 1780852708.674727 mapread 0x00acd3 thru 0x00b402 (0x730 bytes) 000084 1780852708.689700 read 0x008b84 thru 0x008f48 (0x3c5 bytes) 000085 1780852708.700489 MIRROR_EXTEND to 3 mirrors 000086 1780852709.633328 read 0x00dd8a thru 0x013370 (0x55e7 bytes) 000087 1780852709.649437 mapread 0x000e4a thru 0x009691 (0x8848 bytes) 000088 1780852709.681888 mapwrite 0x038a92 thru 0x03ffff (0x756e bytes) 000089 1780852709.855622 read 0x010223 thru 0x010e03 (0xbe1 bytes) truncating to largest ever: 0x3a1cd 000094 1780852709.878329 trunc from 0x040000 to 0x03a1cc (0x5e33 bytes) 000095 1780852709.912141 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1cd): Mirror 2: 0x8446c936 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1cd): Mirror 2: 0x8446c936 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1cd: skipped 000096 1780852710.270767 write 0x03f6a2 thru 0x03ffff (0x95e bytes) 000098 1780852710.284744 trunc from 0x040000 to 0x019213 (0x26dec bytes) 000099 1780852710.309609 read 0x0161a7 thru 0x016d77 (0xbd1 bytes) 000101 1780852710.332417 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x19214): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19214): Mirror 2: 0xc2bfe5b2 (stale) CRC-32 checksum value for chunk [0, 0x19214): Mirror 1: 0xc2bfe5b2 Mirror 3: 0xc2bfe5b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19214: skipped 000102 1780852711.477021 write 0x0369cf thru 0x03ffff (0x9631 bytes) 000103 1780852711.589279 trunc from 0x040000 to 0x031d13 (0xe2ec bytes) 000105 1780852711.651168 write 0x006d12 thru 0x0073cc (0x6bb bytes) 000106 1780852711.668375 mapread 0x00a404 thru 0x019938 (0xf535 bytes) 000109 1780852711.692837 trunc from 0x031d14 to 0x006a1f (0x2b2f4 bytes) 000110 1780852711.716507 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x6a20): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6a20): Mirror 2: 0xd1a04f41 (stale) Mirror 3: 0xd1a04f41 (stale) CRC-32 checksum value for chunk [0, 0x6a20): Mirror 1: 0xd1a04f41 Mirror 4: 0xd1a04f41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6a20: skipped 000111 1780852712.747227 MIRROR_RESYNC 4 mirrors 000112 1780852713.783758 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000116 1780852714.251085 write 0x036df9 thru 0x03871a (0x1922 bytes) 000118 1780852714.358290 trunc from 0x03871b to 0x00ce75 (0x2b8a5 bytes) 000119 1780852714.381016 mapread 0x0001cf thru 0x002016 (0x1e48 bytes) 000120 1780852714.399830 mapwrite 0x035fed thru 0x03ffff (0xa013 bytes) 000121 1780852714.548742 MIRROR_RESYNC 3 mirrors 000122 1780852715.850290 mapread 0x030540 thru 0x03dd55 (0xd816 bytes) 000125 1780852715.903457 read 0x039bb8 thru 0x03ffff (0x6448 bytes) 000126 1780852715.907535 MIRROR_RESYNC 3 mirrors 000130 1780852716.373609 trunc from 0x040000 to 0x02d8e7 (0x12718 bytes) 000131 1780852716.457929 read 0x004fcb thru 0x01464c (0xf682 bytes) 000133 1780852716.472044 write 0x002057 thru 0x0036c0 (0x166a bytes) 000135 1780852716.484474 read 0x0262fc thru 0x02d8e7 (0x75ec bytes) 000136 1780852716.498065 mapread 0x00f6da thru 0x0173eb (0x7d12 bytes) 000138 1780852716.501608 mapread 0x0070da thru 0x0153a9 (0xe2d0 bytes) 000139 1780852716.509068 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d8e8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d8e8): Mirror 3: 0xe7d48265 (stale) Mirror 4: 0xe7d48265 (stale) CRC-32 checksum value for chunk [0, 0x2d8e8): Mirror 1: 0x12712742 Mirror 5: 0x12712742 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d8e8: skipped 000140 1780852717.408101 read 0x02053f thru 0x02bf68 (0xba2a bytes) 000142 1780852717.440375 trunc from 0x02d8e8 to 0x0127a1 (0x1b146 bytes) 000144 1780852717.657002 read 0x00b8b4 thru 0x00c45c (0xba9 bytes) 000146 1780852717.666248 trunc from 0x0127a2 to 0x030467 (0x1dcc6 bytes) 000148 1780852717.687216 write 0x0228d4 thru 0x02db68 (0xb295 bytes) 000149 1780852717.714639 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30468): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30468): Mirror 3: 0x71f2c4d1 (stale) Mirror 4: 0x71f2c4d1 (stale) Mirror 5: 0x6355d484 (stale) CRC-32 checksum value for chunk [0, 0x30468): Mirror 1: 0xf496b5fa Mirror 6: 0xf496b5fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30468: skipped 000150 1780852719.097251 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30468): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30468): Mirror 3: 0x71f2c4d1 (stale) Mirror 4: 0x71f2c4d1 (stale) Mirror 5: 0x6355d484 (stale) CRC-32 checksum value for chunk [0, 0x30468): Mirror 1: 0xf496b5fa Mirror 6: 0xf496b5fa Mirror 7: 0xf496b5fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30468: skipped 000152 1780852720.551146 read 0x01397c thru 0x013d5c (0x3e1 bytes) 000153 1780852720.556206 trunc from 0x030468 to 0x00e105 (0x22362 bytes) 000157 1780852720.736040 mapwrite 0x02b5e9 thru 0x033572 (0x7f8a bytes) 000158 1780852720.960598 mapwrite 0x023f56 thru 0x029f44 (0x5fef bytes) 000160 1780852721.057018 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33573): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33573): Mirror 3: 0x210e00aa (stale) Mirror 4: 0x210e00aa (stale) Mirror 5: 0x62cd7758 (stale) Mirror 6: 0x2019b67d (stale) Mirror 7: 0x2019b67d (stale) CRC-32 checksum value for chunk [0, 0x33573): Mirror 1: 0xa7989f7c Mirror 8: 0xa7989f7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33573: skipped 000162 1780852722.920328 trunc from 0x033573 to 0x0099ff (0x29b73 bytes) 000163 1780852723.055324 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9a00): Mirror 3: 0x3003fd60 (stale) Mirror 5: 0x868945b (stale) Mirror 6: 0x868945b (stale) Mirror 7: 0x868945b (stale) Mirror 8: 0x868945b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a00): Mirror 3: 0x3003fd60 (stale) Mirror 5: 0x868945b (stale) Mirror 6: 0x868945b (stale) Mirror 7: 0x868945b (stale) Mirror 8: 0x868945b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a00: skipped 000164 1780852723.519489 mapread 0x0068f6 thru 0x0099ff (0x310a bytes) 000165 1780852723.545058 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x9a00): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a00): Mirror 3: 0x3003fd60 (stale) Mirror 5: 0x868945b (stale) Mirror 6: 0x868945b (stale) Mirror 7: 0x868945b (stale) Mirror 8: 0x868945b (stale) CRC-32 checksum value for chunk [0, 0x9a00): Mirror 1: 0x868945b Mirror 9: 0x868945b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a00: skipped 000166 1780852725.024531 mapwrite 0x01625e thru 0x01e0ec (0x7e8f bytes) 000167 1780852725.361629 write 0x004100 thru 0x00fc7e (0xbb7f bytes) 000168 1780852725.378006 read 0x00a80a thru 0x011d1e (0x7515 bytes) 000169 1780852725.379784 mapread 0x00e177 thru 0x01def1 (0xfd7b bytes) 000170 1780852725.381602 write 0x01eaea thru 0x01f623 (0xb3a bytes) 000171 1780852725.386952 trunc from 0x01f624 to 0x0275cc (0x7fa9 bytes) 000172 1780852725.456606 MIRROR_RESYNC 7 mirrors 000173 1780852727.677097 mapread 0x00ad0e thru 0x00f75c (0x4a4f bytes) 000174 1780852727.726331 read 0x018637 thru 0x01c36b (0x3d35 bytes) 000177 1780852727.760237 MIRROR_EXTEND to 8 mirrors 000178 1780852728.862024 write 0x011b3f thru 0x0201f7 (0xe6b9 bytes) 000179 1780852729.060481 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x275cd): Mirror 3: 0x7dadc7 (stale) Mirror 6: 0x7dadc7 (stale) Mirror 7: 0x7dadc7 (stale) Mirror 8: 0x7dadc7 (stale) Mirror 9: 0x7dadc7 (stale) Mirror 10: 0x7dadc7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x275cd): Mirror 3: 0x7dadc7 (stale) Mirror 6: 0x7dadc7 (stale) Mirror 7: 0x7dadc7 (stale) Mirror 8: 0x7dadc7 (stale) Mirror 9: 0x7dadc7 (stale) Mirror 10: 0x7dadc7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x275cd: skipped 000180 1780852729.969553 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x275cd): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x275cd): Mirror 3: 0x7dadc7 (stale) Mirror 6: 0x7dadc7 (stale) Mirror 7: 0x7dadc7 (stale) Mirror 8: 0x7dadc7 (stale) Mirror 9: 0x7dadc7 (stale) Mirror 10: 0x7dadc7 (stale) CRC-32 checksum value for chunk [0, 0x275cd): Mirror 1: 0x4b130989 Mirror 11: 0x4b130989 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x275cd: skipped 000188 1780852731.603320 mapread 0x0195c7 thru 0x01be04 (0x283e bytes) 000189 1780852731.691105 mapwrite 0x016269 thru 0x01d237 (0x6fcf bytes) 000190 1780852731.981538 MIRROR_RESYNC 8 mirrors 000191 1780852733.738637 mapread 0x00d419 thru 0x016769 (0x9351 bytes) 000192 1780852733.780324 write 0x03cb53 thru 0x03ffff (0x34ad bytes) 000193 1780852733.915450 write 0x00682f thru 0x00800e (0x17e0 bytes) 000194 1780852733.953196 mapwrite 0x00232b thru 0x007c2c (0x5902 bytes) 000195 1780852734.029239 write 0x014d44 thru 0x01fa4f (0xad0c bytes) 000196 1780852734.049249 mapwrite 0x015898 thru 0x01d9b6 (0x811f bytes) 000197 1780852734.127726 mapwrite 0x0253f0 thru 0x033ccf (0xe8e0 bytes) 000198 1780852734.301551 trunc from 0x040000 to 0x000f50 (0x3f0af bytes) 000200 1780852734.354922 write 0x0301fa thru 0x037882 (0x7689 bytes) 000201 1780852734.367919 mapread 0x02927b thru 0x0303ec (0x7172 bytes) 000202 1780852734.392985 MIRROR_RESYNC 8 mirrors 000203 1780852736.347483 write 0x006cd1 thru 0x00dcd4 (0x7004 bytes) 000204 1780852736.474402 write 0x03f5b2 thru 0x03ffff (0xa4e bytes) 000205 1780852736.483801 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd450c0c8 (stale) Mirror 6: 0xd450c0c8 (stale) Mirror 7: 0xd450c0c8 (stale) Mirror 8: 0xd450c0c8 (stale) Mirror 9: 0xd450c0c8 (stale) Mirror 10: 0xd450c0c8 (stale) Mirror 11: 0xd450c0c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5556aaa5 Mirror 12: 0x5556aaa5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000206 1780852738.385026 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd450c0c8 (stale) Mirror 6: 0xd450c0c8 (stale) Mirror 7: 0xd450c0c8 (stale) Mirror 8: 0xd450c0c8 (stale) Mirror 9: 0xd450c0c8 (stale) Mirror 10: 0xd450c0c8 (stale) Mirror 11: 0xd450c0c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5556aaa5 Mirror 12: 0x5556aaa5 Mirror 13: 0x5556aaa5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000207 1780852739.623643 trunc from 0x040000 to 0x01f478 (0x20b87 bytes) 000208 1780852739.716607 read 0x0153fc thru 0x01a485 (0x508a bytes) 000209 1780852739.735584 mapread 0x012713 thru 0x018c9e (0x658c bytes) 000210 1780852739.757789 mapread 0x014cc1 thru 0x017cd7 (0x3017 bytes) 000211 1780852739.759397 mapwrite 0x013367 thru 0x022708 (0xf3a2 bytes) 000212 1780852739.913553 mapwrite 0x014c34 thru 0x01f0b5 (0xa482 bytes) 000213 1780852739.976620 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22709): Mirror 6: 0x31e3ba12 (stale) Mirror 7: 0x31e3ba12 (stale) Mirror 8: 0x31e3ba12 (stale) Mirror 9: 0x31e3ba12 (stale) Mirror 10: 0x31e3ba12 (stale) Mirror 11: 0x31e3ba12 (stale) Mirror 12: 0xff305f75 (stale) Mirror 13: 0xff305f75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22709): Mirror 6: 0x31e3ba12 (stale) Mirror 7: 0x31e3ba12 (stale) Mirror 8: 0x31e3ba12 (stale) Mirror 9: 0x31e3ba12 (stale) Mirror 10: 0x31e3ba12 (stale) Mirror 11: 0x31e3ba12 (stale) Mirror 12: 0xff305f75 (stale) Mirror 13: 0xff305f75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22709: skipped 000214 1780852740.727112 read 0x00bd2b thru 0x00edae (0x3084 bytes) 000215 1780852740.737744 write 0x019bae thru 0x021315 (0x7768 bytes) 000216 1780852740.752544 write 0x02a3fb thru 0x034124 (0x9d2a bytes) 000219 1780852740.767456 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x34125): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34125): Mirror 6: 0xa62be15a (stale) Mirror 7: 0xa62be15a (stale) Mirror 8: 0xa62be15a (stale) Mirror 9: 0xa62be15a (stale) Mirror 10: 0xa62be15a (stale) Mirror 11: 0xa62be15a (stale) Mirror 12: 0x81cbc11f (stale) Mirror 13: 0x81cbc11f (stale) CRC-32 checksum value for chunk [0, 0x34125): Mirror 1: 0xa4fbb47 Mirror 14: 0xa4fbb47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34125: skipped 000220 1780852742.162140 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34125): Mirror 7: 0xa62be15a (stale) Mirror 8: 0xa62be15a (stale) Mirror 9: 0xa62be15a (stale) Mirror 10: 0xa62be15a (stale) Mirror 11: 0xa62be15a (stale) Mirror 12: 0x81cbc11f (stale) Mirror 13: 0x81cbc11f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34125): Mirror 7: 0xa62be15a (stale) Mirror 8: 0xa62be15a (stale) Mirror 9: 0xa62be15a (stale) Mirror 10: 0xa62be15a (stale) Mirror 11: 0xa62be15a (stale) Mirror 12: 0x81cbc11f (stale) Mirror 13: 0x81cbc11f (stale) CRC-32 checksum value for chunk [0, 0x34125): Mirror 1: 0xa4fbb47 Mirror 14: 0xa4fbb47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34125: skipped 000222 1780852743.094816 trunc from 0x034125 to 0x00e31d (0x25e07 bytes) 000223 1780852743.234375 write 0x03cff0 thru 0x03ffff (0x3010 bytes) 000224 1780852743.235812 mapwrite 0x02cf42 thru 0x032b5d (0x5c1c bytes) 000227 1780852743.354317 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd450c0c8 (stale) Mirror 8: 0xd450c0c8 (stale) Mirror 9: 0xd450c0c8 (stale) Mirror 10: 0xd450c0c8 (stale) Mirror 11: 0xd450c0c8 (stale) Mirror 12: 0x5556aaa5 (stale) Mirror 13: 0x5556aaa5 (stale) Mirror 14: 0x880758d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7fddebb0 Mirror 15: 0x7fddebb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000228 1780852744.576182 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd450c0c8 (stale) Mirror 8: 0xd450c0c8 (stale) Mirror 9: 0xd450c0c8 (stale) Mirror 10: 0xd450c0c8 (stale) Mirror 11: 0xd450c0c8 (stale) Mirror 12: 0x5556aaa5 (stale) Mirror 13: 0x5556aaa5 (stale) Mirror 14: 0x880758d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7fddebb0 Mirror 15: 0x7fddebb0 Mirror 16: 0x7fddebb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000232 1780852745.801314 mapwrite 0x03476d thru 0x03eb32 (0xa3c6 bytes) 000235 1780852745.982971 MIRROR_RESYNC 11 mirrors 000236 1780852747.861750 read 0x003bd5 thru 0x00d895 (0x9cc1 bytes) 000239 1780852747.870434 write 0x004cb6 thru 0x0054b8 (0x803 bytes) 000240 1780852747.957122 mapread 0x027536 thru 0x0280f2 (0xbbd bytes) 000241 1780852747.971519 trunc from 0x040000 to 0x01fdc6 (0x20239 bytes) 000244 1780852747.996971 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fdc7): Mirror 8: 0x188f5ff6 (stale) Mirror 9: 0x188f5ff6 (stale) Mirror 10: 0x188f5ff6 (stale) Mirror 11: 0x188f5ff6 (stale) Mirror 12: 0x188f5ff6 (stale) Mirror 13: 0x188f5ff6 (stale) Mirror 14: 0x188f5ff6 (stale) Mirror 15: 0x188f5ff6 (stale) Mirror 16: 0x188f5ff6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fdc7): Mirror 8: 0x188f5ff6 (stale) Mirror 9: 0x188f5ff6 (stale) Mirror 10: 0x188f5ff6 (stale) Mirror 11: 0x188f5ff6 (stale) Mirror 12: 0x188f5ff6 (stale) Mirror 13: 0x188f5ff6 (stale) Mirror 14: 0x188f5ff6 (stale) Mirror 15: 0x188f5ff6 (stale) Mirror 16: 0x188f5ff6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fdc7: skipped 000245 1780852748.576610 trunc from 0x01fdc7 to 0x011e9a (0xdf2c bytes) 000247 1780852748.618931 mapwrite 0x00a9b7 thru 0x0193fb (0xea45 bytes) 000248 1780852748.729943 write 0x03a9af thru 0x03ffff (0x5651 bytes) 000249 1780852748.736432 MIRROR_RESYNC 10 mirrors 000252 1780852750.099028 mapread 0x016a0f thru 0x01aa6e (0x4060 bytes) 000254 1780852750.124078 MIRROR_EXTEND to 11 mirrors 000255 1780852750.891092 MIRROR_EXTEND to 12 mirrors 000256 1780852751.854024 MIRROR_RESYNC 12 mirrors 000257 1780852752.683394 mapread 0x014760 thru 0x016a30 (0x22d1 bytes) 000258 1780852752.714379 trunc from 0x040000 to 0x01ad6b (0x25294 bytes) 000259 1780852752.833247 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1ad6c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ad6c): Mirror 8: 0xa85448fa (stale) Mirror 9: 0xa85448fa (stale) Mirror 10: 0xa85448fa (stale) Mirror 11: 0xa85448fa (stale) Mirror 12: 0xa85448fa (stale) Mirror 13: 0xa85448fa (stale) Mirror 14: 0xa85448fa (stale) Mirror 15: 0xa85448fa (stale) Mirror 16: 0xa85448fa (stale) Mirror 17: 0xa85448fa (stale) Mirror 18: 0xa85448fa (stale) CRC-32 checksum value for chunk [0, 0x1ad6c): Mirror 1: 0xa85448fa Mirror 19: 0xa85448fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ad6c: skipped 000261 1780852754.059265 read 0x010b88 thru 0x014104 (0x357d bytes) 000263 1780852754.114324 read 0x000662 thru 0x01006b (0xfa0a bytes) 000264 1780852754.146383 read 0x00ec75 thru 0x00ed71 (0xfd bytes) 000265 1780852754.159090 mapwrite 0x001733 thru 0x00c8bd (0xb18b bytes) 000268 1780852754.396311 read 0x011b7d thru 0x01ad6b (0x91ef bytes) 000269 1780852754.397452 read 0x00b1fe thru 0x014343 (0x9146 bytes) 000270 1780852754.398202 mapread 0x004358 thru 0x00d701 (0x93aa bytes) 000275 1780852754.401586 write 0x02b73f thru 0x03ab74 (0xf436 bytes) 000276 1780852754.403894 read 0x02705c thru 0x02f79b (0x8740 bytes) 000278 1780852754.415217 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab75): Mirror 8: 0x6bdfb80a (stale) Mirror 9: 0x6bdfb80a (stale) Mirror 10: 0x6bdfb80a (stale) Mirror 11: 0x6bdfb80a (stale) Mirror 12: 0x6bdfb80a (stale) Mirror 13: 0x6bdfb80a (stale) Mirror 14: 0x6bdfb80a (stale) Mirror 15: 0x6bdfb80a (stale) Mirror 16: 0x6bdfb80a (stale) Mirror 17: 0x6bdfb80a (stale) Mirror 19: 0xaac6ef57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab75): Mirror 8: 0x6bdfb80a (stale) Mirror 9: 0x6bdfb80a (stale) Mirror 10: 0x6bdfb80a (stale) Mirror 11: 0x6bdfb80a (stale) Mirror 12: 0x6bdfb80a (stale) Mirror 13: 0x6bdfb80a (stale) Mirror 14: 0x6bdfb80a (stale) Mirror 15: 0x6bdfb80a (stale) Mirror 16: 0x6bdfb80a (stale) Mirror 17: 0x6bdfb80a (stale) Mirror 19: 0xaac6ef57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab75: skipped 000279 1780852755.272834 mapwrite 0x02777a thru 0x0298db (0x2162 bytes) 000281 1780852755.408619 read 0x01fb5b thru 0x02fa21 (0xfec7 bytes) 000282 1780852755.427598 write 0x01bfcd thru 0x02812e (0xc162 bytes) 000284 1780852755.449714 trunc from 0x03ab75 to 0x0280cb (0x12aa9 bytes) 000287 1780852755.486321 write 0x00b603 thru 0x0166e4 (0xb0e2 bytes) 000288 1780852755.514694 trunc from 0x0280cc to 0x00fbfe (0x184cd bytes) 000289 1780852755.558061 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfbff): Mirror 9: 0xa92cb3ed (stale) Mirror 10: 0xa92cb3ed (stale) Mirror 11: 0xa92cb3ed (stale) Mirror 12: 0xa92cb3ed (stale) Mirror 13: 0xa92cb3ed (stale) Mirror 14: 0xa92cb3ed (stale) Mirror 15: 0xa92cb3ed (stale) Mirror 16: 0xa92cb3ed (stale) Mirror 17: 0xa92cb3ed (stale) Mirror 19: 0xa92cb3ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfbff): Mirror 9: 0xa92cb3ed (stale) Mirror 10: 0xa92cb3ed (stale) Mirror 11: 0xa92cb3ed (stale) Mirror 12: 0xa92cb3ed (stale) Mirror 13: 0xa92cb3ed (stale) Mirror 14: 0xa92cb3ed (stale) Mirror 15: 0xa92cb3ed (stale) Mirror 16: 0xa92cb3ed (stale) Mirror 17: 0xa92cb3ed (stale) Mirror 19: 0xa92cb3ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfbff: skipped 000290 1780852756.248387 trunc from 0x00fbff to 0x013cd9 (0x40db bytes) 000291 1780852756.272514 mapwrite 0x0312eb thru 0x039a50 (0x8766 bytes) 000293 1780852756.418084 mapread 0x026cc2 thru 0x02a947 (0x3c86 bytes) 000294 1780852756.452452 mapread 0x033f33 thru 0x038f15 (0x4fe3 bytes) 000295 1780852756.455255 write 0x0253f1 thru 0x026b72 (0x1782 bytes) 000297 1780852756.473781 read 0x036f5f thru 0x039a50 (0x2af2 bytes) 000299 1780852756.482473 write 0x007812 thru 0x008523 (0xd12 bytes) 000301 1780852756.541641 trunc from 0x039a51 to 0x01edac (0x1aca4 bytes) 000302 1780852756.591559 mapread 0x00f926 thru 0x01b141 (0xb81c bytes) 000304 1780852756.635045 trunc from 0x01edad to 0x003f51 (0x1ae5b bytes) 000307 1780852756.688457 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3f52): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f52): Mirror 9: 0xf1832045 (stale) Mirror 10: 0xf1832045 (stale) Mirror 11: 0xf1832045 (stale) Mirror 12: 0xf1832045 (stale) Mirror 13: 0xf1832045 (stale) Mirror 14: 0xf1832045 (stale) Mirror 15: 0xf1832045 (stale) Mirror 16: 0xf1832045 (stale) Mirror 17: 0xf1832045 (stale) Mirror 19: 0xf1832045 (stale) CRC-32 checksum value for chunk [0, 0x3f52): Mirror 1: 0xe1a2de9a Mirror 20: 0xe1a2de9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f52: skipped 000308 1780852758.167862 write 0x0297bb thru 0x038fb3 (0xf7f9 bytes) 000311 1780852758.335252 read 0x01462d thru 0x01eeb3 (0xa887 bytes) 000312 1780852758.369792 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x38fb4): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fb4): Mirror 9: 0x52cf4212 (stale) Mirror 10: 0x52cf4212 (stale) Mirror 11: 0x52cf4212 (stale) Mirror 12: 0x52cf4212 (stale) Mirror 13: 0x52cf4212 (stale) Mirror 14: 0x52cf4212 (stale) Mirror 15: 0x52cf4212 (stale) Mirror 16: 0x52cf4212 (stale) Mirror 17: 0x52cf4212 (stale) Mirror 19: 0x52cf4212 (stale) Mirror 20: 0xb35bd9ed (stale) CRC-32 checksum value for chunk [0, 0x38fb4): Mirror 1: 0x52f7db30 Mirror 21: 0x52f7db30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fb4: skipped 000313 1780852759.547809 mapwrite 0x02717a thru 0x02c0fc (0x4f83 bytes) 000314 1780852759.827513 trunc from 0x038fb4 to 0x030285 (0x8d2e bytes) 000316 1780852759.901958 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x30286): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30286): Mirror 9: 0x10d7516a (stale) Mirror 10: 0x10d7516a (stale) Mirror 11: 0x10d7516a (stale) Mirror 12: 0x10d7516a (stale) Mirror 13: 0x10d7516a (stale) Mirror 14: 0x10d7516a (stale) Mirror 15: 0x10d7516a (stale) Mirror 16: 0x10d7516a (stale) Mirror 17: 0x10d7516a (stale) Mirror 19: 0x10d7516a (stale) Mirror 20: 0x5aac887b (stale) Mirror 21: 0x5100a002 (stale) CRC-32 checksum value for chunk [0, 0x30286): Mirror 1: 0xcc44f824 Mirror 22: 0xcc44f824 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30286: skipped 000318 1780852761.126254 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x30286): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30286): Mirror 9: 0x10d7516a (stale) Mirror 10: 0x10d7516a (stale) Mirror 11: 0x10d7516a (stale) Mirror 12: 0x10d7516a (stale) Mirror 13: 0x10d7516a (stale) Mirror 14: 0x10d7516a (stale) Mirror 15: 0x10d7516a (stale) Mirror 16: 0x10d7516a (stale) Mirror 17: 0x10d7516a (stale) Mirror 19: 0x10d7516a (stale) Mirror 20: 0x5aac887b (stale) Mirror 21: 0x5100a002 (stale) CRC-32 checksum value for chunk [0, 0x30286): Mirror 1: 0xcc44f824 Mirror 22: 0xcc44f824 Mirror 23: 0xcc44f824 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30286: skipped 000319 1780852762.195607 trunc from 0x030286 to 0x01c9ee (0x13897 bytes) 000320 1780852762.351687 read 0x008ea9 thru 0x0158ca (0xca22 bytes) 000321 1780852762.366733 read 0x00a813 thru 0x019490 (0xec7e bytes) 000324 1780852762.386330 write 0x03095f thru 0x038352 (0x79f4 bytes) 000325 1780852762.392863 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38353): Mirror 10: 0x16b92b53 (stale) Mirror 11: 0x16b92b53 (stale) Mirror 12: 0x16b92b53 (stale) Mirror 13: 0x16b92b53 (stale) Mirror 14: 0x16b92b53 (stale) Mirror 15: 0x16b92b53 (stale) Mirror 16: 0x16b92b53 (stale) Mirror 17: 0x16b92b53 (stale) Mirror 19: 0x16b92b53 (stale) Mirror 20: 0xc9d0c88 (stale) Mirror 21: 0xcb154432 (stale) Mirror 22: 0x60d79e2a (stale) Mirror 23: 0x60d79e2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38353): Mirror 10: 0x16b92b53 (stale) Mirror 11: 0x16b92b53 (stale) Mirror 12: 0x16b92b53 (stale) Mirror 13: 0x16b92b53 (stale) Mirror 14: 0x16b92b53 (stale) Mirror 15: 0x16b92b53 (stale) Mirror 16: 0x16b92b53 (stale) Mirror 17: 0x16b92b53 (stale) Mirror 19: 0x16b92b53 (stale) Mirror 20: 0xc9d0c88 (stale) Mirror 21: 0xcb154432 (stale) Mirror 22: 0x60d79e2a (stale) Mirror 23: 0x60d79e2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38353: skipped 000326 1780852763.222229 mapwrite 0x006964 thru 0x011799 (0xae36 bytes) 000329 1780852763.381466 read 0x00f2df thru 0x01d8cf (0xe5f1 bytes) 000330 1780852763.399546 write 0x015cf4 thru 0x021f16 (0xc223 bytes) 000331 1780852763.408514 read 0x0244c4 thru 0x02ca2d (0x856a bytes) 000332 1780852763.432381 mapwrite 0x018ef5 thru 0x026c7b (0xdd87 bytes) 000333 1780852763.543540 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38353): Mirror 10: 0x16b92b53 (stale) Mirror 11: 0x16b92b53 (stale) Mirror 12: 0x16b92b53 (stale) Mirror 13: 0x16b92b53 (stale) Mirror 15: 0x16b92b53 (stale) Mirror 16: 0x16b92b53 (stale) Mirror 17: 0x16b92b53 (stale) Mirror 19: 0x16b92b53 (stale) Mirror 20: 0xc9d0c88 (stale) Mirror 21: 0xcb154432 (stale) Mirror 22: 0x60d79e2a (stale) Mirror 23: 0x60d79e2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38353): Mirror 10: 0x16b92b53 (stale) Mirror 11: 0x16b92b53 (stale) Mirror 12: 0x16b92b53 (stale) Mirror 13: 0x16b92b53 (stale) Mirror 15: 0x16b92b53 (stale) Mirror 16: 0x16b92b53 (stale) Mirror 17: 0x16b92b53 (stale) Mirror 19: 0x16b92b53 (stale) Mirror 20: 0xc9d0c88 (stale) Mirror 21: 0xcb154432 (stale) Mirror 22: 0x60d79e2a (stale) Mirror 23: 0x60d79e2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38353: skipped 000337 1780852764.343677 mapread 0x013de7 thru 0x01df47 (0xa161 bytes) 000338 1780852764.366737 MIRROR_RESYNC 13 mirrors 000339 1780852766.262770 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000340 1780852766.815466 read 0x02f4e3 thru 0x038352 (0x8e70 bytes) 000342 1780852766.829505 mapread 0x033ca1 thru 0x038352 (0x46b2 bytes) 000343 1780852766.835453 mapread 0x00b152 thru 0x016c84 (0xbb33 bytes) 000344 1780852766.884690 write 0x0158ca thru 0x01f499 (0x9bd0 bytes) 000345 1780852766.999981 mapwrite 0x01a707 thru 0x028a48 (0xe342 bytes) 000346 1780852767.132000 mapread 0x02c155 thru 0x038352 (0xc1fe bytes) 000347 1780852767.162814 write 0x0321e3 thru 0x03c9c8 (0xa7e6 bytes) 000348 1780852767.171149 MIRROR_RESYNC 12 mirrors 000349 1780852769.339202 trunc from 0x03c9c9 to 0x02881a (0x141ae bytes) 000353 1780852769.421130 mapwrite 0x03a54a thru 0x03ffff (0x5ab6 bytes) 000354 1780852769.528613 write 0x003cdc thru 0x0139f1 (0xfd16 bytes) 000355 1780852769.558617 read 0x03a7fb thru 0x03ffff (0x5805 bytes) 000356 1780852769.559747 trunc from 0x040000 to 0x0240dd (0x1bf22 bytes) 000357 1780852769.585644 mapwrite 0x0112bb thru 0x01485c (0x35a2 bytes) 000358 1780852769.684403 mapwrite 0x03e7c4 thru 0x03f53b (0xd78 bytes) 000359 1780852769.774875 trunc from 0x03f53c to 0x01bda4 (0x23797 bytes) 000361 1780852769.802421 trunc from 0x01bda5 to 0x017023 (0x4d81 bytes) 000363 1780852769.830128 read 0x00519c thru 0x00d431 (0x8296 bytes) 000366 1780852769.844594 trunc from 0x017024 to 0x00e3c0 (0x8c63 bytes) truncating to largest ever: 0x3ab4c 000370 1780852769.870563 trunc from 0x00e3c1 to 0x03ab4b (0x2c78b bytes) 000371 1780852769.893468 read 0x028c2e thru 0x029eee (0x12c1 bytes) 000372 1780852769.911573 mapwrite 0x0355ab thru 0x03ffff (0xaa55 bytes) 000373 1780852770.020709 trunc from 0x040000 to 0x02c9e5 (0x1361a bytes) 000374 1780852770.039808 mapread 0x0124bc thru 0x01eb43 (0xc688 bytes) 000375 1780852770.042633 read 0x025e6b thru 0x02c9e5 (0x6b7b bytes) 000377 1780852770.051705 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c9e6): Mirror 10: 0xbebfebf0 (stale) Mirror 11: 0xbebfebf0 (stale) Mirror 12: 0xbebfebf0 (stale) Mirror 13: 0xbebfebf0 (stale) Mirror 15: 0xbebfebf0 (stale) Mirror 16: 0xbebfebf0 (stale) Mirror 17: 0xbebfebf0 (stale) Mirror 19: 0xbebfebf0 (stale) Mirror 22: 0xbebfebf0 (stale) Mirror 23: 0xbebfebf0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c9e6): Mirror 10: 0xbebfebf0 (stale) Mirror 11: 0xbebfebf0 (stale) Mirror 12: 0xbebfebf0 (stale) Mirror 13: 0xbebfebf0 (stale) Mirror 15: 0xbebfebf0 (stale) Mirror 16: 0xbebfebf0 (stale) Mirror 17: 0xbebfebf0 (stale) Mirror 19: 0xbebfebf0 (stale) Mirror 22: 0xbebfebf0 (stale) Mirror 23: 0xbebfebf0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c9e6: skipped 000379 1780852770.698416 trunc from 0x02c9e6 to 0x01fe49 (0xcb9c bytes) 000381 1780852770.725553 MIRROR_RESYNC 11 mirrors 000382 1780852772.898110 mapwrite 0x013f6b thru 0x01bfe2 (0x8078 bytes) 000383 1780852773.156860 read 0x011fba thru 0x01fc42 (0xdc89 bytes) 000384 1780852773.163010 mapread 0x0190a7 thru 0x01fe49 (0x6da3 bytes) 000385 1780852773.164212 write 0x000489 thru 0x006e92 (0x6a0a bytes) 000387 1780852773.175922 trunc from 0x01fe4a to 0x003e5d (0x1bfec bytes) 000388 1780852773.207843 mapwrite 0x01e79a thru 0x027393 (0x8bfa bytes) 000389 1780852773.301353 read 0x013224 thru 0x0197dc (0x65b9 bytes) 000390 1780852773.315507 mapwrite 0x0123f9 thru 0x0174c1 (0x50c9 bytes) 000391 1780852773.386069 mapread 0x002ab1 thru 0x00dd51 (0xb2a1 bytes) 000392 1780852773.389338 trunc from 0x027394 to 0x021f32 (0x5461 bytes) 000393 1780852773.409008 write 0x027b23 thru 0x02fc4e (0x812c bytes) 000394 1780852773.410222 mapread 0x027520 thru 0x02b891 (0x4372 bytes) truncating to largest ever: 0x3bd2c 000395 1780852773.411278 trunc from 0x02fc4f to 0x03bd2b (0xc0dd bytes) 000397 1780852773.433332 trunc from 0x03bd2c to 0x00c290 (0x2fa9b bytes) 000401 1780852773.461133 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc291): Mirror 11: 0x81b10084 (stale) Mirror 12: 0x81b10084 (stale) Mirror 13: 0x81b10084 (stale) Mirror 15: 0x81b10084 (stale) Mirror 16: 0x81b10084 (stale) Mirror 17: 0x81b10084 (stale) Mirror 19: 0x81b10084 (stale) Mirror 22: 0x81b10084 (stale) Mirror 23: 0x81b10084 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc291): Mirror 11: 0x81b10084 (stale) Mirror 12: 0x81b10084 (stale) Mirror 13: 0x81b10084 (stale) Mirror 15: 0x81b10084 (stale) Mirror 16: 0x81b10084 (stale) Mirror 17: 0x81b10084 (stale) Mirror 19: 0x81b10084 (stale) Mirror 22: 0x81b10084 (stale) Mirror 23: 0x81b10084 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc291: skipped 000402 1780852774.012580 MIRROR_RESYNC 10 mirrors 000403 1780852776.042979 mapwrite 0x03898f thru 0x03cc17 (0x4289 bytes) 000404 1780852776.253718 mapwrite 0x0248e6 thru 0x025aa2 (0x11bd bytes) 000406 1780852776.369758 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc18): Mirror 11: 0x9c9ef8a2 (stale) Mirror 12: 0x9c9ef8a2 (stale) Mirror 13: 0x9c9ef8a2 (stale) Mirror 15: 0x9c9ef8a2 (stale) Mirror 17: 0x9c9ef8a2 (stale) Mirror 19: 0x9c9ef8a2 (stale) Mirror 22: 0x9c9ef8a2 (stale) Mirror 23: 0x9c9ef8a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc18): Mirror 11: 0x9c9ef8a2 (stale) Mirror 12: 0x9c9ef8a2 (stale) Mirror 13: 0x9c9ef8a2 (stale) Mirror 15: 0x9c9ef8a2 (stale) Mirror 17: 0x9c9ef8a2 (stale) Mirror 19: 0x9c9ef8a2 (stale) Mirror 22: 0x9c9ef8a2 (stale) Mirror 23: 0x9c9ef8a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc18: skipped 000407 1780852777.103242 write 0x01cb6d thru 0x028ac9 (0xbf5d bytes) 000411 1780852777.121382 read 0x02d613 thru 0x02e54e (0xf3c bytes) 000413 1780852777.135315 mapwrite 0x02ec07 thru 0x030c29 (0x2023 bytes) 000414 1780852777.256863 mapread 0x034d58 thru 0x03cc17 (0x7ec0 bytes) 000416 1780852777.294750 mapread 0x02870c thru 0x02fc2b (0x7520 bytes) 000418 1780852777.305507 write 0x002581 thru 0x00d5a0 (0xb020 bytes) 000420 1780852777.320910 write 0x024b0a thru 0x02b4cc (0x69c3 bytes) 000421 1780852777.322783 trunc from 0x03cc18 to 0x00332d (0x398ea bytes) 000423 1780852777.343064 mapwrite 0x01b986 thru 0x02628f (0xa90a bytes) 000424 1780852777.574723 read 0x01abaa thru 0x021c36 (0x708d bytes) 000425 1780852777.592044 MIRROR_RESYNC 9 mirrors 000427 1780852779.680910 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000430 1780852780.276743 write 0x032778 thru 0x03e649 (0xbed2 bytes) 000432 1780852780.395298 trunc from 0x03e64a to 0x030b93 (0xdab6 bytes) 000433 1780852780.425159 mapread 0x018b02 thru 0x018c13 (0x112 bytes) 000434 1780852780.435350 write 0x02a542 thru 0x02ade8 (0x8a7 bytes) 000435 1780852780.466898 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x30b94): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b94): Mirror 11: 0x9844ef7b (stale) Mirror 12: 0x9844ef7b (stale) Mirror 13: 0x9844ef7b (stale) Mirror 15: 0x9844ef7b (stale) Mirror 17: 0x9844ef7b (stale) Mirror 22: 0x9844ef7b (stale) Mirror 23: 0x9844ef7b (stale) CRC-32 checksum value for chunk [0, 0x30b94): Mirror 1: 0xbca81e9e Mirror 24: 0xbca81e9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b94: skipped 000436 1780852781.688834 read 0x025fa0 thru 0x0289b0 (0x2a11 bytes) 000437 1780852781.696425 mapwrite 0x00272e thru 0x00326a (0xb3d bytes) 000438 1780852782.040474 mapwrite 0x03efa3 thru 0x03ffff (0x105d bytes) 000441 1780852782.291258 trunc from 0x040000 to 0x001cd6 (0x3e329 bytes) 000442 1780852782.333272 trunc from 0x001cd7 to 0x039220 (0x3754a bytes) 000443 1780852782.396807 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39221): Mirror 11: 0xa908e03e (stale) Mirror 12: 0xa908e03e (stale) Mirror 13: 0xa908e03e (stale) Mirror 15: 0xa908e03e (stale) Mirror 17: 0xa908e03e (stale) Mirror 22: 0xa908e03e (stale) Mirror 24: 0xd392db36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39221): Mirror 11: 0xa908e03e (stale) Mirror 12: 0xa908e03e (stale) Mirror 13: 0xa908e03e (stale) Mirror 15: 0xa908e03e (stale) Mirror 17: 0xa908e03e (stale) Mirror 22: 0xa908e03e (stale) Mirror 24: 0xd392db36 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39221: skipped 000444 1780852783.271104 trunc from 0x039221 to 0x0119e2 (0x2783e bytes) 000447 1780852783.313985 write 0x012899 thru 0x017082 (0x47ea bytes) 000448 1780852783.318614 write 0x03e68c thru 0x03ffff (0x1974 bytes) 000449 1780852783.325119 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x794bb4d6 (stale) Mirror 12: 0x794bb4d6 (stale) Mirror 13: 0x794bb4d6 (stale) Mirror 15: 0x794bb4d6 (stale) Mirror 17: 0x794bb4d6 (stale) Mirror 22: 0x794bb4d6 (stale) Mirror 24: 0x6b55a140 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18ebd18e Mirror 25: 0x18ebd18e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000451 1780852784.907782 mapwrite 0x03dc08 thru 0x03ffff (0x23f8 bytes) 000453 1780852785.132890 trunc from 0x040000 to 0x02017a (0x1fe85 bytes) 000455 1780852785.193477 write 0x018dcd thru 0x0219c1 (0x8bf5 bytes) 000458 1780852785.208960 mapread 0x004b77 thru 0x014170 (0xf5fa bytes) 000459 1780852785.221942 write 0x0128b1 thru 0x021873 (0xefc3 bytes) 000461 1780852785.225761 read 0x018665 thru 0x0219c1 (0x935d bytes) 000462 1780852785.226701 write 0x02c4ce thru 0x033d62 (0x7895 bytes) 000464 1780852785.229877 mapwrite 0x016f70 thru 0x022eb7 (0xbf48 bytes) 000465 1780852785.299490 write 0x01df20 thru 0x01eeac (0xf8d bytes) 000466 1780852785.312603 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33d63): Mirror 12: 0xf55b4029 (stale) Mirror 13: 0xf55b4029 (stale) Mirror 15: 0xf55b4029 (stale) Mirror 17: 0xf55b4029 (stale) Mirror 22: 0xf55b4029 (stale) Mirror 24: 0x27d2a51d (stale) Mirror 25: 0x16fdd947 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d63): Mirror 12: 0xf55b4029 (stale) Mirror 13: 0xf55b4029 (stale) Mirror 15: 0xf55b4029 (stale) Mirror 17: 0xf55b4029 (stale) Mirror 22: 0xf55b4029 (stale) Mirror 24: 0x27d2a51d (stale) Mirror 25: 0x16fdd947 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d63: skipped 000467 1780852785.812778 write 0x033db9 thru 0x035481 (0x16c9 bytes) 000468 1780852785.818931 trunc from 0x035482 to 0x0048d6 (0x30bab bytes) 000469 1780852785.843285 read 0x004019 thru 0x0046a0 (0x688 bytes) 000472 1780852785.852523 mapwrite 0x03871f thru 0x03e5bd (0x5e9f bytes) 000473 1780852785.940558 write 0x01e1b7 thru 0x02b279 (0xd0c3 bytes) 000474 1780852785.963111 mapread 0x00df6e thru 0x01d10c (0xf19f bytes) 000475 1780852785.981542 read 0x015591 thru 0x015a69 (0x4d9 bytes) 000477 1780852785.984260 mapread 0x01608c thru 0x01a7f2 (0x4767 bytes) 000478 1780852785.987060 trunc from 0x03e5be to 0x002ca6 (0x3b917 bytes) 000479 1780852786.019929 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ca7): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca7): Mirror 12: 0xdcf71754 (stale) Mirror 13: 0xdcf71754 (stale) Mirror 15: 0xdcf71754 (stale) Mirror 17: 0xdcf71754 (stale) Mirror 22: 0xdcf71754 (stale) Mirror 24: 0xdcf71754 (stale) Mirror 25: 0xc6fe73c7 (stale) CRC-32 checksum value for chunk [0, 0x2ca7): Mirror 1: 0xc6fe73c7 Mirror 26: 0xc6fe73c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca7: skipped 000480 1780852787.203252 mapwrite 0x03cf7e thru 0x03f2c7 (0x234a bytes) 000481 1780852787.453991 mapwrite 0x02ea08 thru 0x0348bc (0x5eb5 bytes) 000482 1780852787.557078 read 0x00ed05 thru 0x012c4d (0x3f49 bytes) 000483 1780852787.581884 mapwrite 0x010173 thru 0x014183 (0x4011 bytes) 000484 1780852787.689126 write 0x01aa9b thru 0x022e26 (0x838c bytes) 000485 1780852787.734075 read 0x02800c thru 0x03018d (0x8182 bytes) 000486 1780852787.738518 mapwrite 0x00720b thru 0x00da9e (0x6894 bytes) 000487 1780852787.894901 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2c8): Mirror 13: 0x76bf5227 (stale) Mirror 15: 0x76bf5227 (stale) Mirror 17: 0x76bf5227 (stale) Mirror 22: 0x76bf5227 (stale) Mirror 24: 0x902b4b02 (stale) Mirror 25: 0xaef98be7 (stale) Mirror 26: 0xbb83d208 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c8): Mirror 13: 0x76bf5227 (stale) Mirror 15: 0x76bf5227 (stale) Mirror 17: 0x76bf5227 (stale) Mirror 22: 0x76bf5227 (stale) Mirror 24: 0x902b4b02 (stale) Mirror 25: 0xaef98be7 (stale) Mirror 26: 0xbb83d208 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c8: skipped 000489 1780852788.572183 write 0x033d96 thru 0x03c244 (0x84af bytes) 000490 1780852788.592034 mapwrite 0x00eeed thru 0x018171 (0x9285 bytes) 000492 1780852788.691819 read 0x03a34c thru 0x03f2c7 (0x4f7c bytes) 000494 1780852788.708907 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f2c8): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c8): Mirror 13: 0x76bf5227 (stale) Mirror 15: 0x76bf5227 (stale) Mirror 17: 0x76bf5227 (stale) Mirror 22: 0x76bf5227 (stale) Mirror 24: 0x902b4b02 (stale) Mirror 25: 0xaef98be7 (stale) Mirror 26: 0xbb83d208 (stale) CRC-32 checksum value for chunk [0, 0x3f2c8): Mirror 1: 0x2c4299b3 Mirror 27: 0x2c4299b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c8: skipped 000497 1780852790.223717 mapread 0x028a2e thru 0x034646 (0xbc19 bytes) 000499 1780852790.296098 mapwrite 0x00771e thru 0x017354 (0xfc37 bytes) 000500 1780852790.699508 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2c8): Mirror 15: 0x76bf5227 (stale) Mirror 17: 0x76bf5227 (stale) Mirror 22: 0x76bf5227 (stale) Mirror 24: 0x902b4b02 (stale) Mirror 25: 0xaef98be7 (stale) Mirror 26: 0xbb83d208 (stale) Mirror 27: 0x2c4299b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c8): Mirror 15: 0x76bf5227 (stale) Mirror 17: 0x76bf5227 (stale) Mirror 22: 0x76bf5227 (stale) Mirror 24: 0x902b4b02 (stale) Mirror 25: 0xaef98be7 (stale) Mirror 26: 0xbb83d208 (stale) Mirror 27: 0x2c4299b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c8: skipped 000502 1780852791.358420 mapread 0x013158 thru 0x020f0d (0xddb6 bytes) 000503 1780852791.421128 trunc from 0x03f2c8 to 0x037f90 (0x7337 bytes) 000505 1780852791.461298 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f91): Mirror 15: 0xbcd81b55 (stale) Mirror 17: 0xbcd81b55 (stale) Mirror 22: 0xbcd81b55 (stale) Mirror 24: 0x94417b05 (stale) Mirror 25: 0x8ec516b (stale) Mirror 26: 0x3abef8b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f91): Mirror 15: 0xbcd81b55 (stale) Mirror 17: 0xbcd81b55 (stale) Mirror 22: 0xbcd81b55 (stale) Mirror 24: 0x94417b05 (stale) Mirror 25: 0x8ec516b (stale) Mirror 26: 0x3abef8b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f91: skipped 000506 1780852792.190961 mapread 0x01255c thru 0x01ec7d (0xc722 bytes) 000507 1780852792.237342 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f91): Mirror 15: 0xbcd81b55 (stale) Mirror 17: 0xbcd81b55 (stale) Mirror 22: 0xbcd81b55 (stale) Mirror 24: 0x94417b05 (stale) Mirror 25: 0x8ec516b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f91): Mirror 15: 0xbcd81b55 (stale) Mirror 17: 0xbcd81b55 (stale) Mirror 22: 0xbcd81b55 (stale) Mirror 24: 0x94417b05 (stale) Mirror 25: 0x8ec516b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f91: skipped 000508 1780852792.744881 trunc from 0x037f91 to 0x03aca6 (0x2d16 bytes) 000509 1780852792.767775 read 0x015826 thru 0x01a2c8 (0x4aa3 bytes) 000510 1780852792.798253 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3aca7): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aca7): Mirror 15: 0x4465c5b4 (stale) Mirror 17: 0x4465c5b4 (stale) Mirror 22: 0x4465c5b4 (stale) Mirror 24: 0x71c75089 (stale) Mirror 25: 0xc6f72ff3 (stale) CRC-32 checksum value for chunk [0, 0x3aca7): Mirror 1: 0xa296354b Mirror 26: 0xa296354b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aca7: skipped 000511 1780852794.085384 read 0x00cc68 thru 0x0159d7 (0x8d70 bytes) 000515 1780852794.136433 trunc from 0x03aca7 to 0x0183d8 (0x228ce bytes) 000516 1780852794.252955 mapread 0x015f89 thru 0x0183d8 (0x2450 bytes) 000517 1780852794.274048 trunc from 0x0183d9 to 0x001340 (0x17098 bytes) 000519 1780852794.296026 read 0x0000e9 thru 0x001340 (0x1258 bytes) 000520 1780852794.301883 write 0x02d729 thru 0x03b8fc (0xe1d4 bytes) 000521 1780852794.304713 read 0x02b7ed thru 0x02b950 (0x164 bytes) 000523 1780852794.312888 write 0x007bf3 thru 0x00b2b4 (0x36c2 bytes) 000524 1780852794.313958 write 0x01a6d6 thru 0x02088d (0x61b8 bytes) 000526 1780852794.316924 read 0x0386c5 thru 0x03b8fc (0x3238 bytes) 000528 1780852794.322154 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8fd): Mirror 15: 0xd6d50df4 (stale) Mirror 17: 0xd6d50df4 (stale) Mirror 22: 0xd6d50df4 (stale) Mirror 24: 0x5b5d2226 (stale) Mirror 25: 0xb3f59866 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8fd): Mirror 15: 0xd6d50df4 (stale) Mirror 17: 0xd6d50df4 (stale) Mirror 22: 0xd6d50df4 (stale) Mirror 24: 0x5b5d2226 (stale) Mirror 25: 0xb3f59866 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8fd: skipped 000530 1780852794.841323 mapread 0x00e272 thru 0x0135e5 (0x5374 bytes) 000532 1780852794.855226 trunc from 0x03b8fd to 0x038484 (0x3478 bytes) 000533 1780852794.875396 read 0x000f62 thru 0x00a4cc (0x956b bytes) 000534 1780852794.882496 write 0x02f8b4 thru 0x0336f5 (0x3e42 bytes) 000535 1780852794.895924 trunc from 0x038485 to 0x0360e1 (0x23a3 bytes) 000536 1780852794.917865 trunc from 0x0360e2 to 0x01c95d (0x19784 bytes) 000537 1780852794.950726 trunc from 0x01c95e to 0x039eda (0x1d57d bytes) 000539 1780852795.003535 write 0x008060 thru 0x00dace (0x5a6f bytes) 000540 1780852795.005907 read 0x029818 thru 0x039038 (0xf821 bytes) 000542 1780852795.021191 read 0x000ef4 thru 0x00979a (0x88a7 bytes) 000543 1780852795.022048 trunc from 0x039edb to 0x00b7c6 (0x2e714 bytes) 000545 1780852795.039328 read 0x006d8c thru 0x00b4bb (0x4730 bytes) 000548 1780852795.041247 mapread 0x008727 thru 0x00b7c6 (0x30a0 bytes) 000549 1780852795.042503 read 0x00124b thru 0x00b7c6 (0xa57c bytes) 000550 1780852795.043030 trunc from 0x00b7c7 to 0x0140c4 (0x88fe bytes) 000551 1780852795.056292 write 0x01a0ac thru 0x02973f (0xf694 bytes) 000552 1780852795.060180 trunc from 0x029740 to 0x02bcc1 (0x2582 bytes) 000556 1780852795.090993 write 0x01c78f thru 0x02091e (0x4190 bytes) 000559 1780852795.093850 mapread 0x00d0ad thru 0x01001a (0x2f6e bytes) 000561 1780852795.120129 write 0x02485e thru 0x032b93 (0xe336 bytes) 000562 1780852795.124083 mapwrite 0x003ccc thru 0x00d12a (0x945f bytes) 000568 1780852795.341708 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32b94): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b94): Mirror 15: 0xebad68b3 (stale) Mirror 17: 0xebad68b3 (stale) Mirror 22: 0xebad68b3 (stale) Mirror 24: 0xbf07587 (stale) Mirror 25: 0x2aa26193 (stale) CRC-32 checksum value for chunk [0, 0x32b94): Mirror 1: 0xeab1159c Mirror 26: 0xeab1159c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b94: skipped 000569 1780852796.710796 read 0x02d92f thru 0x02f26b (0x193d bytes) 000570 1780852796.731868 trunc from 0x032b94 to 0x01f08b (0x13b08 bytes) 000571 1780852796.895197 mapwrite 0x01fcc0 thru 0x02ef05 (0xf246 bytes) 000572 1780852797.072559 mapwrite 0x01c56a thru 0x0206d7 (0x416e bytes) 000573 1780852797.189301 write 0x020585 thru 0x0240d7 (0x3b53 bytes) 000574 1780852797.191434 mapread 0x00d959 thru 0x019d68 (0xc410 bytes) 000575 1780852797.228633 MIRROR_RESYNC 7 mirrors 000576 1780852798.979604 MIRROR_EXTEND to 8 mirrors 000577 1780852800.037047 trunc from 0x02ef06 to 0x01c795 (0x12770 bytes) 000582 1780852800.202411 read 0x015e95 thru 0x018e6d (0x2fd9 bytes) 000583 1780852800.220284 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1c796): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c796): Mirror 15: 0xd8b20260 (stale) Mirror 17: 0xd8b20260 (stale) Mirror 22: 0xd8b20260 (stale) Mirror 24: 0xd8b20260 (stale) Mirror 25: 0xd8b20260 (stale) Mirror 26: 0xd8b20260 (stale) Mirror 27: 0xd8b20260 (stale) CRC-32 checksum value for chunk [0, 0x1c796): Mirror 1: 0xd8b20260 Mirror 28: 0xd8b20260 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c796: skipped 000584 1780852801.445434 mapwrite 0x01de84 thru 0x022e12 (0x4f8f bytes) 000586 1780852801.656872 mapwrite 0x02e395 thru 0x03990d (0xb579 bytes) 000589 1780852801.782893 mapread 0x02aa4a thru 0x03191e (0x6ed5 bytes) 000591 1780852801.785397 read 0x02509f thru 0x02ac32 (0x5b94 bytes) 000594 1780852801.796206 trunc from 0x03990e to 0x015cf6 (0x23c17 bytes) 000596 1780852801.815959 read 0x00361b thru 0x00cc3c (0x9622 bytes) truncating to largest ever: 0x3f7a9 000597 1780852801.826925 trunc from 0x015cf7 to 0x03f7a8 (0x29ab2 bytes) 000598 1780852801.854284 mapread 0x037b4d thru 0x03b837 (0x3ceb bytes) 000599 1780852801.876576 trunc from 0x03f7a9 to 0x0047cb (0x3afdd bytes) 000600 1780852801.896420 mapwrite 0x02c8a9 thru 0x03c80e (0xff66 bytes) 000601 1780852802.072430 mapread 0x03413f thru 0x037e16 (0x3cd8 bytes) 000602 1780852802.074329 mapread 0x014c09 thru 0x021456 (0xc84e bytes) 000603 1780852802.211942 MIRROR_RESYNC 9 mirrors 000604 1780852803.888458 mapread 0x010e8c thru 0x01b883 (0xa9f8 bytes) 000605 1780852803.953833 write 0x0144fa thru 0x01c02d (0x7b34 bytes) 000606 1780852804.094256 mapread 0x038ab5 thru 0x03c80e (0x3d5a bytes) 000609 1780852804.109052 write 0x01ae03 thru 0x028836 (0xda34 bytes) 000610 1780852804.115939 trunc from 0x03c80f to 0x009abe (0x32d50 bytes) 000614 1780852804.147647 mapwrite 0x03fda9 thru 0x03ffff (0x257 bytes) 000615 1780852804.240726 write 0x00180a thru 0x00b33a (0x9b31 bytes) 000616 1780852804.257897 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x84a04e50 (stale) Mirror 17: 0x84a04e50 (stale) Mirror 22: 0x84a04e50 (stale) Mirror 24: 0x84a04e50 (stale) Mirror 25: 0x84a04e50 (stale) Mirror 26: 0x84a04e50 (stale) Mirror 27: 0x84a04e50 (stale) Mirror 28: 0x84a04e50 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeba9cecb Mirror 29: 0xeba9cecb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000617 1780852805.331604 read 0x028911 thru 0x0372f6 (0xe9e6 bytes) 000618 1780852805.356527 trunc from 0x040000 to 0x02d555 (0x12aaa bytes) 000620 1780852805.483937 mapwrite 0x013636 thru 0x013e08 (0x7d3 bytes) 000622 1780852805.537770 trunc from 0x02d556 to 0x013585 (0x19fd0 bytes) 000624 1780852805.561032 write 0x00f3e1 thru 0x0167cb (0x73eb bytes) 000626 1780852805.570147 mapwrite 0x00bd04 thru 0x01aa71 (0xed6e bytes) 000627 1780852805.670505 read 0x00c5a5 thru 0x019e29 (0xd885 bytes) 000628 1780852805.671267 trunc from 0x01aa72 to 0x03b5c4 (0x20b53 bytes) 000630 1780852805.711343 mapread 0x01273f thru 0x019fbb (0x787d bytes) 000632 1780852805.713064 read 0x01d53c thru 0x026e32 (0x98f7 bytes) 000633 1780852805.736337 read 0x020a07 thru 0x028564 (0x7b5e bytes) 000634 1780852805.738327 write 0x00ec83 thru 0x01895b (0x9cd9 bytes) 000635 1780852805.741651 write 0x02fa3a thru 0x03d591 (0xdb58 bytes) 000637 1780852805.746117 trunc from 0x03d592 to 0x0085c2 (0x34fcf bytes) 000638 1780852805.773106 mapwrite 0x035f12 thru 0x038c8f (0x2d7e bytes) 000640 1780852805.858826 write 0x015765 thru 0x016a83 (0x131f bytes) 000643 1780852805.868208 mapwrite 0x027c8c thru 0x028ab3 (0xe28 bytes) 000645 1780852805.985353 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c90): Mirror 15: 0xb9102f85 (stale) Mirror 17: 0xb9102f85 (stale) Mirror 22: 0xb9102f85 (stale) Mirror 25: 0xb9102f85 (stale) Mirror 26: 0xb9102f85 (stale) Mirror 27: 0xb9102f85 (stale) Mirror 28: 0xb9102f85 (stale) Mirror 29: 0x66a962f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c90): Mirror 15: 0xb9102f85 (stale) Mirror 17: 0xb9102f85 (stale) Mirror 22: 0xb9102f85 (stale) Mirror 25: 0xb9102f85 (stale) Mirror 26: 0xb9102f85 (stale) Mirror 27: 0xb9102f85 (stale) Mirror 28: 0xb9102f85 (stale) Mirror 29: 0x66a962f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c90: skipped 000651 1780852806.726371 mapread 0x02ad0d thru 0x02d534 (0x2828 bytes) 000654 1780852806.766502 trunc from 0x038c90 to 0x02bfdb (0xccb4 bytes) 000655 1780852806.796603 mapread 0x0148c9 thru 0x01c04c (0x7784 bytes) 000657 1780852806.820226 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfdc): Mirror 15: 0x9fdcdc70 (stale) Mirror 17: 0x9fdcdc70 (stale) Mirror 25: 0x9fdcdc70 (stale) Mirror 26: 0x9fdcdc70 (stale) Mirror 27: 0x9fdcdc70 (stale) Mirror 28: 0x9fdcdc70 (stale) Mirror 29: 0x5e43d6c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfdc): Mirror 15: 0x9fdcdc70 (stale) Mirror 17: 0x9fdcdc70 (stale) Mirror 25: 0x9fdcdc70 (stale) Mirror 26: 0x9fdcdc70 (stale) Mirror 27: 0x9fdcdc70 (stale) Mirror 28: 0x9fdcdc70 (stale) Mirror 29: 0x5e43d6c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfdc: skipped 000658 1780852807.451521 mapwrite 0x00fdba thru 0x01d481 (0xd6c8 bytes) 000660 1780852807.715374 read 0x016229 thru 0x0224df (0xc2b7 bytes) 000662 1780852807.721088 write 0x0281e1 thru 0x031041 (0x8e61 bytes) 000666 1780852807.770871 read 0x00b5d0 thru 0x019ecb (0xe8fc bytes) 000667 1780852807.784625 MIRROR_RESYNC 8 mirrors 000669 1780852809.837773 write 0x017ebd thru 0x0239b0 (0xbaf4 bytes) 000670 1780852810.042348 mapwrite 0x034b16 thru 0x03f63d (0xab28 bytes) 000671 1780852810.142929 trunc from 0x03f63e to 0x03274c (0xcef1 bytes) 000672 1780852810.181087 trunc from 0x03274d to 0x005984 (0x2cdc8 bytes) 000675 1780852810.231795 write 0x0283fa thru 0x0361f0 (0xddf7 bytes) 000680 1780852810.256138 read 0x018272 thru 0x018d27 (0xab6 bytes) 000683 1780852810.269387 read 0x004d93 thru 0x00e549 (0x97b7 bytes) 000685 1780852810.333818 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x361f1): Mirror 15: 0xd76da085 (stale) Mirror 25: 0xd76da085 (stale) Mirror 26: 0xd76da085 (stale) Mirror 27: 0xd76da085 (stale) Mirror 28: 0xd76da085 (stale) Mirror 29: 0xd76da085 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361f1): Mirror 15: 0xd76da085 (stale) Mirror 25: 0xd76da085 (stale) Mirror 26: 0xd76da085 (stale) Mirror 27: 0xd76da085 (stale) Mirror 28: 0xd76da085 (stale) Mirror 29: 0xd76da085 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361f1: skipped 000686 1780852811.159395 write 0x0387c4 thru 0x03c86a (0x40a7 bytes) 000687 1780852811.160619 mapread 0x00e6f1 thru 0x012826 (0x4136 bytes) 000688 1780852811.194091 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c86b): Mirror 15: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c86b): Mirror 15: 0xba8e8217 (stale) Mirror 25: 0xba8e8217 (stale) Mirror 26: 0xba8e8217 (stale) Mirror 27: 0xba8e8217 (stale) Mirror 28: 0xba8e8217 (stale) Mirror 29: 0xba8e8217 (stale) CRC-32 checksum value for chunk [0, 0x3c86b): Mirror 1: 0x5a1d529a Mirror 30: 0x5a1d529a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c86b: skipped 000690 1780852812.215911 mapwrite 0x00fc0f thru 0x01076c (0xb5e bytes) 000691 1780852812.462350 mapwrite 0x007f9a thru 0x00c13c (0x41a3 bytes) 000692 1780852812.602130 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c86b): Mirror 15: 0xba8e8217 (stale) Mirror 25: 0xba8e8217 (stale) Mirror 26: 0xba8e8217 (stale) Mirror 27: 0xba8e8217 (stale) Mirror 28: 0xba8e8217 (stale) Mirror 29: 0xba8e8217 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c86b): Mirror 15: 0xba8e8217 (stale) Mirror 25: 0xba8e8217 (stale) Mirror 26: 0xba8e8217 (stale) Mirror 27: 0xba8e8217 (stale) Mirror 28: 0xba8e8217 (stale) Mirror 29: 0xba8e8217 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c86b: skipped 000693 1780852813.243765 trunc from 0x03c86b to 0x01eeee (0x1d97c bytes) 000699 1780852813.305858 trunc from 0x01eeef to 0x0012fd (0x1dbf1 bytes) 000702 1780852813.337465 trunc from 0x0012fe to 0x02a1b4 (0x28eb7 bytes) 000703 1780852813.375879 trunc from 0x02a1b5 to 0x0301fe (0x604a bytes) 000704 1780852813.404538 write 0x0025a9 thru 0x007989 (0x53e1 bytes) 000708 1780852813.421650 mapread 0x021899 thru 0x02743a (0x5ba2 bytes) 000709 1780852813.431563 mapread 0x01a544 thru 0x01dfe1 (0x3a9e bytes) 000711 1780852813.433138 write 0x03177a thru 0x03ebc2 (0xd449 bytes) 000712 1780852813.435558 mapread 0x03ac5c thru 0x03b988 (0xd2d bytes) 000714 1780852813.438316 mapwrite 0x032151 thru 0x03ffff (0xdeaf bytes) 000717 1780852813.598245 mapread 0x019fa1 thru 0x023b22 (0x9b82 bytes) 000723 1780852813.615507 read 0x03ffc1 thru 0x03ffff (0x3f bytes) 000724 1780852813.616708 mapwrite 0x01f950 thru 0x02924e (0x98ff bytes) 000725 1780852813.690281 trunc from 0x040000 to 0x03bcad (0x4352 bytes) 000726 1780852813.709060 mapread 0x02d190 thru 0x03190d (0x477e bytes) 000728 1780852813.727976 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bcae): Mirror 15: 0x1c6126a2 (stale) Mirror 25: 0x1c6126a2 (stale) Mirror 26: 0x1c6126a2 (stale) Mirror 27: 0x1c6126a2 (stale) Mirror 28: 0x1c6126a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bcae): Mirror 15: 0x1c6126a2 (stale) Mirror 25: 0x1c6126a2 (stale) Mirror 26: 0x1c6126a2 (stale) Mirror 27: 0x1c6126a2 (stale) Mirror 28: 0x1c6126a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bcae: skipped 000730 1780852814.230092 mapwrite 0x003fa2 thru 0x011c3a (0xdc99 bytes) 000731 1780852814.394159 mapwrite 0x02d95e thru 0x03412d (0x67d0 bytes) 000733 1780852814.482272 trunc from 0x03bcae to 0x027288 (0x14a25 bytes) 000736 1780852814.512779 mapread 0x0116b8 thru 0x014372 (0x2cbb bytes) 000737 1780852814.515879 trunc from 0x027289 to 0x01050c (0x16d7c bytes) 000738 1780852814.541522 read 0x001033 thru 0x007642 (0x6610 bytes) 000739 1780852814.545094 mapwrite 0x001ebf thru 0x00636a (0x44ac bytes) 000741 1780852814.610502 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1050d): Mirror 15: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1050d): Mirror 15: 0x695bbff (stale) Mirror 25: 0x695bbff (stale) Mirror 26: 0x695bbff (stale) Mirror 27: 0x695bbff (stale) Mirror 28: 0x695bbff (stale) CRC-32 checksum value for chunk [0, 0x1050d): Mirror 1: 0x7b3f2460 Mirror 29: 0x7b3f2460 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1050d: skipped 000743 1780852815.732332 mapread 0x00d9cf thru 0x01050c (0x2b3e bytes) 000744 1780852815.772637 read 0x003fc9 thru 0x00b72c (0x7764 bytes) 000745 1780852815.791496 mapread 0x00f705 thru 0x01050c (0xe08 bytes) 000746 1780852815.811368 mapwrite 0x010234 thru 0x01bed8 (0xbca5 bytes) 000748 1780852816.019750 mapwrite 0x013281 thru 0x0133d4 (0x154 bytes) 000749 1780852816.088478 trunc from 0x01bed9 to 0x0032a3 (0x18c35 bytes) 000753 1780852816.116724 mapwrite 0x003357 thru 0x009579 (0x6223 bytes) 000754 1780852816.254834 trunc from 0x00957a to 0x0032f9 (0x6280 bytes) 000755 1780852816.279364 mapwrite 0x00638c thru 0x0142b1 (0xdf26 bytes) 000756 1780852816.379131 mapwrite 0x038cbc thru 0x03ffff (0x7344 bytes) 000758 1780852816.466909 read 0x0321e6 thru 0x03925c (0x7077 bytes) 000759 1780852816.468042 read 0x013011 thru 0x0149ac (0x199c bytes) 000760 1780852816.470546 write 0x0304c7 thru 0x03ffff (0xfb39 bytes) 000761 1780852816.486410 trunc from 0x040000 to 0x0147c6 (0x2b839 bytes) 000762 1780852816.509800 mapwrite 0x02b075 thru 0x03278b (0x7717 bytes) 000765 1780852816.633345 mapwrite 0x01f6e9 thru 0x02c373 (0xcc8b bytes) 000766 1780852816.764482 mapread 0x003c08 thru 0x00904e (0x5447 bytes) 000768 1780852816.766627 write 0x03756c thru 0x03eee0 (0x7975 bytes) 000769 1780852816.768768 write 0x03ca0d thru 0x03ffff (0x35f3 bytes) 000775 1780852816.780146 mapread 0x01577c thru 0x024c0f (0xf494 bytes) 000776 1780852816.871715 mapwrite 0x034d60 thru 0x0354ca (0x76b bytes) 000778 1780852816.978862 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0xd79e558f (stale) Mirror 26: 0xd79e558f (stale) Mirror 27: 0xd79e558f (stale) Mirror 28: 0xd79e558f (stale) Mirror 29: 0xd7878087 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0xd79e558f (stale) Mirror 26: 0xd79e558f (stale) Mirror 27: 0xd79e558f (stale) Mirror 28: 0xd79e558f (stale) Mirror 29: 0xd7878087 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000779 1780852817.275920 write 0x028ac4 thru 0x034311 (0xb84e bytes) 000782 1780852817.294562 mapwrite 0x02997d thru 0x037f8e (0xe612 bytes) 000783 1780852817.373036 read 0x02be2c thru 0x039e4d (0xe022 bytes) 000785 1780852817.374651 read 0x022d1f thru 0x0236aa (0x98c bytes) 000786 1780852817.390794 write 0x00331d thru 0x005ae8 (0x27cc bytes) 000787 1780852817.392313 trunc from 0x040000 to 0x03dbce (0x2431 bytes) 000790 1780852817.411064 mapread 0x01e227 thru 0x02d0e1 (0xeebb bytes) 000791 1780852817.413121 trunc from 0x03dbcf to 0x03af47 (0x2c87 bytes) 000793 1780852817.438710 trunc from 0x03af48 to 0x007d82 (0x331c5 bytes) 000794 1780852817.467167 MIRROR_RESYNC 6 mirrors 000796 1780852818.674141 write 0x0154dc thru 0x01c275 (0x6d9a bytes) 000797 1780852818.728942 read 0x00be44 thru 0x00c0a8 (0x265 bytes) 000798 1780852818.743222 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c276): Mirror 25: 0xb3de38df (stale) Mirror 26: 0xb3de38df (stale) Mirror 27: 0xb3de38df (stale) Mirror 29: 0xb3de38df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c276): Mirror 25: 0xb3de38df (stale) Mirror 26: 0xb3de38df (stale) Mirror 27: 0xb3de38df (stale) Mirror 29: 0xb3de38df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c276: skipped 000799 1780852819.169715 mapread 0x010f1f thru 0x016d27 (0x5e09 bytes) 000801 1780852819.193856 write 0x03be3a thru 0x03ffff (0x41c6 bytes) 000804 1780852819.197338 trunc from 0x040000 to 0x025d34 (0x1a2cb bytes) 000805 1780852819.217064 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25d35): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25d35): Mirror 25: 0x7bc226b8 (stale) Mirror 26: 0x7bc226b8 (stale) Mirror 27: 0x7bc226b8 (stale) Mirror 29: 0x7bc226b8 (stale) CRC-32 checksum value for chunk [0, 0x25d35): Mirror 1: 0x9856c062 Mirror 30: 0x9856c062 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25d35: skipped 000806 1780852819.955954 read 0x01fa05 thru 0x025d34 (0x6330 bytes) 000807 1780852819.971241 write 0x0324cc thru 0x03a077 (0x7bac bytes) 000808 1780852820.077445 read 0x016e01 thru 0x01954a (0x274a bytes) 000811 1780852820.102202 mapread 0x01926e thru 0x01a9eb (0x177e bytes) 000813 1780852820.103401 read 0x005e46 thru 0x011091 (0xb24c bytes) 000814 1780852820.104086 mapread 0x005498 thru 0x00bcab (0x6814 bytes) 000817 1780852820.106133 read 0x030515 thru 0x033673 (0x315f bytes) 000820 1780852820.107627 mapread 0x005011 thru 0x007272 (0x2262 bytes) 000821 1780852820.113208 MIRROR_RESYNC 6 mirrors 000822 1780852821.401346 mapread 0x027532 thru 0x02db3f (0x660e bytes) 000823 1780852821.431823 write 0x02104b thru 0x02ab34 (0x9aea bytes) 000824 1780852821.511513 mapwrite 0x004c94 thru 0x00f790 (0xaafd bytes) 000825 1780852821.608558 mapread 0x030309 thru 0x037ea1 (0x7b99 bytes) 000826 1780852821.636923 mapwrite 0x02a172 thru 0x034b62 (0xa9f1 bytes) 000829 1780852821.708240 read 0x002b2d thru 0x00e935 (0xbe09 bytes) 000833 1780852821.710862 trunc from 0x03a078 to 0x022779 (0x178fe bytes) 000834 1780852821.769256 read 0x013dc3 thru 0x016595 (0x27d3 bytes) 000835 1780852821.793022 mapwrite 0x0190df thru 0x01fd1a (0x6c3c bytes) 000837 1780852821.851453 MIRROR_RESYNC 6 mirrors 000838 1780852822.788544 MIRROR_RESYNC 6 mirrors 000839 1780852823.233107 read 0x005c45 thru 0x00d2f3 (0x76af bytes) 000841 1780852823.255896 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000843 1780852823.868951 write 0x025d5a thru 0x029b92 (0x3e39 bytes) 000844 1780852823.953530 mapwrite 0x006d85 thru 0x01455a (0xd7d6 bytes) 000845 1780852824.112898 read 0x008b03 thru 0x012b66 (0xa064 bytes) 000846 1780852824.115036 mapwrite 0x0023fc thru 0x00aff4 (0x8bf9 bytes) 000848 1780852824.230118 write 0x027a3d thru 0x034499 (0xca5d bytes) 000849 1780852824.234070 read 0x005376 thru 0x00dc7a (0x8905 bytes) 000850 1780852824.235824 mapread 0x002711 thru 0x0095f1 (0x6ee1 bytes) 000853 1780852824.250124 read 0x02ab10 thru 0x034499 (0x998a bytes) 000854 1780852824.252447 read 0x01e83f thru 0x026e65 (0x8627 bytes) 000856 1780852824.270780 trunc from 0x03449a to 0x033cd7 (0x7c2 bytes) 000857 1780852824.297224 mapwrite 0x0342ee thru 0x035d05 (0x1a18 bytes) 000860 1780852824.382600 read 0x004b15 thru 0x010212 (0xb6fe bytes) 000864 1780852824.388315 write 0x021eaa thru 0x02859a (0x66f1 bytes) 000865 1780852824.390477 mapwrite 0x01f78d thru 0x024b72 (0x53e6 bytes) 000866 1780852824.433523 write 0x01c99f thru 0x02b8c8 (0xef2a bytes) 000867 1780852824.436424 write 0x033c25 thru 0x035860 (0x1c3c bytes) 000869 1780852824.438289 mapread 0x011014 thru 0x016966 (0x5953 bytes) 000871 1780852824.440945 mapwrite 0x01fb3a thru 0x022fba (0x3481 bytes) 000873 1780852824.546708 trunc from 0x035d06 to 0x02bd7f (0x9f86 bytes) 000875 1780852824.570633 trunc from 0x02bd80 to 0x028ecd (0x2eb2 bytes) 000877 1780852824.591245 trunc from 0x028ece to 0x01306d (0x15e60 bytes) 000878 1780852824.617722 MIRROR_RESYNC 5 mirrors 000879 1780852825.795181 trunc from 0x01306e to 0x004ca8 (0xe3c5 bytes) 000880 1780852825.886012 write 0x00993c thru 0x00b7c8 (0x1e8d bytes) 000882 1780852825.889518 mapwrite 0x03b15c thru 0x03ffff (0x4ea4 bytes) 000885 1780852825.973018 mapread 0x020dea thru 0x028dab (0x7fc2 bytes) 000887 1780852825.987250 write 0x019f57 thru 0x022ced (0x8d97 bytes) 000889 1780852825.989937 read 0x034dee thru 0x03e689 (0x989c bytes) 000890 1780852825.996203 write 0x02d2aa thru 0x038656 (0xb3ad bytes) 000891 1780852826.008181 trunc from 0x040000 to 0x000c7b (0x3f384 bytes) 000892 1780852826.029148 mapread 0x00062e thru 0x000c7b (0x64e bytes) 000894 1780852826.054325 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc7c): Mirror 26: 0xb9756c10 (stale) Mirror 27: 0xb9756c10 (stale) Mirror 29: 0xb9756c10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc7c): Mirror 26: 0xb9756c10 (stale) Mirror 27: 0xb9756c10 (stale) Mirror 29: 0xb9756c10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc7c: skipped 000896 1780852826.500139 write 0x01d6f1 thru 0x025490 (0x7da0 bytes) 000897 1780852826.502293 mapwrite 0x00456f thru 0x01399e (0xf430 bytes) 000898 1780852826.678058 read 0x01a95c thru 0x025490 (0xab35 bytes) 000899 1780852826.700473 write 0x03eddd thru 0x03ffff (0x1223 bytes) 000900 1780852826.711689 read 0x02366e thru 0x02b557 (0x7eea bytes) 000902 1780852826.733599 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0x7af875ea (stale) Mirror 29: 0x7af875ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0x7af875ea (stale) Mirror 29: 0x7af875ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000904 1780852827.287778 read 0x030635 thru 0x031576 (0xf42 bytes) 000905 1780852827.296067 trunc from 0x040000 to 0x02afd5 (0x1502a bytes) 000907 1780852827.318841 write 0x02bd19 thru 0x0383a9 (0xc691 bytes) 000908 1780852827.323256 mapwrite 0x0264e1 thru 0x0289d3 (0x24f3 bytes) 000909 1780852827.402627 write 0x020a63 thru 0x02db70 (0xd10e bytes) 000910 1780852827.420267 mapwrite 0x01ac48 thru 0x01b156 (0x50f bytes) 000911 1780852827.505228 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x383aa): Mirror 29: 0xeccaf862 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383aa): Mirror 29: 0xeccaf862 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383aa: skipped 000913 1780852827.998025 mapread 0x007307 thru 0x00df8a (0x6c84 bytes) 000916 1780852828.034977 mapwrite 0x034bb2 thru 0x03d5b3 (0x8a02 bytes) 000917 1780852828.178718 write 0x0386e8 thru 0x03ffff (0x7918 bytes) 000918 1780852828.180614 mapwrite 0x0027c6 thru 0x0034e4 (0xd1f bytes) 000921 1780852828.250135 mapread 0x03b2f4 thru 0x03ffff (0x4d0c bytes) 000922 1780852828.252494 mapread 0x00873a thru 0x0112a6 (0x8b6d bytes) 000924 1780852828.255952 mapwrite 0x001600 thru 0x00f6fe (0xe0ff bytes) 000926 1780852828.322985 read 0x017361 thru 0x02117b (0x9e1b bytes) 000927 1780852828.350744 trunc from 0x040000 to 0x0137fc (0x2c803 bytes) 000930 1780852828.389982 trunc from 0x0137fd to 0x000fbd (0x1283f bytes) 000932 1780852828.434175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000933 1780852828.597099 MIRROR_EXTEND to 2 mirrors 000934 1780852829.100162 mapwrite 0x01958f thru 0x01c2c7 (0x2d39 bytes) 000937 1780852829.263752 trunc from 0x01c2c8 to 0x03ebe6 (0x2291f bytes) 000939 1780852829.304178 trunc from 0x03ebe7 to 0x01c789 (0x2245d bytes) 000940 1780852829.333090 trunc from 0x01c78a to 0x0239be (0x7235 bytes) 000941 1780852829.357506 read 0x00163c thru 0x003224 (0x1be9 bytes) 000942 1780852829.364332 read 0x00befa thru 0x01898e (0xca95 bytes) 000943 1780852829.379233 trunc from 0x0239bf to 0x015a0d (0xdfb1 bytes) 000944 1780852829.404941 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000945 1780852829.577088 mapread 0x0083ac thru 0x00abac (0x2801 bytes) 000947 1780852829.595585 mapread 0x001cd7 thru 0x007dda (0x6104 bytes) 000948 1780852829.602507 mapwrite 0x01ced0 thru 0x025d78 (0x8ea9 bytes) 000951 1780852829.678771 trunc from 0x025d79 to 0x02ef27 (0x91af bytes) 000955 1780852829.707805 write 0x010ba1 thru 0x01681f (0x5c7f bytes) 000956 1780852829.710225 mapwrite 0x007c4a thru 0x012f0d (0xb2c4 bytes) 000957 1780852829.803734 mapwrite 0x012f01 thru 0x022b0f (0xfc0f bytes) 000960 1780852829.873176 write 0x0093b1 thru 0x00d8d5 (0x4525 bytes) 000962 1780852829.876200 trunc from 0x02ef28 to 0x033e9b (0x4f74 bytes) 000964 1780852829.905483 trunc from 0x033e9c to 0x039d7a (0x5edf bytes) 000965 1780852829.936637 mapwrite 0x02705b thru 0x031234 (0xa1da bytes) 000966 1780852830.088442 trunc from 0x039d7b to 0x03dbe4 (0x3e6a bytes) 000967 1780852830.111582 trunc from 0x03dbe5 to 0x033718 (0xa4cc bytes) 000968 1780852830.142834 mapread 0x00174a thru 0x009414 (0x7ccb bytes) 000969 1780852830.152284 write 0x011241 thru 0x01df9d (0xcd5d bytes) 000973 1780852830.165078 read 0x02c805 thru 0x02ccf6 (0x4f2 bytes) 000974 1780852830.166075 mapwrite 0x026a82 thru 0x02ca72 (0x5ff1 bytes) 000976 1780852830.232148 mapread 0x019f02 thru 0x0283d3 (0xe4d2 bytes) 000977 1780852830.235113 write 0x020ea8 thru 0x02fd18 (0xee71 bytes) 000978 1780852830.237598 write 0x00b8bc thru 0x00ca88 (0x11cd bytes) 000980 1780852830.243144 trunc from 0x033719 to 0x00992d (0x29deb bytes) 000982 1780852830.263344 mapwrite 0x0396b4 thru 0x03c846 (0x3193 bytes) 000983 1780852830.345878 read 0x03b37e thru 0x03c846 (0x14c9 bytes) 000988 1780852830.350792 read 0x003f6b thru 0x00f6cc (0xb762 bytes) 000989 1780852830.371893 write 0x0054d4 thru 0x0153b0 (0xfedd bytes) 000990 1780852830.373115 mapwrite 0x01c87e thru 0x0293d3 (0xcb56 bytes) 000992 1780852830.465838 write 0x02f6e2 thru 0x037fa1 (0x88c0 bytes) 000993 1780852830.467013 mapread 0x02ed24 thru 0x03b8dc (0xcbb9 bytes) 000995 1780852830.469062 write 0x03f778 thru 0x03ffff (0x888 bytes) 000998 1780852830.477466 MIRROR_EXTEND to 2 mirrors 001000 1780852831.146121 mapread 0x02ed75 thru 0x033985 (0x4c11 bytes) All operations completed A-OK! PASS 70a (133s) == sanity-flr test 71: check mirror extend parent fid ==== 13:20:36 (1780852836) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 13:20:40 (1780852840) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.310724 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178256 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 13:20:49 (1780852849) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4198976 .. Extending file size to 4149792 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7189920 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9282720 .. Extending file size to 7617888 .. Extending file size to 8893568 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9143328 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9333792 .. Waiting 56665 56667 56668 56669 56670 done 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) 192.168.203.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:22:12 (1780852932) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3524128 .. Extending file size to 6788928 .. failed Extending file size to 9079072 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7134528 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9192192 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9328768 .. Waiting 58078 58079 58081 58083 58084 done 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) 192.168.203.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:23:38 (1780853018) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141549 s, 14.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.446319 s, 4.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x47845dd2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:23:57 (1780853037) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0659572 s, 31.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 13:24:03 (1780853043) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1df:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:24:09 (1780853049) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267698 s, 7.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:24:16 (1780853056) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248091 s, 8.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:24:25 (1780853065) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299704 s, 7.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:24:58 (1780853098) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18158 s, 5.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:25:53 (1780853153) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205616 s, 10.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:26:51 (1780853211) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. Creating new pool test_204f oleg302-server: Pool lustre.test_204f created Adding targets to pool oleg302-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg302-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.253524 s, 8.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg302-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg302-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg302-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:28:10 (1780853290) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:28:18 (1780853298) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:28:25 (1780853305) oleg302-server: Pool lustre.test_206 created Adding targets to pool oleg302-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg302-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg302-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg302-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg302-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:28:47 (1780853327) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575074 s, 18.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300303 s, 34.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc5a5f970 Mirror 3: 0xc5a5f970 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:28:54 (1780853334) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:28:57 (1780853337) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:29:00 (1780853340) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.115267 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2360000 Read this (4 bytes): BBBB 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:29:35 (1780853375) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00844744 s, 970 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:29:41 (1780853381) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243805 s, 43.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x204 lmm_fid: [0x200000404:0x204:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x204 lmm_fid: [0x200000404:0x204:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834436 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 13:29:43.000000000 -0400 Modify: 2026-06-07 13:29:43.000000000 -0400 Change: 2026-06-07 13:29:44.000000000 -0400 Birth: 2026-06-07 13:29:42.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:29:49 (1780853389) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:29:52 (1780853392) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.522913 s, 78.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0511495 s, 80.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 7 13:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x206:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 7 13:29 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:29:59 (1780853399) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:30:05 (1780853405) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1757 sec =========== 13:30:13 (1780853413) === sanity-flr: start cleanup 13:30:14 (1780853414) === === sanity-flr: finish cleanup 13:30:20 (1780853420) ===