-----============= acceptance-small: sanity-flr ============----- Wed May 20 16:52:21 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg444-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:52:35 (1779310355) === oleg444-client.virtnet: executing check_config_client /mnt/lustre oleg444-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg444-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f8c42fb7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f8c42fb7000.idle_timeout=debug disable quota as required oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:52:57 (1779310377) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8367] == sanity-flr test 0a: lfs mirror create with -N option == 16:52:59 (1779310379) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:53:10 (1779310390) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:53:13 (1779310393) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:53:15 (1779310395) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:53:34 (1779310414) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:53:36 (1779310416) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 16:53:39 (1779310419) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f8c42fb7000.stats=clear osc.lustre-OST0001-osc-ffff9f8c42fb7000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:53:45 (1779310425) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:53:53 (1779310433) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:53:58 (1779310438) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:54:00 (1779310440) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:54:06 (1779310446) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 16:54:12 (1779310452) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:54:17 (1779310457) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:54:21 (1779310461) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:54:26 (1779310466) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:54:32 (1779310472) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:54:37 (1779310477) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145036 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089343 s, 11.7 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:54:42 (1779310482) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.264491 s, 67.4 MB/s PASS 22 (5s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:54:47 (1779310487) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:54:54 (1779310494) Stopping /mnt/lustre-ost1 (opts:) on oleg444-server oleg444-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg444-server: oleg444-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg444-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg444-server oleg444-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg444-server: oleg444-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg444-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (46s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:55:41 (1779310541) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0864828 s, 24.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg444-server oleg444-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg444-server: oleg444-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg444-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (24s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:56:04 (1779310564) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg444-server oleg444-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg444-server: oleg444-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg444-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg444-server oleg444-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg444-server: oleg444-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg444-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (49s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:56:53 (1779310613) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.61327 s, 14.5 MB/s -rw-r--r-- 1 root root 52428800 May 20 16:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.725s user 0m0.000s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg444-server oleg444-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f8c42fb7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.675s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg444-server: oleg444-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg444-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg444-server oleg444-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.962s user 0m0.000s sys 0m0.175s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg444-server: oleg444-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg444-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f8c42fb7000.ost_server_uuid in FULL state after 0 sec PASS 33b (58s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:57:51 (1779310671) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:57:53 (1779310673) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:57:56 (1779310676) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:57:58 (1779310678) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 16:58:04 (1779310684) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.464263 s, 27.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:58:13 (1779310693) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185336 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312393 s, 33.6 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:58:17 (1779310697) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:58:23 (1779310703) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00156198 s, 656 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00575759 s, 178 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00209936 s, 488 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107498 s, 953 kB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 16:58:30 (1779310710) create test files with size 15755831 .. c48831b6622c73c9282023cbd9ce7017 - ae3857b14416839199a5156eb90da37c - 3cc8fbd080fe095858ce923ada6f1c71 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg444-client.virtnet /mnt/lustre (opts:) Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre PASS 37 (15s) == sanity-flr test 38: resync ============================ 16:58:45 (1779310725) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (10s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:58:56 (1779310736) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:59:00 (1779310740) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583528 s, 18.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178024 s, 58.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 16:59:05 (1779310745) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104256 s, 20.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.259873 s, 16.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138151 s, 15.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (11s) == sanity-flr test 42: lfs mirror verify ================= 16:59:16 (1779310756) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 16:59:19 (1779310759) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:59:21 (1779310761) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017894 s, 58.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 16:59:27 (1779310767) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.2627 s, 12.0 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:59:35 (1779310775) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:59:40 (1779310780) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.154601 s, 67.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg444-client.virtnet /mnt/lustre (opts:) Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (6s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:59:47 (1779310787) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f8c42cf8800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.113431 s, 92.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg444-client.virtnet /mnt/lustre (opts:) Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff9f8c51153000.xattr_cache=1 PASS 44e (8s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:59:55 (1779310795) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (24s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:00:19 (1779310819) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (23s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:00:42 (1779310842) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 17:00:43 (1779310843) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.130012 s, 24.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.107179 s, 29.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3bb73647 (nosync,stale) Mirror 2: 0x3bb73647 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 80ea45b5787241bd47cd42162216af4b - mirror 1 checksum: bd7e7008c75c707cdd46be2ae537a5de - mirror 2 checksum: 80ea45b5787241bd47cd42162216af4b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 20 17:00:45 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 20 17:00:45 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 80ea45b5787241bd47cd42162216af4b - mirror 1 checksum: bd7e7008c75c707cdd46be2ae537a5de - mirror 2 checksum: bd7e7008c75c707cdd46be2ae537a5de - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 20 17:00:45 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 20 17:00:45 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (6s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:00:50 (1779310850) Failing mds1 on oleg444-server Stopping /mnt/lustre-mds1 (opts:) on oleg444-server 17:00:55 (1779310855) shut down facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server Failover mds1 to oleg444-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 17:01:12 (1779310872) targets are mounted 17:01:12 (1779310872) facet_failover done oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020014 s, 52.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:01:21 (1779310881) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:01:23 (1779310883) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:01:25 (1779310885) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0132175 s, 1.2 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:01:30 (1779310890) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:01:32 (1779310892) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:01:34 (1779310894) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) PASS 60b (16s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:01:50 (1779310910) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (23s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:02:13 (1779310933) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:02:33 (1779310953) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:02:54 (1779310974) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 17:02:58 (1779310978) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55528 mirror split pid 55529 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7730: 55528 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6904: 55529 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6904: 55529 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 55529 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 17:04:03 (1779311043) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 1044 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x18e0c 000002 1779311044.729127 trunc from 00000000 to 0x018e0b (0x18e0c bytes) 000005 1779311044.748917 mapread 0x005133 thru 0x01427a (0xf148 bytes) 000006 1779311044.769946 trunc from 0x018e0c to 0x001650 (0x177bb bytes) 000007 1779311044.780933 mapread 0x00002d thru 0x001650 (0x1624 bytes) 000009 1779311044.789160 write 0x01faaa thru 0x02ec78 (0xf1cf bytes) 000014 1779311044.795644 mapwrite 0x00c704 thru 0x0107d4 (0x40d1 bytes) 000015 1779311044.871474 mapwrite 0x0119d7 thru 0x014f7c (0x35a6 bytes) 000017 1779311044.906294 read 0x0049bc thru 0x00f858 (0xae9d bytes) 000019 1779311044.911496 write 0x024a3d thru 0x03238f (0xd953 bytes) 000020 1779311044.918075 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x1e34e 000022 1779311045.524071 trunc from 0x032390 to 0x01e34d (0x14042 bytes) 000023 1779311045.583742 mapwrite 0x009a9b thru 0x013323 (0x9889 bytes) 000025 1779311045.661077 read 0x01606c thru 0x01d389 (0x731e bytes) 000027 1779311045.674738 mapwrite 0x0164ce thru 0x02208b (0xbbbe bytes) 000028 1779311045.762653 trunc from 0x02208c to 0x0074e8 (0x1aba3 bytes) 000029 1779311045.785428 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x74e9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x74e9): Mirror 2: 0xd2fda07b (stale) CRC-32 checksum value for chunk [0, 0x74e9): Mirror 1: 0xd2fda07b Mirror 3: 0xd2fda07b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x74e9: skipped 000030 1779311046.450419 mapread 0x004012 thru 0x005524 (0x1513 bytes) 000032 1779311046.484018 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x74e9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x74e9): Mirror 2: 0xd2fda07b (stale) CRC-32 checksum value for chunk [0, 0x74e9): Mirror 1: 0xd2fda07b Mirror 3: 0xd2fda07b Mirror 4: 0xd2fda07b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x74e9: skipped 000033 1779311047.089907 mapwrite 0x027e32 thru 0x02978d (0x195c bytes) 000034 1779311047.244937 write 0x03e41c thru 0x03ffff (0x1be4 bytes) 000035 1779311047.253566 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x96ebbc3b (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73ef5f1c Mirror 5: 0x73ef5f1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000037 1779311047.876822 read 0x022548 thru 0x030748 (0xe201 bytes) 000039 1779311047.916561 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x96ebbc3b (stale) Mirror 4: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x96ebbc3b (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73ef5f1c Mirror 5: 0x73ef5f1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000045 1779311048.583168 mapwrite 0x02d5ec thru 0x030b40 (0x3555 bytes) 000046 1779311048.746872 read 0x03a778 thru 0x03eb0a (0x4393 bytes) 000047 1779311048.758239 read 0x02c33c thru 0x034c38 (0x88fd bytes) 000048 1779311048.769220 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x96ebbc3b (stale) Mirror 5: 0x73ef5f1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x96ebbc3b (stale) Mirror 5: 0x73ef5f1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000049 1779311049.045624 read 0x02c910 thru 0x03a6ec (0xdddd bytes) 000051 1779311049.058913 mapwrite 0x03c1c0 thru 0x03ffff (0x3e40 bytes) 000052 1779311049.127302 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x26343 000053 1779311049.720855 trunc from 0x040000 to 0x026342 (0x19cbd bytes) 000054 1779311049.818618 write 0x00573b thru 0x008c02 (0x34c8 bytes) 000056 1779311049.832790 read 0x001f73 thru 0x00507e (0x310c bytes) 000059 1779311049.839137 trunc from 0x026343 to 0x004741 (0x21c01 bytes) 000061 1779311049.855837 write 0x019afa thru 0x01d0c9 (0x35d0 bytes) 000066 1779311049.859980 read 0x0175f6 thru 0x01d0c9 (0x5ad4 bytes) 000068 1779311049.868538 write 0x030b68 thru 0x0356fc (0x4b95 bytes) 000069 1779311049.870256 mapwrite 0x000a2b thru 0x0021e9 (0x17bf bytes) 000072 1779311049.922616 mapread 0x004a04 thru 0x00efb7 (0xa5b4 bytes) 000073 1779311049.924634 mapread 0x00a520 thru 0x018077 (0xdb58 bytes) 000076 1779311049.928219 write 0x033e72 thru 0x039519 (0x56a8 bytes) 000077 1779311049.930014 mapwrite 0x02dce5 thru 0x0382a3 (0xa5bf bytes) 000078 1779311049.984056 read 0x01de80 thru 0x01e15e (0x2df bytes) 000079 1779311049.984751 write 0x03ac19 thru 0x03ffff (0x53e7 bytes) 000082 1779311049.987204 mapread 0x00d8f2 thru 0x01b903 (0xe012 bytes) truncating to largest ever: 0x365ea 000083 1779311049.989050 trunc from 0x040000 to 0x0365e9 (0x9a16 bytes) 000084 1779311050.006409 mapwrite 0x003f9d thru 0x0126d3 (0xe737 bytes) 000085 1779311050.042054 MIRROR_RESYNC 3 mirrors 000087 1779311050.512864 mapwrite 0x0005b5 thru 0x00cc7f (0xc6cb bytes) 000088 1779311050.652127 write 0x02bf82 thru 0x031126 (0x51a5 bytes) 000089 1779311050.671577 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x365ea): Mirror 2: 0x857e7d2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x365ea): Mirror 2: 0x857e7d2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x365ea: skipped 000090 1779311050.925362 read 0x016660 thru 0x017cdd (0x167e bytes) 000092 1779311050.931517 mapread 0x00763b thru 0x01579e (0xe164 bytes) truncating to largest ever: 0x3e8c1 000094 1779311050.960652 trunc from 0x0365ea to 0x03e8c0 (0x82d7 bytes) 000096 1779311050.990464 write 0x0168b4 thru 0x023909 (0xd056 bytes) 000097 1779311050.996467 write 0x02338d thru 0x029d22 (0x6996 bytes) 000099 1779311051.003335 mapwrite 0x013238 thru 0x01bb3c (0x8905 bytes) 000100 1779311051.042033 trunc from 0x03e8c1 to 0x00ce29 (0x31a97 bytes) 000102 1779311051.056284 read 0x00a379 thru 0x00ce29 (0x2ab1 bytes) 000103 1779311051.056910 mapwrite 0x004272 thru 0x007b00 (0x388f bytes) 000106 1779311051.098602 mapread 0x00ac68 thru 0x00ce29 (0x21c2 bytes) 000107 1779311051.099964 mapwrite 0x008c9c thru 0x0173b0 (0xe715 bytes) 000108 1779311051.174734 mapwrite 0x018c5d thru 0x023442 (0xa7e6 bytes) 000110 1779311051.255173 mapread 0x00fca3 thru 0x0120f4 (0x2452 bytes) 000111 1779311051.256258 write 0x0171a7 thru 0x02144e (0xa2a8 bytes) 000114 1779311051.259422 mapread 0x015155 thru 0x01e68e (0x953a bytes) 000115 1779311051.260898 trunc from 0x023443 to 0x027de2 (0x49a0 bytes) 000119 1779311051.284830 MIRROR_RESYNC 2 mirrors 000120 1779311051.744255 write 0x026469 thru 0x035988 (0xf520 bytes) 000121 1779311051.797830 read 0x0115bf thru 0x01a258 (0x8c9a bytes) 000122 1779311051.808184 trunc from 0x035989 to 0x02bb75 (0x9e13 bytes) 000125 1779311051.831148 mapwrite 0x03c483 thru 0x03ffff (0x3b7d bytes) 000129 1779311051.906618 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3872196 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xba2f1902 Mirror 3: 0xba2f1902 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000130 1779311052.467829 read 0x0179f2 thru 0x023616 (0xbc25 bytes) 000132 1779311052.489195 mapread 0x03c333 thru 0x03ffff (0x3ccd bytes) 000133 1779311052.519923 trunc from 0x040000 to 0x00b481 (0x34b7e bytes) 000138 1779311052.609084 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xb482): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb482): Mirror 2: 0xa6cf1224 (stale) Mirror 3: 0xa6cf1224 (stale) CRC-32 checksum value for chunk [0, 0xb482): Mirror 1: 0xa6cf1224 Mirror 4: 0xa6cf1224 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb482: skipped 000139 1779311053.174791 trunc from 0x00b482 to 0x001e79 (0x9608 bytes) 000142 1779311053.249013 mapwrite 0x01725b thru 0x01974b (0x24f1 bytes) 000143 1779311053.341383 write 0x004e86 thru 0x007f14 (0x308f bytes) 000144 1779311053.353660 write 0x02e772 thru 0x0314b9 (0x2d48 bytes) 000145 1779311053.354848 write 0x0015e6 thru 0x009090 (0x7aab bytes) 000147 1779311053.356775 trunc from 0x0314ba to 0x0381a9 (0x6cf0 bytes) 000150 1779311053.374122 mapwrite 0x01b6c4 thru 0x02a3e5 (0xed22 bytes) 000151 1779311053.436031 write 0x0381a4 thru 0x0386b6 (0x513 bytes) 000152 1779311053.446264 trunc from 0x0386b7 to 0x00950f (0x2f1a7 bytes) 000154 1779311053.461294 write 0x03ec72 thru 0x03ffff (0x138e bytes) 000155 1779311053.462913 read 0x02821a thru 0x03506d (0xce54 bytes) 000157 1779311053.472354 read 0x0164ea thru 0x017623 (0x113a bytes) 000158 1779311053.479865 trunc from 0x040000 to 0x01969b (0x26964 bytes) 000159 1779311053.496523 trunc from 0x01969c to 0x0358a5 (0x1c20a bytes) 000160 1779311053.517470 MIRROR_RESYNC 4 mirrors 000161 1779311054.094616 write 0x035c46 thru 0x03bcb7 (0x6072 bytes) 000162 1779311054.158631 trunc from 0x03bcb8 to 0x038887 (0x3430 bytes) 000166 1779311054.209293 trunc from 0x038888 to 0x03af7a (0x26f3 bytes) 000167 1779311054.236987 MIRROR_RESYNC 4 mirrors 000168 1779311054.952926 mapwrite 0x01bf24 thru 0x01d1a0 (0x127d bytes) 000170 1779311055.124787 write 0x00b79c thru 0x011d49 (0x65ae bytes) 000172 1779311055.138852 mapwrite 0x03f18c thru 0x03ffff (0xe74 bytes) 000173 1779311055.231251 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb77c1cbb (stale) Mirror 3: 0xb77c1cbb (stale) Mirror 4: 0xb77c1cbb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x322b0fb5 Mirror 5: 0x322b0fb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000175 1779311055.839265 read 0x035dd6 thru 0x03d773 (0x799e bytes) 000176 1779311055.855016 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb77c1cbb (stale) Mirror 3: 0xb77c1cbb (stale) Mirror 4: 0xb77c1cbb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x322b0fb5 Mirror 5: 0x322b0fb5 Mirror 6: 0x322b0fb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000180 1779311056.497086 write 0x02ff60 thru 0x03fbe8 (0xfc89 bytes) 000181 1779311056.566917 read 0x035d03 thru 0x03ffff (0xa2fd bytes) 000182 1779311056.568831 mapwrite 0x02ca15 thru 0x031282 (0x486e bytes) 000183 1779311056.625576 MIRROR_RESYNC 6 mirrors 000184 1779311057.428549 mapwrite 0x01e086 thru 0x022774 (0x46ef bytes) 000185 1779311057.577948 mapread 0x011804 thru 0x018dbb (0x75b8 bytes) 000186 1779311057.595573 mapread 0x01b911 thru 0x01d215 (0x1905 bytes) 000187 1779311057.598679 mapread 0x013a69 thru 0x018b76 (0x510e bytes) 000188 1779311057.604366 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb864c83f (stale) Mirror 4: 0xb864c83f (stale) Mirror 5: 0xb864c83f (stale) Mirror 6: 0xb864c83f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb864c83f (stale) Mirror 4: 0xb864c83f (stale) Mirror 5: 0xb864c83f (stale) Mirror 6: 0xb864c83f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000191 1779311057.922340 read 0x00cc1c thru 0x011d8f (0x5174 bytes) 000192 1779311057.934376 read 0x02ea9f thru 0x039846 (0xada8 bytes) 000193 1779311057.950832 trunc from 0x040000 to 0x02451e (0x1bae1 bytes) 000194 1779311057.971811 read 0x022528 thru 0x02451e (0x1ff7 bytes) 000195 1779311057.972595 mapwrite 0x026c68 thru 0x0356be (0xea57 bytes) 000197 1779311058.069979 mapwrite 0x038441 thru 0x03d557 (0x5117 bytes) 000199 1779311058.129475 mapwrite 0x00ef0c thru 0x01ce44 (0xdf39 bytes) 000200 1779311058.175580 write 0x00a411 thru 0x017173 (0xcd63 bytes) 000201 1779311058.177804 mapwrite 0x005362 thru 0x0070dd (0x1d7c bytes) 000202 1779311058.213005 read 0x0097f4 thru 0x00d30f (0x3b1c bytes) 000203 1779311058.213614 mapread 0x01a83e thru 0x029b21 (0xf2e4 bytes) 000205 1779311058.215763 mapwrite 0x03085b thru 0x03df7e (0xd724 bytes) 000207 1779311058.272066 mapread 0x015f2f thru 0x023f59 (0xe02b bytes) 000210 1779311058.275930 write 0x03702d thru 0x03ffff (0x8fd3 bytes) 000211 1779311058.283779 MIRROR_RESYNC 5 mirrors 000214 1779311058.985908 write 0x00d1eb thru 0x01cdfe (0xfc14 bytes) 000216 1779311059.062926 write 0x00b5f8 thru 0x013cce (0x86d7 bytes) 000218 1779311059.076524 mapwrite 0x034995 thru 0x03ffff (0xb66b bytes) 000219 1779311059.129101 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb2b30168 (stale) Mirror 4: 0xb2b30168 (stale) Mirror 5: 0xb2b30168 (stale) Mirror 6: 0xb2b30168 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a91867c Mirror 7: 0x1a91867c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000220 1779311059.759633 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb2b30168 (stale) Mirror 4: 0xb2b30168 (stale) Mirror 5: 0xb2b30168 (stale) Mirror 6: 0xb2b30168 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a91867c Mirror 7: 0x1a91867c Mirror 8: 0x1a91867c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000221 1779311060.462660 read 0x006006 thru 0x012247 (0xc242 bytes) 000225 1779311060.490851 write 0x02660e thru 0x03598e (0xf381 bytes) 000226 1779311060.576578 mapwrite 0x02dead thru 0x032010 (0x4164 bytes) 000227 1779311060.621598 read 0x0007b8 thru 0x00bb8e (0xb3d7 bytes) 000228 1779311060.634105 mapwrite 0x03fefd thru 0x03ffff (0x103 bytes) 000229 1779311060.664403 mapwrite 0x00998f thru 0x00d9ae (0x4020 bytes) 000231 1779311060.704746 mapwrite 0x03d71d thru 0x03eec1 (0x17a5 bytes) 000234 1779311060.752291 trunc from 0x040000 to 0x017c25 (0x283da bytes) 000235 1779311060.801097 read 0x0055bd thru 0x00e3f3 (0x8e37 bytes) 000236 1779311060.801861 write 0x023f49 thru 0x02c62e (0x86e6 bytes) 000237 1779311060.803462 read 0x018651 thru 0x02167a (0x902a bytes) 000238 1779311060.812332 trunc from 0x02c62f to 0x012da6 (0x19888 bytes) 000239 1779311060.825817 write 0x019bb9 thru 0x01b53b (0x1983 bytes) 000240 1779311060.827093 trunc from 0x01b53c to 0x0102a0 (0xb29b bytes) 000241 1779311060.838886 trunc from 0x0102a1 to 0x03b83a (0x2b59a bytes) 000242 1779311060.857656 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b83b): Mirror 2: 0xb3bd87e6 (stale) Mirror 5: 0xb3bd87e6 (stale) Mirror 6: 0xb3bd87e6 (stale) Mirror 7: 0xdab06980 (stale) Mirror 8: 0xdab06980 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b83b): Mirror 2: 0xb3bd87e6 (stale) Mirror 5: 0xb3bd87e6 (stale) Mirror 6: 0xb3bd87e6 (stale) Mirror 7: 0xdab06980 (stale) Mirror 8: 0xdab06980 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b83b: skipped 000243 1779311061.189805 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b83b): Mirror 2: 0xb3bd87e6 (stale) Mirror 5: 0xb3bd87e6 (stale) Mirror 6: 0xb3bd87e6 (stale) Mirror 8: 0xdab06980 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b83b): Mirror 2: 0xb3bd87e6 (stale) Mirror 5: 0xb3bd87e6 (stale) Mirror 6: 0xb3bd87e6 (stale) Mirror 8: 0xdab06980 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b83b: skipped 000244 1779311061.521838 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b83b): Mirror 2: 0xb3bd87e6 (stale) Mirror 5: 0xb3bd87e6 (stale) Mirror 6: 0xb3bd87e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b83b): Mirror 2: 0xb3bd87e6 (stale) Mirror 5: 0xb3bd87e6 (stale) Mirror 6: 0xb3bd87e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b83b: skipped 000247 1779311061.822966 mapread 0x01f952 thru 0x02bef9 (0xc5a8 bytes) 000248 1779311061.858551 mapwrite 0x03536e thru 0x037378 (0x200b bytes) 000249 1779311061.906411 write 0x01f673 thru 0x02b9b0 (0xc33e bytes) 000250 1779311061.907684 write 0x01b5ab thru 0x02675e (0xb1b4 bytes) 000251 1779311061.909001 mapwrite 0x000f39 thru 0x00eda8 (0xde70 bytes) 000252 1779311061.979672 mapwrite 0x01f762 thru 0x022793 (0x3032 bytes) 000253 1779311062.029109 mapwrite 0x03792f thru 0x0385b0 (0xc82 bytes) 000254 1779311062.069302 mapread 0x011159 thru 0x01b6c7 (0xa56f bytes) 000255 1779311062.087820 trunc from 0x03b83b to 0x01976b (0x220cf bytes) 000256 1779311062.104837 mapread 0x00e1bd thru 0x00f9df (0x1823 bytes) 000258 1779311062.111684 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1976c): Mirror 2: 0x37f5a2b0 (stale) Mirror 5: 0x37f5a2b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1976c): Mirror 2: 0x37f5a2b0 (stale) Mirror 5: 0x37f5a2b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1976c: skipped 000260 1779311062.358230 trunc from 0x01976c to 0x020fb1 (0x7846 bytes) 000262 1779311062.376339 mapread 0x0149ff thru 0x020fb1 (0xc5b3 bytes) 000264 1779311062.414230 write 0x03eaeb thru 0x03ffff (0x1515 bytes) 000268 1779311062.417610 mapwrite 0x001354 thru 0x007d9c (0x6a49 bytes) 000270 1779311062.475295 mapread 0x017daf thru 0x01bb27 (0x3d79 bytes) 000273 1779311062.477445 mapread 0x015361 thru 0x02286e (0xd50e bytes) 000276 1779311062.486438 mapwrite 0x0289d8 thru 0x02f9ed (0x7016 bytes) 000277 1779311062.547883 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb2b30168 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb2b30168 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1779311062.837358 trunc from 0x040000 to 0x0110cb (0x2ef34 bytes) 000286 1779311062.860628 mapread 0x00b6fd thru 0x0110cb (0x59cf bytes) 000287 1779311062.879610 trunc from 0x0110cc to 0x03aa48 (0x2997d bytes) 000288 1779311062.905199 mapwrite 0x0036bf thru 0x00a784 (0x70c6 bytes) 000295 1779311062.969563 mapwrite 0x00e212 thru 0x00f8ef (0x16de bytes) 000296 1779311063.008545 write 0x01ad42 thru 0x02a5c3 (0xf882 bytes) 000297 1779311063.024727 read 0x01df8c thru 0x028257 (0xa2cc bytes) 000298 1779311063.025998 write 0x0043e4 thru 0x01105a (0xcc77 bytes) 000299 1779311063.027739 mapwrite 0x028a0d thru 0x033c72 (0xb266 bytes) 000300 1779311063.083412 trunc from 0x03aa49 to 0x0349ca (0x607e bytes) 000301 1779311063.115344 MIRROR_RESYNC 2 mirrors 000303 1779311063.650023 mapwrite 0x00757d thru 0x00c63d (0x50c1 bytes) 000304 1779311063.788070 MIRROR_RESYNC 2 mirrors 000305 1779311064.203602 trunc from 0x0349cb to 0x03692d (0x1f63 bytes) 000306 1779311064.256806 read 0x00d66e thru 0x01b26d (0xdc00 bytes) 000307 1779311064.267689 mapwrite 0x03e944 thru 0x03ffff (0x16bc bytes) 000308 1779311064.333667 write 0x0283f5 thru 0x02e45c (0x6068 bytes) 000309 1779311064.344320 read 0x03499a thru 0x039828 (0x4e8f bytes) 000312 1779311064.357793 read 0x02eac6 thru 0x03341e (0x4959 bytes) 000313 1779311064.358961 write 0x02bfb5 thru 0x02c122 (0x16e bytes) 000315 1779311064.365058 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) 000316 1779311064.494386 mapread 0x0028cf thru 0x00e362 (0xba94 bytes) 000323 1779311064.523884 trunc from 0x040000 to 0x03cc15 (0x33ea bytes) 000324 1779311064.537441 read 0x017d41 thru 0x01c32f (0x45ef bytes) 000325 1779311064.543413 mapwrite 0x009c67 thru 0x01068f (0x6a29 bytes) 000327 1779311064.597625 MIRROR_EXTEND to 2 mirrors 000329 1779311065.049568 mapwrite 0x0340fc thru 0x03ffff (0xbf04 bytes) 000332 1779311065.199515 write 0x010cea thru 0x016638 (0x594f bytes) 000333 1779311065.222127 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x51395082 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa97c6689 Mirror 3: 0xa97c6689 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000335 1779311065.717807 write 0x038181 thru 0x03ffff (0x7e7f bytes) 000336 1779311065.781430 write 0x014486 thru 0x01df30 (0x9aab bytes) 000337 1779311065.790897 write 0x001a81 thru 0x00d004 (0xb584 bytes) 000341 1779311065.808266 write 0x00f6a1 thru 0x01093e (0x129e bytes) 000343 1779311065.815815 read 0x01c1a2 thru 0x02998a (0xd7e9 bytes) 000344 1779311065.828580 MIRROR_RESYNC 3 mirrors 000346 1779311066.336274 write 0x035afa thru 0x03962c (0x3b33 bytes) 000347 1779311066.392961 trunc from 0x040000 to 0x014c78 (0x2b387 bytes) 000348 1779311066.415539 MIRROR_RESYNC 3 mirrors 000349 1779311066.888881 mapwrite 0x01f047 thru 0x022aa7 (0x3a61 bytes) 000350 1779311067.008418 mapwrite 0x037527 thru 0x03cbf7 (0x56d1 bytes) 000351 1779311067.084807 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cbf8): Mirror 2: 0xc7c690d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbf8): Mirror 2: 0xc7c690d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbf8: skipped 000353 1779311067.305988 trunc from 0x03cbf8 to 0x03d0d9 (0x4e2 bytes) 000354 1779311067.322863 write 0x021197 thru 0x0275d5 (0x643f bytes) 000355 1779311067.334022 write 0x01d6dd thru 0x028b53 (0xb477 bytes) 000356 1779311067.353275 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d0da): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0da): Mirror 2: 0xf8ead1fe (stale) CRC-32 checksum value for chunk [0, 0x3d0da): Mirror 1: 0xf650211f Mirror 3: 0xf650211f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0da: skipped 000358 1779311067.901053 mapread 0x02ccdb thru 0x03bbe1 (0xef07 bytes) 000359 1779311067.946829 write 0x011297 thru 0x01d761 (0xc4cb bytes) 000361 1779311068.002240 MIRROR_RESYNC 3 mirrors 000362 1779311068.480013 mapwrite 0x00830c thru 0x011c85 (0x997a bytes) 000364 1779311068.615576 mapwrite 0x016525 thru 0x01bdd0 (0x58ac bytes) 000367 1779311068.670631 write 0x03d96c thru 0x03ffff (0x2694 bytes) 000368 1779311068.678172 trunc from 0x040000 to 0x037f54 (0x80ab bytes) 000370 1779311068.698348 trunc from 0x037f55 to 0x015b51 (0x22403 bytes) 000371 1779311068.712134 read 0x00d2a6 thru 0x015b51 (0x88ac bytes) 000372 1779311068.712963 mapwrite 0x01f4e1 thru 0x02df6e (0xea8e bytes) 000373 1779311068.780152 write 0x007601 thru 0x01757d (0xff7d bytes) 000374 1779311068.785493 mapread 0x011005 thru 0x01f6cf (0xe6cb bytes) 000375 1779311068.786540 read 0x00f25b thru 0x017c66 (0x8a0c bytes) 000376 1779311068.787482 mapwrite 0x02cebc thru 0x0347a4 (0x78e9 bytes) 000377 1779311068.856204 write 0x01341e thru 0x017992 (0x4575 bytes) 000378 1779311068.857357 mapwrite 0x033c66 thru 0x036aa3 (0x2e3e bytes) 000380 1779311068.920576 trunc from 0x036aa4 to 0x02eb83 (0x7f20 bytes) 000382 1779311068.951649 MIRROR_RESYNC 3 mirrors 000383 1779311069.464357 write 0x00eba3 thru 0x018a6a (0x9ec8 bytes) 000385 1779311069.527593 mapread 0x025650 thru 0x029109 (0x3aba bytes) 000387 1779311069.536533 read 0x01f128 thru 0x02eb83 (0xfa5c bytes) 000388 1779311069.540472 write 0x02f7ab thru 0x036225 (0x6a7b bytes) 000390 1779311069.542232 write 0x02b1d6 thru 0x0361f8 (0xb023 bytes) 000392 1779311069.547930 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36226): Mirror 3: 0x55f555bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36226): Mirror 3: 0x55f555bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36226: skipped 000393 1779311069.811689 read 0x0350b3 thru 0x036225 (0x1173 bytes) 000394 1779311069.816045 mapwrite 0x003e22 thru 0x00ab08 (0x6ce7 bytes) 000395 1779311069.886944 write 0x0310d1 thru 0x03cf90 (0xbec0 bytes) 000396 1779311069.897845 trunc from 0x03cf91 to 0x030f32 (0xc05e bytes) 000397 1779311069.916011 mapread 0x017c09 thru 0x0224cc (0xa8c4 bytes) 000398 1779311069.937546 MIRROR_RESYNC 2 mirrors 000401 1779311070.464075 mapwrite 0x010390 thru 0x016750 (0x63c1 bytes) 000402 1779311070.581439 MIRROR_RESYNC 2 mirrors 000403 1779311070.965304 trunc from 0x030f33 to 0x00570e (0x2b824 bytes) 000404 1779311071.000138 write 0x016b29 thru 0x026636 (0xfb0e bytes) 000411 1779311071.004498 read 0x009b8a thru 0x0102f1 (0x6768 bytes) 000413 1779311071.013232 mapwrite 0x02d436 thru 0x0376d9 (0xa2a4 bytes) 000416 1779311071.091315 write 0x03a2dc thru 0x03ffff (0x5d24 bytes) 000417 1779311071.094605 write 0x01c0e3 thru 0x01fbb0 (0x3ace bytes) 000418 1779311071.098228 MIRROR_RESYNC 2 mirrors 000419 1779311071.500395 write 0x03ba81 thru 0x03ffff (0x457f bytes) 000420 1779311071.550074 write 0x01f6a4 thru 0x02c58d (0xceea bytes) 000421 1779311071.561251 trunc from 0x040000 to 0x00def2 (0x3210d bytes) 000425 1779311071.598214 trunc from 0x00def3 to 0x028ff3 (0x1b101 bytes) 000427 1779311071.616619 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28ff4): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ff4): Mirror 3: 0xcca65db2 (stale) CRC-32 checksum value for chunk [0, 0x28ff4): Mirror 1: 0x46fb90f4 Mirror 4: 0x46fb90f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ff4: skipped 000428 1779311072.067490 read 0x026893 thru 0x028ff3 (0x2761 bytes) 000430 1779311072.084532 MIRROR_RESYNC 3 mirrors 000433 1779311072.494485 write 0x014821 thru 0x0193b1 (0x4b91 bytes) 000435 1779311072.555288 trunc from 0x028ff4 to 0x021f94 (0x705f bytes) 000436 1779311072.581782 trunc from 0x021f95 to 0x02fe30 (0xde9c bytes) 000438 1779311072.596578 write 0x02fa44 thru 0x031f1b (0x24d8 bytes) 000441 1779311072.608250 mapread 0x017865 thru 0x0185e5 (0xd81 bytes) 000442 1779311072.609414 trunc from 0x031f1c to 0x022a75 (0xf4a6 bytes) 000444 1779311072.624354 read 0x003088 thru 0x004a68 (0x19e1 bytes) 000446 1779311072.632482 write 0x015160 thru 0x01974b (0x45ec bytes) 000448 1779311072.637250 read 0x01ad4d thru 0x01cc86 (0x1f3a bytes) 000449 1779311072.658206 MIRROR_RESYNC 3 mirrors 000452 1779311073.107257 trunc from 0x022a76 to 0x02933c (0x68c7 bytes) 000454 1779311073.172097 mapread 0x01bdec thru 0x02933c (0xd551 bytes) 000456 1779311073.198477 trunc from 0x02933d to 0x032b03 (0x97c7 bytes) 000458 1779311073.222571 read 0x028830 thru 0x03141e (0x8bef bytes) 000459 1779311073.233531 mapwrite 0x0190d6 thru 0x027bb3 (0xeade bytes) 000460 1779311073.277215 mapread 0x0136bb thru 0x0181f2 (0x4b38 bytes) 000461 1779311073.293095 mapwrite 0x00c6d2 thru 0x00e9f2 (0x2321 bytes) 000463 1779311073.325545 mapread 0x02ac5d thru 0x032b03 (0x7ea7 bytes) 000465 1779311073.331292 mapwrite 0x020467 thru 0x02d751 (0xd2eb bytes) 000466 1779311073.377313 mapwrite 0x02a717 thru 0x03933c (0xec26 bytes) 000467 1779311073.450520 trunc from 0x03933d to 0x03b96a (0x262e bytes) 000468 1779311073.461418 trunc from 0x03b96b to 0x021c26 (0x19d44 bytes) 000469 1779311073.476893 mapwrite 0x032dc3 thru 0x039eb3 (0x70f1 bytes) 000472 1779311073.537207 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39eb4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39eb4): Mirror 3: 0xa41cc3ac (stale) Mirror 4: 0xa41cc3ac (stale) CRC-32 checksum value for chunk [0, 0x39eb4): Mirror 1: 0x1851eab6 Mirror 5: 0x1851eab6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39eb4: skipped 000473 1779311074.015097 trunc from 0x039eb4 to 0x01bd50 (0x1e163 bytes) 000475 1779311074.095072 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1bd51): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd51): Mirror 3: 0x7bbba17 (stale) Mirror 4: 0x7bbba17 (stale) Mirror 5: 0x33cf5f20 (stale) CRC-32 checksum value for chunk [0, 0x1bd51): Mirror 1: 0x33cf5f20 Mirror 6: 0x33cf5f20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd51: skipped 000476 1779311074.631543 MIRROR_RESYNC 5 mirrors 000478 1779311075.120176 write 0x0320fb thru 0x0323bf (0x2c5 bytes) 000479 1779311075.174247 read 0x022556 thru 0x02b9be (0x9469 bytes) 000482 1779311075.180878 mapwrite 0x01d970 thru 0x02a4ca (0xcb5b bytes) 000483 1779311075.243257 trunc from 0x0323c0 to 0x01b776 (0x16c49 bytes) 000484 1779311075.260122 mapread 0x015e82 thru 0x01b776 (0x58f5 bytes) 000486 1779311075.268136 trunc from 0x01b777 to 0x012159 (0x961d bytes) 000488 1779311075.283390 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1215a): Mirror 3: 0x30f68fcc (stale) Mirror 4: 0x30f68fcc (stale) Mirror 6: 0x30f68fcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1215a): Mirror 3: 0x30f68fcc (stale) Mirror 4: 0x30f68fcc (stale) Mirror 6: 0x30f68fcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1215a: skipped 000489 1779311075.525086 mapread 0x00e94d thru 0x012159 (0x380d bytes) 000490 1779311075.538465 mapwrite 0x00af92 thru 0x011deb (0x6e5a bytes) 000491 1779311075.590460 trunc from 0x01215a to 0x02f84d (0x1d6f4 bytes) 000494 1779311075.602275 write 0x02fd35 thru 0x03dcba (0xdf86 bytes) 000496 1779311075.611594 mapread 0x003498 thru 0x008e63 (0x59cc bytes) 000497 1779311075.626200 read 0x00ba75 thru 0x01a129 (0xe6b5 bytes) 000498 1779311075.634441 mapread 0x011de0 thru 0x011faf (0x1d0 bytes) 000501 1779311075.642532 mapwrite 0x034044 thru 0x03ffff (0xbfbc bytes) 000503 1779311075.699891 trunc from 0x040000 to 0x014777 (0x2b888 bytes) 000505 1779311075.713308 read 0x0088fe thru 0x0089d3 (0xd6 bytes) 000506 1779311075.713835 mapread 0x012474 thru 0x014777 (0x2304 bytes) 000508 1779311075.715337 mapread 0x014246 thru 0x014777 (0x532 bytes) 000509 1779311075.716208 mapread 0x001636 thru 0x00c762 (0xb12d bytes) 000510 1779311075.717473 write 0x03b2ba thru 0x03ffff (0x4d46 bytes) 000512 1779311075.719149 mapread 0x015d2a thru 0x01d945 (0x7c1c bytes) 000517 1779311075.732540 mapread 0x0342f0 thru 0x03fb19 (0xb82a bytes) 000518 1779311075.733838 write 0x01ff43 thru 0x025664 (0x5722 bytes) 000519 1779311075.734909 trunc from 0x040000 to 0x005b7c (0x3a483 bytes) 000520 1779311075.750250 trunc from 0x005b7d to 0x03029f (0x2a723 bytes) 000522 1779311075.765146 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302a0): Mirror 4: 0x116b7b1e (stale) Mirror 6: 0x116b7b1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302a0): Mirror 4: 0x116b7b1e (stale) Mirror 6: 0x116b7b1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302a0: skipped 000526 1779311075.985106 write 0x016fc9 thru 0x02026b (0x92a3 bytes) 000528 1779311075.995867 read 0x02e005 thru 0x03029f (0x229b bytes) 000529 1779311076.003347 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x302a0): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302a0): Mirror 4: 0x116b7b1e (stale) Mirror 6: 0x116b7b1e (stale) CRC-32 checksum value for chunk [0, 0x302a0): Mirror 1: 0x78676337 Mirror 7: 0x78676337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302a0: skipped 000532 1779311076.538167 MIRROR_RESYNC 4 mirrors 000533 1779311077.030271 read 0x016ca5 thru 0x01cce2 (0x603e bytes) 000534 1779311077.042459 mapread 0x007f6f thru 0x00aada (0x2b6c bytes) 000535 1779311077.067128 MIRROR_EXTEND to 5 mirrors 000536 1779311077.471342 read 0x011ecb thru 0x01bef7 (0xa02d bytes) 000537 1779311077.478348 read 0x00a867 thru 0x011d1f (0x74b9 bytes) 000540 1779311077.490839 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) 000542 1779311077.736296 trunc from 0x0302a0 to 0x016d6d (0x19532 bytes) 000543 1779311077.802672 write 0x00bee0 thru 0x00d0d1 (0x11f2 bytes) 000544 1779311077.815168 trunc from 0x016d6e to 0x03479a (0x1da2d bytes) 000545 1779311077.838845 trunc from 0x03479b to 0x01bae3 (0x18cb7 bytes) 000547 1779311077.855755 mapwrite 0x02ceec thru 0x032c36 (0x5d4b bytes) 000548 1779311077.910390 mapwrite 0x032ef5 thru 0x038945 (0x5a51 bytes) 000549 1779311077.960962 write 0x019f01 thru 0x01bf78 (0x2078 bytes) 000551 1779311077.973164 MIRROR_RESYNC 4 mirrors 000552 1779311078.599824 read 0x0122bd thru 0x01645a (0x419e bytes) 000556 1779311078.608834 write 0x034a8c thru 0x03ffff (0xb574 bytes) 000557 1779311078.660565 read 0x00162c thru 0x0076d9 (0x60ae bytes) 000558 1779311078.673436 MIRROR_RESYNC 4 mirrors 000560 1779311079.166913 trunc from 0x040000 to 0x00719d (0x38e62 bytes) 000561 1779311079.236074 trunc from 0x00719e to 0x033db0 (0x2cc13 bytes) 000562 1779311079.251884 mapwrite 0x0391c7 thru 0x03ffff (0x6e39 bytes) 000563 1779311079.317075 read 0x015a6f thru 0x01a80f (0x4da1 bytes) 000568 1779311079.329211 write 0x00222e thru 0x007bcf (0x59a2 bytes) 000569 1779311079.342464 mapwrite 0x01bdf3 thru 0x026a85 (0xac93 bytes) 000570 1779311079.400805 read 0x01c524 thru 0x027f33 (0xba10 bytes) 000571 1779311079.401622 read 0x025881 thru 0x02dba8 (0x8328 bytes) 000574 1779311079.410165 mapwrite 0x0083bc thru 0x009c35 (0x187a bytes) 000575 1779311079.447914 mapwrite 0x0317cf thru 0x03ffff (0xe831 bytes) 000576 1779311079.501244 mapwrite 0x0004ee thru 0x008fe6 (0x8af9 bytes) 000577 1779311079.536766 mapread 0x01a7b2 thru 0x01e903 (0x4152 bytes) 000580 1779311079.538747 mapread 0x02be5e thru 0x034605 (0x87a8 bytes) 000581 1779311079.545037 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x13f9b0ee (stale) Mirror 6: 0x13f9b0ee (stale) Mirror 8: 0x13f9b0ee (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x44be3156 Mirror 9: 0x44be3156 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000584 1779311079.980392 write 0x00d450 thru 0x00d948 (0x4f9 bytes) 000585 1779311080.049800 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x13f9b0ee (stale) Mirror 6: 0x13f9b0ee (stale) Mirror 9: 0x44be3156 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x13f9b0ee (stale) Mirror 6: 0x13f9b0ee (stale) Mirror 9: 0x44be3156 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1779311080.393353 mapread 0x001a8c thru 0x004aa3 (0x3018 bytes) 000589 1779311080.415475 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x13f9b0ee (stale) Mirror 6: 0x13f9b0ee (stale) Mirror 9: 0x44be3156 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x44b7c78c Mirror 10: 0x44b7c78c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000590 1779311080.968799 mapwrite 0x00803e thru 0x00cb60 (0x4b23 bytes) 000591 1779311081.113754 read 0x0255d3 thru 0x02a2c8 (0x4cf6 bytes) 000592 1779311081.123077 mapread 0x009d70 thru 0x00a489 (0x71a bytes) 000594 1779311081.124274 mapwrite 0x005de4 thru 0x006a2d (0xc4a bytes) 000596 1779311081.151832 mapwrite 0x021a93 thru 0x02ae78 (0x93e6 bytes) 000597 1779311081.193812 write 0x01c78a thru 0x02165c (0x4ed3 bytes) 000599 1779311081.195496 mapwrite 0x03275f thru 0x035944 (0x31e6 bytes) 000600 1779311081.249333 read 0x02fa5c thru 0x035d7a (0x631f bytes) 000601 1779311081.253368 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x13f9b0ee (stale) Mirror 6: 0x13f9b0ee (stale) Mirror 9: 0x44be3156 (stale) Mirror 10: 0x44b7c78c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xab890e02 Mirror 11: 0xab890e02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000602 1779311081.728469 mapwrite 0x036137 thru 0x03b588 (0x5452 bytes) 000603 1779311081.848749 write 0x03c195 thru 0x03ffff (0x3e6b bytes) 000604 1779311081.849723 read 0x018622 thru 0x018ca4 (0x683 bytes) 000605 1779311081.854374 mapread 0x036283 thru 0x03855b (0x22d9 bytes) 000606 1779311081.856874 mapwrite 0x0330eb thru 0x0363a3 (0x32b9 bytes) 000609 1779311081.892908 trunc from 0x040000 to 0x012e2f (0x2d1d0 bytes) 000610 1779311081.922904 write 0x002ca6 thru 0x00ad61 (0x80bc bytes) 000611 1779311081.930882 mapwrite 0x03d288 thru 0x03ea50 (0x17c9 bytes) 000613 1779311081.988277 trunc from 0x03ea51 to 0x01a41d (0x24633 bytes) 000614 1779311082.007010 mapwrite 0x033a5e thru 0x0386ce (0x4c71 bytes) 000616 1779311082.075466 trunc from 0x0386cf to 0x025fa2 (0x1272c bytes) 000617 1779311082.091579 write 0x01d431 thru 0x02d08f (0xfc5f bytes) 000618 1779311082.106937 write 0x029a9a thru 0x037ea9 (0xe410 bytes) 000619 1779311082.108174 mapread 0x0253e3 thru 0x02cbf9 (0x7817 bytes) 000620 1779311082.109548 mapwrite 0x026f55 thru 0x0307b2 (0x985e bytes) 000621 1779311082.164276 read 0x02939f thru 0x03796c (0xe5ce bytes) 000622 1779311082.170234 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x37eaa): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37eaa): Mirror 4: 0xcb2bd67 (stale) Mirror 6: 0xcb2bd67 (stale) Mirror 9: 0x9fa4f92f (stale) Mirror 10: 0xf4a607fd (stale) Mirror 11: 0xe4e7e3e5 (stale) CRC-32 checksum value for chunk [0, 0x37eaa): Mirror 1: 0x92ef0067 Mirror 12: 0x92ef0067 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37eaa: skipped 000627 1779311082.747880 read 0x012e7c thru 0x014f66 (0x20eb bytes) 000630 1779311082.764484 write 0x02512f thru 0x0321fc (0xd0ce bytes) 000634 1779311082.826515 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37eaa): Mirror 6: 0xcb2bd67 (stale) Mirror 9: 0x9fa4f92f (stale) Mirror 10: 0xf4a607fd (stale) Mirror 11: 0xe4e7e3e5 (stale) Mirror 12: 0x92ef0067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37eaa): Mirror 6: 0xcb2bd67 (stale) Mirror 9: 0x9fa4f92f (stale) Mirror 10: 0xf4a607fd (stale) Mirror 11: 0xe4e7e3e5 (stale) Mirror 12: 0x92ef0067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37eaa: skipped 000636 1779311083.135264 mapread 0x001e22 thru 0x00b746 (0x9925 bytes) 000637 1779311083.149397 write 0x0008f5 thru 0x00d91c (0xd028 bytes) 000638 1779311083.155209 mapread 0x015ee4 thru 0x01c0be (0x61db bytes) 000639 1779311083.172183 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37eaa): Mirror 6: 0xcb2bd67 (stale) Mirror 10: 0xf4a607fd (stale) Mirror 11: 0xe4e7e3e5 (stale) Mirror 12: 0x92ef0067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37eaa): Mirror 6: 0xcb2bd67 (stale) Mirror 10: 0xf4a607fd (stale) Mirror 11: 0xe4e7e3e5 (stale) Mirror 12: 0x92ef0067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37eaa: skipped 000640 1779311083.475190 trunc from 0x037eaa to 0x02597d (0x1252c bytes) 000641 1779311083.503350 mapread 0x000417 thru 0x009849 (0x9433 bytes) 000642 1779311083.531097 MIRROR_RESYNC 5 mirrors 000643 1779311084.169273 read 0x01a206 thru 0x02597d (0xb778 bytes) 000644 1779311084.175277 trunc from 0x02597e to 0x0292ea (0x396d bytes) 000647 1779311084.238878 write 0x035400 thru 0x037620 (0x2221 bytes) 000651 1779311084.242848 mapwrite 0x018d9a thru 0x026946 (0xdbad bytes) 000652 1779311084.312509 read 0x00c5fc thru 0x01180b (0x5210 bytes) 000653 1779311084.319907 mapread 0x0249a0 thru 0x02c662 (0x7cc3 bytes) 000654 1779311084.321169 mapwrite 0x032f37 thru 0x03ffff (0xd0c9 bytes) 000655 1779311084.379265 read 0x0110dd thru 0x01a863 (0x9787 bytes) 000656 1779311084.379852 mapwrite 0x010129 thru 0x0194aa (0x9382 bytes) 000657 1779311084.416197 write 0x02a93f thru 0x037e27 (0xd4e9 bytes) 000658 1779311084.418322 trunc from 0x040000 to 0x016a9e (0x29561 bytes) 000659 1779311084.433124 mapwrite 0x02ee7f thru 0x039e8b (0xb00d bytes) 000662 1779311084.494344 mapwrite 0x02008a thru 0x023258 (0x31cf bytes) 000664 1779311084.532243 trunc from 0x039e8c to 0x00ab4f (0x2f33c bytes) 000666 1779311084.549045 trunc from 0x00ab50 to 0x022575 (0x17a26 bytes) 000669 1779311084.562492 mapread 0x015799 thru 0x01f7fb (0xa063 bytes) 000672 1779311084.611059 mapread 0x01a460 thru 0x01e6e5 (0x4286 bytes) 000673 1779311084.612134 mapread 0x022454 thru 0x022575 (0x122 bytes) 000674 1779311084.618558 mapread 0x015265 thru 0x01c280 (0x701c bytes) 000675 1779311084.620180 mapwrite 0x023d96 thru 0x033484 (0xf6ef bytes) 000678 1779311084.773218 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33485): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33485): Mirror 6: 0x215dc6aa (stale) Mirror 10: 0x215dc6aa (stale) Mirror 11: 0x215dc6aa (stale) Mirror 12: 0x215dc6aa (stale) CRC-32 checksum value for chunk [0, 0x33485): Mirror 1: 0x3705e734 Mirror 13: 0x3705e734 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33485: skipped 000682 1779311085.318620 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33485): Mirror 6: 0x215dc6aa (stale) Mirror 10: 0x215dc6aa (stale) Mirror 11: 0x215dc6aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33485): Mirror 6: 0x215dc6aa (stale) Mirror 10: 0x215dc6aa (stale) Mirror 11: 0x215dc6aa (stale) CRC-32 checksum value for chunk [0, 0x33485): Mirror 1: 0x3705e734 Mirror 13: 0x3705e734 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33485: skipped 000684 1779311085.792117 mapread 0x0128b8 thru 0x01de9d (0xb5e6 bytes) 000686 1779311085.845475 trunc from 0x033485 to 0x0300c2 (0x33c2 bytes) 000688 1779311085.924399 write 0x007d14 thru 0x0080af (0x39c bytes) 000689 1779311085.934259 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x300c3): Mirror 10: 0xf96fe75e (stale) Mirror 11: 0xf96fe75e (stale) Mirror 13: 0xe548fa7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x300c3): Mirror 10: 0xf96fe75e (stale) Mirror 11: 0xf96fe75e (stale) Mirror 13: 0xe548fa7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x300c3: skipped 000690 1779311086.186721 mapwrite 0x02be12 thru 0x03780a (0xb9f9 bytes) 000691 1779311086.260720 read 0x02e963 thru 0x031b1d (0x31bb bytes) 000692 1779311086.261227 mapread 0x02b465 thru 0x03780a (0xc3a6 bytes) 000693 1779311086.262838 read 0x026a41 thru 0x02e941 (0x7f01 bytes) 000694 1779311086.267685 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3780b): Mirror 10: 0xe63271f (stale) Mirror 11: 0xe63271f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3780b): Mirror 10: 0xe63271f (stale) Mirror 11: 0xe63271f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3780b: skipped 000695 1779311086.487683 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3780b): Mirror 11: 0xe63271f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3780b): Mirror 11: 0xe63271f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3780b: skipped 000697 1779311086.682309 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3780b): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3780b): Mirror 11: 0xe63271f (stale) CRC-32 checksum value for chunk [0, 0x3780b): Mirror 1: 0xf304bc08 Mirror 12: 0xf304bc08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3780b: skipped 000699 1779311087.181399 mapwrite 0x0111d4 thru 0x01ce51 (0xbc7e bytes) 000700 1779311087.333061 write 0x031865 thru 0x035167 (0x3903 bytes) 000703 1779311087.343499 read 0x0048b9 thru 0x00ac0d (0x6355 bytes) 000704 1779311087.351631 read 0x023180 thru 0x02e53a (0xb3bb bytes) 000705 1779311087.363010 MIRROR_RESYNC 3 mirrors 000707 1779311087.799242 mapwrite 0x007456 thru 0x00ae05 (0x39b0 bytes) 000708 1779311087.924783 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3780b): Mirror 12: 0x82b76ddb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3780b): Mirror 12: 0x82b76ddb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3780b: skipped 000709 1779311088.154822 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) 000711 1779311088.242633 write 0x004ad1 thru 0x011672 (0xcba2 bytes) 000716 1779311088.262328 read 0x01fb22 thru 0x025eb6 (0x6395 bytes) 000718 1779311088.269520 read 0x030139 thru 0x03780a (0x76d2 bytes) 000719 1779311088.280625 mapwrite 0x02b0a8 thru 0x036155 (0xb0ae bytes) 000720 1779311088.329098 trunc from 0x03780b to 0x0057db (0x3202f bytes) 000721 1779311088.360106 mapwrite 0x000352 thru 0x007397 (0x7046 bytes) 000722 1779311088.423328 mapwrite 0x00933d thru 0x018a4f (0xf713 bytes) 000723 1779311088.552909 read 0x00a3ac thru 0x00bd54 (0x19a9 bytes) 000724 1779311088.553544 read 0x012476 thru 0x018a4f (0x65da bytes) 000726 1779311088.554820 mapwrite 0x02fbf9 thru 0x03350f (0x3917 bytes) 000728 1779311088.610899 write 0x026ae5 thru 0x02e247 (0x7763 bytes) 000731 1779311088.613267 mapwrite 0x00871b thru 0x0108ea (0x81d0 bytes) 000732 1779311088.650187 write 0x00c695 thru 0x01b3ec (0xed58 bytes) 000735 1779311088.654186 write 0x0241fe thru 0x024f15 (0xd18 bytes) 000736 1779311088.655165 trunc from 0x033510 to 0x00774a (0x2bdc5 bytes) 000737 1779311088.666083 mapwrite 0x039efc thru 0x03a296 (0x39b bytes) 000738 1779311088.706053 mapread 0x02bdc1 thru 0x02c142 (0x382 bytes) 000739 1779311088.712862 mapwrite 0x02115f thru 0x028f77 (0x7e19 bytes) 000740 1779311088.792563 mapread 0x00d376 thru 0x01811c (0xada7 bytes) 000743 1779311088.849402 mapwrite 0x02cbe6 thru 0x03231d (0x5738 bytes) 000744 1779311088.919546 mapread 0x019074 thru 0x0276c0 (0xe64d bytes) 000746 1779311088.947469 read 0x02c53a thru 0x0329f3 (0x64ba bytes) 000749 1779311088.949201 trunc from 0x03a297 to 0x001834 (0x38a62 bytes) 000750 1779311088.969328 MIRROR_EXTEND to 2 mirrors 000752 1779311089.355428 read 0x0009f2 thru 0x001834 (0xe43 bytes) 000753 1779311089.367445 read 0x001474 thru 0x001834 (0x3c1 bytes) 000754 1779311089.379583 MIRROR_RESYNC 2 mirrors 000755 1779311089.637889 mapwrite 0x0023aa thru 0x003f8a (0x1be1 bytes) 000756 1779311089.755805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f8b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8b): Mirror 2: 0x7105827a (stale) CRC-32 checksum value for chunk [0, 0x3f8b): Mirror 1: 0x2e355b33 Mirror 3: 0x2e355b33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8b: skipped 000758 1779311090.193197 read 0x00195a thru 0x003f8a (0x2631 bytes) 000760 1779311090.212143 write 0x00f1ab thru 0x00f969 (0x7bf bytes) 000761 1779311090.265632 trunc from 0x00f96a to 0x0394c0 (0x29b57 bytes) 000762 1779311090.313848 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x394c1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394c1): Mirror 2: 0xd99314f7 (stale) Mirror 3: 0x96d70fb5 (stale) CRC-32 checksum value for chunk [0, 0x394c1): Mirror 1: 0xb7ee2c9c Mirror 4: 0xb7ee2c9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394c1: skipped 000763 1779311090.815335 read 0x035ee9 thru 0x037a82 (0x1b9a bytes) 000764 1779311090.833818 trunc from 0x0394c1 to 0x03b9d8 (0x2518 bytes) 000765 1779311090.906557 mapread 0x00ec76 thru 0x01bbf3 (0xcf7e bytes) 000768 1779311090.929885 trunc from 0x03b9d9 to 0x027812 (0x141c6 bytes) 000770 1779311090.941941 write 0x02d333 thru 0x03bf6a (0xec38 bytes) 000771 1779311090.946760 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bf6b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf6b): Mirror 2: 0x77f3f5b5 (stale) Mirror 3: 0x7af07092 (stale) Mirror 4: 0xe9c6c3ae (stale) CRC-32 checksum value for chunk [0, 0x3bf6b): Mirror 1: 0xea06f705 Mirror 5: 0xea06f705 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf6b: skipped 000772 1779311091.536189 mapwrite 0x0026c4 thru 0x011be6 (0xf523 bytes) 000773 1779311091.697998 read 0x001592 thru 0x003e0e (0x287d bytes) 000775 1779311091.699244 mapread 0x01d1e5 thru 0x0238dc (0x66f8 bytes) 000776 1779311091.707240 mapwrite 0x039add thru 0x03ffff (0x6523 bytes) 000777 1779311091.785819 read 0x00a8ec thru 0x0134c6 (0x8bdb bytes) 000780 1779311091.788260 mapwrite 0x018b03 thru 0x01b4a5 (0x29a3 bytes) 000782 1779311091.824789 trunc from 0x040000 to 0x03221a (0xdde5 bytes) 000783 1779311091.842187 mapread 0x02e507 thru 0x02e7de (0x2d8 bytes) 000785 1779311091.852053 mapread 0x001a83 thru 0x00e656 (0xcbd4 bytes) 000786 1779311091.853421 trunc from 0x03221b to 0x01aeb7 (0x17363 bytes) 000787 1779311091.865788 write 0x0105f6 thru 0x012db8 (0x27c3 bytes) 000788 1779311091.867957 read 0x00dc2c thru 0x01a211 (0xc5e6 bytes) 000792 1779311091.870966 read 0x00422d thru 0x00e835 (0xa609 bytes) 000794 1779311091.872078 write 0x01e056 thru 0x01e3f3 (0x39e bytes) 000795 1779311091.873217 mapread 0x0045a4 thru 0x00b88d (0x72ea bytes) 000799 1779311091.876147 write 0x003f87 thru 0x00a887 (0x6901 bytes) 000800 1779311091.877352 write 0x00ecc6 thru 0x0125a0 (0x38db bytes) 000801 1779311091.878592 read 0x00da05 thru 0x01293b (0x4f37 bytes) 000802 1779311091.879423 mapwrite 0x00a547 thru 0x012983 (0x843d bytes) 000803 1779311091.920869 trunc from 0x01e3f4 to 0x02b014 (0xcc21 bytes) 000804 1779311091.936473 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b015): Mirror 3: 0x9d766703 (stale) Mirror 4: 0x7f1dae8b (stale) Mirror 5: 0x7f1dae8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b015): Mirror 3: 0x9d766703 (stale) Mirror 4: 0x7f1dae8b (stale) Mirror 5: 0x7f1dae8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b015: skipped 000805 1779311092.192529 MIRROR_RESYNC 4 mirrors 000808 1779311092.816617 read 0x0102ed thru 0x019afa (0x980e bytes) 000810 1779311092.829572 mapwrite 0x01cf57 thru 0x026e40 (0x9eea bytes) 000813 1779311092.995125 read 0x0005ac thru 0x00dbde (0xd633 bytes) 000814 1779311093.002421 mapread 0x019740 thru 0x02942e (0xfcef bytes) 000815 1779311093.007944 MIRROR_RESYNC 4 mirrors 000817 1779311093.638842 MIRROR_EXTEND to 5 mirrors 000819 1779311094.119479 write 0x001dcf thru 0x005245 (0x3477 bytes) 000820 1779311094.194129 mapwrite 0x03ecf6 thru 0x03ffff (0x130a bytes) 000823 1779311094.279301 mapread 0x024547 thru 0x0279f4 (0x34ae bytes) 000824 1779311094.298018 mapread 0x035560 thru 0x03eb4a (0x95eb bytes) 000829 1779311094.314827 MIRROR_RESYNC 5 mirrors 000831 1779311094.993991 write 0x0324da thru 0x033c08 (0x172f bytes) 000832 1779311095.047961 read 0x00e726 thru 0x0150d3 (0x69ae bytes) 000833 1779311095.059376 write 0x00d2d7 thru 0x01a2d3 (0xcffd bytes) 000834 1779311095.074327 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdd4d557d (stale) Mirror 4: 0xdd4d557d (stale) Mirror 5: 0xdd4d557d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdd4d557d (stale) Mirror 4: 0xdd4d557d (stale) Mirror 5: 0xdd4d557d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000835 1779311095.331946 write 0x015953 thru 0x0240b8 (0xe766 bytes) 000837 1779311095.340846 write 0x010e94 thru 0x01462b (0x3798 bytes) 000839 1779311095.348912 write 0x014c41 thru 0x02196f (0xcd2f bytes) 000841 1779311095.355592 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdd4d557d (stale) Mirror 4: 0xdd4d557d (stale) Mirror 5: 0xdd4d557d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb0d55d66 Mirror 6: 0xb0d55d66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000842 1779311095.880462 MIRROR_RESYNC 5 mirrors 000843 1779311096.412394 write 0x02ec4f thru 0x03cc38 (0xdfea bytes) 000844 1779311096.499946 write 0x00acf2 thru 0x01615a (0xb469 bytes) 000846 1779311096.520098 mapread 0x0133c8 thru 0x021f95 (0xebce bytes) 000848 1779311096.544496 mapread 0x02fbf6 thru 0x03e17b (0xe586 bytes) 000849 1779311096.554448 MIRROR_RESYNC 5 mirrors 000850 1779311097.040163 trunc from 0x040000 to 0x00341b (0x3cbe4 bytes) 000851 1779311097.102019 trunc from 0x00341c to 0x0356a5 (0x3228a bytes) 000852 1779311097.120471 write 0x033215 thru 0x035988 (0x2774 bytes) 000853 1779311097.127740 read 0x01bf2b thru 0x020f0a (0x4fe0 bytes) 000856 1779311097.138418 trunc from 0x035989 to 0x00760d (0x2e37b bytes) 000859 1779311097.153556 read 0x001a1b thru 0x00760d (0x5bf3 bytes) 000860 1779311097.162356 mapread 0x0047ff thru 0x00760d (0x2e0f bytes) 000861 1779311097.163646 mapread 0x00640f thru 0x00760d (0x11ff bytes) 000862 1779311097.166067 mapwrite 0x02588c thru 0x02dccf (0x8444 bytes) 000864 1779311097.230564 mapwrite 0x03dcd2 thru 0x03f1fe (0x152d bytes) 000865 1779311097.294112 mapwrite 0x00e856 thru 0x01c4dc (0xdc87 bytes) 000867 1779311097.331491 trunc from 0x03f1ff to 0x01de18 (0x213e6 bytes) 000869 1779311097.347225 mapread 0x014f3b thru 0x01de18 (0x8ede bytes) 000870 1779311097.348503 write 0x006c8d thru 0x00b29d (0x4611 bytes) 000872 1779311097.354530 mapread 0x008fde thru 0x013037 (0xa05a bytes) 000874 1779311097.357463 mapwrite 0x0010f2 thru 0x008d69 (0x7c78 bytes) 000875 1779311097.393017 mapread 0x01d6fa thru 0x01de18 (0x71f bytes) 000878 1779311097.395076 mapwrite 0x005ba9 thru 0x007748 (0x1ba0 bytes) 000880 1779311097.426942 mapread 0x009709 thru 0x0099e4 (0x2dc bytes) 000881 1779311097.427937 mapread 0x00d919 thru 0x012708 (0x4df0 bytes) 000882 1779311097.429152 write 0x02ecf9 thru 0x03bbd1 (0xced9 bytes) 000883 1779311097.430727 write 0x023870 thru 0x03201e (0xe7af bytes) 000885 1779311097.437367 mapwrite 0x027910 thru 0x02aa49 (0x313a bytes) 000886 1779311097.476618 write 0x0276cb thru 0x02e304 (0x6c3a bytes) 000888 1779311097.478560 mapwrite 0x0385a8 thru 0x0398ff (0x1358 bytes) 000890 1779311097.520770 mapwrite 0x030921 thru 0x03901a (0x86fa bytes) 000891 1779311097.568607 write 0x03f02f thru 0x03ffff (0xfd1 bytes) 000892 1779311097.570249 read 0x020a0b thru 0x0249ab (0x3fa1 bytes) 000893 1779311097.579238 mapread 0x00c2c6 thru 0x0106ec (0x4427 bytes) 000894 1779311097.580423 mapwrite 0x00f039 thru 0x01128c (0x2254 bytes) 000897 1779311097.621328 read 0x034c4d thru 0x03ffff (0xb3b3 bytes) 000899 1779311097.623379 trunc from 0x040000 to 0x00d889 (0x32776 bytes) 000900 1779311097.639444 mapwrite 0x025fc1 thru 0x02bcbd (0x5cfd bytes) 000902 1779311097.701857 mapwrite 0x024c21 thru 0x026fde (0x23be bytes) 000904 1779311097.733184 mapread 0x014569 thru 0x01bbe3 (0x767b bytes) 000905 1779311097.734291 mapread 0x018159 thru 0x0225b0 (0xa458 bytes) 000910 1779311097.740017 MIRROR_RESYNC 5 mirrors 000911 1779311098.351606 MIRROR_RESYNC 5 mirrors 000913 1779311098.664159 read 0x00d868 thru 0x010eb1 (0x364a bytes) 000914 1779311098.670441 write 0x02812c thru 0x02c5d1 (0x44a6 bytes) 000915 1779311098.727633 MIRROR_RESYNC 5 mirrors 000918 1779311099.357213 mapread 0x012a6f thru 0x01eb1f (0xc0b1 bytes) 000919 1779311099.383599 trunc from 0x02c5d2 to 0x022ef4 (0x96dd bytes) 000920 1779311099.436929 read 0x00f3cb thru 0x011f9e (0x2bd4 bytes) 000922 1779311099.442935 mapread 0x006e5d thru 0x007978 (0xb1c bytes) 000924 1779311099.453212 trunc from 0x022ef5 to 0x0190f4 (0x9e00 bytes) 000926 1779311099.469151 mapread 0x004aaa thru 0x00abe3 (0x613a bytes) 000927 1779311099.470271 mapread 0x0027ac thru 0x008452 (0x5ca7 bytes) 000928 1779311099.472199 read 0x00b0a4 thru 0x010b52 (0x5aaf bytes) 000929 1779311099.472905 write 0x024250 thru 0x02e299 (0xa04a bytes) 000931 1779311099.479177 MIRROR_RESYNC 5 mirrors 000934 1779311100.088102 MIRROR_RESYNC 5 mirrors 000935 1779311100.359013 write 0x002367 thru 0x00efc4 (0xcc5e bytes) 000936 1779311100.424997 mapwrite 0x00e29e thru 0x0105aa (0x230d bytes) 000938 1779311100.471844 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e29a): Mirror 3: 0x23dd285c (stale) Mirror 5: 0x23dd285c (stale) Mirror 6: 0x23dd285c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e29a): Mirror 3: 0x23dd285c (stale) Mirror 5: 0x23dd285c (stale) Mirror 6: 0x23dd285c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e29a: skipped 000947 1779311100.739390 MIRROR_RESYNC 4 mirrors 000948 1779311101.349102 trunc from 0x02e29a to 0x0261f8 (0x80a1 bytes) 000949 1779311101.401025 mapread 0x01cf87 thru 0x0261f8 (0x9272 bytes) 000952 1779311101.425475 trunc from 0x0261f9 to 0x007b16 (0x1e6e2 bytes) 000954 1779311101.446360 write 0x012086 thru 0x01e70a (0xc685 bytes) 000956 1779311101.450048 mapread 0x01018c thru 0x01a55e (0xa3d3 bytes) 000957 1779311101.464023 MIRROR_RESYNC 4 mirrors 000958 1779311101.948578 write 0x015c95 thru 0x0210ed (0xb459 bytes) 000959 1779311102.011295 mapread 0x01e2a1 thru 0x0210ed (0x2e4d bytes) 000961 1779311102.016339 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x210ee): Mirror 5: 0xce2d37ba (stale) Mirror 6: 0xce2d37ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210ee): Mirror 5: 0xce2d37ba (stale) Mirror 6: 0xce2d37ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210ee: skipped 000965 1779311102.252298 trunc from 0x0210ee to 0x01fbe2 (0x150b bytes) 000966 1779311102.288364 trunc from 0x01fbe3 to 0x02c6c2 (0xcae0 bytes) 000967 1779311102.302119 write 0x0393d2 thru 0x03ffff (0x6c2e bytes) 000968 1779311102.303385 read 0x007f72 thru 0x00f2ed (0x737c bytes) 000970 1779311102.310105 read 0x0005f3 thru 0x00a7a7 (0xa1b5 bytes) 000971 1779311102.321365 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x143a97e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x143a97e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000972 1779311102.559065 mapwrite 0x00fbf3 thru 0x01a826 (0xac34 bytes) 000973 1779311102.637982 trunc from 0x040000 to 0x00927b (0x36d84 bytes) 000975 1779311102.650890 mapread 0x005eda thru 0x00927b (0x33a2 bytes) 000976 1779311102.670229 MIRROR_RESYNC 2 mirrors 000977 1779311103.077093 mapwrite 0x015ac1 thru 0x01e2e2 (0x8822 bytes) 000980 1779311103.226612 mapread 0x01ada9 thru 0x01e2e2 (0x353a bytes) 000981 1779311103.232842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) 000982 1779311103.348289 trunc from 0x01e2e3 to 0x03cce0 (0x1e9fe bytes) 000983 1779311103.362510 mapwrite 0x026e39 thru 0x02f293 (0x845b bytes) 000989 1779311103.421363 read 0x01926f thru 0x023d63 (0xaaf5 bytes) 000992 1779311103.438967 read 0x038935 thru 0x03cce0 (0x43ac bytes) 000993 1779311103.447195 mapread 0x00a33a thru 0x00b13f (0xe06 bytes) 000996 1779311103.449950 trunc from 0x03cce1 to 0x030aea (0xc1f6 bytes) 000997 1779311103.467763 mapread 0x02625a thru 0x02cafe (0x68a5 bytes) 001000 1779311103.471267 write 0x00cd00 thru 0x01160b (0x490c bytes) All operations completed A-OK! PASS 70a (63s) == sanity-flr test 71: check mirror extend parent fid ==== 17:05:06 (1779311106) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 17:05:08 (1779311108) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144333 s, 14.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11997 s, 8.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 17:05:12 (1779311112) Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre2 Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3071680 .. Extending file size to 2796288 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6039360 .. Extending file size to 6226176 .. Extending file size to 7971200 .. Extending file size to 7397728 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8401088 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8899776 .. Extending file size to 8197024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9161152 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8198976 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8416704 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8450784 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 59203 59204 59205 59206 59209 192.168.204.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg444-client.virtnet /mnt/lustre2 (opts:) 192.168.204.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg444-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:06:23 (1779311183) Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre2 Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2755680 .. Extending file size to 3962848 .. done Extending file size to 6450144 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8470336 .. Extending file size to 7038528 .. Extending file size to 7536224 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8928832 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8952576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9225184 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9280672 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9142400 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61018 61019 61020 61022 61024 done 192.168.204.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg444-client.virtnet /mnt/lustre2 (opts:) 192.168.204.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg444-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:07:30 (1779311250) Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3742720 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 2432 2206080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 26624 3742720 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0494607 s, 42.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183469 s, 11.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd24533ee (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg444-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:07:39 (1779311259) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0379566 s, 55.3 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 17:07:42 (1779311262) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x6e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x121:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:07:46 (1779311266) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0759656 s, 27.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:07:49 (1779311269) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10889 s, 19.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x140:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x141:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x140:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x141:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:07:53 (1779311273) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0880799 s, 23.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:08:19 (1779311299) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg444-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538701 s, 19.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2cb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2cb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2cc:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:09:09 (1779311349) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg444-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181929 s, 11.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:09:56 (1779311396) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg444-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg444-server: Pool lustre.test_204f created Adding targets to pool oleg444-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg444-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182754 s, 11.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x145:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x145:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg444-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg444-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg444-server: Pool lustre.test_204f destroyed PASS 204f (62s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:10:58 (1779311458) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:11:01 (1779311461) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:11:05 (1779311465) oleg444-server: Pool lustre.test_206 created Adding targets to pool oleg444-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg444-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg444-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg444-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg444-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:11:23 (1779311483) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303983 s, 34.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151377 s, 69.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x344542bf Mirror 3: 0x344542bf PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:11:28 (1779311488) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:11:30 (1779311490) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:11:32 (1779311492) Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0616739 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2290000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1600000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1600000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1600000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b00000 Read this (4 bytes): BBBB 192.168.204.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg444-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:11:54 (1779311514) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00659902 s, 1.2 MB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:11:58 (1779311518) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255187 s, 41.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8f lmm_fid: [0x200000405:0x8f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8f lmm_fid: [0x200000405:0x8f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611279 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 17:11:58.000000000 -0400 Modify: 2026-05-20 17:11:58.000000000 -0400 Change: 2026-05-20 17:11:59.000000000 -0400 Birth: 2026-05-20 17:11:58.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:12:02 (1779311522) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:12:03 (1779311523) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.279859 s, 146 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0460582 s, 88.9 kB/s size after second write -rw-r--r-- 1 root root 4096 May 20 17:12 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x91:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 20 17:12 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:12:07 (1779311527) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:12:11 (1779311531) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1194 sec =========== 17:12:16 (1779311536) === sanity-flr: start cleanup 17:12:17 (1779311537) === === sanity-flr: finish cleanup 17:12:21 (1779311541) ===