-----============= acceptance-small: sanity-flr ============----- Mon May 18 13:36:28 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg249-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:36:49 (1779125809) === oleg249-client.virtnet: executing check_config_client /mnt/lustre oleg249-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg249-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b8ca09fa800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b8ca09fa800.idle_timeout=debug disable quota as required oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:37:38 (1779125858) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7652] == sanity-flr test 0a: lfs mirror create with -N option == 13:37:40 (1779125860) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:38:01 (1779125881) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:38:05 (1779125885) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:38:09 (1779125889) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:38:37 (1779125917) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:38:41 (1779125921) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 13:38:45 (1779125925) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b8ca09fa800.stats=clear osc.lustre-OST0001-osc-ffff8b8ca09fa800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:38:54 (1779125934) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:39:06 (1779125946) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:39:17 (1779125957) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:39:21 (1779125961) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:39:29 (1779125969) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 13:39:39 (1779125979) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:39:47 (1779125987) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:39:51 (1779125991) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:39:59 (1779125999) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:40:11 (1779126011) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:40:20 (1779126020) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 2.09829 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278593 s, 3.8 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:40:33 (1779126033) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.606554 s, 31.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:40:43 (1779126043) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:40:53 (1779126053) Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg249-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (65s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:41:59 (1779126119) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.568253 s, 18.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:42:40 (1779126160) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg249-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:44:01 (1779126241) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.54072 s, 7.0 MB/s -rw-r--r-- 1 root root 52428800 May 18 13:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.779s user 0m0.000s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b8ca09fa800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.656s user 0m0.000s sys 0m0.226s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg249-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg249-server oleg249-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.553s user 0m0.001s sys 0m0.207s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg249-server: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg249-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg249-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b8ca09fa800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:45:38 (1779126338) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:45:42 (1779126342) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:45:45 (1779126345) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 13:45:49 (1779126349) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 13:45:58 (1779126358) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.914102 s, 13.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:46:12 (1779126372) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852469 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307951 s, 34.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:46:20 (1779126380) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:46:29 (1779126389) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00398538 s, 257 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104131 s, 983 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115718 s, 885 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00472276 s, 217 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 13:46:40 (1779126400) create test files with size 15743429 .. aeb8c1765573bb174257f9451dd87b9e - 6ae329681a08552f26c43ffd6d4059fa - 53d8ea16d4db476895d05fe83f0536d0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg249-client.virtnet /mnt/lustre (opts:) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 13:47:03 (1779126423) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:47:20 (1779126440) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:47:28 (1779126448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131053 s, 8.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707874 s, 14.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 13:47:41 (1779126461) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201222 s, 10.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.361691 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222888 s, 9.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 13:47:58 (1779126478) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 13:48:01 (1779126481) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:48:05 (1779126485) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268308 s, 39.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 13:48:14 (1779126494) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:48:17 (1779126497) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:48:28 (1779126508) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.398719 s, 26.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg249-client.virtnet /mnt/lustre (opts:) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:48:44 (1779126524) llite.lustre-ffff8b8c82d8c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.455825 s, 23.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg249-client.virtnet /mnt/lustre (opts:) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8b8c87f7a800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:49:00 (1779126540) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:49:50 (1779126590) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:50:30 (1779126630) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 13:50:33 (1779126633) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.335511 s, 9.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153489 s, 20.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xdbd9f105 (nosync,stale) Mirror 2: 0xdbd9f105 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 70aef873b0f5c93f36d0fa20e6ab44c0 - mirror 1 checksum: ef18ace6da1a1ee3886ba804c065ad79 - mirror 2 checksum: 70aef873b0f5c93f36d0fa20e6ab44c0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 18 13:50:37 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 18 13:50:37 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 70aef873b0f5c93f36d0fa20e6ab44c0 - mirror 1 checksum: ef18ace6da1a1ee3886ba804c065ad79 - mirror 2 checksum: ef18ace6da1a1ee3886ba804c065ad79 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 18 13:50:37 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 18 13:50:37 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:50:48 (1779126648) Failing mds1 on oleg249-server Stopping /mnt/lustre-mds1 (opts:) on oleg249-server 13:50:56 (1779126656) shut down facet: mds1 facet_host: oleg249-server facet_failover_host: oleg249-server Failover mds1 to oleg249-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 13:51:30 (1779126690) targets are mounted 13:51:30 (1779126690) facet_failover done oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270434 s, 38.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:51:46 (1779126706) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:51:50 (1779126710) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:51:54 (1779126714) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0117594 s, 1.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:52:04 (1779126724) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:52:08 (1779126728) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:52:12 (1779126732) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:52:41 (1779126761) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:53:12 (1779126792) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:53:37 (1779126817) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:54:02 (1779126842) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 13:54:11 (1779126851) mirror create pid 53618 mirror split pid 53619 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53618 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53619 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 13:55:19 (1779126919) Seed set to 6921 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1779126922.109007 write 0x031209 thru 0x033e1b (0x2c13 bytes) 000003 1779126922.116145 write 0x021478 thru 0x02c7eb (0xb374 bytes) 000004 1779126922.139621 MIRROR_EXTEND to 2 mirrors 000008 1779126923.374904 read 0x02b95c thru 0x031fd8 (0x667d bytes) 000010 1779126923.429507 mapread 0x011035 thru 0x015791 (0x475d bytes) 000011 1779126923.492893 read 0x0004e8 thru 0x009b4a (0x9663 bytes) 000012 1779126923.520880 mapwrite 0x017f56 thru 0x027777 (0xf822 bytes) 000013 1779126923.844253 write 0x026300 thru 0x031a33 (0xb734 bytes) 000017 1779126923.857878 mapread 0x002a4f thru 0x0113c6 (0xe978 bytes) 000020 1779126924.055932 MIRROR_RESYNC 2 mirrors 000022 1779126925.023259 write 0x0017a2 thru 0x0101ca (0xea29 bytes) truncating to largest ever: 0x2ded7 000023 1779126925.124809 trunc from 0x033e1c to 0x02ded6 (0x5f45 bytes) 000025 1779126925.161750 mapread 0x000f63 thru 0x00bc78 (0xad16 bytes) 000028 1779126925.181141 mapwrite 0x019437 thru 0x01affb (0x1bc5 bytes) 000029 1779126925.261210 mapread 0x001057 thru 0x0062d9 (0x5283 bytes) 000031 1779126925.265131 read 0x0048f2 thru 0x00a415 (0x5b24 bytes) 000033 1779126925.268944 mapwrite 0x016571 thru 0x02018e (0x9c1e bytes) 000034 1779126925.334111 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ded7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ded7): Mirror 2: 0xd6b787b8 (stale) CRC-32 checksum value for chunk [0, 0x2ded7): Mirror 1: 0xb276a293 Mirror 3: 0xb276a293 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ded7: skipped 000036 1779126926.697022 mapread 0x00a413 thru 0x010fe0 (0x6bce bytes) truncating to largest ever: 0x397da 000038 1779126926.724005 trunc from 0x02ded7 to 0x0397d9 (0xb903 bytes) 000039 1779126926.905781 read 0x008152 thru 0x016c6c (0xeb1b bytes) 000040 1779126926.920352 mapwrite 0x036e09 thru 0x03bb34 (0x4d2c bytes) 000041 1779126927.055991 read 0x01cc2b thru 0x028e4b (0xc221 bytes) 000043 1779126927.079383 read 0x02b70a thru 0x0340fd (0x89f4 bytes) 000044 1779126927.081179 trunc from 0x03bb35 to 0x0153c2 (0x26772 bytes) 000045 1779126927.107700 write 0x027d2e thru 0x0289ce (0xca1 bytes) 000046 1779126927.109022 mapwrite 0x016d0d thru 0x019848 (0x2b3c bytes) 000049 1779126927.171299 MIRROR_RESYNC 3 mirrors 000052 1779126928.261786 read 0x0022d4 thru 0x0104e1 (0xe20e bytes) 000053 1779126928.284682 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000054 1779126928.865812 read 0x01d48e thru 0x01f412 (0x1f85 bytes) 000056 1779126928.875148 trunc from 0x0289cf to 0x021430 (0x759e bytes) 000057 1779126929.104587 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000058 1779126929.311246 write 0x00ee03 thru 0x013a47 (0x4c45 bytes) 000061 1779126929.338121 mapwrite 0x03bff5 thru 0x03ffff (0x400b bytes) 000063 1779126929.469795 mapwrite 0x015358 thru 0x01e960 (0x9609 bytes) 000064 1779126929.560091 read 0x021b3c thru 0x0249ba (0x2e7f bytes) 000065 1779126929.560925 mapread 0x007eb0 thru 0x00c11f (0x4270 bytes) 000066 1779126929.576801 write 0x019de3 thru 0x0225d9 (0x87f7 bytes) 000067 1779126929.582042 read 0x012e52 thru 0x02129e (0xe44d bytes) 000068 1779126929.589044 MIRROR_EXTEND to 2 mirrors 000069 1779126930.267777 write 0x01d417 thru 0x0291ac (0xbd96 bytes) 000070 1779126930.394459 read 0x02246e thru 0x02b111 (0x8ca4 bytes) 000071 1779126930.402379 mapread 0x00cb4e thru 0x0124e4 (0x5997 bytes) 000072 1779126930.416133 write 0x02397c thru 0x02f43d (0xbac2 bytes) 000073 1779126930.423975 read 0x03ff3c thru 0x03ffff (0xc4 bytes) 000074 1779126930.464225 mapread 0x02c1af thru 0x0302f8 (0x414a bytes) 000075 1779126930.478531 MIRROR_RESYNC 2 mirrors 000076 1779126931.254631 trunc from 0x040000 to 0x014d3b (0x2b2c4 bytes) 000078 1779126931.383906 MIRROR_RESYNC 2 mirrors 000079 1779126932.286439 mapwrite 0x014b5b thru 0x015667 (0xb0d bytes) 000080 1779126932.419913 write 0x021206 thru 0x02d3a8 (0xc1a3 bytes) 000083 1779126932.425859 mapread 0x02a824 thru 0x02d3a8 (0x2b85 bytes) 000085 1779126932.429136 mapwrite 0x01905e thru 0x01c24e (0x31f1 bytes) 000086 1779126932.501966 mapread 0x00e9f1 thru 0x015470 (0x6a80 bytes) 000087 1779126932.521315 write 0x0231a3 thru 0x02cb49 (0x99a7 bytes) 000088 1779126932.524159 trunc from 0x02d3a9 to 0x00057b (0x2ce2d bytes) 000089 1779126932.557294 mapread 0x00046a thru 0x00057b (0x112 bytes) 000090 1779126932.566261 write 0x0266f1 thru 0x032c0a (0xc51a bytes) 000091 1779126932.575871 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32c0b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c0b): Mirror 2: 0x13d7e9af (stale) CRC-32 checksum value for chunk [0, 0x32c0b): Mirror 1: 0x64aca1b9 Mirror 3: 0x64aca1b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c0b: skipped 000092 1779126933.636499 write 0x03b8b6 thru 0x03ffff (0x474a bytes) 000094 1779126933.748038 mapwrite 0x02185e thru 0x02ac47 (0x93ea bytes) 000097 1779126933.844983 write 0x037eb8 thru 0x03ffff (0x8148 bytes) 000100 1779126933.890125 write 0x030624 thru 0x03e98d (0xe36a bytes) 000101 1779126933.891883 trunc from 0x040000 to 0x011f9e (0x2e061 bytes) 000102 1779126933.928055 mapread 0x00fbcf thru 0x011f9e (0x23d0 bytes) 000104 1779126933.945799 MIRROR_RESYNC 3 mirrors 000106 1779126934.875589 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000107 1779126935.195969 write 0x032ce7 thru 0x0385b6 (0x58d0 bytes) 000109 1779126935.274637 trunc from 0x0385b7 to 0x000252 (0x38364 bytes) 000110 1779126935.297842 mapwrite 0x03cccf thru 0x03f9f2 (0x2d24 bytes) 000111 1779126935.383864 mapread 0x0120e3 thru 0x01786e (0x578c bytes) 000112 1779126935.396864 write 0x0028c5 thru 0x011d5d (0xf499 bytes) 000114 1779126935.406852 read 0x007093 thru 0x00879b (0x1709 bytes) 000115 1779126935.407551 read 0x0135b5 thru 0x016b57 (0x35a3 bytes) 000116 1779126935.409064 mapread 0x0121d3 thru 0x0212bc (0xf0ea bytes) 000118 1779126935.420455 read 0x026061 thru 0x027bd9 (0x1b79 bytes) 000120 1779126935.422350 mapread 0x01c486 thru 0x021df7 (0x5972 bytes) 000128 1779126935.433925 trunc from 0x03f9f3 to 0x01bb6b (0x23e87 bytes) 000129 1779126935.467061 read 0x018634 thru 0x01bb6b (0x3538 bytes) 000131 1779126935.476200 trunc from 0x01bb6c to 0x000aaf (0x1b0bc bytes) 000133 1779126935.501044 mapread 0x000217 thru 0x000aaf (0x899 bytes) 000134 1779126935.502969 trunc from 0x000ab0 to 0x02e427 (0x2d978 bytes) 000135 1779126935.522360 write 0x00a7e5 thru 0x016a80 (0xc29c bytes) truncating to largest ever: 0x3f3de 000137 1779126935.543202 trunc from 0x02e428 to 0x03f3dd (0x10fb6 bytes) 000138 1779126935.565966 write 0x00b0db thru 0x016a07 (0xb92d bytes) 000140 1779126935.569862 trunc from 0x03f3de to 0x00b2a3 (0x3413a bytes) 000144 1779126935.594072 read 0x0085f0 thru 0x00b2a3 (0x2cb4 bytes) 000145 1779126935.596323 trunc from 0x00b2a4 to 0x03a213 (0x2ef70 bytes) 000146 1779126935.639404 MIRROR_RESYNC 2 mirrors 000148 1779126936.555781 mapread 0x0364b5 thru 0x03a213 (0x3d5f bytes) 000149 1779126936.585199 mapwrite 0x005f9b thru 0x013ca3 (0xdd09 bytes) 000150 1779126936.824860 trunc from 0x03a214 to 0x00afca (0x2f249 bytes) 000152 1779126936.893855 mapread 0x00232a thru 0x004418 (0x20ef bytes) 000157 1779126936.910743 read 0x004621 thru 0x00afca (0x69aa bytes) 000158 1779126936.914088 mapwrite 0x024983 thru 0x02f5bf (0xac3d bytes) 000159 1779126937.085276 mapwrite 0x020a12 thru 0x024258 (0x3847 bytes) 000160 1779126937.164059 mapread 0x021417 thru 0x024d9c (0x3986 bytes) 000161 1779126937.166108 trunc from 0x02f5c0 to 0x03082c (0x126d bytes) 000163 1779126937.204377 read 0x018573 thru 0x01b9e0 (0x346e bytes) 000164 1779126937.221960 write 0x01da46 thru 0x01fc80 (0x223b bytes) 000166 1779126937.225819 read 0x01522c thru 0x0200af (0xae84 bytes) 000167 1779126937.227972 write 0x03d297 thru 0x03ffff (0x2d69 bytes) 000168 1779126937.230947 mapread 0x03199c thru 0x038e00 (0x7465 bytes) 000169 1779126937.246065 mapread 0x028544 thru 0x032d58 (0xa815 bytes) 000171 1779126937.255108 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9fd67a0e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7bf7d319 Mirror 4: 0x7bf7d319 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000173 1779126938.231358 write 0x013090 thru 0x022207 (0xf178 bytes) 000175 1779126938.401481 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9fd67a0e (stale) Mirror 4: 0x7bf7d319 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcd5db91 Mirror 5: 0xcd5db91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000177 1779126939.810663 trunc from 0x040000 to 0x02581b (0x1a7e4 bytes) 000178 1779126940.089883 MIRROR_RESYNC 4 mirrors 000180 1779126941.152815 MIRROR_RESYNC 4 mirrors 000181 1779126941.716112 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000183 1779126942.145961 read 0x01bf74 thru 0x02581b (0x98a8 bytes) 000189 1779126942.172263 MIRROR_EXTEND to 4 mirrors 000190 1779126943.007571 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000191 1779126943.575261 mapread 0x01b8c3 thru 0x01c6da (0xe18 bytes) 000192 1779126943.612962 MIRROR_EXTEND to 4 mirrors 000193 1779126944.658662 write 0x01a5a2 thru 0x01b3bc (0xe1b bytes) 000196 1779126944.745595 mapread 0x0097d7 thru 0x0107bc (0x6fe6 bytes) 000199 1779126944.755315 mapwrite 0x003102 thru 0x00fd8c (0xcc8b bytes) 000201 1779126944.804847 mapread 0x00c721 thru 0x00fc6a (0x354a bytes) 000202 1779126944.806091 mapread 0x01fa63 thru 0x02581b (0x5db9 bytes) 000204 1779126944.822620 MIRROR_RESYNC 4 mirrors 000205 1779126945.841305 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000206 1779126946.196064 mapwrite 0x0253be thru 0x027298 (0x1edb bytes) 000208 1779126946.361018 write 0x0135cb thru 0x016693 (0x30c9 bytes) 000209 1779126946.369682 trunc from 0x027299 to 0x007e15 (0x1f483 bytes) 000210 1779126946.406598 MIRROR_RESYNC 3 mirrors 000213 1779126947.270275 MIRROR_RESYNC 3 mirrors 000214 1779126947.753109 mapread 0x00639f thru 0x007e15 (0x1a77 bytes) 000215 1779126947.770275 trunc from 0x007e16 to 0x039569 (0x31754 bytes) 000216 1779126947.887276 mapwrite 0x028d9f thru 0x036578 (0xd7da bytes) 000218 1779126948.135235 trunc from 0x03956a to 0x0058bc (0x33cad bytes) 000219 1779126948.218059 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x58bd): Mirror 4: 0x4d3af770 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x58bd): Mirror 4: 0x4d3af770 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x58bd: skipped 000220 1779126948.769692 mapread 0x001e4d thru 0x0058bc (0x3a70 bytes) 000221 1779126948.807059 mapwrite 0x007fc9 thru 0x009f4e (0x1f86 bytes) 000223 1779126948.930593 read 0x005f31 thru 0x009f4e (0x401e bytes) 000225 1779126948.934369 read 0x0049fe thru 0x009f4e (0x5551 bytes) 000228 1779126948.938757 write 0x01a4de thru 0x029630 (0xf153 bytes) 000232 1779126948.945522 read 0x01b779 thru 0x01b790 (0x18 bytes) 000234 1779126948.947803 trunc from 0x029631 to 0x012f06 (0x1672a bytes) 000235 1779126948.972218 trunc from 0x012f07 to 0x00f672 (0x3894 bytes) 000237 1779126948.992896 write 0x01f9bb thru 0x023092 (0x36d8 bytes) 000238 1779126948.995341 read 0x0199ce thru 0x019ae6 (0x119 bytes) 000240 1779126949.013404 mapwrite 0x024f08 thru 0x02ffae (0xb0a7 bytes) 000241 1779126949.103722 read 0x00ff15 thru 0x01b2dd (0xb3c9 bytes) 000242 1779126949.105120 mapread 0x0064bb thru 0x00f333 (0x8e79 bytes) 000243 1779126949.108849 trunc from 0x02ffaf to 0x021883 (0xe72b bytes) 000244 1779126949.144640 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x21884): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21884): Mirror 4: 0xe28f7068 (stale) CRC-32 checksum value for chunk [0, 0x21884): Mirror 1: 0xefd55e5a Mirror 5: 0xefd55e5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21884: skipped 000245 1779126950.414503 mapwrite 0x02efbc thru 0x033a41 (0x4a86 bytes) 000247 1779126950.820404 read 0x0193a4 thru 0x027169 (0xddc6 bytes) 000249 1779126950.870576 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a42): Mirror 5: 0x927a4a0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a42): Mirror 5: 0x927a4a0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a42: skipped 000250 1779126951.428520 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33a42): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a42): Mirror 5: 0x927a4a0d (stale) CRC-32 checksum value for chunk [0, 0x33a42): Mirror 1: 0x474faf03 Mirror 6: 0x474faf03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a42: skipped 000251 1779126952.955079 mapwrite 0x03ed36 thru 0x03ffff (0x12ca bytes) 000253 1779126953.151628 read 0x026e2f thru 0x02ed16 (0x7ee8 bytes) 000255 1779126953.169388 MIRROR_RESYNC 3 mirrors 000257 1779126954.102727 read 0x03b809 thru 0x03ffff (0x47f7 bytes) 000258 1779126954.141944 write 0x017eac thru 0x02106f (0x91c4 bytes) 000260 1779126954.449974 mapread 0x034eda thru 0x03ffff (0xb126 bytes) 000261 1779126954.480992 mapwrite 0x02ee25 thru 0x038a98 (0x9c74 bytes) 000263 1779126954.604385 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5b64ce9a (stale) Mirror 6: 0x5b64ce9a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfe4f133 Mirror 7: 0xcfe4f133 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000264 1779126955.543399 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5b64ce9a (stale) Mirror 6: 0x5b64ce9a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfe4f133 Mirror 7: 0xcfe4f133 Mirror 8: 0xcfe4f133 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000265 1779126956.715275 read 0x03f947 thru 0x03ffff (0x6b9 bytes) 000269 1779126956.728176 MIRROR_RESYNC 5 mirrors 000271 1779126958.095078 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000273 1779126958.937095 mapread 0x0088ca thru 0x00aa5f (0x2196 bytes) 000274 1779126959.033772 read 0x012c0d thru 0x015c87 (0x307b bytes) 000277 1779126959.167047 MIRROR_RESYNC 4 mirrors 000278 1779126960.074891 trunc from 0x040000 to 0x02d17b (0x12e84 bytes) 000286 1779126960.187862 mapwrite 0x014d42 thru 0x023961 (0xec20 bytes) 000287 1779126960.250603 write 0x028f2f thru 0x033bf0 (0xacc2 bytes) 000289 1779126960.269252 MIRROR_RESYNC 4 mirrors 000290 1779126961.498643 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000293 1779126962.173725 write 0x00582f thru 0x00a91a (0x50ec bytes) 000295 1779126962.302329 write 0x03d436 thru 0x03ffff (0x2bca bytes) 000298 1779126962.320737 write 0x018125 thru 0x026a42 (0xe91e bytes) 000299 1779126962.345533 mapwrite 0x00ab0c thru 0x01485f (0x9d54 bytes) 000300 1779126962.528349 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x10880d93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x10880d93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1779126963.098862 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000303 1779126963.412339 mapwrite 0x02dd5f thru 0x034262 (0x6504 bytes) 000307 1779126963.625380 read 0x0338c0 thru 0x0398fd (0x603e bytes) 000308 1779126963.663258 write 0x03007e thru 0x03b3bd (0xb340 bytes) 000309 1779126963.697395 MIRROR_EXTEND to 2 mirrors 000310 1779126965.295271 mapwrite 0x00e53b thru 0x01bf8b (0xda51 bytes) 000311 1779126965.672797 mapwrite 0x00426f thru 0x01300a (0xed9c bytes) 000313 1779126965.898002 read 0x02a941 thru 0x02f2fa (0x49ba bytes) 000316 1779126965.932675 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8c0fc455 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe2ed7dcf Mirror 3: 0xe2ed7dcf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000318 1779126968.144257 mapread 0x026014 thru 0x033b81 (0xdb6e bytes) 000320 1779126968.222351 mapwrite 0x001f19 thru 0x0053e7 (0x34cf bytes) 000321 1779126968.581966 trunc from 0x040000 to 0x00c601 (0x339fe bytes) 000325 1779126968.638209 trunc from 0x00c602 to 0x000634 (0xbfcd bytes) 000328 1779126968.667591 write 0x020c31 thru 0x02bff1 (0xb3c1 bytes) 000329 1779126968.673714 write 0x0020f5 thru 0x00fe31 (0xdd3d bytes) 000331 1779126968.695483 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bff2): Mirror 2: 0xbd48545d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bff2): Mirror 2: 0xbd48545d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bff2: skipped 000332 1779126969.043008 trunc from 0x02bff2 to 0x00c121 (0x1fed0 bytes) 000333 1779126969.060665 write 0x019e86 thru 0x01f019 (0x5194 bytes) 000334 1779126969.064098 mapread 0x01d6e6 thru 0x01f019 (0x1934 bytes) 000335 1779126969.066286 read 0x0086bf thru 0x00f366 (0x6ca8 bytes) 000336 1779126969.078792 mapwrite 0x011162 thru 0x020c61 (0xfb00 bytes) 000337 1779126969.169860 mapread 0x010d1f thru 0x0113d8 (0x6ba bytes) 000338 1779126969.172804 read 0x010a44 thru 0x013f4b (0x3508 bytes) 000339 1779126969.179333 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000342 1779126969.333331 MIRROR_EXTEND to 2 mirrors 000343 1779126970.244377 trunc from 0x020c62 to 0x01ad58 (0x5f09 bytes) 000345 1779126970.500799 trunc from 0x01ad59 to 0x014470 (0x68e8 bytes) 000347 1779126970.551708 write 0x003d23 thru 0x005692 (0x1970 bytes) 000348 1779126970.574607 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000349 1779126970.962745 MIRROR_EXTEND to 2 mirrors 000351 1779126972.286183 mapread 0x00f1dd thru 0x014470 (0x5294 bytes) 000352 1779126972.346368 read 0x00756d thru 0x00893b (0x13cf bytes) 000355 1779126972.395006 trunc from 0x014471 to 0x03b688 (0x27218 bytes) 000356 1779126972.495996 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000357 1779126972.722205 trunc from 0x03b689 to 0x036faf (0x46d9 bytes) 000358 1779126972.744320 read 0x016196 thru 0x024e26 (0xec91 bytes) 000359 1779126972.771323 mapwrite 0x03fc24 thru 0x03ffff (0x3dc bytes) 000362 1779126972.953210 mapread 0x03851f thru 0x03cf8f (0x4a71 bytes) 000363 1779126972.999909 MIRROR_EXTEND to 2 mirrors 000364 1779126974.095328 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000365 1779126974.444110 mapwrite 0x01f1d7 thru 0x02cf34 (0xdd5e bytes) 000366 1779126974.601038 read 0x02bd7f thru 0x03233d (0x65bf bytes) 000368 1779126974.605652 write 0x038c97 thru 0x03999c (0xd06 bytes) 000370 1779126974.623465 read 0x0121b1 thru 0x019c6c (0x7abc bytes) 000371 1779126974.636521 mapread 0x0284c1 thru 0x0285ab (0xeb bytes) 000374 1779126974.639913 write 0x00a126 thru 0x01827d (0xe158 bytes) 000375 1779126974.643257 trunc from 0x040000 to 0x021e68 (0x1e197 bytes) 000376 1779126974.669686 read 0x01aca5 thru 0x01faa6 (0x4e02 bytes) 000377 1779126974.671167 write 0x03036e thru 0x036565 (0x61f8 bytes) 000381 1779126974.679152 trunc from 0x036566 to 0x028e7b (0xd6ea bytes) 000384 1779126974.730984 MIRROR_EXTEND to 2 mirrors 000386 1779126976.081333 mapwrite 0x011551 thru 0x01ea39 (0xd4e9 bytes) 000387 1779126976.352799 write 0x0264d9 thru 0x02f62e (0x9156 bytes) 000389 1779126976.375566 read 0x0146b0 thru 0x01d280 (0x8bd1 bytes) 000390 1779126976.376904 trunc from 0x02f62f to 0x002161 (0x2d4cd bytes) 000394 1779126976.415088 write 0x03e719 thru 0x03ffff (0x18e7 bytes) 000395 1779126976.416518 mapwrite 0x01a678 thru 0x01ccec (0x2675 bytes) 000396 1779126976.529080 MIRROR_RESYNC 2 mirrors 000397 1779126977.614884 write 0x00b099 thru 0x018bfa (0xdb62 bytes) 000398 1779126977.758579 mapwrite 0x024ec3 thru 0x02689e (0x19dc bytes) 000399 1779126977.902324 read 0x006b1c thru 0x009aa6 (0x2f8b bytes) 000400 1779126977.930017 mapread 0x0374d4 thru 0x03ad0b (0x3838 bytes) 000401 1779126977.950137 mapwrite 0x03863b thru 0x03ffff (0x79c5 bytes) 000404 1779126978.057289 read 0x01e489 thru 0x0220ca (0x3c42 bytes) 000405 1779126978.058789 read 0x00a680 thru 0x01794f (0xd2d0 bytes) 000406 1779126978.082576 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000410 1779126978.373567 trunc from 0x040000 to 0x028fae (0x17051 bytes) 000411 1779126978.408266 mapread 0x005b6e thru 0x012aa5 (0xcf38 bytes) 000412 1779126978.447454 trunc from 0x028faf to 0x0117cf (0x177df bytes) 000413 1779126978.474052 mapwrite 0x002e03 thru 0x00569d (0x289b bytes) 000415 1779126978.521479 trunc from 0x0117d0 to 0x00d7ad (0x4022 bytes) 000419 1779126978.548551 read 0x007172 thru 0x00d7ad (0x663c bytes) 000424 1779126978.555273 mapwrite 0x0177b8 thru 0x01acbf (0x3508 bytes) 000426 1779126978.695841 trunc from 0x01acc0 to 0x02c0cf (0x11410 bytes) 000427 1779126978.735781 trunc from 0x02c0d0 to 0x002187 (0x29f48 bytes) 000428 1779126978.791093 MIRROR_EXTEND to 2 mirrors 000429 1779126979.723729 mapread 0x000a75 thru 0x002187 (0x1713 bytes) 000431 1779126979.785313 mapread 0x000fd2 thru 0x002187 (0x11b6 bytes) 000432 1779126979.804470 mapread 0x000ba3 thru 0x002187 (0x15e5 bytes) 000434 1779126979.833746 write 0x03fb27 thru 0x03ffff (0x4d9 bytes) 000436 1779126979.906565 write 0x00ab5e thru 0x00b714 (0xbb7 bytes) 000437 1779126979.916573 trunc from 0x040000 to 0x028c06 (0x173f9 bytes) 000439 1779126979.940343 trunc from 0x028c07 to 0x015b66 (0x130a0 bytes) 000440 1779126979.960592 mapread 0x009143 thru 0x015b66 (0xca24 bytes) 000441 1779126979.972320 read 0x004633 thru 0x00b5fb (0x6fc9 bytes) 000442 1779126979.973941 mapwrite 0x002edb thru 0x004678 (0x179e bytes) 000445 1779126980.029152 mapwrite 0x00f97b thru 0x01c8b1 (0xcf37 bytes) 000446 1779126980.188538 mapread 0x0047fb thru 0x00dd6d (0x9573 bytes) 000447 1779126980.189925 mapread 0x002d8f thru 0x011593 (0xe805 bytes) 000448 1779126980.207634 trunc from 0x01c8b2 to 0x02da6b (0x111ba bytes) 000449 1779126980.239733 read 0x002197 thru 0x00ff96 (0xde00 bytes) 000452 1779126980.249557 write 0x00335f thru 0x00c534 (0x91d6 bytes) 000454 1779126980.251798 write 0x00498c thru 0x00cc23 (0x8298 bytes) 000456 1779126980.254196 mapread 0x0001a8 thru 0x00de04 (0xdc5d bytes) 000458 1779126980.268250 mapread 0x00dff8 thru 0x0190a0 (0xb0a9 bytes) 000465 1779126980.276732 mapwrite 0x00bc36 thru 0x01101d (0x53e8 bytes) 000466 1779126980.359158 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2da6c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da6c): Mirror 2: 0x8729d011 (stale) CRC-32 checksum value for chunk [0, 0x2da6c): Mirror 1: 0xa308165b Mirror 3: 0xa308165b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da6c: skipped 000467 1779126981.605713 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2da6c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da6c): Mirror 2: 0x8729d011 (stale) CRC-32 checksum value for chunk [0, 0x2da6c): Mirror 1: 0xa308165b Mirror 3: 0xa308165b Mirror 4: 0xa308165b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da6c: skipped 000469 1779126983.057395 mapread 0x000d85 thru 0x0055a5 (0x4821 bytes) 000470 1779126983.113847 read 0x02ce3f thru 0x02da6b (0xc2d bytes) 000471 1779126983.128684 mapread 0x021770 thru 0x024725 (0x2fb6 bytes) 000473 1779126983.171588 mapwrite 0x03ca7f thru 0x03e051 (0x15d3 bytes) 000476 1779126983.383040 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e052): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e052): Mirror 2: 0xcf41ee67 (stale) Mirror 3: 0x1e9189af (stale) Mirror 4: 0x1e9189af (stale) CRC-32 checksum value for chunk [0, 0x3e052): Mirror 1: 0xafa6b731 Mirror 5: 0xafa6b731 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e052: skipped 000477 1779126984.702685 mapread 0x03cff1 thru 0x03e051 (0x1061 bytes) 000478 1779126984.752270 mapread 0x01f4d7 thru 0x02da7e (0xe5a8 bytes) 000480 1779126984.804902 mapread 0x006ea4 thru 0x00856b (0x16c8 bytes) 000481 1779126984.840287 trunc from 0x03e052 to 0x00bfb8 (0x32099 bytes) 000484 1779126985.013589 mapread 0x000cff thru 0x00bfb8 (0xb2ba bytes) 000485 1779126985.024054 mapwrite 0x01550d thru 0x024c8e (0xf782 bytes) 000488 1779126985.196790 read 0x0245f2 thru 0x024c8e (0x69d bytes) 000489 1779126985.201493 write 0x00cf25 thru 0x00d470 (0x54c bytes) 000490 1779126985.226850 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24c8f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c8f): Mirror 2: 0x2f0966b8 (stale) Mirror 3: 0x6002d6b7 (stale) Mirror 4: 0x6002d6b7 (stale) Mirror 5: 0x6002d6b7 (stale) CRC-32 checksum value for chunk [0, 0x24c8f): Mirror 1: 0xd82c5a18 Mirror 6: 0xd82c5a18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c8f: skipped 000491 1779126986.859847 mapread 0x01605c thru 0x01ee0e (0x8db3 bytes) 000492 1779126986.951502 read 0x0191cb thru 0x024c8e (0xbac4 bytes) 000494 1779126986.994147 write 0x003d67 thru 0x013637 (0xf8d1 bytes) 000495 1779126987.236242 mapwrite 0x01a03d thru 0x0224c2 (0x8486 bytes) 000496 1779126987.411925 read 0x024beb thru 0x024c8e (0xa4 bytes) 000497 1779126987.428206 mapwrite 0x00e5dd thru 0x01c8a6 (0xe2ca bytes) 000498 1779126987.598914 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c8f): Mirror 3: 0x6002d6b7 (stale) Mirror 4: 0x6002d6b7 (stale) Mirror 5: 0x6002d6b7 (stale) Mirror 6: 0xd82c5a18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c8f): Mirror 3: 0x6002d6b7 (stale) Mirror 4: 0x6002d6b7 (stale) Mirror 5: 0x6002d6b7 (stale) Mirror 6: 0xd82c5a18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c8f: skipped 000499 1779126988.433458 mapwrite 0x0088ac thru 0x00e1fb (0x5950 bytes) 000502 1779126988.575953 read 0x00fc0e thru 0x012cf5 (0x30e8 bytes) 000504 1779126988.590771 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c8f): Mirror 3: 0x6002d6b7 (stale) Mirror 4: 0x6002d6b7 (stale) Mirror 5: 0x6002d6b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c8f): Mirror 3: 0x6002d6b7 (stale) Mirror 4: 0x6002d6b7 (stale) Mirror 5: 0x6002d6b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c8f: skipped 000506 1779126989.095319 mapread 0x00af41 thru 0x00c12b (0x11eb bytes) 000507 1779126989.112231 read 0x000d69 thru 0x01084b (0xfae3 bytes) 000509 1779126989.124363 trunc from 0x024c8f to 0x032196 (0xd508 bytes) 000512 1779126989.201241 mapread 0x030956 thru 0x03112d (0x7d8 bytes) 000513 1779126989.219838 read 0x00a230 thru 0x015168 (0xaf39 bytes) 000514 1779126989.240244 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32197): Mirror 4: 0xb33c5a76 (stale) Mirror 5: 0xb33c5a76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32197): Mirror 4: 0xb33c5a76 (stale) Mirror 5: 0xb33c5a76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32197: skipped 000515 1779126989.751657 write 0x007c80 thru 0x0098ff (0x1c80 bytes) 000516 1779126989.767145 mapread 0x023938 thru 0x032196 (0xe85f bytes) truncating to largest ever: 0x3fdc1 000518 1779126989.809700 trunc from 0x032197 to 0x03fdc0 (0xdc2a bytes) 000522 1779126989.875066 mapwrite 0x01f0c7 thru 0x029a08 (0xa942 bytes) 000523 1779126990.075028 mapread 0x02065c thru 0x02984d (0x91f2 bytes) 000526 1779126990.077790 trunc from 0x03fdc1 to 0x00e7d4 (0x315ec bytes) 000527 1779126990.118664 write 0x03ebae thru 0x03ffff (0x1452 bytes) 000532 1779126990.130269 mapwrite 0x0071ca thru 0x00e4c9 (0x7300 bytes) 000533 1779126990.236667 trunc from 0x040000 to 0x008a88 (0x37577 bytes) 000535 1779126990.255568 mapread 0x0001fb thru 0x008a88 (0x888e bytes) 000536 1779126990.263842 mapread 0x007510 thru 0x008a88 (0x1579 bytes) 000537 1779126990.265108 write 0x025689 thru 0x025f8e (0x906 bytes) 000540 1779126990.268248 mapwrite 0x00fa46 thru 0x0170b7 (0x7672 bytes) 000542 1779126990.345158 mapwrite 0x019c52 thru 0x02162c (0x79db bytes) 000544 1779126990.420116 trunc from 0x025f8f to 0x005a0d (0x20581 bytes) 000545 1779126990.448387 write 0x01146d thru 0x01836c (0x6f00 bytes) 000546 1779126990.449793 trunc from 0x01836d to 0x004f40 (0x1342c bytes) 000547 1779126990.485622 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4f41): Mirror 5: 0x10f941ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4f41): Mirror 5: 0x10f941ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4f41: skipped 000548 1779126990.785625 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x4f41): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4f41): Mirror 5: 0x10f941ce (stale) CRC-32 checksum value for chunk [0, 0x4f41): Mirror 1: 0x3f7018fa Mirror 6: 0x3f7018fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4f41: skipped 000551 1779126992.242213 mapread 0x00238e thru 0x004f40 (0x2bb3 bytes) 000552 1779126992.281832 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x4f41): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4f41): Mirror 5: 0x10f941ce (stale) CRC-32 checksum value for chunk [0, 0x4f41): Mirror 1: 0x3f7018fa Mirror 6: 0x3f7018fa Mirror 7: 0x3f7018fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4f41: skipped 000555 1779126993.462379 mapread 0x00441c thru 0x004f40 (0xb25 bytes) 000558 1779126993.521280 mapread 0x0023d3 thru 0x0040a9 (0x1cd7 bytes) 000559 1779126993.628186 MIRROR_RESYNC 4 mirrors 000560 1779126994.688900 read 0x003af9 thru 0x004f40 (0x1448 bytes) 000561 1779126994.702349 mapread 0x00249d thru 0x004f40 (0x2aa4 bytes) 000562 1779126994.724161 mapwrite 0x01aa66 thru 0x026c53 (0xc1ee bytes) 000563 1779126995.059075 write 0x00a615 thru 0x019e33 (0xf81f bytes) 000564 1779126995.124800 trunc from 0x026c54 to 0x01f678 (0x75db bytes) 000565 1779126995.196381 mapwrite 0x0120d0 thru 0x021178 (0xf0a9 bytes) 000566 1779126995.471520 mapread 0x00491d thru 0x009972 (0x5056 bytes) 000567 1779126995.489740 mapread 0x008de2 thru 0x00a82e (0x1a4d bytes) 000568 1779126995.495819 write 0x0227f8 thru 0x029799 (0x6fa2 bytes) 000571 1779126995.516386 trunc from 0x02979a to 0x035ed4 (0xc73b bytes) 000572 1779126995.568463 write 0x01273d thru 0x0177d9 (0x509d bytes) 000574 1779126995.586454 write 0x00599b thru 0x00732c (0x1992 bytes) 000575 1779126995.595023 MIRROR_RESYNC 4 mirrors 000576 1779126997.695073 MIRROR_RESYNC 4 mirrors 000577 1779126998.924053 trunc from 0x035ed5 to 0x00c775 (0x2975f bytes) 000579 1779126999.026065 mapread 0x0040c8 thru 0x005f9e (0x1ed7 bytes) 000580 1779126999.037526 mapread 0x001e78 thru 0x00c775 (0xa8fe bytes) 000582 1779126999.053335 read 0x00c44c thru 0x00c775 (0x32a bytes) 000583 1779126999.054038 mapread 0x009ab6 thru 0x00c775 (0x2cc0 bytes) 000584 1779126999.060188 mapwrite 0x018220 thru 0x019900 (0x16e1 bytes) 000591 1779126999.124403 mapwrite 0x01e28f thru 0x020ee9 (0x2c5b bytes) 000592 1779126999.188841 write 0x009665 thru 0x00bcad (0x2649 bytes) 000593 1779126999.190249 write 0x02f285 thru 0x038bd0 (0x994c bytes) 000595 1779126999.196682 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38bd1): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bd1): Mirror 5: 0x1a703a02 (stale) Mirror 6: 0x1a703a02 (stale) Mirror 7: 0x1a703a02 (stale) CRC-32 checksum value for chunk [0, 0x38bd1): Mirror 1: 0x9f0828a6 Mirror 8: 0x9f0828a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bd1: skipped 000598 1779127000.380677 trunc from 0x038bd1 to 0x03f00a (0x643a bytes) 000599 1779127000.715724 read 0x02af72 thru 0x0323b8 (0x7447 bytes) 000601 1779127000.769612 write 0x02f375 thru 0x0358f2 (0x657e bytes) 000603 1779127000.793041 read 0x0223c4 thru 0x02f0a9 (0xcce6 bytes) 000604 1779127000.878649 read 0x023600 thru 0x0328d1 (0xf2d2 bytes) 000605 1779127000.900948 MIRROR_RESYNC 5 mirrors 000606 1779127003.297979 write 0x030e91 thru 0x03f940 (0xeab0 bytes) 000609 1779127003.531931 mapwrite 0x01dde8 thru 0x01f320 (0x1539 bytes) 000610 1779127003.784839 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f941): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f941): Mirror 5: 0xabe4cd1c (stale) Mirror 6: 0xabe4cd1c (stale) Mirror 7: 0xabe4cd1c (stale) Mirror 8: 0xabe4cd1c (stale) CRC-32 checksum value for chunk [0, 0x3f941): Mirror 1: 0x8e56153a Mirror 9: 0x8e56153a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f941: skipped 000612 1779127005.691332 read 0x02d463 thru 0x03663d (0x91db bytes) 000613 1779127005.789933 MIRROR_RESYNC 6 mirrors 000614 1779127006.993618 trunc from 0x03f941 to 0x000695 (0x3f2ab bytes) 000618 1779127007.098031 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x696): Mirror 6: 0xddc1812 (stale) Mirror 7: 0xddc1812 (stale) Mirror 8: 0xddc1812 (stale) Mirror 9: 0xddc1812 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x696): Mirror 6: 0xddc1812 (stale) Mirror 7: 0xddc1812 (stale) Mirror 8: 0xddc1812 (stale) Mirror 9: 0xddc1812 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x696: skipped 000619 1779127007.519253 trunc from 0x000696 to 0x009738 (0x90a3 bytes) 000621 1779127007.549306 read 0x001537 thru 0x006650 (0x511a bytes) 000627 1779127007.562600 read 0x0035a4 thru 0x009738 (0x6195 bytes) 000629 1779127007.576749 read 0x0093b5 thru 0x009738 (0x384 bytes) 000631 1779127007.578038 mapwrite 0x011a41 thru 0x01dbdb (0xc19b bytes) 000632 1779127007.675598 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1dbdc): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbdc): Mirror 6: 0xe9088203 (stale) Mirror 7: 0xe9088203 (stale) Mirror 8: 0xe9088203 (stale) Mirror 9: 0xe9088203 (stale) CRC-32 checksum value for chunk [0, 0x1dbdc): Mirror 1: 0x3c7d14d6 Mirror 10: 0x3c7d14d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbdc: skipped 000633 1779127008.891818 trunc from 0x01dbdc to 0x0202bc (0x26e1 bytes) 000635 1779127009.018144 trunc from 0x0202bd to 0x02ba34 (0xb778 bytes) 000636 1779127009.051953 read 0x005487 thru 0x007588 (0x2102 bytes) 000640 1779127009.064371 trunc from 0x02ba35 to 0x02cfd5 (0x15a1 bytes) 000641 1779127009.089319 trunc from 0x02cfd6 to 0x0004bc (0x2cb19 bytes) 000642 1779127009.111786 read 0x0000a8 thru 0x0004bc (0x415 bytes) 000644 1779127009.120206 write 0x02663b thru 0x02d035 (0x69fb bytes) 000646 1779127009.124146 write 0x00a12b thru 0x012eda (0x8db0 bytes) 000648 1779127009.133871 mapwrite 0x0316fa thru 0x03f3b3 (0xdcba bytes) 000650 1779127009.221574 trunc from 0x03f3b4 to 0x02b064 (0x1434f bytes) 000651 1779127009.240841 write 0x0366c3 thru 0x03e7cb (0x8109 bytes) 000653 1779127009.243459 write 0x02a664 thru 0x02dfa2 (0x393f bytes) 000656 1779127009.249069 mapread 0x025060 thru 0x034f60 (0xff01 bytes) 000657 1779127009.303154 mapwrite 0x0328bb thru 0x0375b7 (0x4cfd bytes) 000658 1779127009.450681 write 0x01638b thru 0x01f0aa (0x8d20 bytes) 000659 1779127009.457189 read 0x01dc6b thru 0x02c48c (0xe822 bytes) 000660 1779127009.470660 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e7cc): Mirror 7: 0xefc5f66d (stale) Mirror 8: 0xefc5f66d (stale) Mirror 9: 0xefc5f66d (stale) Mirror 10: 0x44ead83b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7cc): Mirror 7: 0xefc5f66d (stale) Mirror 8: 0xefc5f66d (stale) Mirror 9: 0xefc5f66d (stale) Mirror 10: 0x44ead83b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7cc: skipped 000664 1779127010.001864 read 0x003cd9 thru 0x005015 (0x133d bytes) 000665 1779127010.011579 mapread 0x026b07 thru 0x035f1b (0xf415 bytes) 000667 1779127010.056614 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e7cc): Mirror 8: 0xefc5f66d (stale) Mirror 9: 0xefc5f66d (stale) Mirror 10: 0x44ead83b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7cc): Mirror 8: 0xefc5f66d (stale) Mirror 9: 0xefc5f66d (stale) Mirror 10: 0x44ead83b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7cc: skipped 000668 1779127010.475616 read 0x00a914 thru 0x0168f9 (0xbfe6 bytes) 000670 1779127010.488894 trunc from 0x03e7cc to 0x01e827 (0x1ffa4 bytes) 000671 1779127010.510315 write 0x0206aa thru 0x02719e (0x6af5 bytes) 000674 1779127010.515182 read 0x01acbd thru 0x026eef (0xc233 bytes) 000676 1779127010.532532 trunc from 0x02719f to 0x0274ca (0x32c bytes) 000677 1779127010.547467 mapread 0x01fb75 thru 0x0274ca (0x7956 bytes) 000678 1779127010.549837 trunc from 0x0274cb to 0x028870 (0x13a6 bytes) 000679 1779127010.570470 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28871): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28871): Mirror 8: 0x73a5f047 (stale) Mirror 9: 0x73a5f047 (stale) Mirror 10: 0x6593d516 (stale) CRC-32 checksum value for chunk [0, 0x28871): Mirror 1: 0x27f1941b Mirror 11: 0x27f1941b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28871: skipped 000680 1779127011.751710 trunc from 0x028871 to 0x0257ee (0x3082 bytes) 000681 1779127011.907919 mapread 0x00a84a thru 0x00cab0 (0x2267 bytes) 000684 1779127011.952017 mapread 0x00a12e thru 0x00d542 (0x3415 bytes) 000686 1779127011.956362 mapread 0x002e79 thru 0x00e799 (0xb921 bytes) 000688 1779127011.976631 trunc from 0x0257ef to 0x03259d (0xcdaf bytes) 000689 1779127012.021847 trunc from 0x03259e to 0x02a4a9 (0x80f4 bytes) 000694 1779127012.070927 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a4aa): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a4aa): Mirror 8: 0xd9262bfb (stale) Mirror 9: 0xd9262bfb (stale) Mirror 10: 0x3e262625 (stale) Mirror 11: 0x9e99dae8 (stale) CRC-32 checksum value for chunk [0, 0x2a4aa): Mirror 1: 0xa3fc8b12 Mirror 12: 0xa3fc8b12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a4aa: skipped 000696 1779127013.492337 read 0x021c6b thru 0x022623 (0x9b9 bytes) 000697 1779127013.531955 MIRROR_RESYNC 6 mirrors 000698 1779127014.796664 write 0x02333e thru 0x03029e (0xcf61 bytes) 000702 1779127014.947126 MIRROR_RESYNC 6 mirrors 000707 1779127016.291277 write 0x01e993 thru 0x02309b (0x4709 bytes) 000709 1779127016.402192 read 0x02542f thru 0x027a7a (0x264c bytes) 000710 1779127016.410034 write 0x02348e thru 0x026b5c (0x36cf bytes) 000711 1779127016.411177 trunc from 0x03029f to 0x018dee (0x174b0 bytes) 000712 1779127016.438559 trunc from 0x018def to 0x02617f (0xd391 bytes) 000713 1779127016.479114 mapread 0x011279 thru 0x0155a3 (0x432b bytes) 000714 1779127016.508653 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26180): Mirror 9: 0x29c536c2 (stale) Mirror 10: 0x29c536c2 (stale) Mirror 11: 0x29c536c2 (stale) Mirror 12: 0x29c536c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26180): Mirror 9: 0x29c536c2 (stale) Mirror 10: 0x29c536c2 (stale) Mirror 11: 0x29c536c2 (stale) Mirror 12: 0x29c536c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26180: skipped 000716 1779127017.177000 write 0x029264 thru 0x02b296 (0x2033 bytes) 000717 1779127017.178163 read 0x00856f thru 0x017fda (0xfa6c bytes) 000719 1779127017.194289 mapread 0x01a584 thru 0x022532 (0x7faf bytes) 000720 1779127017.270615 MIRROR_RESYNC 5 mirrors 000721 1779127019.074206 write 0x01fa1a thru 0x02a742 (0xad29 bytes) 000724 1779127019.241842 trunc from 0x02b297 to 0x03b6ee (0x10458 bytes) 000725 1779127019.271980 mapwrite 0x02c118 thru 0x03a181 (0xe06a bytes) 000727 1779127019.545652 write 0x0038ea thru 0x010601 (0xcd18 bytes) 000730 1779127019.595142 read 0x0349de thru 0x03b6ee (0x6d11 bytes) 000731 1779127019.597151 write 0x02fe9b thru 0x03017e (0x2e4 bytes) 000732 1779127019.608128 mapread 0x031eaa thru 0x03b6ee (0x9845 bytes) 000733 1779127019.622159 MIRROR_RESYNC 5 mirrors 000734 1779127021.706493 MIRROR_RESYNC 5 mirrors 000735 1779127022.429775 mapwrite 0x028317 thru 0x02e7b9 (0x64a3 bytes) 000737 1779127022.654817 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b6ef): Mirror 9: 0x480a158c (stale) Mirror 10: 0x480a158c (stale) Mirror 11: 0x480a158c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6ef): Mirror 9: 0x480a158c (stale) Mirror 10: 0x480a158c (stale) Mirror 11: 0x480a158c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6ef: skipped 000740 1779127023.235380 read 0x00adb8 thru 0x015997 (0xabe0 bytes) 000741 1779127023.286214 MIRROR_RESYNC 4 mirrors 000742 1779127024.810833 mapread 0x02b54f thru 0x03b48d (0xff3f bytes) 000743 1779127024.855999 read 0x0275d3 thru 0x03671d (0xf14b bytes) 000745 1779127024.857776 mapread 0x038627 thru 0x03b6ee (0x30c8 bytes) 000746 1779127024.858831 mapread 0x01a6e6 thru 0x01bbe8 (0x1503 bytes) 000747 1779127024.870140 mapread 0x02ade1 thru 0x031487 (0x66a7 bytes) 000748 1779127024.876737 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000750 1779127025.230324 trunc from 0x03b6ef to 0x032ce4 (0x8a0a bytes) 000752 1779127025.325813 write 0x03093f thru 0x03365f (0x2d21 bytes) 000754 1779127025.334911 mapread 0x00b18d thru 0x014486 (0x92fa bytes) 000756 1779127025.353174 read 0x010c7e thru 0x0121dc (0x155f bytes) 000757 1779127025.354250 trunc from 0x033660 to 0x0155b8 (0x1e0a7 bytes) 000761 1779127025.378197 trunc from 0x0155b9 to 0x0169f0 (0x1438 bytes) 000763 1779127025.402011 trunc from 0x0169f1 to 0x01a820 (0x3e30 bytes) 000764 1779127025.419938 write 0x002057 thru 0x005e58 (0x3e02 bytes) 000766 1779127025.429761 write 0x00f54f thru 0x0122d7 (0x2d89 bytes) 000768 1779127025.432791 write 0x01d719 thru 0x02454f (0x6e37 bytes) 000771 1779127025.436240 mapread 0x0176f9 thru 0x01a98b (0x3293 bytes) 000773 1779127025.447907 trunc from 0x024550 to 0x026a30 (0x24e1 bytes) 000774 1779127025.480244 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a31): Mirror 11: 0x499775 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a31): Mirror 11: 0x499775 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a31: skipped 000775 1779127025.769166 mapread 0x0092ad thru 0x00cb4b (0x389f bytes) 000776 1779127025.791095 mapread 0x001d99 thru 0x007a8f (0x5cf7 bytes) 000777 1779127025.799334 mapwrite 0x01f530 thru 0x02086f (0x1340 bytes) 000778 1779127025.849984 write 0x020316 thru 0x027b6e (0x7859 bytes) 000779 1779127025.853926 mapread 0x0108ea thru 0x015e01 (0x5518 bytes) 000782 1779127025.870817 mapread 0x001fb8 thru 0x0046f7 (0x2740 bytes) 000783 1779127025.872487 mapwrite 0x03a33f thru 0x03a699 (0x35b bytes) 000784 1779127025.969771 read 0x009a2b thru 0x0169f1 (0xcfc7 bytes) 000785 1779127025.977272 write 0x01fcef thru 0x022f8a (0x329c bytes) 000786 1779127025.978539 write 0x026985 thru 0x02de74 (0x74f0 bytes) 000788 1779127026.001247 mapread 0x022f1f thru 0x02a932 (0x7a14 bytes) 000790 1779127026.018580 trunc from 0x03a69a to 0x008bf6 (0x31aa3 bytes) 000791 1779127026.037486 mapwrite 0x02262f thru 0x0230ee (0xac0 bytes) 000793 1779127026.114687 write 0x02155e thru 0x027775 (0x6218 bytes) 000794 1779127026.122900 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000795 1779127026.292753 mapread 0x015295 thru 0x01abae (0x591a bytes) 000796 1779127026.313864 mapread 0x019c3f thru 0x027004 (0xd3c6 bytes) 000799 1779127026.332296 trunc from 0x027776 to 0x01c716 (0xb05f bytes) 000802 1779127026.351750 write 0x025d5e thru 0x02f1be (0x9461 bytes) 000805 1779127026.356844 write 0x03a128 thru 0x03d6a7 (0x3580 bytes) 000806 1779127026.358998 mapwrite 0x027691 thru 0x036b48 (0xf4b8 bytes) 000808 1779127026.411937 trunc from 0x03d6a8 to 0x00dad9 (0x2fbce bytes) 000809 1779127026.435886 read 0x00a2e1 thru 0x00b37b (0x109b bytes) 000810 1779127026.441500 read 0x000101 thru 0x00b682 (0xb582 bytes) 000811 1779127026.456407 mapwrite 0x026819 thru 0x02c12c (0x5914 bytes) 000813 1779127026.556164 read 0x0074d6 thru 0x016b0b (0xf636 bytes) 000816 1779127026.559943 read 0x012232 thru 0x01605b (0x3e2a bytes) 000818 1779127026.562510 read 0x0218fa thru 0x022d9e (0x14a5 bytes) 000819 1779127026.563752 read 0x01c3be thru 0x02463f (0x8282 bytes) 000821 1779127026.565947 mapread 0x0127bd thru 0x012c1d (0x461 bytes) 000822 1779127026.567773 mapread 0x014ac8 thru 0x02245c (0xd995 bytes) 000823 1779127026.569927 mapread 0x01fa29 thru 0x02ad89 (0xb361 bytes) 000827 1779127026.573254 trunc from 0x02c12d to 0x0375d1 (0xb4a5 bytes) 000831 1779127026.594787 mapwrite 0x03f1b0 thru 0x03ffff (0xe50 bytes) 000832 1779127026.666314 trunc from 0x040000 to 0x033562 (0xca9d bytes) 000833 1779127026.683828 mapread 0x01470f thru 0x0213e6 (0xccd8 bytes) 000835 1779127026.685867 write 0x0160b5 thru 0x02100a (0xaf56 bytes) 000836 1779127026.692763 mapread 0x002caa thru 0x00f850 (0xcba7 bytes) 000837 1779127026.695272 mapwrite 0x0017b7 thru 0x010035 (0xe87f bytes) 000840 1779127026.757943 trunc from 0x033563 to 0x03a8b3 (0x7351 bytes) 000841 1779127026.777359 read 0x039e10 thru 0x03a8b3 (0xaa4 bytes) 000842 1779127026.789155 write 0x0190f5 thru 0x022558 (0x9464 bytes) 000845 1779127026.794193 write 0x017fd1 thru 0x026bab (0xebdb bytes) 000847 1779127026.798009 trunc from 0x03a8b4 to 0x01dbad (0x1cd06 bytes) 000848 1779127026.819168 trunc from 0x01dbae to 0x00eda8 (0xee05 bytes) 000850 1779127026.838838 write 0x02e4a3 thru 0x03d67f (0xf1dd bytes) 000853 1779127026.844166 mapwrite 0x03f404 thru 0x03ffff (0xbfc bytes) 000856 1779127026.959296 mapread 0x03966e thru 0x03ffff (0x6992 bytes) 000858 1779127026.961237 write 0x01c887 thru 0x01d5a6 (0xd20 bytes) 000859 1779127027.004240 write 0x01b530 thru 0x01f31b (0x3dec bytes) 000861 1779127027.028942 read 0x01a779 thru 0x01ae33 (0x6bb bytes) 000864 1779127027.055880 trunc from 0x040000 to 0x02a0c6 (0x15f39 bytes) 000866 1779127027.104584 mapwrite 0x03aa34 thru 0x03ffff (0x55cc bytes) 000867 1779127027.264039 mapread 0x0041dc thru 0x008327 (0x414c bytes) 000868 1779127027.266205 write 0x008426 thru 0x013dae (0xb989 bytes) 000871 1779127027.278197 mapwrite 0x003995 thru 0x008df3 (0x545f bytes) 000875 1779127027.391252 mapwrite 0x021ac4 thru 0x02f4fd (0xda3a bytes) 000877 1779127027.574992 trunc from 0x040000 to 0x036a92 (0x956d bytes) 000878 1779127027.595268 mapread 0x01734a thru 0x0191e0 (0x1e97 bytes) 000881 1779127027.599168 mapwrite 0x00332b thru 0x00abda (0x78b0 bytes) 000883 1779127027.676283 write 0x01b6c8 thru 0x024299 (0x8bd2 bytes) 000884 1779127027.677676 mapread 0x02b9fb thru 0x03069f (0x4ca5 bytes) 000885 1779127027.687343 write 0x0252ae thru 0x029431 (0x4184 bytes) 000887 1779127027.689068 read 0x0198a4 thru 0x01dfed (0x474a bytes) 000889 1779127027.692286 trunc from 0x036a93 to 0x031675 (0x541d bytes) 000890 1779127027.713230 mapread 0x02347c thru 0x02c66d (0x91f2 bytes) 000891 1779127027.716202 mapread 0x016669 thru 0x01d95e (0x72f6 bytes) 000892 1779127027.718840 trunc from 0x031676 to 0x01804a (0x1962b bytes) 000893 1779127027.750836 MIRROR_EXTEND to 2 mirrors 000894 1779127028.640134 MIRROR_EXTEND to 3 mirrors 000895 1779127030.043299 trunc from 0x01804b to 0x004c74 (0x133d6 bytes) 000896 1779127030.165209 mapwrite 0x03fd6b thru 0x03ffff (0x295 bytes) 000898 1779127030.339506 trunc from 0x040000 to 0x03231a (0xdce5 bytes) 000900 1779127030.364318 mapread 0x0230b6 thru 0x0321d1 (0xf11c bytes) 000901 1779127030.392797 trunc from 0x03231b to 0x006c39 (0x2b6e1 bytes) 000903 1779127030.422815 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x6c3a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6c3a): Mirror 2: 0x251b3462 (stale) Mirror 3: 0x251b3462 (stale) CRC-32 checksum value for chunk [0, 0x6c3a): Mirror 1: 0x2bdf3734 Mirror 4: 0x2bdf3734 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6c3a: skipped 000904 1779127031.450141 write 0x00bd95 thru 0x00cdd0 (0x103c bytes) 000906 1779127031.529486 mapwrite 0x035f5d thru 0x03ffff (0xa0a3 bytes) 000908 1779127031.646284 trunc from 0x040000 to 0x01ac10 (0x253ef bytes) 000910 1779127031.674734 read 0x011c08 thru 0x01ac10 (0x9009 bytes) 000911 1779127031.684986 mapread 0x00a995 thru 0x017f19 (0xd585 bytes) 000912 1779127031.701686 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1ac11): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac11): Mirror 2: 0xfe958743 (stale) Mirror 3: 0xfe958743 (stale) Mirror 4: 0xed56a870 (stale) CRC-32 checksum value for chunk [0, 0x1ac11): Mirror 1: 0x72d88f33 Mirror 5: 0x72d88f33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac11: skipped 000915 1779127032.794692 mapread 0x009197 thru 0x01692b (0xd795 bytes) 000916 1779127032.844864 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1ac11): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac11): Mirror 2: 0xfe958743 (stale) Mirror 3: 0xfe958743 (stale) Mirror 4: 0xed56a870 (stale) CRC-32 checksum value for chunk [0, 0x1ac11): Mirror 1: 0x72d88f33 Mirror 5: 0x72d88f33 Mirror 6: 0x72d88f33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac11: skipped 000918 1779127033.981935 MIRROR_RESYNC 6 mirrors 000919 1779127035.176793 read 0x01a40e thru 0x01ac10 (0x803 bytes) 000921 1779127035.192440 read 0x00503e thru 0x0138c5 (0xe888 bytes) 000923 1779127035.225164 mapread 0x00192c thru 0x005a48 (0x411d bytes) 000926 1779127035.238408 write 0x034b01 thru 0x038bab (0x40ab bytes) 000928 1779127035.544987 mapread 0x0144da thru 0x01ec38 (0xa75f bytes) 000930 1779127035.646891 read 0x01b6c3 thru 0x01d70c (0x204a bytes) 000931 1779127035.648112 mapread 0x01af25 thru 0x02261e (0x76fa bytes) 000932 1779127035.653030 trunc from 0x038bac to 0x021e25 (0x16d86 bytes) 000933 1779127035.690025 read 0x017049 thru 0x021e25 (0xaddd bytes) 000934 1779127035.691314 mapread 0x00c1fd thru 0x0121c1 (0x5fc5 bytes) 000935 1779127035.724714 MIRROR_RESYNC 6 mirrors 000936 1779127037.471137 MIRROR_EXTEND to 7 mirrors 000937 1779127038.461829 write 0x03bae1 thru 0x03ffff (0x451f bytes) 000940 1779127038.560154 mapwrite 0x0118da thru 0x017285 (0x59ac bytes) 000941 1779127038.697688 MIRROR_RESYNC 7 mirrors 000942 1779127040.556576 mapwrite 0x02a562 thru 0x0367b3 (0xc252 bytes) 000943 1779127040.877596 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8e2fcb2c (stale) Mirror 3: 0x8e2fcb2c (stale) Mirror 4: 0x8e2fcb2c (stale) Mirror 5: 0x8e2fcb2c (stale) Mirror 6: 0x8e2fcb2c (stale) Mirror 7: 0x8e2fcb2c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcb33649b Mirror 8: 0xcb33649b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000944 1779127042.209227 MIRROR_RESYNC 8 mirrors 000946 1779127043.669923 trunc from 0x040000 to 0x0156ea (0x2a915 bytes) 000947 1779127043.804081 read 0x00d546 thru 0x00e3fd (0xeb8 bytes) 000948 1779127043.812044 mapwrite 0x012cd5 thru 0x01a3eb (0x7717 bytes) 000949 1779127044.015325 mapread 0x0135d0 thru 0x014915 (0x1346 bytes) 000950 1779127044.016470 mapwrite 0x034ef8 thru 0x03ffff (0xb108 bytes) 000951 1779127044.238178 write 0x03f1e4 thru 0x03ffff (0xe1c bytes) 000952 1779127044.248655 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcb33649b (stale) Mirror 3: 0xcb33649b (stale) Mirror 4: 0xcb33649b (stale) Mirror 5: 0xcb33649b (stale) Mirror 6: 0xcb33649b (stale) Mirror 7: 0xcb33649b (stale) Mirror 8: 0xcb33649b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7fb2547c Mirror 9: 0x7fb2547c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000953 1779127045.746635 write 0x00e205 thru 0x00edb9 (0xbb5 bytes) 000957 1779127045.859627 mapread 0x02f745 thru 0x0370dc (0x7998 bytes) 000958 1779127045.900943 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcb33649b (stale) Mirror 3: 0xcb33649b (stale) Mirror 4: 0xcb33649b (stale) Mirror 5: 0xcb33649b (stale) Mirror 6: 0xcb33649b (stale) Mirror 7: 0xcb33649b (stale) Mirror 8: 0xcb33649b (stale) Mirror 9: 0x7fb2547c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x15114d2 Mirror 10: 0x15114d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000960 1779127047.463090 read 0x01b3f4 thru 0x0251ab (0x9db8 bytes) 000961 1779127047.484070 write 0x03b5a9 thru 0x03e291 (0x2ce9 bytes) 000962 1779127047.626139 mapread 0x0379fc thru 0x03ffff (0x8604 bytes) 000963 1779127047.639967 read 0x020909 thru 0x02db94 (0xd28c bytes) 000964 1779127047.653704 mapread 0x031152 thru 0x03a512 (0x93c1 bytes) 000965 1779127047.657656 mapwrite 0x00544c thru 0x009743 (0x42f8 bytes) 000967 1779127047.713088 read 0x00f3f6 thru 0x00fa3a (0x645 bytes) 000968 1779127047.720623 MIRROR_RESYNC 10 mirrors 000969 1779127050.767510 mapwrite 0x008ac7 thru 0x01801f (0xf559 bytes) 000971 1779127051.114627 read 0x03afdc thru 0x03ffff (0x5024 bytes) 000972 1779127051.135931 read 0x004248 thru 0x00cd90 (0x8b49 bytes) 000976 1779127051.138342 trunc from 0x040000 to 0x016133 (0x29ecc bytes) 000978 1779127051.197941 mapwrite 0x01cc04 thru 0x029519 (0xc916 bytes) 000981 1779127051.415796 trunc from 0x02951a to 0x0311ab (0x7c92 bytes) 000988 1779127051.497285 trunc from 0x0311ac to 0x038c53 (0x7aa8 bytes) 000989 1779127051.521245 mapwrite 0x014ec2 thru 0x019966 (0x4aa5 bytes) 000990 1779127051.686399 mapread 0x01188a thru 0x017ea0 (0x6617 bytes) 000991 1779127051.687966 write 0x01f215 thru 0x026cc1 (0x7aad bytes) 000992 1779127051.692717 read 0x014307 thru 0x018a74 (0x476e bytes) 000993 1779127051.694840 write 0x0105ee thru 0x01f699 (0xf0ac bytes) 000994 1779127051.713598 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38c54): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c54): Mirror 2: 0x67bab3bb (stale) Mirror 3: 0x67bab3bb (stale) Mirror 4: 0x67bab3bb (stale) Mirror 5: 0x67bab3bb (stale) Mirror 6: 0x67bab3bb (stale) Mirror 7: 0x67bab3bb (stale) Mirror 8: 0x67bab3bb (stale) Mirror 9: 0x67bab3bb (stale) Mirror 10: 0x67bab3bb (stale) CRC-32 checksum value for chunk [0, 0x38c54): Mirror 1: 0x83f421aa Mirror 11: 0x83f421aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c54: skipped 000997 1779127053.498675 mapread 0x00026c thru 0x00068b (0x420 bytes) 000998 1779127053.525363 write 0x01e2c9 thru 0x02d602 (0xf33a bytes) 000999 1779127053.668986 mapwrite 0x03be7f thru 0x03ffff (0x4181 bytes) 001000 1779127053.836803 mapwrite 0x00ce9a thru 0x0146bc (0x7823 bytes) All operations completed A-OK! PASS 70a (140s) == sanity-flr test 71: check mirror extend parent fid ==== 13:57:39 (1779127059) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 13:57:43 (1779127063) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.234507 s, 8.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328446 s, 3.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 13:57:54 (1779127074) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1437152 .. Extending file size to 4020800 .. done Extending file size to 2680736 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6472928 .. Extending file size to 4213280 .. failed Extending file size to 8232608 .. Extending file size to 7376224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9130080 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9084256 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9091808 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56705 56706 56708 56711 56713 done 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) 192.168.202.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:59:20 (1779127160) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 829568 .. Extending file size to 4165312 .. Extending file size to 9183680 .. Extending file size to 5020736 .. Extending file size to 8114752 .. Extending file size to 8981696 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58138 58139 58140 58142 58144 done 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) 192.168.202.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (93s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:00:53 (1779127253) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113724 s, 18.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.323296 s, 6.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4ffda0ea (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:01:15 (1779127275) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143295 s, 14.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 14:01:23 (1779127283) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:01:30 (1779127290) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.324104 s, 6.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:01:39 (1779127299) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213252 s, 9.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:01:49 (1779127309) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.445985 s, 4.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:02:23 (1779127343) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg249-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111895 s, 9.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:03:25 (1779127405) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg249-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.395239 s, 5.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:04:30 (1779127470) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg249-server mds-ost sync done. Creating new pool test_204f oleg249-server: Pool lustre.test_204f created Adding targets to pool oleg249-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg249-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.376846 s, 5.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg249-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg249-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg249-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:05:50 (1779127550) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:05:58 (1779127558) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:06:07 (1779127567) oleg249-server: Pool lustre.test_206 created Adding targets to pool oleg249-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg249-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg249-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg249-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg249-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:06:35 (1779127595) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694927 s, 15.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400273 s, 26.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4d2b19ac Mirror 3: 0x4d2b19ac PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:06:46 (1779127606) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:06:50 (1779127610) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:06:54 (1779127614) Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.135766 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x930000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x930000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x930000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d30000 Read this (4 bytes): BBBB 192.168.202.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg249-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:07:33 (1779127653) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0409051 s, 200 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:07:40 (1779127660) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365737 s, 28.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ef lmm_fid: [0x200000404:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ef lmm_fid: [0x200000404:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834415 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-18 14:07:42.000000000 -0400 Modify: 2026-05-18 14:07:42.000000000 -0400 Change: 2026-05-18 14:07:43.000000000 -0400 Birth: 2026-05-18 14:07:42.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:07:47 (1779127667) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:07:50 (1779127670) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.685079 s, 59.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0527801 s, 77.6 kB/s size after second write -rw-r--r-- 1 root root 4096 May 18 14:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 18 14:07 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:07:58 (1779127678) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:08:05 (1779127685) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1904 sec =========== 14:08:13 (1779127693) === sanity-flr: start cleanup 14:08:14 (1779127694) === === sanity-flr: finish cleanup 14:08:20 (1779127700) ===