-----============= acceptance-small: sanity-flr ============----- Tue Sep 8 06:25:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg139-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:25:45 (1788863145) === oleg139-client.virtnet: executing check_config_client /mnt/lustre oleg139-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg139-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9490847b3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9490847b3000.idle_timeout=debug disable quota as required oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:26:28 (1788863188) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7635] == sanity-flr test 0a: lfs mirror create with -N option == 06:26:31 (1788863191) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:26:51 (1788863211) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:26:56 (1788863216) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:26:59 (1788863219) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:27:30 (1788863250) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:27:36 (1788863256) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (6s) == sanity-flr test 0g: lfs mirror create flags support === 06:27:41 (1788863261) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9490847b3000.stats=clear osc.lustre-OST0001-osc-ffff9490847b3000.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:27:53 (1788863273) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:28:05 (1788863285) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:28:15 (1788863295) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:28:19 (1788863299) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:28:29 (1788863309) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 06:28:40 (1788863320) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:28:48 (1788863328) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:28:51 (1788863331) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:29:00 (1788863340) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:29:11 (1788863351) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:29:19 (1788863359) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.08088 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109189 s, 9.6 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:29:30 (1788863370) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.150791 s, 27.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:29:38 (1788863378) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:29:50 (1788863390) Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg139-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (88s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:31:18 (1788863478) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.420246 s, 17.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:32:06 (1788863526) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg139-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:33:33 (1788863613) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.79329 s, 10.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 8 06:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.258s user 0m0.002s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9490847b3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.257s user 0m0.000s sys 0m0.185s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.345s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg139-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg139-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9490847b3000.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:35:18 (1788863718) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:35:21 (1788863721) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:35:25 (1788863725) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 06:35:28 (1788863728) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 06:35:37 (1788863737) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.672419 s, 18.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:35:50 (1788863750) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225874 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334517 s, 31.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:35:58 (1788863758) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:36:07 (1788863767) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104163 s, 983 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00159464 s, 642 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010762 s, 951 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000755449 s, 1.4 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 06:36:17 (1788863777) create test files with size 15751693 .. 8e16bdfc5729e736cb39fafe0a2e4358 - 7de4a2b8c2cdda7401088760582ad5f4 - 09559222b102ffefe50a45b74556f4bb - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg139-client.virtnet /mnt/lustre (opts:) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 06:36:41 (1788863801) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:36:55 (1788863815) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:37:02 (1788863822) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128779 s, 8.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523443 s, 20.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 06:37:13 (1788863833) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138806 s, 15.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.491538 s, 8.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25776 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 06:37:31 (1788863851) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:37:34 (1788863854) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:37:38 (1788863858) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865566 s, 12.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 06:37:49 (1788863869) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:37:53 (1788863873) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:38:04 (1788863884) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.430527 s, 24.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg139-client.virtnet /mnt/lustre (opts:) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:38:16 (1788863896) llite.lustre-ffff949089311000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.406397 s, 25.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg139-client.virtnet /mnt/lustre (opts:) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff949084c36800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:38:33 (1788863913) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:39:15 (1788863955) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:39:59 (1788863999) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 06:40:02 (1788864002) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.265281 s, 11.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0938917 s, 33.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7980c17 (nosync,stale) Mirror 2: 0x7980c17 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d98d6d41f0e058d756a1d814bf19d246 - mirror 1 checksum: 7548969b1ce89505fa82e0eb6599d809 - mirror 2 checksum: d98d6d41f0e058d756a1d814bf19d246 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T06:40:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T06:40:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d98d6d41f0e058d756a1d814bf19d246 - mirror 1 checksum: 7548969b1ce89505fa82e0eb6599d809 - mirror 2 checksum: 7548969b1ce89505fa82e0eb6599d809 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T06:40:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T06:40:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:40:15 (1788864015) Failing mds1 on oleg139-server Stopping /mnt/lustre-mds1 (opts:) on oleg139-server 06:40:21 (1788864021) shut down facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server Failover mds1 to oleg139-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 06:40:46 (1788864046) targets are mounted 06:40:46 (1788864046) facet_failover done oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267655 s, 39.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:41:04 (1788864064) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:41:07 (1788864067) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:41:11 (1788864071) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0056912 s, 2.9 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:41:19 (1788864079) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:41:22 (1788864082) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0131327 s, 312 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0672571 s, 60.9 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:41:30 (1788864090) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:41:54 (1788864114) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:42:25 (1788864145) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:42:50 (1788864170) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:43:16 (1788864196) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 06:43:24 (1788864204) mirror create pid 54352 mirror split pid 54354 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54352 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54354 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 06:44:33 (1788864273) Seed set to 4276 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read skipping zero size read truncating to largest ever: 0x26dbe 000007 1788864276.761496 trunc from 00000000 to 0x026dbd (0x26dbe bytes) 000012 1788864276.792152 mapread 0x021f3c thru 0x026dbd (0x4e82 bytes) 000015 1788864276.818019 mapwrite 0x008b2a thru 0x010dee (0x82c5 bytes) 000016 1788864276.912964 mapwrite 0x026734 thru 0x033748 (0xd015 bytes) 000017 1788864277.034188 trunc from 0x033749 to 0x00a37d (0x293cb bytes) 000018 1788864277.054033 write 0x02e59c thru 0x034aed (0x6552 bytes) 000019 1788864277.056244 read 0x0157d5 thru 0x01f662 (0x9e8e bytes) 000020 1788864277.065098 trunc from 0x034aee to 0x001cfd (0x32df0 bytes) 000021 1788864277.084674 mapread 0x0012c5 thru 0x001cfd (0xa39 bytes) 000024 1788864277.086729 read 0x00138b thru 0x001cfd (0x973 bytes) 000028 1788864277.091013 mapread 0x001942 thru 0x001cfd (0x3bc bytes) 000029 1788864277.092159 punch from 0x001638 to 0x001cfd (0x6c6 bytes) 000032 1788864277.099163 mapwrite 0x02a4cc thru 0x039cb0 (0xf7e5 bytes) 000033 1788864277.178410 read 0x000d28 thru 0x001100 (0x3d9 bytes) 000034 1788864277.179625 read 0x039a9c thru 0x039cb0 (0x215 bytes) 000036 1788864277.182652 write 0x02ca1a thru 0x034525 (0x7b0c bytes) 000037 1788864277.189473 trunc from 0x039cb1 to 0x0144a0 (0x25810 bytes) truncating to largest ever: 0x2f2d9 000038 1788864277.210826 trunc from 0x0144a1 to 0x02f2d8 (0x1ae38 bytes) 000040 1788864277.246697 write 0x010b70 thru 0x01eb45 (0xdfd6 bytes) 000041 1788864277.256116 mapread 0x02ca6d thru 0x02f2d8 (0x286c bytes) 000042 1788864277.273111 MIRROR_EXTEND to 2 mirrors 000043 1788864278.460874 mapwrite 0x0040c3 thru 0x0041fd (0x13b bytes) 000044 1788864278.629715 mapread 0x022049 thru 0x02cc17 (0xabcf bytes) 000046 1788864278.648588 mapwrite 0x00067f thru 0x0029b8 (0x233a bytes) 000047 1788864278.691445 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000049 1788864278.914875 mapwrite 0x02ad4c thru 0x03139a (0x664f bytes) 000053 1788864279.040953 read 0x020643 thru 0x02bc45 (0xb603 bytes) 000055 1788864279.053033 mapread 0x02e716 thru 0x03139a (0x2c85 bytes) 000056 1788864279.054159 write 0x039801 thru 0x03d955 (0x4155 bytes) 000057 1788864279.055139 mapwrite 0x0334ee thru 0x03e56d (0xb080 bytes) 000059 1788864279.217316 read 0x00ba3d thru 0x016aba (0xb07e bytes) 000061 1788864279.230863 read 0x033e1d thru 0x03a3a0 (0x6584 bytes) 000062 1788864279.231673 punch from 0x007d30 to 0x011667 (0x9938 bytes) 000063 1788864279.247894 mapread 0x023007 thru 0x0240d7 (0x10d1 bytes) 000065 1788864279.249448 read 0x0054e1 thru 0x00b424 (0x5f44 bytes) 000067 1788864279.269647 read 0x00ee4f thru 0x01b24c (0xc3fe bytes) 000068 1788864279.270345 mapread 0x000fd5 thru 0x010977 (0xf9a3 bytes) 000069 1788864279.274335 mapread 0x020ecd thru 0x02ac74 (0x9da8 bytes) 000070 1788864279.276213 mapwrite 0x0064c0 thru 0x01196a (0xb4ab bytes) 000072 1788864279.326353 mapwrite 0x0352a2 thru 0x03ffff (0xad5e bytes) 000074 1788864279.427420 mapread 0x0121b2 thru 0x01345c (0x12ab bytes) 000075 1788864279.430027 mapwrite 0x00129b thru 0x00c693 (0xb3f9 bytes) 000077 1788864279.489448 MIRROR_EXTEND to 2 mirrors 000078 1788864280.797956 mapwrite 0x0349e8 thru 0x03a501 (0x5b1a bytes) 000079 1788864281.088537 punch from 0x01c45b to 0x02930a (0xceb0 bytes) 000082 1788864281.138235 mapwrite 0x00f17a thru 0x012aa3 (0x392a bytes) 000085 1788864281.296247 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000086 1788864281.547882 mapread 0x00572b thru 0x00ab01 (0x53d7 bytes) 000087 1788864281.586240 trunc from 0x040000 to 0x00c1e5 (0x33e1a bytes) 000089 1788864281.618487 mapread 0x00117d thru 0x00c1e5 (0xb069 bytes) 000092 1788864281.623366 punch from 0x0037f9 to 0x00c1e5 (0x89ed bytes) 000093 1788864281.644118 mapwrite 0x0099f1 thru 0x011bbd (0x81cd bytes) 000094 1788864281.855348 mapwrite 0x002866 thru 0x00da10 (0xb1ab bytes) 000096 1788864282.038847 read 0x005c17 thru 0x011bbd (0xbfa7 bytes) 000097 1788864282.039554 read 0x00023f thru 0x0091ca (0x8f8c bytes) 000098 1788864282.053448 read 0x0063c4 thru 0x009361 (0x2f9e bytes) 000099 1788864282.061427 mapread 0x009f08 thru 0x00bfab (0x20a4 bytes) 000101 1788864282.080287 punch from 0x0025a8 to 0x009168 (0x6bc1 bytes) 000102 1788864282.095267 read 0x00db07 thru 0x011bbd (0x40b7 bytes) 000103 1788864282.097572 write 0x03c485 thru 0x03ffff (0x3b7b bytes) 000104 1788864282.101854 punch from 0x03c10a to 0x03ffff (0x3ef6 bytes) 000107 1788864282.127463 mapread 0x011be3 thru 0x017fb2 (0x63d0 bytes) 000109 1788864282.186927 write 0x02aa5b thru 0x03490d (0x9eb3 bytes) 000113 1788864282.204492 punch from 0x00356f to 0x00e78a (0xb21c bytes) 000115 1788864282.259627 read 0x021c04 thru 0x027f7c (0x6379 bytes) 000116 1788864282.263833 read 0x0057cd thru 0x014d70 (0xf5a4 bytes) 000117 1788864282.309383 mapwrite 0x03b119 thru 0x03ffff (0x4ee7 bytes) truncating to largest ever: 0x2ffbb 000118 1788864282.477438 trunc from 0x040000 to 0x02ffba (0x10045 bytes) 000121 1788864282.512478 mapwrite 0x033b3d thru 0x03b124 (0x75e8 bytes) 000122 1788864282.733062 mapread 0x02ea1f thru 0x0307c6 (0x1da8 bytes) 000124 1788864282.745139 mapread 0x023804 thru 0x0285ee (0x4deb bytes) 000125 1788864282.752337 mapread 0x017460 thru 0x01f020 (0x7bc1 bytes) truncating to largest ever: 0x3b193 000126 1788864282.754022 trunc from 0x03b125 to 0x03b192 (0x6e bytes) 000127 1788864282.817016 MIRROR_EXTEND to 2 mirrors 000128 1788864284.721018 MIRROR_EXTEND to 3 mirrors 000130 1788864286.502416 read 0x02792a thru 0x02dce4 (0x63bb bytes) 000131 1788864286.551408 mapwrite 0x030f98 thru 0x03213f (0x11a8 bytes) 000132 1788864286.891401 punch from 0x01acde to 0x01b433 (0x756 bytes) 000133 1788864286.901909 punch from 0x00b1b3 to 0x00e207 (0x3055 bytes) 000134 1788864286.907346 write 0x020520 thru 0x029b0a (0x95eb bytes) 000135 1788864286.921251 read 0x03a473 thru 0x03b192 (0xd20 bytes) 000136 1788864286.935408 mapwrite 0x01b42e thru 0x02093d (0x5510 bytes) 000138 1788864287.056410 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b193): Mirror 2: 0x3edf526c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b193): Mirror 2: 0x3edf526c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b193: skipped truncating to largest ever: 0x3bdb8 000139 1788864287.618270 trunc from 0x03b193 to 0x03bdb7 (0xc25 bytes) 000140 1788864287.656238 write 0x02dc82 thru 0x03c251 (0xe5d0 bytes) 000142 1788864287.674286 read 0x03579c thru 0x03b6cb (0x5f30 bytes) 000143 1788864287.677125 punch from 0x018754 to 0x019f5b (0x1808 bytes) 000145 1788864287.742827 trunc from 0x03c252 to 0x02968c (0x12bc5 bytes) 000146 1788864287.783290 mapread 0x021b3b thru 0x02968c (0x7b52 bytes) 000149 1788864287.819980 punch from 0x024e5f to 0x02968c (0x482e bytes) 000150 1788864287.829421 write 0x0147cf thru 0x017b77 (0x33a9 bytes) 000151 1788864287.879034 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2968d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2968d): Mirror 2: 0xbfba731e (stale) CRC-32 checksum value for chunk [0, 0x2968d): Mirror 1: 0x2bc7c04c Mirror 3: 0x2bc7c04c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2968d: skipped 000153 1788864290.264158 mapread 0x02018e thru 0x0253c1 (0x5234 bytes) 000154 1788864290.302809 mapread 0x0144b2 thru 0x01e2e9 (0x9e38 bytes) 000156 1788864290.350144 mapread 0x0239e5 thru 0x023e7e (0x49a bytes) 000157 1788864290.374859 mapwrite 0x006840 thru 0x015faa (0xf76b bytes) 000159 1788864290.632119 mapwrite 0x006132 thru 0x00a2ed (0x41bc bytes) 000160 1788864290.703388 punch from 0x000545 to 0x003b81 (0x363d bytes) 000161 1788864290.710900 punch from 0x003a98 to 0x012168 (0xe6d1 bytes) 000164 1788864290.721686 mapread 0x01c801 thru 0x02968c (0xce8c bytes) 000165 1788864290.761816 read 0x020555 thru 0x0291c3 (0x8c6f bytes) 000166 1788864290.771225 mapread 0x003b81 thru 0x00a27b (0x66fb bytes) 000168 1788864290.798933 mapwrite 0x0275ee thru 0x0284f5 (0xf08 bytes) 000170 1788864290.968811 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2968d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2968d): Mirror 2: 0xbfba731e (stale) Mirror 3: 0x2bc7c04c (stale) CRC-32 checksum value for chunk [0, 0x2968d): Mirror 1: 0x71339f45 Mirror 4: 0x71339f45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2968d: skipped 000172 1788864293.138372 mapwrite 0x0300a3 thru 0x035605 (0x5563 bytes) 000173 1788864293.636254 mapwrite 0x03e8da thru 0x03ffff (0x1726 bytes) 000174 1788864293.763220 punch from 0x008a24 to 0x016d31 (0xe30e bytes) 000175 1788864293.801134 write 0x003a3b thru 0x0134aa (0xfa70 bytes) 000176 1788864293.835841 MIRROR_RESYNC 4 mirrors 000178 1788864298.443410 mapwrite 0x003f54 thru 0x01029f (0xc34c bytes) 000181 1788864298.719248 mapwrite 0x034a16 thru 0x03d377 (0x8962 bytes) 000182 1788864298.968667 MIRROR_RESYNC 4 mirrors 000184 1788864301.904479 trunc from 0x040000 to 0x01fded (0x20212 bytes) 000186 1788864302.092101 trunc from 0x01fdee to 0x00f8e2 (0x1050b bytes) 000187 1788864302.137103 read 0x002303 thru 0x008c1c (0x691a bytes) 000188 1788864302.161879 trunc from 0x00f8e3 to 0x02add5 (0x1b4f3 bytes) 000189 1788864302.195209 mapwrite 0x01955d thru 0x01b82d (0x22d1 bytes) 000190 1788864302.415028 mapread 0x01e31c thru 0x029cbe (0xb9a3 bytes) truncating to largest ever: 0x3be62 000191 1788864302.446764 trunc from 0x02add6 to 0x03be61 (0x1108c bytes) 000193 1788864302.484323 read 0x030478 thru 0x036adf (0x6668 bytes) 000196 1788864302.511466 punch from 0x02acc4 to 0x0384bd (0xd7fa bytes) 000197 1788864302.518136 trunc from 0x03be62 to 0x030561 (0xb900 bytes) 000198 1788864302.581985 read 0x00028f thru 0x0023cf (0x2141 bytes) 000199 1788864302.590112 read 0x02c361 thru 0x030561 (0x4201 bytes) 000200 1788864302.625788 MIRROR_RESYNC 4 mirrors 000201 1788864304.850889 read 0x01fd21 thru 0x02d1cf (0xd4af bytes) 000202 1788864304.873347 read 0x02f772 thru 0x030561 (0xdf0 bytes) 000205 1788864304.900490 trunc from 0x030562 to 0x01a423 (0x1613e bytes) 000206 1788864305.111478 write 0x03c96f thru 0x03ffff (0x3691 bytes) 000207 1788864305.112604 punch from 0x028608 to 0x02b3ff (0x2df8 bytes) 000208 1788864305.188009 read 0x010384 thru 0x01d62e (0xd2ab bytes) 000210 1788864305.207113 write 0x0357ce thru 0x03ffff (0xa832 bytes) 000211 1788864305.208294 mapread 0x00a121 thru 0x00d39f (0x327f bytes) 000212 1788864305.212993 read 0x020d43 thru 0x021b5c (0xe1a bytes) 000213 1788864305.215041 read 0x02a5f5 thru 0x02a63a (0x46 bytes) 000214 1788864305.215737 punch from 0x01ec59 to 0x029d1e (0xb0c6 bytes) 000215 1788864305.231140 mapread 0x019813 thru 0x019d35 (0x523 bytes) 000216 1788864305.236964 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x32ccaa18 (stale) Mirror 3: 0x32ccaa18 (stale) Mirror 4: 0x32ccaa18 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd8400c0f Mirror 5: 0xd8400c0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000217 1788864307.431955 write 0x01651b thru 0x0170a0 (0xb86 bytes) 000218 1788864307.574007 punch from 0x03d953 to 0x03e122 (0x7d0 bytes) 000220 1788864307.586075 punch from 0x030c04 to 0x03139f (0x79c bytes) 000222 1788864307.597354 trunc from 0x040000 to 0x02c98f (0x13670 bytes) 000223 1788864307.649748 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c990): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c990): Mirror 2: 0xd2e2bcfa (stale) Mirror 3: 0xd2e2bcfa (stale) Mirror 4: 0xd2e2bcfa (stale) Mirror 5: 0xf178bf43 (stale) CRC-32 checksum value for chunk [0, 0x2c990): Mirror 1: 0xc43f507 Mirror 6: 0xc43f507 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c990: skipped 000226 1788864310.137705 punch from 0x006814 to 0x0077c3 (0xfb0 bytes) 000227 1788864310.294590 trunc from 0x02c990 to 0x02041e (0xc571 bytes) 000228 1788864310.338399 mapread 0x0063ef thru 0x01143b (0xb04d bytes) 000230 1788864310.417841 write 0x02f55c thru 0x02f825 (0x2ca bytes) 000231 1788864310.419909 mapwrite 0x0149de thru 0x01c815 (0x7e38 bytes) 000232 1788864310.553751 mapwrite 0x00bb8f thru 0x018cd2 (0xd144 bytes) 000233 1788864310.678396 mapwrite 0x03a447 thru 0x03ead7 (0x4691 bytes) 000235 1788864310.817855 mapwrite 0x00b9ea thru 0x0198db (0xdef2 bytes) 000236 1788864310.993925 write 0x02451e thru 0x02db63 (0x9646 bytes) 000237 1788864311.028982 mapread 0x010227 thru 0x010bac (0x986 bytes) 000239 1788864311.046480 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ead8): Mirror 2: 0xac967563 (stale) Mirror 3: 0xac967563 (stale) Mirror 4: 0xac967563 (stale) Mirror 6: 0x28f84fcb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ead8): Mirror 2: 0xac967563 (stale) Mirror 3: 0xac967563 (stale) Mirror 4: 0xac967563 (stale) Mirror 6: 0x28f84fcb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ead8: skipped 000240 1788864311.825187 read 0x012bf1 thru 0x02223d (0xf64d bytes) truncating to largest ever: 0x3f48f 000242 1788864311.868429 trunc from 0x03ead8 to 0x03f48e (0x9b7 bytes) 000244 1788864311.919459 mapwrite 0x020cd6 thru 0x026ae2 (0x5e0d bytes) 000247 1788864312.053625 read 0x001162 thru 0x0020f2 (0xf91 bytes) 000248 1788864312.065667 trunc from 0x03f48f to 0x0349a8 (0xaae6 bytes) 000251 1788864312.089971 read 0x025e63 thru 0x031801 (0xb99f bytes) 000252 1788864312.095939 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x349a9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349a9): Mirror 2: 0xbf871bc8 (stale) Mirror 3: 0xbf871bc8 (stale) Mirror 4: 0xbf871bc8 (stale) Mirror 6: 0xd494f4f6 (stale) CRC-32 checksum value for chunk [0, 0x349a9): Mirror 1: 0x9e7192a2 Mirror 7: 0x9e7192a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349a9: skipped 000253 1788864313.362052 mapread 0x00fd36 thru 0x01af55 (0xb220 bytes) 000254 1788864313.428349 mapwrite 0x023b3e thru 0x02c3b8 (0x887b bytes) 000255 1788864313.842628 write 0x037510 thru 0x03ffff (0x8af0 bytes) 000256 1788864313.844100 write 0x02ec37 thru 0x032aa0 (0x3e6a bytes) 000257 1788864313.852588 mapread 0x037106 thru 0x03ffff (0x8efa bytes) 000258 1788864313.857629 trunc from 0x040000 to 0x006fdb (0x39024 bytes) 000259 1788864313.986685 trunc from 0x006fdc to 0x036128 (0x2f14d bytes) 000261 1788864314.065213 mapread 0x00a592 thru 0x00ed01 (0x4770 bytes) 000262 1788864314.089958 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36129): Mirror 2: 0x2aaf8c71 (stale) Mirror 3: 0x2aaf8c71 (stale) Mirror 4: 0x2aaf8c71 (stale) Mirror 6: 0xdf2dd2f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36129): Mirror 2: 0x2aaf8c71 (stale) Mirror 3: 0x2aaf8c71 (stale) Mirror 4: 0x2aaf8c71 (stale) Mirror 6: 0xdf2dd2f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36129: skipped 000264 1788864314.566653 punch from 0x01fc0b to 0x0217c1 (0x1bb7 bytes) 000265 1788864314.576025 mapwrite 0x03cc54 thru 0x03ffff (0x33ac bytes) 000267 1788864314.679189 MIRROR_RESYNC 5 mirrors 000268 1788864316.599841 read 0x01c12a thru 0x026047 (0x9f1e bytes) 000269 1788864316.613574 write 0x028c76 thru 0x031ff4 (0x937f bytes) 000270 1788864316.681748 write 0x00be34 thru 0x01270d (0x68da bytes) 000272 1788864316.697265 trunc from 0x040000 to 0x008816 (0x377e9 bytes) 000274 1788864316.735956 mapwrite 0x02ff21 thru 0x037964 (0x7a44 bytes) 000275 1788864316.836435 mapwrite 0x0146cb thru 0x01e79a (0xa0d0 bytes) 000276 1788864316.930700 punch from 0x037170 to 0x037964 (0x7f5 bytes) 000278 1788864316.938934 mapread 0x0205a9 thru 0x02ead6 (0xe52e bytes) 000279 1788864316.951276 read 0x00374c thru 0x004ad1 (0x1386 bytes) 000280 1788864316.961286 punch from 0x027e8f to 0x02bab4 (0x3c26 bytes) 000282 1788864316.970113 write 0x0004de thru 0x0072c9 (0x6dec bytes) 000283 1788864316.971386 mapread 0x00f7da thru 0x01d868 (0xe08f bytes) 000284 1788864316.974837 mapread 0x01e9ba thru 0x02024b (0x1892 bytes) 000285 1788864316.976328 mapwrite 0x032cb0 thru 0x03da4d (0xad9e bytes) 000287 1788864317.086929 read 0x0033ea thru 0x006081 (0x2c98 bytes) 000289 1788864317.088937 punch from 0x012558 to 0x021448 (0xeef1 bytes) 000290 1788864317.095578 write 0x02a060 thru 0x034f03 (0xaea4 bytes) 000291 1788864317.108499 mapread 0x031c0d thru 0x033ffc (0x23f0 bytes) 000292 1788864317.109842 mapwrite 0x01406b thru 0x014a8f (0xa25 bytes) 000294 1788864317.186616 write 0x03fdce thru 0x03ffff (0x232 bytes) 000297 1788864317.201950 MIRROR_RESYNC 5 mirrors 000298 1788864319.609250 trunc from 0x040000 to 0x0319eb (0xe614 bytes) 000299 1788864319.777946 punch from 0x0077b0 to 0x009d92 (0x25e3 bytes) 000301 1788864319.792614 mapwrite 0x01359a thru 0x01ec65 (0xb6cc bytes) 000302 1788864319.912784 mapwrite 0x00c7e6 thru 0x012e76 (0x6691 bytes) 000303 1788864319.980712 write 0x03c9a9 thru 0x03ffff (0x3657 bytes) 000305 1788864319.987246 mapwrite 0x005f86 thru 0x00a26c (0x42e7 bytes) 000307 1788864320.068914 trunc from 0x040000 to 0x02337b (0x1cc84 bytes) 000308 1788864320.096658 trunc from 0x02337c to 0x029b9a (0x681f bytes) 000309 1788864320.126789 write 0x03d84c thru 0x03fbd4 (0x2389 bytes) 000310 1788864320.130127 mapread 0x038008 thru 0x03bc72 (0x3c6b bytes) 000311 1788864320.149294 trunc from 0x03fbd5 to 0x0071d3 (0x38a01 bytes) 000312 1788864320.179059 mapread 0x007154 thru 0x0071d3 (0x80 bytes) 000314 1788864320.180566 read 0x002acd thru 0x0071d3 (0x4707 bytes) 000315 1788864320.195846 punch from 0x005bd0 to 0x0071d3 (0x1604 bytes) 000316 1788864320.203947 punch from 0x0022ca to 0x0071d3 (0x4f0a bytes) 000317 1788864320.216386 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x71d4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x71d4): Mirror 2: 0xa57a151c (stale) Mirror 3: 0xa57a151c (stale) Mirror 4: 0xa57a151c (stale) Mirror 6: 0xa57a151c (stale) CRC-32 checksum value for chunk [0, 0x71d4): Mirror 1: 0xc4411309 Mirror 7: 0xc4411309 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x71d4: skipped 000318 1788864321.591758 mapwrite 0x0221e9 thru 0x023d05 (0x1b1d bytes) 000320 1788864321.755713 write 0x03662f thru 0x037bfc (0x15ce bytes) 000321 1788864321.758428 mapread 0x003ae5 thru 0x0078a4 (0x3dc0 bytes) 000322 1788864321.771263 trunc from 0x037bfd to 0x0065db (0x31621 bytes) 000323 1788864321.791947 trunc from 0x0065dc to 0x0115b9 (0xafde bytes) 000324 1788864321.812344 mapwrite 0x003037 thru 0x008e84 (0x5e4e bytes) 000327 1788864321.870248 mapread 0x000bda thru 0x00b9c9 (0xadf0 bytes) 000328 1788864321.877623 trunc from 0x0115ba to 0x0398de (0x28325 bytes) 000329 1788864321.894356 mapwrite 0x015ae4 thru 0x01e44f (0x896c bytes) 000330 1788864321.963481 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x398df): Mirror 2: 0x2e5a51ce (stale) Mirror 3: 0x2e5a51ce (stale) Mirror 6: 0x2e5a51ce (stale) Mirror 7: 0xeeb7750 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398df): Mirror 2: 0x2e5a51ce (stale) Mirror 3: 0x2e5a51ce (stale) Mirror 6: 0x2e5a51ce (stale) Mirror 7: 0xeeb7750 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398df: skipped 000331 1788864322.368212 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x398df): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398df): Mirror 2: 0x2e5a51ce (stale) Mirror 3: 0x2e5a51ce (stale) Mirror 6: 0x2e5a51ce (stale) Mirror 7: 0xeeb7750 (stale) CRC-32 checksum value for chunk [0, 0x398df): Mirror 1: 0x2149623a Mirror 8: 0x2149623a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398df: skipped 000332 1788864323.595464 read 0x0162bf thru 0x0181be (0x1f00 bytes) 000333 1788864323.606835 read 0x02ef48 thru 0x02f74b (0x804 bytes) 000336 1788864323.638054 mapread 0x02c75e thru 0x0331d7 (0x6a7a bytes) 000337 1788864323.650454 mapread 0x0135a5 thru 0x014140 (0xb9c bytes) 000338 1788864323.663006 mapwrite 0x028e3c thru 0x02fb63 (0x6d28 bytes) 000340 1788864323.909096 read 0x01e9ec thru 0x02d96a (0xef7f bytes) 000342 1788864323.926543 trunc from 0x0398df to 0x01f5d2 (0x1a30c bytes) 000343 1788864323.967425 write 0x02dc3e thru 0x03b8ec (0xdcaf bytes) 000344 1788864323.972564 trunc from 0x03b8ed to 0x03a1a1 (0x174b bytes) 000346 1788864324.009343 mapwrite 0x011c20 thru 0x01b5d6 (0x99b7 bytes) 000348 1788864324.088608 write 0x012676 thru 0x012c6d (0x5f8 bytes) 000349 1788864324.106715 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a1a2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1a2): Mirror 2: 0x93b3598d (stale) Mirror 3: 0x93b3598d (stale) Mirror 6: 0x93b3598d (stale) Mirror 7: 0xece9f0aa (stale) Mirror 8: 0x52ff492c (stale) CRC-32 checksum value for chunk [0, 0x3a1a2): Mirror 1: 0x65570c7a Mirror 9: 0x65570c7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1a2: skipped 000350 1788864325.467636 punch from 0x00afbb to 0x017a8c (0xcad2 bytes) 000352 1788864325.595213 mapwrite 0x03ac85 thru 0x03ffff (0x537b bytes) 000353 1788864325.734031 write 0x02fddf thru 0x035e1e (0x6040 bytes) 000354 1788864325.743781 mapread 0x013619 thru 0x0213e9 (0xddd1 bytes) 000355 1788864325.826179 read 0x03e067 thru 0x03ffe3 (0x1f7d bytes) 000356 1788864325.826841 read 0x01f59f thru 0x02740d (0x7e6f bytes) 000358 1788864325.845976 write 0x00cef5 thru 0x0181f4 (0xb300 bytes) 000360 1788864325.847549 punch from 0x028cab to 0x031f7a (0x92d0 bytes) 000361 1788864325.865050 trunc from 0x040000 to 0x032d97 (0xd268 bytes) 000363 1788864325.883721 mapwrite 0x018533 thru 0x01e416 (0x5ee4 bytes) 000367 1788864325.976763 mapwrite 0x03031e thru 0x036c4f (0x6932 bytes) 000369 1788864326.088079 punch from 0x002a7b to 0x004239 (0x17bf bytes) 000370 1788864326.099776 read 0x02d70c thru 0x03076b (0x3060 bytes) 000373 1788864326.133435 MIRROR_RESYNC 7 mirrors 000375 1788864329.815150 punch from 0x024f5e to 0x033b97 (0xec3a bytes) 000376 1788864329.899366 trunc from 0x036c50 to 0x024937 (0x12318 bytes) 000378 1788864329.931672 mapread 0x02368f thru 0x024937 (0x12a9 bytes) 000379 1788864329.944262 punch from 0x020e84 to 0x024937 (0x3ab4 bytes) 000381 1788864329.950288 write 0x00e207 thru 0x011576 (0x3370 bytes) 000382 1788864329.965032 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24938): Mirror 2: 0x6d4ee93c (stale) Mirror 6: 0x6d4ee93c (stale) Mirror 7: 0x6d4ee93c (stale) Mirror 8: 0x6d4ee93c (stale) Mirror 9: 0x6d4ee93c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24938): Mirror 2: 0x6d4ee93c (stale) Mirror 6: 0x6d4ee93c (stale) Mirror 7: 0x6d4ee93c (stale) Mirror 8: 0x6d4ee93c (stale) Mirror 9: 0x6d4ee93c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24938: skipped 000383 1788864330.444172 read 0x00a294 thru 0x00f38e (0x50fb bytes) 000384 1788864330.455618 trunc from 0x024938 to 0x010c45 (0x13cf2 bytes) 000386 1788864330.478434 read 0x00ef0f thru 0x010178 (0x126a bytes) 000387 1788864330.506369 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10c46): Mirror 2: 0xbafb0357 (stale) Mirror 6: 0xbafb0357 (stale) Mirror 8: 0xbafb0357 (stale) Mirror 9: 0xbafb0357 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10c46): Mirror 2: 0xbafb0357 (stale) Mirror 6: 0xbafb0357 (stale) Mirror 8: 0xbafb0357 (stale) Mirror 9: 0xbafb0357 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10c46: skipped 000388 1788864331.062720 write 0x001322 thru 0x004338 (0x3017 bytes) 000390 1788864331.077440 punch from 0x000686 to 0x00a520 (0x9e9b bytes) 000393 1788864331.100023 mapwrite 0x0328fc thru 0x03ffff (0xd704 bytes) 000394 1788864331.221122 mapwrite 0x004f17 thru 0x00676a (0x1854 bytes) 000395 1788864331.284678 write 0x00a11f thru 0x00cef0 (0x2dd2 bytes) 000396 1788864331.293130 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa44cd61e (stale) Mirror 8: 0xa44cd61e (stale) Mirror 9: 0xa44cd61e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa44cd61e (stale) Mirror 8: 0xa44cd61e (stale) Mirror 9: 0xa44cd61e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000397 1788864331.809772 trunc from 0x040000 to 0x0107a6 (0x2f859 bytes) 000398 1788864331.833682 write 0x003e7e thru 0x0117b3 (0xd936 bytes) 000399 1788864331.842001 mapwrite 0x032bda thru 0x03c0ab (0x94d2 bytes) 000401 1788864332.051193 write 0x02eea2 thru 0x038876 (0x99d5 bytes) 000402 1788864332.052290 read 0x025eab thru 0x0358cd (0xfa23 bytes) 000406 1788864332.090122 MIRROR_RESYNC 4 mirrors 000407 1788864334.294046 MIRROR_EXTEND to 5 mirrors 000408 1788864335.156217 mapwrite 0x0071d1 thru 0x01554e (0xe37e bytes) 000409 1788864335.389839 mapread 0x036d64 thru 0x03c0ab (0x5348 bytes) 000410 1788864335.400008 mapwrite 0x010d5d thru 0x01b3e4 (0xa688 bytes) 000411 1788864335.516074 write 0x03b388 thru 0x03ffff (0x4c78 bytes) 000413 1788864335.528269 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xd65cb74b (stale) Mirror 8: 0xd65cb74b (stale) Mirror 9: 0xd65cb74b (stale) Mirror 10: 0xd65cb74b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x42199265 Mirror 11: 0x42199265 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000414 1788864337.182472 write 0x02c25a thru 0x0360d3 (0x9e7a bytes) 000417 1788864337.295868 write 0x016ea4 thru 0x01a824 (0x3981 bytes) 000419 1788864337.309220 write 0x01f0ac thru 0x02b68f (0xc5e4 bytes) 000420 1788864337.323463 punch from 0x003469 to 0x00c7bd (0x9355 bytes) 000421 1788864337.346822 MIRROR_RESYNC 6 mirrors 000423 1788864341.054089 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000424 1788864341.431204 mapread 0x0270d9 thru 0x034a1c (0xd944 bytes) 000425 1788864341.455121 mapwrite 0x038881 thru 0x03ffff (0x777f bytes) 000427 1788864341.726895 trunc from 0x040000 to 0x02e3dc (0x11c23 bytes) 000428 1788864341.797047 trunc from 0x02e3dd to 0x02a03b (0x43a1 bytes) 000429 1788864341.847429 mapread 0x01a789 thru 0x0211b0 (0x6a28 bytes) 000430 1788864341.867406 mapwrite 0x014281 thru 0x01f74c (0xb4cc bytes) 000432 1788864341.945927 read 0x02409c thru 0x026cbe (0x2c23 bytes) 000433 1788864341.961224 mapread 0x012c51 thru 0x01487b (0x1c2b bytes) 000435 1788864341.985965 read 0x007169 thru 0x009ae1 (0x2979 bytes) 000437 1788864342.010494 MIRROR_RESYNC 5 mirrors 000438 1788864344.242938 trunc from 0x02a03c to 0x02ee5c (0x4e21 bytes) 000439 1788864344.331212 mapwrite 0x010515 thru 0x0121a5 (0x1c91 bytes) 000440 1788864344.409410 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ee5d): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee5d): Mirror 8: 0xaf5b2b6 (stale) Mirror 9: 0xaf5b2b6 (stale) Mirror 10: 0xaf5b2b6 (stale) Mirror 11: 0xaf5b2b6 (stale) CRC-32 checksum value for chunk [0, 0x2ee5d): Mirror 1: 0xf3d270d1 Mirror 12: 0xf3d270d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee5d: skipped 000441 1788864345.900181 trunc from 0x02ee5d to 0x005e61 (0x28ffb bytes) 000443 1788864346.022187 punch from 0x002870 to 0x002e32 (0x5c3 bytes) 000445 1788864346.028445 read 0x001e7b thru 0x0058fb (0x3a81 bytes) 000446 1788864346.042955 write 0x039a8f thru 0x03ffff (0x6571 bytes) 000447 1788864346.044434 punch from 0x02193d to 0x02afcd (0x9691 bytes) 000448 1788864346.064193 write 0x0393f8 thru 0x03eb48 (0x5751 bytes) 000449 1788864346.068925 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb5ca3c0 (stale) Mirror 10: 0xb5ca3c0 (stale) Mirror 11: 0xb5ca3c0 (stale) Mirror 12: 0x1702f549 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb5ca3c0 (stale) Mirror 10: 0xb5ca3c0 (stale) Mirror 11: 0xb5ca3c0 (stale) Mirror 12: 0x1702f549 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000451 1788864346.320822 punch from 0x0047c2 to 0x00c1ef (0x7a2e bytes) 000452 1788864346.324440 write 0x02f0fa thru 0x034ca3 (0x5baa bytes) 000453 1788864346.331380 punch from 0x03ac50 to 0x03ffff (0x53b0 bytes) 000454 1788864346.339466 trunc from 0x040000 to 0x0339cc (0xc633 bytes) 000455 1788864346.349855 mapwrite 0x00c881 thru 0x0135ce (0x6d4e bytes) 000456 1788864346.406082 punch from 0x0077a8 to 0x015b38 (0xe391 bytes) 000457 1788864346.411269 write 0x03e117 thru 0x03f093 (0xf7d bytes) 000458 1788864346.413801 mapread 0x03d47b thru 0x03f093 (0x1c19 bytes) 000459 1788864346.422097 read 0x01f1e7 thru 0x022aab (0x38c5 bytes) 000460 1788864346.428741 trunc from 0x03f094 to 0x01bb47 (0x2354c bytes) 000462 1788864346.450820 MIRROR_RESYNC 5 mirrors 000463 1788864348.291813 mapread 0x0084a8 thru 0x012f3e (0xaa97 bytes) 000465 1788864348.311463 punch from 0x00cd1d to 0x017ace (0xadb2 bytes) 000466 1788864348.370842 mapread 0x006db1 thru 0x00d6fa (0x694a bytes) 000467 1788864348.381141 punch from 0x007c2d to 0x016295 (0xe669 bytes) 000471 1788864348.386799 punch from 0x009e28 to 0x019cf7 (0xfed0 bytes) 000473 1788864348.390952 trunc from 0x01bb48 to 0x03cb0c (0x20fc5 bytes) 000475 1788864348.420291 MIRROR_RESYNC 5 mirrors 000477 1788864349.963246 read 0x00b781 thru 0x017e02 (0xc682 bytes) 000479 1788864349.973042 write 0x0187e0 thru 0x027324 (0xeb45 bytes) 000480 1788864350.043752 trunc from 0x03cb0d to 0x00d742 (0x2f3ca bytes) 000481 1788864350.092554 write 0x03c376 thru 0x03ffff (0x3c8a bytes) 000482 1788864350.102211 trunc from 0x040000 to 0x02526c (0x1ad93 bytes) 000483 1788864350.133179 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2526d): Mirror 9: 0x138d1a3 (stale) Mirror 10: 0x138d1a3 (stale) Mirror 12: 0x138d1a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2526d): Mirror 9: 0x138d1a3 (stale) Mirror 10: 0x138d1a3 (stale) Mirror 12: 0x138d1a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2526d: skipped 000484 1788864350.583477 trunc from 0x02526d to 0x011cfa (0x13572 bytes) 000485 1788864350.604729 mapread 0x002425 thru 0x010f6c (0xeb48 bytes) 000486 1788864350.627404 punch from 0x0026a9 to 0x00e382 (0xbcda bytes) 000487 1788864350.632442 write 0x021f0a thru 0x021f33 (0x2a bytes) 000488 1788864350.634665 read 0x00ec0f thru 0x01b58f (0xc981 bytes) 000490 1788864350.646192 mapread 0x01b7d7 thru 0x021f33 (0x675d bytes) 000491 1788864350.656260 trunc from 0x021f34 to 0x00b126 (0x16e0d bytes) 000492 1788864350.676058 punch from 0x0097ee to 0x00b126 (0x1939 bytes) 000493 1788864350.683079 trunc from 0x00b127 to 0x020a40 (0x1591a bytes) 000494 1788864350.701137 trunc from 0x020a41 to 0x030778 (0xfd38 bytes) 000498 1788864350.722332 mapwrite 0x03e813 thru 0x03ffff (0x17ed bytes) 000499 1788864350.795198 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x65590519 (stale) Mirror 12: 0x65590519 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x65590519 (stale) Mirror 12: 0x65590519 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000500 1788864351.226135 mapwrite 0x016e3b thru 0x01f8f0 (0x8ab6 bytes) 000503 1788864351.323538 mapwrite 0x0123ff thru 0x012dac (0x9ae bytes) 000505 1788864351.373212 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x65590519 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x65590519 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000507 1788864351.685582 read 0x027455 thru 0x027bea (0x796 bytes) 000510 1788864351.693054 punch from 0x03841f to 0x03b4a9 (0x308b bytes) 000512 1788864351.705062 punch from 0x013733 to 0x020637 (0xcf05 bytes) 000513 1788864351.710880 mapwrite 0x02136f thru 0x021728 (0x3ba bytes) 000515 1788864351.777115 mapwrite 0x00879f thru 0x00beaa (0x370c bytes) 000516 1788864351.868322 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000517 1788864351.980785 trunc from 0x040000 to 0x036d8c (0x9273 bytes) 000518 1788864352.004106 mapread 0x01414e thru 0x022a96 (0xe949 bytes) 000522 1788864352.049942 mapwrite 0x02175a thru 0x024bcf (0x3476 bytes) 000523 1788864352.101337 read 0x002242 thru 0x00abae (0x896d bytes) 000525 1788864352.111839 punch from 0x03182e to 0x036d8c (0x555f bytes) 000526 1788864352.117896 mapwrite 0x029906 thru 0x029e23 (0x51e bytes) 000528 1788864352.165147 mapwrite 0x009f61 thru 0x00d840 (0x38e0 bytes) 000532 1788864352.230437 punch from 0x003d3f to 0x004a65 (0xd27 bytes) 000535 1788864352.238026 read 0x023e67 thru 0x033448 (0xf5e2 bytes) 000537 1788864352.254096 MIRROR_EXTEND to 2 mirrors 000538 1788864353.426942 read 0x0094de thru 0x00eb94 (0x56b7 bytes) 000540 1788864353.463919 write 0x00dae4 thru 0x012ff4 (0x5511 bytes) 000544 1788864353.615284 mapread 0x001b69 thru 0x005449 (0x38e1 bytes) 000546 1788864353.644966 read 0x00c3db thru 0x0147d8 (0x83fe bytes) 000547 1788864353.659896 read 0x030520 thru 0x036d8c (0x686d bytes) 000548 1788864353.665990 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36d8d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d8d): Mirror 2: 0x8c96ffea (stale) CRC-32 checksum value for chunk [0, 0x36d8d): Mirror 1: 0xbf969836 Mirror 3: 0xbf969836 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d8d: skipped 000551 1788864355.341261 trunc from 0x036d8d to 0x03babe (0x4d32 bytes) 000552 1788864355.438987 write 0x00096e thru 0x00f718 (0xedab bytes) 000553 1788864355.450265 trunc from 0x03babf to 0x0318ef (0xa1cf bytes) 000554 1788864355.475215 write 0x032169 thru 0x03e9c6 (0xc85e bytes) 000555 1788864355.477630 read 0x00ca89 thru 0x013398 (0x6910 bytes) 000556 1788864355.485030 punch from 0x039ccd to 0x03e9c6 (0x4cfa bytes) 000557 1788864355.512274 mapread 0x01a143 thru 0x01e02b (0x3ee9 bytes) 000559 1788864355.543041 trunc from 0x03e9c7 to 0x0357d8 (0x91ee bytes) 000560 1788864355.581852 MIRROR_RESYNC 3 mirrors 000561 1788864356.961915 punch from 0x007187 to 0x00b41b (0x4295 bytes) 000563 1788864357.003808 write 0x008a57 thru 0x010d16 (0x82c0 bytes) 000564 1788864357.014796 write 0x013d04 thru 0x02370f (0xfa0c bytes) 000565 1788864357.027954 read 0x001d5a thru 0x003e69 (0x2110 bytes) 000566 1788864357.033624 mapread 0x01068a thru 0x010b57 (0x4ce bytes) 000567 1788864357.035974 punch from 0x02d13d to 0x032638 (0x54fc bytes) 000568 1788864357.061294 read 0x0268d9 thru 0x02dbaf (0x72d7 bytes) 000571 1788864357.072808 trunc from 0x0357d9 to 0x02d89a (0x7f3e bytes) 000572 1788864357.098998 write 0x012761 thru 0x019d64 (0x7604 bytes) 000574 1788864357.101806 mapread 0x01bcfc thru 0x0229fe (0x6d03 bytes) 000576 1788864357.103406 write 0x025e7c thru 0x02eceb (0x8e70 bytes) 000577 1788864357.105457 punch from 0x02764a to 0x02c5ef (0x4fa6 bytes) 000578 1788864357.122973 mapread 0x000f35 thru 0x00cfb3 (0xc07f bytes) 000579 1788864357.124428 read 0x0013bd thru 0x00940f (0x8053 bytes) 000580 1788864357.125077 mapread 0x01d5d1 thru 0x02ad48 (0xd778 bytes) 000581 1788864357.137157 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ecec): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecec): Mirror 2: 0xcc91eee (stale) Mirror 3: 0xcc91eee (stale) CRC-32 checksum value for chunk [0, 0x2ecec): Mirror 1: 0x22539ebb Mirror 4: 0x22539ebb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecec: skipped 000585 1788864358.202947 mapwrite 0x009860 thru 0x00a245 (0x9e6 bytes) 000586 1788864358.380341 mapread 0x023f52 thru 0x02bd7d (0x7e2c bytes) 000589 1788864358.406649 write 0x014800 thru 0x01fabd (0xb2be bytes) 000593 1788864358.421695 mapwrite 0x03053c thru 0x03acb5 (0xa77a bytes) 000594 1788864358.587644 mapread 0x028b04 thru 0x034015 (0xb512 bytes) 000595 1788864358.590923 mapread 0x016b4f thru 0x025d48 (0xf1fa bytes) 000596 1788864358.592619 mapwrite 0x02503a thru 0x02f623 (0xa5ea bytes) 000597 1788864358.654080 mapwrite 0x00ff9b thru 0x0189a1 (0x8a07 bytes) 000598 1788864358.722424 write 0x023917 thru 0x02b2e7 (0x79d1 bytes) 000599 1788864358.724683 mapwrite 0x019e67 thru 0x026139 (0xc2d3 bytes) 000600 1788864358.781923 mapwrite 0x00a9e3 thru 0x00dd9d (0x33bb bytes) 000601 1788864358.834992 write 0x03c363 thru 0x03ffff (0x3c9d bytes) 000602 1788864358.836639 punch from 0x038aee to 0x03ffff (0x7512 bytes) 000603 1788864358.847861 write 0x03d6b3 thru 0x03e854 (0x11a2 bytes) 000604 1788864358.859128 mapwrite 0x027202 thru 0x031236 (0xa035 bytes) 000605 1788864358.905969 write 0x002e37 thru 0x0100e8 (0xd2b2 bytes) 000606 1788864358.915957 read 0x010aee thru 0x018115 (0x7628 bytes) 000608 1788864358.917273 mapwrite 0x011cc7 thru 0x012dae (0x10e8 bytes) 000610 1788864358.963707 punch from 0x03e9dc to 0x03ffff (0x1624 bytes) 000611 1788864358.970209 trunc from 0x040000 to 0x0161b9 (0x29e46 bytes) 000613 1788864359.001658 trunc from 0x0161ba to 0x009477 (0xcd42 bytes) 000614 1788864359.024155 read 0x003e5c thru 0x006397 (0x253c bytes) 000615 1788864359.031470 MIRROR_RESYNC 4 mirrors 000616 1788864359.963308 read 0x002cac thru 0x009477 (0x67cc bytes) 000617 1788864359.973073 punch from 0x004dd1 to 0x009477 (0x46a7 bytes) 000618 1788864360.044060 mapread 0x0041ea thru 0x0057e6 (0x15fd bytes) 000620 1788864360.057536 trunc from 0x009478 to 0x022052 (0x18bdb bytes) 000621 1788864360.114139 trunc from 0x022053 to 0x016270 (0xbde2 bytes) 000622 1788864360.136952 read 0x004df7 thru 0x008081 (0x328b bytes) 000624 1788864360.163063 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16271): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16271): Mirror 2: 0x2a277652 (stale) Mirror 3: 0x2a277652 (stale) Mirror 4: 0x2a277652 (stale) CRC-32 checksum value for chunk [0, 0x16271): Mirror 1: 0xe1b8c258 Mirror 5: 0xe1b8c258 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16271: skipped 000628 1788864361.310589 trunc from 0x016271 to 0x01c57f (0x630f bytes) 000629 1788864361.419477 mapread 0x00e24d thru 0x00f923 (0x16d7 bytes) 000631 1788864361.432919 read 0x011a36 thru 0x01c57f (0xab4a bytes) 000632 1788864361.442834 trunc from 0x01c580 to 0x00deb4 (0xe6cb bytes) 000633 1788864361.466558 read 0x002f16 thru 0x00deb4 (0xaf9f bytes) 000634 1788864361.482395 write 0x021c1c thru 0x02fbe3 (0xdfc8 bytes) 000635 1788864361.484167 punch from 0x015a0c to 0x0239fd (0xdff2 bytes) 000636 1788864361.496906 mapread 0x029ab3 thru 0x02fbe3 (0x6131 bytes) 000637 1788864361.497994 write 0x01124e thru 0x019d4b (0x8afe bytes) 000638 1788864361.511396 mapread 0x026e99 thru 0x02fbe3 (0x8d4b bytes) 000639 1788864361.512561 read 0x007cc4 thru 0x015bb7 (0xdef4 bytes) 000641 1788864361.514822 punch from 0x01d0f3 to 0x0246be (0x75cc bytes) 000642 1788864361.526774 mapwrite 0x034b30 thru 0x03acc7 (0x6198 bytes) 000643 1788864361.598066 mapwrite 0x02c766 thru 0x02cb5f (0x3fa bytes) 000644 1788864361.634869 read 0x00dcdc thru 0x014cde (0x7003 bytes) 000645 1788864361.644400 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3acc8): Mirror 2: 0xdcc0e1f (stale) Mirror 3: 0xdcc0e1f (stale) Mirror 4: 0xdcc0e1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3acc8): Mirror 2: 0xdcc0e1f (stale) Mirror 3: 0xdcc0e1f (stale) Mirror 4: 0xdcc0e1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3acc8: skipped 000646 1788864362.309464 mapwrite 0x027680 thru 0x027699 (0x1a bytes) 000647 1788864362.446158 punch from 0x014f35 to 0x01ddee (0x8eba bytes) 000648 1788864362.471938 read 0x000f60 thru 0x00a0dc (0x917d bytes) 000649 1788864362.494963 trunc from 0x03acc8 to 0x0223e4 (0x188e3 bytes) 000650 1788864362.527754 read 0x003578 thru 0x00ca5c (0x94e5 bytes) 000651 1788864362.540770 trunc from 0x0223e5 to 0x01ff35 (0x24af bytes) 000652 1788864362.589913 MIRROR_RESYNC 4 mirrors 000653 1788864364.210663 mapwrite 0x025dd8 thru 0x02ca6f (0x6c98 bytes) 000656 1788864364.460248 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ca70): Mirror 3: 0xa566c735 (stale) Mirror 4: 0xa566c735 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca70): Mirror 3: 0xa566c735 (stale) Mirror 4: 0xa566c735 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca70: skipped 000657 1788864365.134204 mapwrite 0x03f719 thru 0x03ffff (0x8e7 bytes) 000658 1788864365.195715 write 0x026738 thru 0x030229 (0x9af2 bytes) 000661 1788864365.209052 read 0x00cd93 thru 0x013d65 (0x6fd3 bytes) 000663 1788864365.224077 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x76a81cf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x76a81cf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1788864365.597711 punch from 0x02c3e8 to 0x031e41 (0x5a5a bytes) 000665 1788864365.604913 mapwrite 0x036161 thru 0x03e1d7 (0x8077 bytes) 000666 1788864365.676863 mapread 0x011f1d thru 0x01da9b (0xbb7f bytes) 000667 1788864365.703320 read 0x0231f3 thru 0x02bcca (0x8ad8 bytes) 000668 1788864365.738770 mapwrite 0x010d7d thru 0x01eea8 (0xe12c bytes) 000669 1788864365.832797 trunc from 0x040000 to 0x0132e9 (0x2cd16 bytes) 000670 1788864365.872363 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000671 1788864366.096754 trunc from 0x0132ea to 0x00f778 (0x3b71 bytes) 000672 1788864366.128415 MIRROR_EXTEND to 2 mirrors 000673 1788864367.006311 mapwrite 0x028499 thru 0x02c0a0 (0x3c08 bytes) 000675 1788864367.176136 trunc from 0x02c0a1 to 0x03777e (0xb6de bytes) 000676 1788864367.213651 read 0x0171fe thru 0x0225c0 (0xb3c3 bytes) 000677 1788864367.231102 mapread 0x035038 thru 0x03777e (0x2747 bytes) 000678 1788864367.250115 MIRROR_RESYNC 2 mirrors 000679 1788864368.416914 write 0x03f631 thru 0x03fac8 (0x498 bytes) 000680 1788864368.485720 read 0x03445a thru 0x03ae4c (0x69f3 bytes) 000681 1788864368.495014 read 0x025469 thru 0x02f06d (0x9c05 bytes) 000682 1788864368.508255 mapwrite 0x01aa53 thru 0x01d146 (0x26f4 bytes) 000683 1788864368.581916 write 0x01c7a0 thru 0x02442a (0x7c8b bytes) 000684 1788864368.583915 mapwrite 0x03fef6 thru 0x03ffff (0x10a bytes) 000686 1788864368.682370 punch from 0x030d5f to 0x03e9be (0xdc60 bytes) 000687 1788864368.687008 trunc from 0x040000 to 0x015b91 (0x2a46e bytes) 000688 1788864368.707956 mapwrite 0x00e92b thru 0x01bfa2 (0xd678 bytes) 000689 1788864368.836384 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000690 1788864369.021113 read 0x014587 thru 0x01ad98 (0x6812 bytes) 000691 1788864369.031916 trunc from 0x01bfa3 to 0x025c91 (0x9cef bytes) 000692 1788864369.058888 MIRROR_EXTEND to 2 mirrors 000694 1788864370.154870 mapwrite 0x024c15 thru 0x02d1a3 (0x858f bytes) 000695 1788864370.366208 mapread 0x026c7d thru 0x02d1a3 (0x6527 bytes) 000698 1788864370.378477 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000699 1788864370.550076 write 0x00132e thru 0x00ab07 (0x97da bytes) 000700 1788864370.563474 mapwrite 0x036f22 thru 0x038a98 (0x1b77 bytes) 000702 1788864370.660932 MIRROR_EXTEND to 2 mirrors 000705 1788864371.647918 read 0x02c4e4 thru 0x03486f (0x838c bytes) 000708 1788864371.670794 mapread 0x01d2e4 thru 0x02ba46 (0xe763 bytes) 000709 1788864371.734796 mapwrite 0x01e843 thru 0x025e82 (0x7640 bytes) 000711 1788864371.984385 punch from 0x031ab2 to 0x03528b (0x37da bytes) 000713 1788864371.993767 read 0x003ecb thru 0x0045c4 (0x6fa bytes) 000714 1788864372.021636 mapwrite 0x012993 thru 0x01983f (0x6ead bytes) 000715 1788864372.213839 trunc from 0x038a99 to 0x034a60 (0x4038 bytes) 000717 1788864372.288889 mapread 0x01fdf2 thru 0x026632 (0x6841 bytes) 000719 1788864372.293031 read 0x01cba7 thru 0x025f96 (0x93f0 bytes) 000721 1788864372.308182 trunc from 0x034a61 to 0x033940 (0x1120 bytes) 000722 1788864372.361988 mapread 0x02701a thru 0x0318da (0xa8c1 bytes) 000723 1788864372.363421 mapwrite 0x035ed1 thru 0x03ffff (0xa12f bytes) 000724 1788864372.505602 mapwrite 0x02157d thru 0x02eecf (0xd953 bytes) 000726 1788864372.600849 read 0x026259 thru 0x029c08 (0x39b0 bytes) 000727 1788864372.601651 trunc from 0x040000 to 0x036b58 (0x94a7 bytes) 000728 1788864372.629200 mapwrite 0x02475e thru 0x02cf3f (0x87e2 bytes) 000731 1788864372.684340 mapwrite 0x019b25 thru 0x023e13 (0xa2ef bytes) 000732 1788864372.735065 write 0x00a157 thru 0x0148df (0xa789 bytes) 000734 1788864372.738365 mapread 0x01cd5a thru 0x01dd68 (0x100f bytes) 000736 1788864372.740436 mapwrite 0x0314d4 thru 0x03ae41 (0x996e bytes) 000737 1788864372.824467 mapread 0x029467 thru 0x02c0d1 (0x2c6b bytes) 000738 1788864372.827873 write 0x02f104 thru 0x0385c2 (0x94bf bytes) 000739 1788864372.831852 trunc from 0x03ae42 to 0x0103f1 (0x2aa50 bytes) 000741 1788864372.863649 punch from 0x0010bd to 0x00bdc3 (0xad07 bytes) 000743 1788864372.869761 mapread 0x006ce6 thru 0x0103f1 (0x970c bytes) 000744 1788864372.901227 trunc from 0x0103f2 to 0x00a97c (0x5a75 bytes) 000745 1788864372.921856 trunc from 0x00a97d to 0x01e0f8 (0x1377c bytes) 000747 1788864372.944902 read 0x014180 thru 0x0186cf (0x4550 bytes) 000748 1788864372.960858 write 0x01d05d thru 0x01e911 (0x18b5 bytes) 000749 1788864372.962136 write 0x0125b0 thru 0x0202f4 (0xdd45 bytes) 000750 1788864372.965239 write 0x0075c0 thru 0x016a4e (0xf48f bytes) 000753 1788864372.974125 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000755 1788864373.131384 mapread 0x0100ac thru 0x01d8d8 (0xd82d bytes) 000757 1788864373.166805 mapwrite 0x01874c thru 0x01e0a4 (0x5959 bytes) 000758 1788864373.208874 mapwrite 0x012efd thru 0x018cc6 (0x5dca bytes) 000759 1788864373.254696 mapread 0x017f44 thru 0x018ac1 (0xb7e bytes) 000760 1788864373.255645 trunc from 0x0202f5 to 0x039bb7 (0x198c3 bytes) 000761 1788864373.281623 MIRROR_EXTEND to 2 mirrors 000763 1788864374.206032 mapread 0x0119a0 thru 0x01486b (0x2ecc bytes) 000765 1788864374.229341 MIRROR_RESYNC 2 mirrors 000766 1788864374.756561 mapread 0x000044 thru 0x006a9e (0x6a5b bytes) 000768 1788864374.785174 trunc from 0x039bb8 to 0x036087 (0x3b30 bytes) 000770 1788864374.952299 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36088): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36088): Mirror 2: 0xa4981efc (stale) CRC-32 checksum value for chunk [0, 0x36088): Mirror 1: 0xa4981efc Mirror 3: 0xa4981efc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36088: skipped 000771 1788864376.413127 MIRROR_RESYNC 3 mirrors 000772 1788864377.443497 punch from 0x00bc35 to 0x01a6bc (0xea88 bytes) 000773 1788864377.566576 punch from 0x0323d0 to 0x036087 (0x3cb8 bytes) 000775 1788864377.576222 write 0x03c4c5 thru 0x03ffff (0x3b3b bytes) 000777 1788864377.580724 punch from 0x019539 to 0x0226a7 (0x916f bytes) 000778 1788864377.601595 mapread 0x025c4c thru 0x0295bc (0x3971 bytes) 000779 1788864377.657957 write 0x018449 thru 0x020d47 (0x88ff bytes) 000780 1788864377.669941 punch from 0x0390b3 to 0x03ffff (0x6f4d bytes) 000781 1788864377.686837 mapread 0x02fdcd thru 0x03c4be (0xc6f2 bytes) 000784 1788864377.742681 mapread 0x0301ad thru 0x039b73 (0x99c7 bytes) 000785 1788864377.743834 trunc from 0x040000 to 0x00c5bc (0x33a43 bytes) 000787 1788864377.795319 trunc from 0x00c5bd to 0x02d522 (0x20f66 bytes) 000788 1788864377.837782 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d523): Mirror 3: 0x65cf025b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d523): Mirror 3: 0x65cf025b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d523: skipped 000789 1788864378.240437 mapwrite 0x0280b2 thru 0x0322f4 (0xa243 bytes) 000790 1788864378.376691 mapread 0x01a946 thru 0x01d830 (0x2eeb bytes) 000791 1788864378.394275 mapread 0x02598f thru 0x02e805 (0x8e77 bytes) 000793 1788864378.402712 mapwrite 0x02d3e1 thru 0x037d8b (0xa9ab bytes) 000794 1788864378.497881 write 0x01561f thru 0x01d210 (0x7bf2 bytes) 000796 1788864378.505860 MIRROR_RESYNC 2 mirrors 000798 1788864379.516752 mapwrite 0x003334 thru 0x00ab8f (0x785c bytes) 000800 1788864379.776589 read 0x022252 thru 0x0269b2 (0x4761 bytes) 000801 1788864379.790932 mapread 0x02e1b0 thru 0x033750 (0x55a1 bytes) 000802 1788864379.802426 trunc from 0x037d8c to 0x003095 (0x34cf6 bytes) 000803 1788864379.838438 punch from 0x001f12 to 0x003095 (0x1184 bytes) 000804 1788864379.844431 write 0x0228fb thru 0x027128 (0x482e bytes) 000805 1788864379.857491 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000807 1788864380.016763 write 0x037dd1 thru 0x03ffff (0x822f bytes) 000809 1788864380.021835 mapwrite 0x00684f thru 0x012bff (0xc3b1 bytes) 000810 1788864380.094865 trunc from 0x040000 to 0x008eb4 (0x3714b bytes) 000812 1788864380.111869 trunc from 0x008eb5 to 0x01fded (0x16f39 bytes) 000813 1788864380.128950 MIRROR_EXTEND to 2 mirrors 000814 1788864381.295273 mapread 0x00cd6b thru 0x01bb7f (0xee15 bytes) 000815 1788864381.335429 trunc from 0x01fdee to 0x023c12 (0x3e25 bytes) 000818 1788864381.482288 write 0x03a7ab thru 0x03ffff (0x5855 bytes) 000819 1788864381.488774 mapread 0x03c065 thru 0x03ffff (0x3f9b bytes) 000820 1788864381.498093 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000822 1788864381.779750 read 0x03528a thru 0x03ba46 (0x67bd bytes) 000824 1788864381.792943 mapwrite 0x0140d6 thru 0x02314b (0xf076 bytes) 000825 1788864381.977114 read 0x0073a2 thru 0x0117f5 (0xa454 bytes) 000828 1788864381.988842 read 0x01e532 thru 0x02a18c (0xbc5b bytes) 000829 1788864381.989991 write 0x013bef thru 0x01993b (0x5d4d bytes) 000830 1788864381.996589 MIRROR_EXTEND to 2 mirrors 000831 1788864382.923064 mapread 0x01aa1c thru 0x02726d (0xc852 bytes) 000833 1788864382.950017 punch from 0x03c94d to 0x03ffff (0x36b3 bytes) 000834 1788864383.015394 mapread 0x02a3f1 thru 0x02ac81 (0x891 bytes) 000835 1788864383.027303 read 0x03b478 thru 0x03ffff (0x4b88 bytes) 000836 1788864383.036003 mapwrite 0x00fae0 thru 0x01c0e0 (0xc601 bytes) 000837 1788864383.102640 write 0x022908 thru 0x02a0f5 (0x77ee bytes) 000838 1788864383.108440 punch from 0x02416c to 0x02cfc3 (0x8e58 bytes) 000839 1788864383.121079 punch from 0x01f753 to 0x026e70 (0x771e bytes) 000840 1788864383.125912 mapread 0x03c70a thru 0x03fa34 (0x332b bytes) 000842 1788864383.127434 punch from 0x03dbea to 0x03ffff (0x2416 bytes) 000843 1788864383.136645 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000844 1788864383.268235 mapwrite 0x004075 thru 0x00c404 (0x8390 bytes) 000845 1788864383.346316 MIRROR_EXTEND to 2 mirrors 000847 1788864384.165456 punch from 0x009cd9 to 0x01204a (0x8372 bytes) 000848 1788864384.237722 write 0x03b67b thru 0x03ffff (0x4985 bytes) 000849 1788864384.244372 trunc from 0x040000 to 0x01c5e8 (0x23a17 bytes) 000850 1788864384.290784 read 0x000beb thru 0x0069f6 (0x5e0c bytes) 000851 1788864384.299174 trunc from 0x01c5e9 to 0x02f991 (0x133a9 bytes) 000852 1788864384.314161 trunc from 0x02f992 to 0x0080af (0x278e2 bytes) 000853 1788864384.338814 trunc from 0x0080b0 to 0x0007a2 (0x790d bytes) 000854 1788864384.360980 write 0x01d369 thru 0x01fa94 (0x272c bytes) 000855 1788864384.371116 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000856 1788864384.513181 read 0x0049e0 thru 0x00f4b8 (0xaad9 bytes) 000858 1788864384.524237 mapread 0x0085c2 thru 0x0160ef (0xdb2e bytes) 000859 1788864384.542868 mapread 0x00ffca thru 0x01baf0 (0xbb27 bytes) 000860 1788864384.551798 mapwrite 0x003882 thru 0x009aa8 (0x6227 bytes) 000861 1788864384.599389 trunc from 0x01fa95 to 0x00090d (0x1f187 bytes) 000864 1788864384.617996 mapread 0x0007b8 thru 0x00090d (0x156 bytes) 000865 1788864384.629306 mapread 0x000067 thru 0x00090d (0x8a7 bytes) 000866 1788864384.630276 write 0x0200ae thru 0x029f4f (0x9ea2 bytes) 000867 1788864384.633244 mapread 0x007877 thru 0x00e2a6 (0x6a30 bytes) 000869 1788864384.646054 read 0x01834a thru 0x01a9a7 (0x265e bytes) 000870 1788864384.659114 MIRROR_EXTEND to 2 mirrors 000871 1788864385.546621 mapwrite 0x03f5be thru 0x03ffff (0xa42 bytes) 000872 1788864385.762986 MIRROR_RESYNC 2 mirrors 000874 1788864386.632999 mapread 0x014993 thru 0x01f34c (0xa9ba bytes) 000875 1788864386.656785 trunc from 0x040000 to 0x010f92 (0x2f06d bytes) 000876 1788864386.741418 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000877 1788864386.920281 mapwrite 0x038697 thru 0x03b5ef (0x2f59 bytes) 000878 1788864387.071611 mapwrite 0x013b28 thru 0x017a90 (0x3f69 bytes) 000879 1788864387.192146 punch from 0x018d9e to 0x01cefa (0x415d bytes) 000880 1788864387.203166 MIRROR_EXTEND to 2 mirrors 000882 1788864388.116790 MIRROR_EXTEND to 3 mirrors 000883 1788864389.246740 mapread 0x02c52e thru 0x034c49 (0x871c bytes) 000884 1788864389.302305 read 0x02d538 thru 0x031d03 (0x47cc bytes) 000886 1788864389.320090 read 0x030199 thru 0x03b5ef (0xb457 bytes) 000887 1788864389.342011 mapwrite 0x00c4bb thru 0x017176 (0xacbc bytes) 000888 1788864389.618144 punch from 0x026f27 to 0x029642 (0x271c bytes) 000890 1788864389.626032 read 0x00bece thru 0x014c0b (0x8d3e bytes) 000893 1788864389.642242 write 0x002e53 thru 0x00a1cf (0x737d bytes) 000894 1788864389.648117 write 0x01825d thru 0x01b9d8 (0x377c bytes) 000895 1788864389.649494 mapwrite 0x014886 thru 0x020e59 (0xc5d4 bytes) 000896 1788864389.742991 write 0x02b059 thru 0x032439 (0x73e1 bytes) 000897 1788864389.744479 mapwrite 0x006194 thru 0x00d598 (0x7405 bytes) 000899 1788864389.792439 punch from 0x02c0dd to 0x02dfb8 (0x1edc bytes) 000901 1788864389.798623 write 0x0148de thru 0x01e6da (0x9dfd bytes) 000902 1788864389.805938 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b5f0): Mirror 3: 0x97cc04cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5f0): Mirror 3: 0x97cc04cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5f0: skipped 000903 1788864390.098212 punch from 0x01fe28 to 0x020f3c (0x1115 bytes) 000905 1788864390.109533 mapread 0x0061db thru 0x0064d5 (0x2fb bytes) 000906 1788864390.115737 mapread 0x021be4 thru 0x02af60 (0x937d bytes) 000908 1788864390.133470 mapwrite 0x02fc19 thru 0x03c3ae (0xc796 bytes) 000909 1788864390.230234 punch from 0x026b2a to 0x0306ff (0x9bd6 bytes) 000910 1788864390.238988 read 0x02cdea thru 0x03ad67 (0xdf7e bytes) 000911 1788864390.250072 trunc from 0x03c3af to 0x00c42e (0x2ff80 bytes) 000914 1788864390.280341 MIRROR_RESYNC 2 mirrors 000917 1788864391.219289 mapread 0x00442c thru 0x00c42e (0x8003 bytes) 000918 1788864391.273979 trunc from 0x00c42f to 0x03b7ca (0x2f39c bytes) 000921 1788864391.559122 mapwrite 0x012064 thru 0x01c303 (0xa2a0 bytes) 000922 1788864391.809326 mapread 0x039725 thru 0x03b7ca (0x20a6 bytes) 000924 1788864391.829344 mapread 0x02c877 thru 0x03194d (0x50d7 bytes) 000925 1788864391.857648 mapread 0x001c71 thru 0x0045b0 (0x2940 bytes) 000926 1788864391.919437 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b7cb): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7cb): Mirror 3: 0xd977f025 (stale) CRC-32 checksum value for chunk [0, 0x3b7cb): Mirror 1: 0xed461ac6 Mirror 4: 0xed461ac6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7cb: skipped 000927 1788864393.476933 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b7cb): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7cb): Mirror 3: 0xd977f025 (stale) CRC-32 checksum value for chunk [0, 0x3b7cb): Mirror 1: 0xed461ac6 Mirror 4: 0xed461ac6 Mirror 5: 0xed461ac6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7cb: skipped 000928 1788864394.554667 punch from 0x015bf8 to 0x018932 (0x2d3b bytes) 000929 1788864394.645164 write 0x02c967 thru 0x0323cb (0x5a65 bytes) 000930 1788864394.658241 mapwrite 0x03f20f thru 0x03ffff (0xdf1 bytes) 000932 1788864394.804226 write 0x003c07 thru 0x0103f0 (0xc7ea bytes) 000933 1788864394.842256 write 0x011798 thru 0x018245 (0x6aae bytes) 000936 1788864394.880941 write 0x012479 thru 0x016c11 (0x4799 bytes) 000937 1788864394.882591 write 0x038ff1 thru 0x03e18b (0x519b bytes) 000938 1788864394.919054 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3bddd9ae (stale) Mirror 4: 0xee618a6c (stale) Mirror 5: 0xee618a6c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x75a13eca Mirror 6: 0x75a13eca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000939 1788864396.570410 punch from 0x001f45 to 0x009ca4 (0x7d60 bytes) 000940 1788864396.639937 punch from 0x00a3cd to 0x0169e4 (0xc618 bytes) 000941 1788864396.647111 write 0x03c1da thru 0x03ffff (0x3e26 bytes) 000942 1788864396.653418 read 0x002314 thru 0x00df5c (0xbc49 bytes) 000943 1788864396.661397 write 0x00e1eb thru 0x017181 (0x8f97 bytes) 000944 1788864396.676481 punch from 0x00b34e to 0x01083a (0x54ed bytes) 000945 1788864396.700362 mapread 0x02ed65 thru 0x0324d6 (0x3772 bytes) 000946 1788864396.727765 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3bddd9ae (stale) Mirror 4: 0xee618a6c (stale) Mirror 5: 0xee618a6c (stale) Mirror 6: 0x75a13eca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc238ba40 Mirror 7: 0xc238ba40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000947 1788864398.364086 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3bddd9ae (stale) Mirror 4: 0xee618a6c (stale) Mirror 5: 0xee618a6c (stale) Mirror 6: 0x75a13eca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc238ba40 Mirror 7: 0xc238ba40 Mirror 8: 0xc238ba40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000948 1788864400.118353 write 0x02e4ca thru 0x033317 (0x4e4e bytes) 000949 1788864400.310233 trunc from 0x040000 to 0x0139bb (0x2c644 bytes) 000950 1788864400.333835 trunc from 0x0139bc to 0x01471a (0xd5f bytes) 000951 1788864400.352709 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1471b): Mirror 4: 0x92bc2216 (stale) Mirror 5: 0x92bc2216 (stale) Mirror 6: 0xf267ea30 (stale) Mirror 7: 0x1752e208 (stale) Mirror 8: 0x1752e208 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1471b): Mirror 4: 0x92bc2216 (stale) Mirror 5: 0x92bc2216 (stale) Mirror 6: 0xf267ea30 (stale) Mirror 7: 0x1752e208 (stale) Mirror 8: 0x1752e208 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1471b: skipped 000952 1788864400.797313 mapread 0x0084bc thru 0x01471a (0xc25f bytes) 000953 1788864400.818193 punch from 0x00a88e to 0x00e5d1 (0x3d44 bytes) 000954 1788864400.834082 trunc from 0x01471b to 0x012900 (0x1e1a bytes) 000955 1788864400.852712 trunc from 0x012901 to 0x01a75b (0x7e5b bytes) 000956 1788864400.869077 mapwrite 0x00917c thru 0x0101aa (0x702f bytes) 000957 1788864400.989454 mapwrite 0x0174a0 thru 0x02015b (0x8cbc bytes) 000959 1788864401.119792 mapwrite 0x022a0e thru 0x02fab4 (0xd0a7 bytes) 000960 1788864401.194377 read 0x00b3fa thru 0x017505 (0xc10c bytes) 000961 1788864401.195202 mapwrite 0x03212e thru 0x03bd3b (0x9c0e bytes) 000963 1788864401.257378 punch from 0x010fe2 to 0x013846 (0x2865 bytes) 000964 1788864401.261375 punch from 0x00797b to 0x01732e (0xf9b4 bytes) 000965 1788864401.265251 mapwrite 0x035dd9 thru 0x03f07c (0x92a4 bytes) 000966 1788864401.318697 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f07d): Mirror 4: 0xf24cb74d (stale) Mirror 5: 0xf24cb74d (stale) Mirror 7: 0xb49ee911 (stale) Mirror 8: 0xb49ee911 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f07d): Mirror 4: 0xf24cb74d (stale) Mirror 5: 0xf24cb74d (stale) Mirror 7: 0xb49ee911 (stale) Mirror 8: 0xb49ee911 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f07d: skipped 000967 1788864401.675011 punch from 0x022f3a to 0x023991 (0xa58 bytes) 000973 1788864401.683369 punch from 0x0301cb to 0x0313d1 (0x1207 bytes) 000974 1788864401.688788 read 0x0153ce thru 0x01c1b2 (0x6de5 bytes) 000975 1788864401.695539 read 0x02189f thru 0x024dd1 (0x3533 bytes) 000976 1788864401.707968 trunc from 0x03f07d to 0x02214f (0x1cf2d bytes) 000977 1788864401.728393 MIRROR_RESYNC 5 mirrors 000978 1788864403.515257 punch from 0x01eed6 to 0x02214f (0x327a bytes) 000979 1788864403.581082 read 0x0103e6 thru 0x01f73b (0xf356 bytes) 000980 1788864403.592160 read 0x00dbb9 thru 0x013844 (0x5c8c bytes) 000981 1788864403.603911 read 0x0219b7 thru 0x02214f (0x799 bytes) 000982 1788864403.604598 read 0x00fce0 thru 0x011c47 (0x1f68 bytes) 000983 1788864403.605145 trunc from 0x022150 to 0x011c2c (0x10523 bytes) 000985 1788864403.628650 write 0x0302d0 thru 0x03c28d (0xbfbe bytes) 000988 1788864403.637014 MIRROR_RESYNC 5 mirrors 000989 1788864405.348283 MIRROR_EXTEND to 6 mirrors 000990 1788864406.257255 mapread 0x00f00d thru 0x010ff7 (0x1feb bytes) 000991 1788864406.271048 punch from 0x0322e1 to 0x03c28d (0x9fad bytes) 000992 1788864406.392747 mapread 0x0177c5 thru 0x025c0c (0xe448 bytes) 000994 1788864406.415034 read 0x024432 thru 0x0265c8 (0x2197 bytes) 000995 1788864406.415837 mapwrite 0x0051e3 thru 0x014609 (0xf427 bytes) 000996 1788864406.488971 trunc from 0x03c28e to 0x036c86 (0x5607 bytes) 000997 1788864406.519155 read 0x00eb16 thru 0x010861 (0x1d4c bytes) 000999 1788864406.526900 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36c87): Mirror 4: 0xd88675cf (stale) Mirror 5: 0xd88675cf (stale) Mirror 7: 0xd88675cf (stale) Mirror 9: 0xd88675cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c87): Mirror 4: 0xd88675cf (stale) Mirror 5: 0xd88675cf (stale) Mirror 7: 0xd88675cf (stale) Mirror 9: 0xd88675cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c87: skipped All operations completed A-OK! PASS 70a (140s) == sanity-flr test 71: check mirror extend parent fid ==== 06:46:53 (1788864413) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 06:46:57 (1788864417) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284985 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197111 s, 5.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 06:47:06 (1788864426) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5319648 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3306912 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5875680 .. Extending file size to 8098496 .. Extending file size to 8426560 .. Extending file size to 7673344 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9284544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9309696 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57389 57390 57393 57395 57396 done 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) 192.168.201.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (96s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:48:42 (1788864522) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1291296 .. Extending file size to 5286272 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8205568 .. Extending file size to 7907104 .. Extending file size to 8461408 .. Extending file size to 8866592 .. Extending file size to 9014976 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58771 58773 58774 58777 58778 failed 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) 192.168.201.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (89s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:50:11 (1788864611) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152569 s, 13.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.362496 s, 5.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xed9e35d4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:50:33 (1788864633) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197902 s, 10.6 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 06:50:43 (1788864643) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:50:52 (1788864652) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150875 s, 13.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:51:01 (1788864661) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199897 s, 10.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:51:09 (1788864669) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.458683 s, 4.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:51:44 (1788864704) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg139-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11386 s, 9.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:52:47 (1788864767) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg139-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342135 s, 6.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:53:51 (1788864831) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg139-server mds-ost sync done. Creating new pool test_204f oleg139-server: Pool lustre.test_204f created Adding targets to pool oleg139-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg139-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.434025 s, 4.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg139-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg139-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg139-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:55:14 (1788864914) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:55:22 (1788864922) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:55:30 (1788864930) oleg139-server: Pool lustre.test_206 created Adding targets to pool oleg139-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg139-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg139-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg139-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg139-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:55:57 (1788864957) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120293 s, 8.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435343 s, 24.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf621da03 Mirror 3: 0xf621da03 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:56:09 (1788864969) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:56:14 (1788864974) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:56:18 (1788864978) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.177131 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x8c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x8c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x8c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1030000 Read this (4 bytes): BBBB 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:56:59 (1788865019) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0306026 s, 268 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:57:07 (1788865027) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05447 s, 19.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fa lmm_fid: [0x200000404:0x1fa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fa lmm_fid: [0x200000404:0x1fa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834426 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 06:57:09.000000000 -0400 Modify: 2026-09-08 06:57:09.000000000 -0400 Change: 2026-09-08 06:57:11.000000000 -0400 Birth: 2026-09-08 06:57:09.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:57:17 (1788865037) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:57:21 (1788865041) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.413011 s, 99.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0415053 s, 98.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 06:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 06:57 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:57:30 (1788865050) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:57:40 (1788865060) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1950 sec =========== 06:57:52 (1788865072) === sanity-flr: start cleanup 06:57:54 (1788865074) === === sanity-flr: finish cleanup 06:58:04 (1788865084) ===