-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 22:43:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg635-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:43:19 (1789180999) === oleg635-client.virtnet: executing check_config_client /mnt/lustre oleg635-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg635-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff921d91540000.idle_timeout=debug osc.lustre-OST0001-osc-ffff921d91540000.idle_timeout=debug disable quota as required oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:43:49 (1789181029) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanity-flr test 0a: lfs mirror create with -N option == 22:43:51 (1789181031) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:44:06 (1789181046) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:44:09 (1789181049) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:44:12 (1789181052) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:44:34 (1789181074) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:44:36 (1789181076) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:44:39 (1789181079) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff921d91540000.stats=clear osc.lustre-OST0001-osc-ffff921d91540000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:44:47 (1789181087) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:44:57 (1789181097) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:45:05 (1789181105) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:45:08 (1789181108) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:45:15 (1789181115) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 22:45:23 (1789181123) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:45:29 (1789181129) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:45:32 (1789181132) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:45:39 (1789181139) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:45:49 (1789181149) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:45:56 (1789181156) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.89051 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225865 s, 4.6 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:46:07 (1789181167) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.367635 s, 51.3 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:46:14 (1789181174) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:46:23 (1789181183) Stopping /mnt/lustre-ost1 (opts:) on oleg635-server oleg635-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg635-server: oleg635-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg635-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg635-server oleg635-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg635-server: oleg635-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg635-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (63s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:47:26 (1789181246) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.623827 s, 16.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg635-server oleg635-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg635-server: oleg635-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg635-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:48:00 (1789181280) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg635-server oleg635-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg635-server: oleg635-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg635-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg635-server oleg635-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg635-server: oleg635-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg635-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (73s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:49:13 (1789181353) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.64241 s, 11.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 22:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.979s user 0m0.001s sys 0m0.166s Stopping /mnt/lustre-ost1 (opts:) on oleg635-server oleg635-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff921d91540000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.783s user 0m0.000s sys 0m0.187s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg635-server: oleg635-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg635-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg635-server oleg635-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.943s user 0m0.001s sys 0m0.171s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg635-server: oleg635-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg635-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg635-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff921d91540000.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:50:44 (1789181444) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:50:48 (1789181448) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:50:51 (1789181451) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:50:55 (1789181455) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:51:02 (1789181462) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.574087 s, 21.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:51:14 (1789181474) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332644 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415978 s, 25.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:51:21 (1789181481) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:51:29 (1789181489) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000950417 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00252244 s, 406 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00131897 s, 776 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00137012 s, 747 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 22:51:39 (1789181499) create test files with size 15733239 .. 29d4a7dccb520da7a5c451240de0b7fb - 14c056986f833c2319fe1e5bed65520d - c22403bfa2329625a646fe792a6162a4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg635-client.virtnet /mnt/lustre (opts:) Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 22:51:59 (1789181519) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (23s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:52:23 (1789181543) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:52:31 (1789181551) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138696 s, 7.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314675 s, 33.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 22:52:42 (1789181562) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154342 s, 13.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.477268 s, 8.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235182 s, 8.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 22:53:01 (1789181581) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:53:04 (1789181584) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:53:09 (1789181589) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534187 s, 19.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 22:53:17 (1789181597) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:53:21 (1789181601) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:53:30 (1789181610) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.22625 s, 46.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg635-client.virtnet /mnt/lustre (opts:) Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:53:38 (1789181618) llite.lustre-ffff921d87d45000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.175084 s, 59.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg635-client.virtnet /mnt/lustre (opts:) Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff921d92746000.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:53:49 (1789181629) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:54:20 (1789181660) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:54:51 (1789181691) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 22:54:54 (1789181694) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.202781 s, 15.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.127475 s, 24.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe31101f1 (nosync,stale) Mirror 2: 0xe31101f1 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 010f453c07dcd7491d8ba5a550209eb9 - mirror 1 checksum: 48d7182eb15d37e7a54a362e3a857648 - mirror 2 checksum: 010f453c07dcd7491d8ba5a550209eb9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T22:54:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T22:54:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 010f453c07dcd7491d8ba5a550209eb9 - mirror 1 checksum: 48d7182eb15d37e7a54a362e3a857648 - mirror 2 checksum: 48d7182eb15d37e7a54a362e3a857648 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T22:54:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T22:54:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:55:04 (1789181704) Failing mds1 on oleg635-server Stopping /mnt/lustre-mds1 (opts:) on oleg635-server 22:55:09 (1789181709) shut down facet: mds1 facet_host: oleg635-server facet_failover_host: oleg635-server Failover mds1 to oleg635-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg635-server: oleg635-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0000 22:55:30 (1789181730) targets are mounted 22:55:30 (1789181730) facet_failover done oleg635-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234486 s, 44.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:55:46 (1789181746) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:55:49 (1789181749) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:55:52 (1789181752) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0125426 s, 1.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:55:59 (1789181759) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:56:02 (1789181762) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00966349 s, 424 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0920925 s, 44.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:56:10 (1789181770) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:56:31 (1789181791) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:56:57 (1789181817) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:57:19 (1789181839) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:57:42 (1789181862) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 22:57:47 (1789181867) mirror create pid 54315 mirror split pid 54316 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54315 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54316 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 22:58:53 (1789181933) Seed set to 1934 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole skipping zero size read 000005 1789181934.644726 mapwrite 0x02d23b thru 0x035770 (0x8536 bytes) 000006 1789181934.765355 read 0x02c1e3 thru 0x0353b1 (0x91cf bytes) 000008 1789181934.766498 write 0x01ca07 thru 0x022f9a (0x6594 bytes) 000011 1789181934.785765 mapread 0x010f08 thru 0x01e420 (0xd519 bytes) 000012 1789181934.803677 punch from 0x00b7af to 0x013540 (0x7d92 bytes) 000013 1789181934.811812 mapwrite 0x01b027 thru 0x0264a5 (0xb47f bytes) 000014 1789181934.892366 read 0x018bea thru 0x019cd7 (0x10ee bytes) 000015 1789181934.893093 write 0x0329b7 thru 0x03d242 (0xa88c bytes) 000016 1789181934.897583 mapread 0x017cc3 thru 0x027778 (0xfab6 bytes) 000019 1789181934.900539 mapread 0x013ddc thru 0x01e754 (0xa979 bytes) 000020 1789181934.919641 mapread 0x037757 thru 0x03a862 (0x310c bytes) 000021 1789181934.942305 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3b4c8 000022 1789181935.722050 trunc from 0x03d243 to 0x03b4c7 (0x1d7b bytes) 000024 1789181935.812550 write 0x02c9bc thru 0x033061 (0x66a6 bytes) 000025 1789181935.823416 trunc from 0x03b4c8 to 0x00bcb3 (0x2f814 bytes) 000026 1789181935.844280 punch from 0x00533c to 0x00bcb3 (0x6978 bytes) 000027 1789181935.849715 trunc from 0x00bcb4 to 0x03a001 (0x2e34e bytes) 000029 1789181935.864359 punch from 0x0396ec to 0x03a001 (0x916 bytes) 000030 1789181935.874455 mapread 0x039483 thru 0x03a001 (0xb7f bytes) 000032 1789181935.896086 read 0x02f463 thru 0x0370bf (0x7c5d bytes) 000033 1789181935.914531 mapread 0x0181de thru 0x01ddee (0x5c11 bytes) 000034 1789181935.945974 mapwrite 0x03ad62 thru 0x03ffff (0x529e bytes) 000035 1789181936.032090 mapwrite 0x03814b thru 0x03ffff (0x7eb5 bytes) 000037 1789181936.082349 mapread 0x00c6e5 thru 0x0107cd (0x40e9 bytes) 000039 1789181936.095162 punch from 0x004a5e to 0x01220c (0xd7af bytes) 000040 1789181936.106896 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcb6940d9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ef54706 Mirror 3: 0x8ef54706 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000041 1789181936.985043 MIRROR_RESYNC 3 mirrors 000042 1789181937.743926 mapwrite 0x00e206 thru 0x017a45 (0x9840 bytes) 000043 1789181937.923095 write 0x00645f thru 0x00cfb8 (0x6b5a bytes) 000045 1789181937.933891 write 0x01e704 thru 0x023226 (0x4b23 bytes) 000049 1789181937.953776 write 0x00781f thru 0x0150ce (0xd8b0 bytes) 000050 1789181937.955627 mapread 0x031de6 thru 0x03bb5c (0x9d77 bytes) 000051 1789181937.976098 write 0x01e44d thru 0x025611 (0x71c5 bytes) 000053 1789181937.982462 trunc from 0x040000 to 0x0125b7 (0x2da48 bytes) 000054 1789181938.029314 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x125b8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x125b8): Mirror 2: 0xfe7dc4d3 (stale) Mirror 3: 0xfe7dc4d3 (stale) CRC-32 checksum value for chunk [0, 0x125b8): Mirror 1: 0xa97e327f Mirror 4: 0xa97e327f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x125b8: skipped 000055 1789181938.915441 write 0x03abca thru 0x03ffff (0x5436 bytes) 000057 1789181939.027895 read 0x01b832 thru 0x01f6f5 (0x3ec4 bytes) 000058 1789181939.041941 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8ef54706 (stale) Mirror 3: 0x8ef54706 (stale) Mirror 4: 0x479c09d0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85b6752b Mirror 5: 0x85b6752b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000060 1789181940.073027 mapwrite 0x03ec44 thru 0x03ffff (0x13bc bytes) 000061 1789181940.302057 read 0x00e2ea thru 0x017a4c (0x9763 bytes) 000062 1789181940.313333 mapread 0x01ac96 thru 0x01f969 (0x4cd4 bytes) 000063 1789181940.325798 trunc from 0x040000 to 0x000348 (0x3fcb7 bytes) 000064 1789181940.374138 mapwrite 0x0147d3 thru 0x01ee26 (0xa654 bytes) 000065 1789181940.500269 punch from 0x01d831 to 0x01ee26 (0x15f6 bytes) 000066 1789181940.510412 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1ee27): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee27): Mirror 2: 0x9882da45 (stale) Mirror 3: 0x9882da45 (stale) Mirror 4: 0xd68bbc59 (stale) Mirror 5: 0xd68bbc59 (stale) CRC-32 checksum value for chunk [0, 0x1ee27): Mirror 1: 0x7f85da2c Mirror 6: 0x7f85da2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee27: skipped 000068 1789181941.371606 punch from 0x00b643 to 0x0184c2 (0xce80 bytes) 000070 1789181941.447664 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ee27): Mirror 3: 0x9882da45 (stale) Mirror 4: 0xd68bbc59 (stale) Mirror 5: 0xd68bbc59 (stale) Mirror 6: 0x7f85da2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee27): Mirror 3: 0x9882da45 (stale) Mirror 4: 0xd68bbc59 (stale) Mirror 5: 0xd68bbc59 (stale) Mirror 6: 0x7f85da2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee27: skipped 000071 1789181941.941508 write 0x01a7f0 thru 0x01fd48 (0x5559 bytes) 000074 1789181941.952867 read 0x0104ac thru 0x0137ec (0x3341 bytes) 000076 1789181941.963951 read 0x01be03 thru 0x01fd48 (0x3f46 bytes) 000078 1789181941.967497 mapwrite 0x03dabe thru 0x03e628 (0xb6b bytes) 000079 1789181942.047612 read 0x00ee12 thru 0x0115a2 (0x2791 bytes) 000080 1789181942.049178 write 0x02ad4a thru 0x02b288 (0x53f bytes) 000081 1789181942.050319 mapwrite 0x00a4d0 thru 0x00c996 (0x24c7 bytes) 000082 1789181942.102671 trunc from 0x03e629 to 0x02a764 (0x13ec4 bytes) 000084 1789181942.119146 mapread 0x018138 thru 0x021a13 (0x98dc bytes) 000085 1789181942.120723 trunc from 0x02a765 to 0x033cb0 (0x954c bytes) 000088 1789181942.143498 MIRROR_RESYNC 5 mirrors 000090 1789181943.282311 mapread 0x0261dd thru 0x027a2f (0x1853 bytes) truncating to largest ever: 0x3cf70 000092 1789181943.294632 trunc from 0x033cb1 to 0x03cf6f (0x92bf bytes) 000093 1789181943.429507 write 0x037af9 thru 0x03ffff (0x8507 bytes) 000094 1789181943.449645 mapwrite 0x03f1b9 thru 0x03ffff (0xe47 bytes) 000096 1789181943.501945 read 0x007b15 thru 0x00e2da (0x67c6 bytes) 000097 1789181943.511293 write 0x020468 thru 0x02682e (0x63c7 bytes) 000098 1789181943.523472 punch from 0x037cd4 to 0x03ffff (0x832c bytes) 000099 1789181943.536930 trunc from 0x040000 to 0x016f97 (0x29068 bytes) 000102 1789181943.562585 punch from 0x013dbd to 0x016f97 (0x31db bytes) 000105 1789181943.568663 mapread 0x01364c thru 0x016f97 (0x394c bytes) 000106 1789181943.577981 trunc from 0x016f98 to 0x004fc6 (0x11fd1 bytes) 000107 1789181943.598705 mapread 0x00043a thru 0x004fc6 (0x4b8d bytes) 000108 1789181943.617428 mapread 0x00261c thru 0x004fc6 (0x29ab bytes) 000109 1789181943.618793 trunc from 0x004fc7 to 0x00877d (0x37b7 bytes) 000111 1789181943.642700 read 0x006bed thru 0x0083c1 (0x17d5 bytes) 000112 1789181943.648191 mapread 0x006736 thru 0x00877d (0x2048 bytes) 000113 1789181943.650190 punch from 0x0063b5 to 0x00877d (0x23c9 bytes) 000115 1789181943.656906 mapwrite 0x026b8b thru 0x027c25 (0x109b bytes) 000116 1789181943.727800 mapwrite 0x006eae thru 0x014dee (0xdf41 bytes) 000117 1789181943.819838 punch from 0x004624 to 0x005717 (0x10f4 bytes) 000118 1789181943.824372 trunc from 0x027c26 to 0x008cd6 (0x1ef4f bytes) 000120 1789181943.842168 read 0x002aad thru 0x008cd6 (0x622a bytes) 000121 1789181943.847983 mapwrite 0x02f824 thru 0x0339e2 (0x41bf bytes) 000122 1789181943.976018 punch from 0x0013f7 to 0x001906 (0x510 bytes) 000123 1789181943.995449 MIRROR_RESYNC 5 mirrors 000124 1789181946.208560 mapread 0x02777d thru 0x02fd81 (0x8605 bytes) 000125 1789181946.255759 mapread 0x000184 thru 0x00ca2d (0xc8aa bytes) 000127 1789181946.276824 read 0x00f02b thru 0x01b5ea (0xc5c0 bytes) 000128 1789181946.303758 MIRROR_RESYNC 5 mirrors 000130 1789181946.855547 read 0x015fac thru 0x021b8b (0xbbe0 bytes) 000131 1789181946.865097 mapread 0x01cf51 thru 0x02b4c0 (0xe570 bytes) 000132 1789181946.870292 MIRROR_RESYNC 5 mirrors 000133 1789181947.421000 punch from 0x01cfb8 to 0x02c987 (0xf9d0 bytes) 000134 1789181947.476538 mapread 0x022a70 thru 0x024154 (0x16e5 bytes) truncating to largest ever: 0x3d3d3 000135 1789181947.497405 trunc from 0x0339e3 to 0x03d3d2 (0x99f0 bytes) 000136 1789181947.524064 read 0x01e017 thru 0x01ed0c (0xcf6 bytes) 000137 1789181947.524832 punch from 0x032f5a to 0x03d3d2 (0xa479 bytes) 000138 1789181947.531401 punch from 0x028ff8 to 0x0362cf (0xd2d8 bytes) 000139 1789181947.535659 read 0x005df5 thru 0x0149a9 (0xebb5 bytes) 000141 1789181947.546600 mapread 0x03abe4 thru 0x03d3d2 (0x27ef bytes) 000144 1789181947.553440 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d3d3): Mirror 4: 0xe3bc5e83 (stale) Mirror 5: 0xe3bc5e83 (stale) Mirror 6: 0xe3bc5e83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3d3): Mirror 4: 0xe3bc5e83 (stale) Mirror 5: 0xe3bc5e83 (stale) Mirror 6: 0xe3bc5e83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3d3: skipped 000145 1789181947.961526 punch from 0x01119b to 0x017bbe (0x6a24 bytes) 000147 1789181947.976662 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d3d3): Mirror 4: 0xe3bc5e83 (stale) Mirror 6: 0xe3bc5e83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3d3): Mirror 4: 0xe3bc5e83 (stale) Mirror 6: 0xe3bc5e83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3d3: skipped 000148 1789181948.313924 punch from 0x01ba27 to 0x01cd5a (0x1334 bytes) 000149 1789181948.319429 mapread 0x032677 thru 0x03d3d2 (0xad5c bytes) 000150 1789181948.352042 punch from 0x02c252 to 0x0349f4 (0x87a3 bytes) 000151 1789181948.357760 trunc from 0x03d3d3 to 0x03761f (0x5db3 bytes) 000153 1789181948.395610 mapwrite 0x034fe5 thru 0x03b357 (0x6373 bytes) 000154 1789181948.472599 read 0x02bb91 thru 0x031f69 (0x63d9 bytes) 000156 1789181948.497933 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b358): Mirror 4: 0x2be996b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b358): Mirror 4: 0x2be996b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b358: skipped 000158 1789181948.838256 write 0x00fa10 thru 0x01561b (0x5c0c bytes) 000159 1789181948.867959 mapread 0x0085dc thru 0x0108b8 (0x82dd bytes) 000160 1789181948.899299 mapread 0x00f980 thru 0x01da9d (0xe11e bytes) 000161 1789181948.929203 write 0x037ebf thru 0x03ffff (0x8141 bytes) 000162 1789181948.936300 punch from 0x01c8a4 to 0x025cb7 (0x9414 bytes) 000163 1789181948.976475 trunc from 0x040000 to 0x03771f (0x88e0 bytes) 000164 1789181949.006762 trunc from 0x037720 to 0x004e77 (0x328a8 bytes) 000165 1789181949.038297 punch from 0x003a7c to 0x004e77 (0x13fc bytes) 000166 1789181949.052698 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000168 1789181949.253674 mapwrite 0x021f09 thru 0x02557d (0x3675 bytes) 000169 1789181949.396524 mapwrite 0x02090c thru 0x02adba (0xa4af bytes) 000170 1789181949.475820 mapwrite 0x01c64b thru 0x01f068 (0x2a1e bytes) 000171 1789181949.526435 punch from 0x0106cf to 0x02042a (0xfd5c bytes) 000172 1789181949.532539 mapread 0x00f204 thru 0x01ab22 (0xb91f bytes) 000173 1789181949.552826 mapread 0x016884 thru 0x019f51 (0x36ce bytes) 000174 1789181949.554562 mapwrite 0x002298 thru 0x004a6f (0x27d8 bytes) 000175 1789181949.644782 mapread 0x01efe8 thru 0x01efea (0x3 bytes) 000176 1789181949.645555 trunc from 0x02adbb to 0x03a852 (0xfa98 bytes) 000180 1789181949.667501 mapwrite 0x037f10 thru 0x03ffff (0x80f0 bytes) 000182 1789181949.751707 write 0x0381ab thru 0x039750 (0x15a6 bytes) 000185 1789181949.755001 write 0x033ed7 thru 0x03d1a0 (0x92ca bytes) 000187 1789181949.757803 mapwrite 0x01c444 thru 0x01f80f (0x33cc bytes) 000188 1789181949.805902 trunc from 0x040000 to 0x025c9e (0x1a361 bytes) 000190 1789181949.819661 read 0x00d1d1 thru 0x01a1dc (0xd00c bytes) 000191 1789181949.823099 punch from 0x006dff to 0x015a8f (0xec91 bytes) 000195 1789181949.831514 read 0x017103 thru 0x021b13 (0xaa11 bytes) 000198 1789181949.834433 write 0x00d510 thru 0x018ea8 (0xb999 bytes) 000200 1789181949.847490 mapread 0x01f3b2 thru 0x024e62 (0x5ab1 bytes) 000204 1789181949.852449 read 0x025c61 thru 0x025c9e (0x3e bytes) 000207 1789181949.855864 mapread 0x017662 thru 0x025c9e (0xe63d bytes) 000209 1789181949.857378 read 0x00f6bc thru 0x013839 (0x417e bytes) 000211 1789181949.859918 mapread 0x01cbb3 thru 0x022cd1 (0x611f bytes) 000212 1789181949.861436 read 0x01a789 thru 0x01ae4d (0x6c5 bytes) 000213 1789181949.862466 punch from 0x0164a8 to 0x022933 (0xc48c bytes) 000214 1789181949.899558 trunc from 0x025c9f to 0x012efe (0x12da0 bytes) 000216 1789181949.921727 mapwrite 0x038c18 thru 0x03c6d3 (0x3abc bytes) 000217 1789181950.046126 write 0x00db50 thru 0x012c0a (0x50bb bytes) 000219 1789181950.049447 mapread 0x00132c thru 0x0049be (0x3693 bytes) 000220 1789181950.051099 punch from 0x010dad to 0x017cd0 (0x6f24 bytes) 000222 1789181950.067151 punch from 0x03469f to 0x03c6d3 (0x8035 bytes) 000223 1789181950.071498 trunc from 0x03c6d4 to 0x0179dd (0x24cf6 bytes) 000224 1789181950.089630 mapread 0x00c806 thru 0x0179dd (0xb1d8 bytes) 000225 1789181950.136144 mapwrite 0x00460c thru 0x00a6da (0x60cf bytes) 000226 1789181950.192306 mapread 0x0123b0 thru 0x0179dd (0x562e bytes) 000229 1789181950.194342 trunc from 0x0179de to 0x006192 (0x1184b bytes) 000232 1789181950.215111 mapread 0x0017a8 thru 0x006192 (0x49eb bytes) 000233 1789181950.216345 write 0x019798 thru 0x025e48 (0xc6b1 bytes) 000234 1789181950.217867 trunc from 0x025e49 to 0x01657b (0xf8cd bytes) 000235 1789181950.249302 write 0x0057a0 thru 0x013018 (0xd879 bytes) 000236 1789181950.255892 mapread 0x015f53 thru 0x01657b (0x629 bytes) 000237 1789181950.270095 mapread 0x011b51 thru 0x01657b (0x4a2b bytes) 000238 1789181950.280807 mapread 0x0115be thru 0x01657b (0x4fbe bytes) 000240 1789181950.283471 punch from 0x005f9b to 0x013bcb (0xdc31 bytes) 000241 1789181950.308637 mapread 0x002c6a thru 0x00fff3 (0xd38a bytes) 000242 1789181950.376608 punch from 0x0051a0 to 0x00dc1b (0x8a7c bytes) 000248 1789181950.387626 mapwrite 0x024805 thru 0x02e4d6 (0x9cd2 bytes) 000251 1789181950.528455 read 0x00ac97 thru 0x017507 (0xc871 bytes) 000253 1789181950.541844 write 0x024c3e thru 0x024c4c (0xf bytes) 000254 1789181950.545187 read 0x00b4f4 thru 0x00ea43 (0x3550 bytes) 000255 1789181950.546902 punch from 0x00b506 to 0x016b48 (0xb643 bytes) 000256 1789181950.560402 punch from 0x011d3d to 0x0134bd (0x1781 bytes) 000258 1789181950.577036 MIRROR_EXTEND to 2 mirrors 000259 1789181951.153344 mapwrite 0x01db38 thru 0x0220f8 (0x45c1 bytes) 000260 1789181951.329318 punch from 0x0023cf to 0x006648 (0x427a bytes) 000261 1789181951.338792 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000262 1789181951.466809 punch from 0x02bd13 to 0x02e4d6 (0x27c4 bytes) 000263 1789181951.472035 trunc from 0x02e4d7 to 0x02b987 (0x2b4f bytes) 000264 1789181951.511424 mapread 0x022d05 thru 0x02b987 (0x8c83 bytes) 000265 1789181951.550048 mapwrite 0x02f19a thru 0x03b260 (0xc0c7 bytes) 000266 1789181951.684282 read 0x03778e thru 0x038d29 (0x159c bytes) 000267 1789181951.685515 read 0x0277cf thru 0x02bd58 (0x458a bytes) 000268 1789181951.686320 trunc from 0x03b261 to 0x034ba8 (0x66b8 bytes) 000270 1789181951.701816 punch from 0x01afca to 0x02971f (0xe756 bytes) 000272 1789181951.707924 read 0x004684 thru 0x005d89 (0x1706 bytes) 000273 1789181951.732022 write 0x01842d thru 0x02190e (0x94e2 bytes) 000274 1789181951.735608 punch from 0x026cca to 0x0330e8 (0xc41f bytes) 000275 1789181951.755345 read 0x02d59d thru 0x034ba8 (0x760c bytes) truncating to largest ever: 0x3ef26 000276 1789181951.767131 trunc from 0x034ba9 to 0x03ef25 (0xa37d bytes) 000278 1789181951.791259 trunc from 0x03ef26 to 0x01c100 (0x22e25 bytes) 000279 1789181951.808896 mapwrite 0x0241a5 thru 0x02d4d6 (0x9332 bytes) 000280 1789181952.063374 mapwrite 0x03564e thru 0x03b0b8 (0x5a6b bytes) 000281 1789181952.255250 mapwrite 0x03c0cd thru 0x03ffff (0x3f33 bytes) 000282 1789181952.402945 mapwrite 0x00492d thru 0x00f179 (0xa84d bytes) 000287 1789181952.508691 mapwrite 0x030f9c thru 0x03ffff (0xf064 bytes) 000288 1789181952.647134 write 0x02c570 thru 0x02fdec (0x387d bytes) 000290 1789181952.660000 punch from 0x021d76 to 0x026d2e (0x4fb9 bytes) 000292 1789181952.679884 MIRROR_EXTEND to 2 mirrors 000293 1789181953.427197 mapread 0x00996d thru 0x00ea35 (0x50c9 bytes) 000294 1789181953.465808 mapwrite 0x01549d thru 0x01a7bc (0x5320 bytes) 000295 1789181953.688457 punch from 0x01a218 to 0x02062f (0x6418 bytes) 000296 1789181953.694037 punch from 0x018ce6 to 0x026e1c (0xe137 bytes) 000298 1789181953.704735 punch from 0x01416d to 0x01eb10 (0xa9a4 bytes) 000299 1789181953.710049 write 0x03386b thru 0x03ffff (0xc795 bytes) 000302 1789181953.722540 punch from 0x010696 to 0x01ce73 (0xc7de bytes) 000303 1789181953.745645 trunc from 0x040000 to 0x018e0a (0x271f5 bytes) 000305 1789181953.789295 mapwrite 0x0246de thru 0x0330a8 (0xe9cb bytes) 000306 1789181953.945380 mapread 0x00d8a3 thru 0x016ed6 (0x9634 bytes) 000308 1789181953.974053 trunc from 0x0330a9 to 0x030c4a (0x245e bytes) 000309 1789181953.995732 trunc from 0x030c4b to 0x014c81 (0x1bfc9 bytes) 000310 1789181954.018824 punch from 0x00deb4 to 0x014c81 (0x6dce bytes) 000311 1789181954.032531 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000312 1789181954.177197 punch from 0x014614 to 0x014c81 (0x66e bytes) 000314 1789181954.190905 mapread 0x00b73f thru 0x00fd46 (0x4608 bytes) 000315 1789181954.209124 write 0x038327 thru 0x03d429 (0x5103 bytes) 000316 1789181954.211144 read 0x00934b thru 0x00d4da (0x4190 bytes) 000317 1789181954.211863 punch from 0x023527 to 0x025bd6 (0x26b0 bytes) 000318 1789181954.224594 write 0x03cd90 thru 0x03ffff (0x3270 bytes) 000320 1789181954.228172 read 0x011e7c thru 0x01d80b (0xb990 bytes) 000321 1789181954.244037 read 0x00fd1c thru 0x017099 (0x737e bytes) 000322 1789181954.244978 read 0x03cd35 thru 0x03ffff (0x32cb bytes) truncating to largest ever: 0x3f542 000323 1789181954.249263 trunc from 0x040000 to 0x03f541 (0xabe bytes) 000325 1789181954.267401 trunc from 0x03f542 to 0x03cd99 (0x27a8 bytes) 000326 1789181954.288577 punch from 0x00348c to 0x0105a0 (0xd115 bytes) 000327 1789181954.301572 MIRROR_EXTEND to 2 mirrors 000329 1789181954.912863 trunc from 0x03cd9a to 0x01636d (0x26a2c bytes) 000331 1789181955.025927 write 0x034375 thru 0x03ffff (0xbc8b bytes) 000332 1789181955.027385 mapwrite 0x011028 thru 0x013144 (0x211d bytes) 000333 1789181955.097412 read 0x030ef2 thru 0x032316 (0x1425 bytes) 000334 1789181955.103538 punch from 0x033bcc to 0x03ffff (0xc434 bytes) 000335 1789181955.113764 MIRROR_RESYNC 2 mirrors 000336 1789181955.992488 mapread 0x00e6bd thru 0x00f459 (0xd9d bytes) 000338 1789181956.011174 mapread 0x0133f3 thru 0x017808 (0x4416 bytes) 000339 1789181956.026539 punch from 0x011e36 to 0x015692 (0x385d bytes) 000340 1789181956.099492 mapwrite 0x016c57 thru 0x0195f0 (0x299a bytes) 000343 1789181956.142121 punch from 0x03cb96 to 0x03ffff (0x346a bytes) 000344 1789181956.149569 trunc from 0x040000 to 0x015f4b (0x2a0b4 bytes) 000345 1789181956.216575 mapread 0x003168 thru 0x00f71a (0xc5b3 bytes) 000346 1789181956.236367 punch from 0x00d4ae to 0x010e52 (0x39a5 bytes) 000348 1789181956.241124 write 0x004cd6 thru 0x005bc8 (0xef3 bytes) 000350 1789181956.247364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15f4c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15f4c): Mirror 2: 0x949dea68 (stale) CRC-32 checksum value for chunk [0, 0x15f4c): Mirror 1: 0xb223c15 Mirror 3: 0xb223c15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15f4c: skipped 000351 1789181957.248298 MIRROR_RESYNC 3 mirrors 000352 1789181958.140349 MIRROR_EXTEND to 4 mirrors 000354 1789181959.070127 trunc from 0x015f4c to 0x0198bc (0x3971 bytes) 000355 1789181959.282033 mapwrite 0x026e24 thru 0x02a5b5 (0x3792 bytes) 000357 1789181959.412653 mapwrite 0x027d0b thru 0x0307b5 (0x8aab bytes) 000359 1789181959.527191 trunc from 0x0307b6 to 0x02bfc6 (0x47ef bytes) 000360 1789181959.544533 mapwrite 0x03ea91 thru 0x03ffff (0x156f bytes) 000361 1789181959.626099 mapwrite 0x034d61 thru 0x03ffff (0xb29f bytes) 000362 1789181959.697403 trunc from 0x040000 to 0x00bb1c (0x344e3 bytes) 000364 1789181959.726460 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbb1d): Mirror 2: 0x1fbe911e (stale) Mirror 3: 0x1fbe911e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbb1d): Mirror 2: 0x1fbe911e (stale) Mirror 3: 0x1fbe911e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbb1d: skipped 000365 1789181960.156345 mapread 0x00252a thru 0x00a877 (0x834e bytes) 000368 1789181960.217549 mapwrite 0x016f1a thru 0x021837 (0xa91e bytes) 000370 1789181960.394879 punch from 0x0158ad to 0x021837 (0xbf8b bytes) 000371 1789181960.400064 mapwrite 0x02c253 thru 0x036d27 (0xaad5 bytes) 000372 1789181960.558580 write 0x027d65 thru 0x030ba8 (0x8e44 bytes) 000376 1789181960.565084 read 0x024ed9 thru 0x031f2b (0xd053 bytes) 000377 1789181960.580308 mapread 0x00ab96 thru 0x00f288 (0x46f3 bytes) 000378 1789181960.594747 mapread 0x0067a8 thru 0x0091d2 (0x2a2b bytes) 000379 1789181960.597157 mapread 0x0102b1 thru 0x01d4d8 (0xd228 bytes) 000381 1789181960.616964 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d28): Mirror 3: 0xcc46454c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d28): Mirror 3: 0xcc46454c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d28: skipped 000383 1789181961.018115 punch from 0x02d31d to 0x036d27 (0x9a0b bytes) 000384 1789181961.023486 mapwrite 0x02c736 thru 0x02d4f6 (0xdc1 bytes) 000385 1789181961.082393 read 0x026a74 thru 0x028e65 (0x23f2 bytes) 000386 1789181961.093851 read 0x00e03f thru 0x01d391 (0xf353 bytes) 000389 1789181961.123916 write 0x005a8c thru 0x013801 (0xdd76 bytes) 000390 1789181961.127645 mapwrite 0x00472d thru 0x007a99 (0x336d bytes) 000391 1789181961.186395 write 0x036b54 thru 0x03ffff (0x94ac bytes) 000392 1789181961.193391 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d7869b7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x997dd8eb Mirror 4: 0x997dd8eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000394 1789181962.097655 write 0x00b478 thru 0x018c52 (0xd7db bytes) 000395 1789181962.227405 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d7869b7 (stale) Mirror 4: 0x997dd8eb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa6ec1f9 Mirror 5: 0xa6ec1f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000396 1789181963.488853 mapwrite 0x01d62e thru 0x01d961 (0x334 bytes) 000397 1789181963.787750 read 0x0310a4 thru 0x036d12 (0x5c6f bytes) 000398 1789181963.813012 mapread 0x02b9f7 thru 0x0386d3 (0xccdd bytes) 000399 1789181963.849770 punch from 0x00ad76 to 0x018ddd (0xe068 bytes) 000402 1789181963.856821 punch from 0x00052d to 0x00d10a (0xcbde bytes) 000406 1789181963.876518 mapwrite 0x012d9b thru 0x015262 (0x24c8 bytes) 000408 1789181964.004624 trunc from 0x040000 to 0x009dac (0x36253 bytes) 000410 1789181964.073476 mapread 0x008b5f thru 0x009dac (0x124e bytes) 000411 1789181964.080668 punch from 0x0027fd to 0x009dac (0x75b0 bytes) 000412 1789181964.085469 write 0x019d81 thru 0x01b77f (0x19ff bytes) 000414 1789181964.087291 write 0x01f6fe thru 0x02288a (0x318d bytes) 000417 1789181964.096003 trunc from 0x02288b to 0x014ae2 (0xdda8 bytes) 000418 1789181964.118837 MIRROR_RESYNC 4 mirrors 000419 1789181965.313519 read 0x004e62 thru 0x013f4f (0xf0ee bytes) 000420 1789181965.324815 read 0x00ecd1 thru 0x014ae2 (0x5e12 bytes) 000421 1789181965.338500 MIRROR_EXTEND to 5 mirrors 000423 1789181966.191307 mapwrite 0x02cf80 thru 0x039d43 (0xcdc4 bytes) 000424 1789181966.395397 read 0x011397 thru 0x0179d5 (0x663f bytes) 000425 1789181966.410309 trunc from 0x039d44 to 0x013ef9 (0x25e4a bytes) 000426 1789181966.438081 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x13efa): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13efa): Mirror 3: 0xb9e1c62e (stale) Mirror 4: 0xb9e1c62e (stale) Mirror 5: 0xb9e1c62e (stale) Mirror 6: 0xb9e1c62e (stale) CRC-32 checksum value for chunk [0, 0x13efa): Mirror 1: 0xb9e1c62e Mirror 7: 0xb9e1c62e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13efa: skipped 000427 1789181967.612708 read 0x008c79 thru 0x013ef9 (0xb281 bytes) 000428 1789181967.646742 trunc from 0x013efa to 0x03e687 (0x2a78e bytes) 000429 1789181967.793573 trunc from 0x03e688 to 0x00e264 (0x30423 bytes) 000433 1789181967.848513 mapread 0x00308e thru 0x006e02 (0x3d75 bytes) 000434 1789181967.886109 read 0x008a7c thru 0x00e264 (0x57e9 bytes) 000435 1789181967.887142 read 0x0078a1 thru 0x00c45d (0x4bbd bytes) 000437 1789181967.890431 mapwrite 0x033a9f thru 0x038248 (0x47aa bytes) 000438 1789181968.028504 trunc from 0x038249 to 0x0010ce (0x3717a bytes) 000439 1789181968.055159 mapread 0x000f6a thru 0x0010ce (0x165 bytes) 000441 1789181968.058053 mapwrite 0x03bddc thru 0x03ffff (0x4224 bytes) 000443 1789181968.136199 punch from 0x03200b to 0x03df7c (0xbf72 bytes) 000444 1789181968.142243 trunc from 0x040000 to 0x0088af (0x37750 bytes) 000445 1789181968.160966 mapread 0x004667 thru 0x0088af (0x4249 bytes) 000446 1789181968.192859 write 0x005c20 thru 0x0068cb (0xcac bytes) 000448 1789181968.194388 write 0x033c1a thru 0x03ffff (0xc3e6 bytes) 000449 1789181968.198184 trunc from 0x040000 to 0x03acf1 (0x530e bytes) 000450 1789181968.217577 mapwrite 0x0106d8 thru 0x01840a (0x7d33 bytes) 000452 1789181968.311332 mapread 0x037e80 thru 0x03acf1 (0x2e72 bytes) 000453 1789181968.312519 read 0x012dd9 thru 0x01db52 (0xad7a bytes) 000454 1789181968.334112 read 0x00d48f thru 0x00e8da (0x144c bytes) 000455 1789181968.335263 write 0x009fb0 thru 0x00c500 (0x2551 bytes) 000456 1789181968.337662 mapread 0x026015 thru 0x030018 (0xa004 bytes) 000457 1789181968.339357 punch from 0x00c594 to 0x00fc57 (0x36c4 bytes) 000458 1789181968.358491 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3acf2): Mirror 3: 0x5d20e4ff (stale) Mirror 4: 0x5d20e4ff (stale) Mirror 6: 0x5d20e4ff (stale) Mirror 7: 0x5d20e4ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3acf2): Mirror 3: 0x5d20e4ff (stale) Mirror 4: 0x5d20e4ff (stale) Mirror 6: 0x5d20e4ff (stale) Mirror 7: 0x5d20e4ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3acf2: skipped 000459 1789181968.756724 mapwrite 0x02ab8e thru 0x02e220 (0x3693 bytes) 000460 1789181968.853824 write 0x0214b0 thru 0x02731b (0x5e6c bytes) 000462 1789181968.861538 trunc from 0x03acf2 to 0x00f507 (0x2b7ea bytes) 000463 1789181968.875790 trunc from 0x00f508 to 0x009560 (0x5fa7 bytes) 000464 1789181968.893898 write 0x02e5bc thru 0x02fcd5 (0x171a bytes) 000466 1789181968.896964 mapread 0x02a0a2 thru 0x02aa8f (0x9ee bytes) 000467 1789181968.905421 write 0x01d056 thru 0x02a9b9 (0xd964 bytes) 000468 1789181968.912454 punch from 0x0155b6 to 0x01e28d (0x8cd8 bytes) 000469 1789181968.933481 mapread 0x02ddec thru 0x02fcd5 (0x1eea bytes) 000471 1789181968.935155 mapwrite 0x03b2d5 thru 0x03ffff (0x4d2b bytes) 000472 1789181969.039372 read 0x00b22e thru 0x010a6d (0x5840 bytes) 000474 1789181969.052633 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd34a1e1 Mirror 8: 0xfd34a1e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1789181970.152977 trunc from 0x040000 to 0x024f29 (0x1b0d6 bytes) 000476 1789181970.285057 trunc from 0x024f2a to 0x02d099 (0x8170 bytes) 000477 1789181970.326728 mapwrite 0x0268cd thru 0x0353e3 (0xeb17 bytes) 000478 1789181970.440392 mapread 0x01627c thru 0x016711 (0x496 bytes) 000480 1789181970.458766 write 0x028901 thru 0x032cd5 (0xa3d5 bytes) 000481 1789181970.459923 trunc from 0x0353e4 to 0x014cd7 (0x2070c bytes) 000486 1789181970.487553 read 0x00bc5e thru 0x00e764 (0x2b07 bytes) 000489 1789181970.498437 write 0x036aca thru 0x039cab (0x31e2 bytes) 000491 1789181970.502927 mapread 0x0363ff thru 0x039cab (0x38ad bytes) 000493 1789181970.504253 write 0x005fd7 thru 0x0085ca (0x25f4 bytes) 000494 1789181970.514310 mapread 0x01164f thru 0x01a7e5 (0x9197 bytes) 000495 1789181970.526435 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39cac): Mirror 3: 0xe7890f21 (stale) Mirror 4: 0xe7890f21 (stale) Mirror 6: 0xe7890f21 (stale) Mirror 7: 0xe7890f21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39cac): Mirror 3: 0xe7890f21 (stale) Mirror 4: 0xe7890f21 (stale) Mirror 6: 0xe7890f21 (stale) Mirror 7: 0xe7890f21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39cac: skipped 000497 1789181970.985996 trunc from 0x039cac to 0x037c7c (0x202f bytes) 000498 1789181971.017577 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37c7d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c7d): Mirror 3: 0x9553b324 (stale) Mirror 4: 0x9553b324 (stale) Mirror 6: 0x9553b324 (stale) Mirror 7: 0x9553b324 (stale) CRC-32 checksum value for chunk [0, 0x37c7d): Mirror 1: 0x302c40c5 Mirror 8: 0x302c40c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c7d: skipped 000500 1789181972.309332 mapwrite 0x02375f thru 0x030fc4 (0xd866 bytes) 000502 1789181972.581430 mapwrite 0x01e26f thru 0x02c3f6 (0xe188 bytes) 000504 1789181972.662604 mapwrite 0x02eb73 thru 0x0378ff (0x8d8d bytes) 000505 1789181972.729832 mapread 0x01465e thru 0x01812c (0x3acf bytes) 000506 1789181972.750462 trunc from 0x037c7d to 0x017660 (0x2061c bytes) 000507 1789181972.771970 read 0x00723c thru 0x00aebb (0x3c80 bytes) 000509 1789181972.779036 trunc from 0x017661 to 0x038dbe (0x2175e bytes) 000510 1789181972.797106 mapwrite 0x03d098 thru 0x03ffff (0x2f68 bytes) 000511 1789181972.899139 write 0x029fa9 thru 0x02f2e9 (0x5341 bytes) 000512 1789181972.912916 mapwrite 0x00aebc thru 0x013f53 (0x9098 bytes) 000514 1789181972.996179 read 0x00cbec thru 0x01bd0e (0xf123 bytes) 000515 1789181973.003745 write 0x024ed4 thru 0x026c70 (0x1d9d bytes) 000516 1789181973.004628 read 0x014ada thru 0x01cb90 (0x80b7 bytes) 000518 1789181973.005705 trunc from 0x040000 to 0x017102 (0x28efd bytes) 000521 1789181973.030139 trunc from 0x017103 to 0x03ebfa (0x27af8 bytes) 000522 1789181973.061246 read 0x030dcd thru 0x031ec1 (0x10f5 bytes) 000523 1789181973.079060 mapread 0x00e368 thru 0x010bf2 (0x288b bytes) 000524 1789181973.080135 punch from 0x00f02c to 0x014200 (0x51d5 bytes) 000525 1789181973.092614 punch from 0x003af3 to 0x00c723 (0x8c31 bytes) 000527 1789181973.098849 mapwrite 0x01cedd thru 0x0277dc (0xa900 bytes) 000528 1789181973.166707 read 0x029a53 thru 0x02e9d1 (0x4f7f bytes) 000533 1789181973.171968 trunc from 0x03ebfb to 0x012fb9 (0x2bc41 bytes) 000534 1789181973.196417 MIRROR_RESYNC 6 mirrors 000535 1789181974.372419 read 0x003676 thru 0x00ae95 (0x7820 bytes) 000539 1789181974.394532 mapwrite 0x0140f9 thru 0x01b018 (0x6f20 bytes) 000540 1789181974.641830 write 0x0398c6 thru 0x03be3f (0x257a bytes) 000541 1789181974.657005 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3be40): Mirror 3: 0xf667ad1d (stale) Mirror 4: 0xf667ad1d (stale) Mirror 6: 0xf667ad1d (stale) Mirror 7: 0xf667ad1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be40): Mirror 3: 0xf667ad1d (stale) Mirror 4: 0xf667ad1d (stale) Mirror 6: 0xf667ad1d (stale) Mirror 7: 0xf667ad1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be40: skipped 000542 1789181975.082380 mapwrite 0x03d998 thru 0x03ffff (0x2668 bytes) 000543 1789181975.178586 trunc from 0x040000 to 0x03e26d (0x1d92 bytes) 000544 1789181975.203368 read 0x00eae3 thru 0x00f476 (0x994 bytes) 000545 1789181975.210454 mapwrite 0x024016 thru 0x028814 (0x47ff bytes) 000546 1789181975.334509 MIRROR_RESYNC 5 mirrors 000547 1789181976.439207 mapwrite 0x010e3a thru 0x011b55 (0xd1c bytes) 000548 1789181976.635121 mapread 0x007eb4 thru 0x017d78 (0xfec5 bytes) 000549 1789181976.644726 punch from 0x01826d to 0x021611 (0x93a5 bytes) 000550 1789181976.649110 write 0x028d41 thru 0x035c64 (0xcf24 bytes) 000551 1789181976.664106 punch from 0x0244b8 to 0x031836 (0xd37f bytes) 000553 1789181976.692364 mapread 0x011609 thru 0x01d6d2 (0xc0ca bytes) 000554 1789181976.709032 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 3: 0x19f68644 (stale) Mirror 4: 0x19f68644 (stale) Mirror 7: 0x19f68644 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 3: 0x19f68644 (stale) Mirror 4: 0x19f68644 (stale) Mirror 7: 0x19f68644 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e26e: skipped 000555 1789181977.068774 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 4: 0x19f68644 (stale) Mirror 7: 0x19f68644 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 4: 0x19f68644 (stale) Mirror 7: 0x19f68644 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e26e: skipped 000556 1789181977.490524 MIRROR_RESYNC 3 mirrors 000557 1789181978.381445 punch from 0x032be9 to 0x03ac47 (0x805f bytes) 000558 1789181978.448786 MIRROR_RESYNC 3 mirrors 000560 1789181979.357476 mapread 0x009905 thru 0x019380 (0xfa7c bytes) 000561 1789181979.416608 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000562 1789181979.811238 mapwrite 0x015d0e thru 0x022012 (0xc305 bytes) 000565 1789181980.043551 mapread 0x01867b thru 0x0260b1 (0xda37 bytes) 000566 1789181980.050721 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 7: 0x81741839 (stale) CRC-32 checksum value for chunk [0, 0x3e26e): Mirror 1: 0xbba975d9 Mirror 8: 0xbba975d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e26e: skipped 000568 1789181980.973748 mapread 0x00b359 thru 0x01adfc (0xfaa4 bytes) 000569 1789181981.023521 punch from 0x02629c to 0x02a7f5 (0x455a bytes) 000570 1789181981.123972 read 0x027acc thru 0x02e5d1 (0x6b06 bytes) 000571 1789181981.136517 read 0x014f17 thru 0x01bd56 (0x6e40 bytes) 000572 1789181981.146530 trunc from 0x03e26e to 0x010b6e (0x2d6ff bytes) 000573 1789181981.175736 trunc from 0x010b6f to 0x000021 (0x10b4d bytes) 000576 1789181981.194133 write 0x009440 thru 0x00bce5 (0x28a6 bytes) 000577 1789181981.195560 read 0x00429d thru 0x009f4f (0x5cb3 bytes) 000579 1789181981.207505 mapwrite 0x004f65 thru 0x00de40 (0x8edc bytes) 000580 1789181981.287113 write 0x024e06 thru 0x02ac5a (0x5e55 bytes) 000581 1789181981.288622 trunc from 0x02ac5b to 0x01381a (0x17440 bytes) 000582 1789181981.306302 trunc from 0x01381b to 0x00c38e (0x748c bytes) 000583 1789181981.326936 mapwrite 0x0074ef thru 0x015601 (0xe113 bytes) 000584 1789181981.404493 mapwrite 0x01b49d thru 0x0285e1 (0xd145 bytes) 000585 1789181981.483570 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x285e2): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285e2): Mirror 7: 0x7b0b316c (stale) Mirror 8: 0x342d0aff (stale) CRC-32 checksum value for chunk [0, 0x285e2): Mirror 1: 0x7a2fff58 Mirror 9: 0x7a2fff58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285e2: skipped 000586 1789181982.269736 mapread 0x0205fc thru 0x022ab2 (0x24b7 bytes) 000588 1789181982.298995 mapread 0x017291 thru 0x018d20 (0x1a90 bytes) 000590 1789181982.324954 mapwrite 0x025419 thru 0x0342da (0xeec2 bytes) 000591 1789181982.547344 write 0x03c4b7 thru 0x03ffff (0x3b49 bytes) 000592 1789181982.555433 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd1845936 (stale) Mirror 8: 0x61740210 (stale) Mirror 9: 0x551f8f3f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7a4ca27d Mirror 10: 0x7a4ca27d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000593 1789181983.397700 read 0x026ba2 thru 0x02b787 (0x4be6 bytes) 000596 1789181983.425112 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd1845936 (stale) Mirror 8: 0x61740210 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd1845936 (stale) Mirror 8: 0x61740210 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7a4ca27d Mirror 10: 0x7a4ca27d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1789181984.003464 mapread 0x033ccf thru 0x03ffff (0xc331 bytes) 000598 1789181984.048685 mapwrite 0x00aa7a thru 0x0134e6 (0x8a6d bytes) 000600 1789181984.281615 mapread 0x0135b1 thru 0x0178e1 (0x4331 bytes) 000601 1789181984.283819 read 0x022584 thru 0x0283e2 (0x5e5f bytes) 000602 1789181984.292424 mapwrite 0x02b6ec thru 0x037e38 (0xc74d bytes) 000604 1789181984.359007 punch from 0x0377b3 to 0x03fadc (0x832a bytes) 000605 1789181984.371009 mapwrite 0x022a43 thru 0x025139 (0x26f7 bytes) 000607 1789181984.435173 mapwrite 0x029438 thru 0x033093 (0x9c5c bytes) 000608 1789181984.545117 mapread 0x015ac9 thru 0x01bbf1 (0x6129 bytes) 000609 1789181984.546665 write 0x02a12f thru 0x0305fa (0x64cc bytes) 000610 1789181984.562485 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd1845936 (stale) Mirror 8: 0x61740210 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd1845936 (stale) Mirror 8: 0x61740210 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000613 1789181984.911643 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x61740210 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x61740210 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000615 1789181985.215084 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x61740210 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xce1e3bc8 Mirror 9: 0xce1e3bc8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000616 1789181986.109859 read 0x037963 thru 0x03e262 (0x6900 bytes) 000617 1789181986.127830 trunc from 0x040000 to 0x0268e2 (0x1971d bytes) 000618 1789181986.250819 mapread 0x00c70f thru 0x01c6ac (0xff9e bytes) 000619 1789181986.282410 mapread 0x00ac33 thru 0x00bf15 (0x12e3 bytes) 000620 1789181986.283569 mapread 0x001698 thru 0x00acf2 (0x965b bytes) 000622 1789181986.295448 write 0x0108ef thru 0x01aafd (0xa20f bytes) 000624 1789181986.305674 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x268e3): Mirror 8: 0xc0bcacdd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268e3): Mirror 8: 0xc0bcacdd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268e3: skipped 000625 1789181986.738057 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000628 1789181986.891772 read 0x020deb thru 0x0235bf (0x27d5 bytes) 000629 1789181986.897366 mapwrite 0x0238aa thru 0x02c6b9 (0x8e10 bytes) 000630 1789181987.009732 write 0x02acf4 thru 0x02b420 (0x72d bytes) 000632 1789181987.011510 write 0x01b3b0 thru 0x02656e (0xb1bf bytes) 000633 1789181987.020926 mapread 0x028c5a thru 0x02c6b9 (0x3a60 bytes) 000634 1789181987.022744 punch from 0x02ba31 to 0x02c6b9 (0xc89 bytes) 000635 1789181987.044580 write 0x02757d thru 0x034d27 (0xd7ab bytes) 000636 1789181987.048261 mapwrite 0x027b73 thru 0x02fb27 (0x7fb5 bytes) 000637 1789181987.111008 punch from 0x014f14 to 0x016fa8 (0x2095 bytes) 000638 1789181987.118346 punch from 0x0171fe to 0x02110b (0x9f0e bytes) 000639 1789181987.125273 write 0x034031 thru 0x03ffff (0xbfcf bytes) 000641 1789181987.130386 punch from 0x018460 to 0x0218a8 (0x9449 bytes) 000642 1789181987.151606 trunc from 0x040000 to 0x03d2a8 (0x2d57 bytes) 000644 1789181987.176088 write 0x02e7e6 thru 0x038b4f (0xa36a bytes) 000647 1789181987.181178 trunc from 0x03d2a9 to 0x031ce5 (0xb5c3 bytes) 000650 1789181987.199463 trunc from 0x031ce6 to 0x00744e (0x2a897 bytes) 000651 1789181987.215469 punch from 0x00301d to 0x006e45 (0x3e29 bytes) 000652 1789181987.226526 MIRROR_EXTEND to 2 mirrors 000653 1789181987.881329 mapread 0x006bdd thru 0x00744e (0x872 bytes) 000654 1789181987.903803 write 0x01c31d thru 0x020176 (0x3e5a bytes) 000656 1789181987.993257 mapwrite 0x00fa01 thru 0x011942 (0x1f42 bytes) 000657 1789181988.064703 punch from 0x012848 to 0x016441 (0x3bfa bytes) 000658 1789181988.071165 mapwrite 0x0285c6 thru 0x032b99 (0xa5d4 bytes) 000659 1789181988.163703 MIRROR_RESYNC 2 mirrors 000660 1789181988.756857 read 0x01f821 thru 0x02d634 (0xde14 bytes) 000661 1789181988.768999 read 0x00e157 thru 0x012476 (0x4320 bytes) 000662 1789181988.778983 mapread 0x019405 thru 0x01d5f5 (0x41f1 bytes) 000663 1789181988.780560 punch from 0x02bb50 to 0x032b99 (0x704a bytes) 000666 1789181988.866834 read 0x004ed0 thru 0x00c59f (0x76d0 bytes) 000667 1789181988.881628 write 0x02981d thru 0x038e6d (0xf651 bytes) 000668 1789181988.884020 read 0x02f9e7 thru 0x032c39 (0x3253 bytes) 000670 1789181988.886518 read 0x00754d thru 0x0112ad (0x9d61 bytes) 000671 1789181988.887855 mapread 0x033d06 thru 0x038441 (0x473c bytes) 000672 1789181988.888979 trunc from 0x038e6e to 0x011e8b (0x26fe2 bytes) 000673 1789181988.922420 mapread 0x00eb5e thru 0x011e8b (0x332e bytes) 000675 1789181988.924609 write 0x02b99a thru 0x02eefa (0x3561 bytes) 000677 1789181988.927805 punch from 0x002069 to 0x00e530 (0xc4c8 bytes) 000678 1789181988.942561 mapwrite 0x008268 thru 0x015a3f (0xd7d8 bytes) 000679 1789181989.042314 MIRROR_RESYNC 2 mirrors 000681 1789181989.676411 write 0x007c55 thru 0x014a85 (0xce31 bytes) 000683 1789181989.785092 mapwrite 0x038924 thru 0x03ffff (0x76dc bytes) 000684 1789181989.899402 write 0x008a9e thru 0x015a27 (0xcf8a bytes) 000685 1789181989.909099 read 0x032460 thru 0x03ffff (0xdba0 bytes) 000686 1789181989.918197 mapread 0x01965a thru 0x024f7b (0xb922 bytes) 000687 1789181989.945625 mapwrite 0x0059f0 thru 0x0149c3 (0xefd4 bytes) 000688 1789181990.026022 trunc from 0x040000 to 0x00dadc (0x32523 bytes) 000690 1789181990.055082 punch from 0x006850 to 0x00c81d (0x5fce bytes) 000691 1789181990.062037 mapread 0x00758b thru 0x00dadc (0x6552 bytes) 000692 1789181990.072058 punch from 0x009b79 to 0x00d15c (0x35e4 bytes) 000693 1789181990.077036 trunc from 0x00dadd to 0x00efc4 (0x14e8 bytes) 000695 1789181990.112314 MIRROR_RESYNC 2 mirrors 000696 1789181990.903193 mapread 0x006472 thru 0x0091ff (0x2d8e bytes) 000698 1789181990.942882 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000699 1789181991.120559 punch from 0x002a24 to 0x00eca4 (0xc281 bytes) 000700 1789181991.139124 read 0x00cd72 thru 0x00efc4 (0x2253 bytes) 000701 1789181991.149113 punch from 0x00a519 to 0x00d5ae (0x3096 bytes) 000702 1789181991.155184 mapwrite 0x038e4f thru 0x03ffff (0x71b1 bytes) 000704 1789181991.286497 trunc from 0x040000 to 0x03446d (0xbb92 bytes) 000706 1789181991.307782 write 0x0392a5 thru 0x039cb9 (0xa15 bytes) 000708 1789181991.311165 read 0x014648 thru 0x020580 (0xbf39 bytes) 000709 1789181991.331353 mapwrite 0x022b85 thru 0x02a165 (0x75e1 bytes) 000710 1789181991.378026 trunc from 0x039cba to 0x033c42 (0x6077 bytes) 000711 1789181991.394561 write 0x0355af thru 0x03cf46 (0x7998 bytes) 000713 1789181991.396537 mapread 0x0379a3 thru 0x03cf46 (0x55a4 bytes) 000715 1789181991.398448 trunc from 0x03cf47 to 0x02f305 (0xdc41 bytes) 000716 1789181991.414702 mapwrite 0x036a81 thru 0x03ffff (0x957f bytes) 000717 1789181991.528237 trunc from 0x040000 to 0x01e904 (0x216fb bytes) 000719 1789181991.551845 mapread 0x00826a thru 0x00de52 (0x5be9 bytes) 000721 1789181991.553662 read 0x0064a9 thru 0x015c39 (0xf791 bytes) 000723 1789181991.555080 read 0x00f1f5 thru 0x01a953 (0xb75f bytes) 000724 1789181991.555847 trunc from 0x01e905 to 0x006281 (0x18683 bytes) 000726 1789181991.574513 trunc from 0x006282 to 0x0122f3 (0xc072 bytes) 000728 1789181991.596093 mapread 0x00a490 thru 0x0122f3 (0x7e64 bytes) 000729 1789181991.605023 trunc from 0x0122f4 to 0x010cb4 (0x163f bytes) 000730 1789181991.626082 trunc from 0x010cb5 to 0x02d156 (0x1c4a2 bytes) 000731 1789181991.646733 mapread 0x018992 thru 0x02501b (0xc68a bytes) 000732 1789181991.659575 read 0x00426c thru 0x0101b9 (0xbf4e bytes) 000733 1789181991.660523 mapread 0x007f01 thru 0x01026e (0x836e bytes) 000734 1789181991.661899 mapread 0x01d391 thru 0x02b5c3 (0xe233 bytes) 000735 1789181991.663206 trunc from 0x02d157 to 0x012212 (0x1af44 bytes) 000737 1789181991.694578 punch from 0x003e97 to 0x0084ba (0x4624 bytes) 000742 1789181991.705157 write 0x02af0a thru 0x02f223 (0x431a bytes) 000743 1789181991.706944 read 0x022320 thru 0x027a0f (0x56f0 bytes) 000745 1789181991.718039 read 0x017aa6 thru 0x019f15 (0x2470 bytes) 000746 1789181991.718613 mapwrite 0x0222a7 thru 0x025e16 (0x3b70 bytes) 000747 1789181991.767875 mapread 0x028fcd thru 0x02f223 (0x6257 bytes) 000750 1789181991.771336 mapwrite 0x0172f7 thru 0x0263d9 (0xf0e3 bytes) 000751 1789181991.840160 punch from 0x01c84c to 0x02bd52 (0xf507 bytes) 000752 1789181991.852201 write 0x03a2ef thru 0x03ffff (0x5d11 bytes) 000754 1789181991.857522 mapwrite 0x01d31a thru 0x027cb9 (0xa9a0 bytes) 000755 1789181991.951279 trunc from 0x040000 to 0x0265d2 (0x19a2d bytes) 000756 1789181991.970009 mapwrite 0x020f0e thru 0x02f5fd (0xe6f0 bytes) 000757 1789181992.120319 write 0x03579d thru 0x03c47e (0x6ce2 bytes) 000758 1789181992.122371 mapread 0x02a070 thru 0x02ce65 (0x2df6 bytes) 000759 1789181992.124243 read 0x02583b thru 0x0286db (0x2ea1 bytes) 000761 1789181992.126515 mapwrite 0x032b0f thru 0x035ab8 (0x2faa bytes) 000762 1789181992.194077 read 0x01eb36 thru 0x01fc9b (0x1166 bytes) 000763 1789181992.194983 write 0x030731 thru 0x034c42 (0x4512 bytes) 000764 1789181992.196595 write 0x007aba thru 0x0174e8 (0xfa2f bytes) 000765 1789181992.198862 trunc from 0x03c47f to 0x02b028 (0x11456 bytes) 000766 1789181992.217081 mapread 0x0151e5 thru 0x020057 (0xae73 bytes) 000767 1789181992.220140 trunc from 0x02b029 to 0x02b2c0 (0x298 bytes) 000768 1789181992.236898 trunc from 0x02b2c1 to 0x00c3ea (0x1eed6 bytes) 000769 1789181992.257947 write 0x036f93 thru 0x03ffff (0x906d bytes) 000771 1789181992.261339 trunc from 0x040000 to 0x017cc6 (0x28339 bytes) 000775 1789181992.284078 mapwrite 0x02948a thru 0x03190d (0x8484 bytes) 000776 1789181992.395834 write 0x010b68 thru 0x0203d5 (0xf86e bytes) 000777 1789181992.398077 mapread 0x01c719 thru 0x02b32d (0xec15 bytes) 000778 1789181992.400068 trunc from 0x03190e to 0x00c446 (0x254c7 bytes) 000780 1789181992.423023 mapread 0x008bca thru 0x00c446 (0x387d bytes) 000781 1789181992.426273 read 0x007112 thru 0x00884b (0x173a bytes) 000782 1789181992.426925 write 0x01081b thru 0x014501 (0x3ce7 bytes) 000783 1789181992.434616 mapread 0x00555a thru 0x011fef (0xca96 bytes) 000786 1789181992.445916 mapread 0x004fc3 thru 0x00eb60 (0x9b9e bytes) 000787 1789181992.446770 mapread 0x009264 thru 0x013ded (0xab8a bytes) 000788 1789181992.449391 read 0x00e40d thru 0x014501 (0x60f5 bytes) 000789 1789181992.450609 read 0x00f2c5 thru 0x014501 (0x523d bytes) 000792 1789181992.463042 mapread 0x00a9bd thru 0x00daf2 (0x3136 bytes) 000793 1789181992.464584 read 0x013620 thru 0x014501 (0xee2 bytes) 000795 1789181992.465991 write 0x0265e1 thru 0x035c7b (0xf69b bytes) 000796 1789181992.474348 mapwrite 0x0066cb thru 0x010ccf (0xa605 bytes) 000797 1789181992.532861 trunc from 0x035c7c to 0x03a4d1 (0x4856 bytes) 000798 1789181992.559173 write 0x01c62c thru 0x0231af (0x6b84 bytes) 000799 1789181992.567990 write 0x01d57f thru 0x028ef3 (0xb975 bytes) 000802 1789181992.570957 trunc from 0x03a4d2 to 0x022970 (0x17b61 bytes) 000804 1789181992.590798 read 0x002107 thru 0x004f3b (0x2e35 bytes) 000805 1789181992.591544 write 0x03398c thru 0x03c463 (0x8ad8 bytes) 000806 1789181992.593873 mapwrite 0x01f9e7 thru 0x025d69 (0x6383 bytes) 000807 1789181992.646790 mapwrite 0x01f3b3 thru 0x0219f4 (0x2642 bytes) 000808 1789181992.712988 mapread 0x02bc1a thru 0x02e9b1 (0x2d98 bytes) 000809 1789181992.714779 mapread 0x007d6c thru 0x00bbaa (0x3e3f bytes) 000810 1789181992.716054 punch from 0x027f79 to 0x033b28 (0xbbb0 bytes) 000811 1789181992.724977 mapwrite 0x03a1f7 thru 0x03ba51 (0x185b bytes) 000813 1789181992.776799 read 0x00086e thru 0x00df26 (0xd6b9 bytes) 000814 1789181992.777932 read 0x017425 thru 0x020d59 (0x9935 bytes) 000817 1789181992.781297 trunc from 0x03c464 to 0x00a470 (0x31ff3 bytes) 000818 1789181992.807166 write 0x022d19 thru 0x027c78 (0x4f60 bytes) 000819 1789181992.808514 read 0x025144 thru 0x027c78 (0x2b35 bytes) 000824 1789181992.815558 punch from 0x01a57f to 0x01f193 (0x4c15 bytes) 000825 1789181992.831855 mapwrite 0x0399cf thru 0x03e3ea (0x4a1c bytes) 000826 1789181992.919737 trunc from 0x03e3eb to 0x01aa76 (0x23974 bytes) 000827 1789181992.947591 read 0x00c90d thru 0x012a48 (0x613c bytes) 000828 1789181992.948393 punch from 0x01a6dd to 0x01aa76 (0x39a bytes) 000832 1789181992.959926 read 0x00672a thru 0x007bc3 (0x149a bytes) 000833 1789181992.965652 trunc from 0x01aa77 to 0x01221f (0x8857 bytes) 000834 1789181992.993456 mapwrite 0x032fa3 thru 0x035f56 (0x2fb4 bytes) 000835 1789181993.102195 mapwrite 0x020d79 thru 0x025f55 (0x51dd bytes) 000836 1789181993.167906 mapwrite 0x029446 thru 0x02c393 (0x2f4e bytes) 000837 1789181993.226028 mapread 0x0252ed thru 0x02ce61 (0x7b75 bytes) 000841 1789181993.231041 read 0x023031 thru 0x0312e1 (0xe2b1 bytes) 000845 1789181993.234373 punch from 0x015f47 to 0x02186a (0xb924 bytes) 000846 1789181993.240687 trunc from 0x035f57 to 0x00f949 (0x2660d bytes) 000847 1789181993.259265 write 0x038804 thru 0x03ffff (0x77fc bytes) 000848 1789181993.260760 read 0x01ff65 thru 0x02b8e0 (0xb97c bytes) 000853 1789181993.277488 mapwrite 0x026b6d thru 0x02d7a2 (0x6c36 bytes) 000854 1789181993.327968 punch from 0x00ff8e to 0x01c8d3 (0xc946 bytes) 000855 1789181993.333131 mapread 0x02702c thru 0x02e1b4 (0x7189 bytes) 000856 1789181993.334514 punch from 0x005101 to 0x011c89 (0xcb89 bytes) 000857 1789181993.339549 write 0x03db46 thru 0x03ffff (0x24ba bytes) 000858 1789181993.341261 read 0x03e3ff thru 0x03ffff (0x1c01 bytes) 000859 1789181993.342656 write 0x033508 thru 0x0368d2 (0x33cb bytes) 000860 1789181993.344264 mapwrite 0x01a02b thru 0x01bf63 (0x1f39 bytes) 000862 1789181993.472636 read 0x0243bf thru 0x029cf2 (0x5934 bytes) 000867 1789181993.478647 mapread 0x014a3a thru 0x0163db (0x19a2 bytes) 000869 1789181993.483287 punch from 0x03617d to 0x03ffff (0x9e83 bytes) 000870 1789181993.493979 mapwrite 0x02a40e thru 0x031b85 (0x7778 bytes) 000872 1789181993.591899 punch from 0x038a4a to 0x03de39 (0x53f0 bytes) 000875 1789181993.608035 write 0x03c84d thru 0x03ffff (0x37b3 bytes) 000876 1789181993.630076 mapwrite 0x021447 thru 0x029ca8 (0x8862 bytes) 000877 1789181993.714589 punch from 0x011ec1 to 0x02154d (0xf68d bytes) 000878 1789181993.722740 punch from 0x0340b8 to 0x0346ee (0x637 bytes) 000879 1789181993.731174 read 0x03d4ed thru 0x03ffff (0x2b13 bytes) 000880 1789181993.731876 trunc from 0x040000 to 0x026d26 (0x192d9 bytes) 000882 1789181993.764030 mapread 0x0239bd thru 0x0244b8 (0xafc bytes) 000884 1789181993.766664 trunc from 0x026d27 to 0x027e29 (0x1103 bytes) 000885 1789181993.798077 mapwrite 0x025ba4 thru 0x031604 (0xba61 bytes) 000886 1789181993.993085 read 0x0278ce thru 0x02e5ff (0x6d32 bytes) 000891 1789181994.002185 mapread 0x016762 thru 0x01f595 (0x8e34 bytes) 000892 1789181994.004457 punch from 0x006d0d to 0x0151e8 (0xe4dc bytes) 000895 1789181994.019480 trunc from 0x031605 to 0x00b4cd (0x26137 bytes) 000899 1789181994.075326 punch from 0x009412 to 0x00b4cd (0x20bc bytes) 000900 1789181994.084172 trunc from 0x00b4ce to 0x037dd2 (0x2c905 bytes) 000903 1789181994.119462 write 0x008af8 thru 0x014a81 (0xbf8a bytes) 000904 1789181994.137161 read 0x0197cd thru 0x01d4a0 (0x3cd4 bytes) 000905 1789181994.138093 mapread 0x01f127 thru 0x022395 (0x326f bytes) 000906 1789181994.140569 read 0x024824 thru 0x02b459 (0x6c36 bytes) 000910 1789181994.146523 mapread 0x003063 thru 0x00829b (0x5239 bytes) 000911 1789181994.148930 mapwrite 0x031af6 thru 0x03d861 (0xbd6c bytes) 000912 1789181994.237850 punch from 0x0006bc to 0x00c6eb (0xc030 bytes) 000913 1789181994.243674 mapwrite 0x038f3c thru 0x03b61f (0x26e4 bytes) 000914 1789181994.313571 punch from 0x0310ad to 0x039158 (0x80ac bytes) 000915 1789181994.323244 trunc from 0x03d862 to 0x01765d (0x26204 bytes) 000917 1789181994.344710 trunc from 0x01765e to 0x0338d1 (0x1c274 bytes) 000918 1789181994.372546 trunc from 0x0338d2 to 0x02f015 (0x48bc bytes) 000919 1789181994.399616 write 0x025d1b thru 0x02cc2f (0x6f15 bytes) 000920 1789181994.412695 trunc from 0x02f016 to 0x03b307 (0xc2f2 bytes) 000921 1789181994.436406 mapwrite 0x011173 thru 0x01e25c (0xd0ea bytes) 000924 1789181994.553065 write 0x026aac thru 0x02a65d (0x3bb2 bytes) 000925 1789181994.554787 write 0x0322a6 thru 0x03ffff (0xdd5a bytes) 000928 1789181994.564664 trunc from 0x040000 to 0x03797a (0x8685 bytes) 000930 1789181994.582925 trunc from 0x03797b to 0x018fb3 (0x1e9c7 bytes) 000931 1789181994.608696 trunc from 0x018fb4 to 0x0026f3 (0x168c0 bytes) 000932 1789181994.629084 trunc from 0x0026f4 to 0x009060 (0x696d bytes) 000933 1789181994.648150 punch from 0x004902 to 0x008cba (0x43b9 bytes) 000934 1789181994.653019 write 0x03ab12 thru 0x03ffff (0x54ee bytes) 000937 1789181994.658454 mapwrite 0x0303b1 thru 0x03d00f (0xcc5f bytes) 000939 1789181994.759357 read 0x00f829 thru 0x0164a5 (0x6c7d bytes) 000940 1789181994.770207 mapwrite 0x03d743 thru 0x03ffff (0x28bd bytes) 000941 1789181994.815182 read 0x022c31 thru 0x028f9f (0x636f bytes) 000945 1789181994.819301 trunc from 0x040000 to 0x01ab0f (0x254f0 bytes) 000947 1789181994.842112 mapwrite 0x032fe8 thru 0x0368b8 (0x38d1 bytes) 000949 1789181994.943649 mapread 0x005814 thru 0x005eb1 (0x69e bytes) 000951 1789181994.949409 punch from 0x0314e2 to 0x0368b8 (0x53d7 bytes) 000953 1789181994.961932 mapread 0x005dfc thru 0x00e2e1 (0x84e6 bytes) 000956 1789181994.968365 write 0x00adea thru 0x014cc1 (0x9ed8 bytes) 000958 1789181994.970393 mapwrite 0x026592 thru 0x035af6 (0xf565 bytes) 000960 1789181995.105078 read 0x01c181 thru 0x01ea39 (0x28b9 bytes) 000961 1789181995.106284 write 0x0101ba thru 0x01a02a (0x9e71 bytes) 000962 1789181995.107880 punch from 0x01e5a9 to 0x028213 (0x9c6b bytes) 000963 1789181995.124818 mapwrite 0x009134 thru 0x011541 (0x840e bytes) 000964 1789181995.171172 write 0x034149 thru 0x03ffff (0xbeb7 bytes) 000968 1789181995.176945 mapwrite 0x0120e9 thru 0x0183f4 (0x630c bytes) 000969 1789181995.220263 trunc from 0x040000 to 0x0343f5 (0xbc0a bytes) 000971 1789181995.243386 punch from 0x0000d3 to 0x00a17b (0xa0a9 bytes) 000975 1789181995.252754 read 0x0117f1 thru 0x01fd75 (0xe585 bytes) 000976 1789181995.278333 read 0x030828 thru 0x0343f5 (0x3bce bytes) 000977 1789181995.279133 write 0x019d25 thru 0x01f1c4 (0x54a0 bytes) 000980 1789181995.283900 mapread 0x00c788 thru 0x017775 (0xafee bytes) 000981 1789181995.289083 punch from 0x026a7f to 0x0326b8 (0xbc3a bytes) 000982 1789181995.303171 mapwrite 0x0177e1 thru 0x01a2dd (0x2afd bytes) 000983 1789181995.354693 mapwrite 0x00afa8 thru 0x00ce77 (0x1ed0 bytes) 000984 1789181995.414931 mapread 0x0176fa thru 0x019c68 (0x256f bytes) 000985 1789181995.416111 read 0x02fa42 thru 0x0343f5 (0x49b4 bytes) 000987 1789181995.430080 trunc from 0x0343f6 to 0x021be2 (0x12813 bytes) 000988 1789181995.454013 trunc from 0x021be3 to 0x016293 (0xb94f bytes) 000990 1789181995.476247 punch from 0x004e2d to 0x008edb (0x40af bytes) 000993 1789181995.484130 punch from 0x004d1c to 0x007663 (0x2948 bytes) 000998 1789181995.500121 MIRROR_EXTEND to 2 mirrors 000999 1789181996.118669 mapwrite 0x03d764 thru 0x03ffff (0x289c bytes) 001000 1789181996.352424 write 0x038aa0 thru 0x03a715 (0x1c76 bytes) All operations completed A-OK! PASS 70a (68s) == sanity-flr test 71: check mirror extend parent fid ==== 23:00:01 (1789182001) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 23:00:04 (1789182004) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242131 s, 8.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177714 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 23:00:12 (1789182012) Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre2 Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2248672 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6722272 .. Extending file size to 6641216 .. Extending file size to 7925600 .. Extending file size to 8433888 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8679680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8531872 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8919808 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8916000 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57507 57508 57510 57512 57513 done 192.168.206.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg635-client.virtnet /mnt/lustre2 (opts:) 192.168.206.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg635-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:01:31 (1789182091) Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre2 Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8324608 .. Extending file size to 2314720 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8971392 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7016672 .. Extending file size to 7676448 .. Extending file size to 8650048 .. Extending file size to 8675200 .. Extending file size to 8858624 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9131936 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59075 59076 59078 59080 59081 done 192.168.206.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg635-client.virtnet /mnt/lustre2 (opts:) 192.168.206.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg635-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:02:48 (1789182168) Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126264 s, 16.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.29552 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x93471bc9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg635-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:03:08 (1789182188) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0816028 s, 25.7 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 23:03:14 (1789182194) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x213:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x338:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:03:21 (1789182201) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150588 s, 13.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x339:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:03:29 (1789182209) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160391 s, 13.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:03:36 (1789182216) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206866 s, 10.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x338:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x338:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x339:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:04:08 (1789182248) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg635-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198697 s, 5.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33e:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:05:07 (1789182307) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg635-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.288851 s, 7.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:06:07 (1789182367) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg635-server mds-ost sync done. Creating new pool test_204f oleg635-server: Pool lustre.test_204f created Adding targets to pool oleg635-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg635-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.419532 s, 5.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x340:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x340:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg635-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg635-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg635-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:07:32 (1789182452) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:07:41 (1789182461) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:07:49 (1789182469) oleg635-server: Pool lustre.test_206 created Adding targets to pool oleg635-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg635-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg635-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg635-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg635-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:08:12 (1789182492) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628545 s, 16.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370609 s, 28.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5c9df558 Mirror 3: 0x5c9df558 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:08:22 (1789182502) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:08:25 (1789182505) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:08:28 (1789182508) Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0944693 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xe40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1c20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1c20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1c20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d40000 Read this (4 bytes): BBBB 192.168.206.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg635-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:09:02 (1789182542) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0120938 s, 677 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:09:07 (1789182547) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726007 s, 14.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834488 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 23:09:08.000000000 -0400 Modify: 2026-09-11 23:09:08.000000000 -0400 Change: 2026-09-11 23:09:10.000000000 -0400 Birth: 2026-09-11 23:09:10.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:09:14 (1789182554) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:09:17 (1789182557) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.475408 s, 86.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0501222 s, 81.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 23:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 23:09 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:09:26 (1789182566) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:09:34 (1789182574) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1599 sec =========== 23:09:43 (1789182583) === sanity-flr: start cleanup 23:09:44 (1789182584) === === sanity-flr: finish cleanup 23:09:51 (1789182591) ===