-----============= acceptance-small: sanity-flr ============----- Sun Sep 6 19:39:02 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:39:08 (1788737948) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a7b43e49800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a7b43e49800.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:39:20 (1788737960) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7648] == sanity-flr test 0a: lfs mirror create with -N option == 19:39:20 (1788737960) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:39:26 (1788737966) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:39:27 (1788737967) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:39:28 (1788737968) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:39:36 (1788737976) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:39:38 (1788737978) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 19:39:39 (1788737979) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a7b43e49800.stats=clear osc.lustre-OST0001-osc-ffff9a7b43e49800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:39:42 (1788737982) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:39:46 (1788737986) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:39:49 (1788737989) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:39:50 (1788737990) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:39:53 (1788737993) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 19:39:56 (1788737996) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:39:59 (1788737999) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:40:00 (1788738000) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:40:03 (1788738003) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:40:07 (1788738007) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:40:10 (1788738010) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.720996 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548925 s, 19.1 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:40:14 (1788738014) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.0687993 s, 107 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:40:17 (1788738017) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:40:21 (1788738021) Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:40:49 (1788738049) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.152516 s, 55.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:41:04 (1788738064) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (32s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:41:36 (1788738096) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49274 s, 35.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 6 19:41 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.290s user 0m0.001s sys 0m0.147s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7b43e49800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.388s user 0m0.002s sys 0m0.170s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.359s user 0m0.001s sys 0m0.148s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7b43e49800.ost_server_uuid in FULL state after 0 sec PASS 33b (36s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:42:12 (1788738132) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:42:13 (1788738133) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:42:14 (1788738134) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 19:42:16 (1788738136) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 19:42:18 (1788738138) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.146706 s, 85.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:42:22 (1788738142) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102551 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102176 s, 103 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:42:24 (1788738144) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:42:27 (1788738147) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00069154 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00066439 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000689007 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000658197 s, 1.6 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 19:42:30 (1788738150) create test files with size 15757442 .. 979c2a2d8c8dd9bc4b48083cb781545a - 4bbbba8e4d2d08177999c4b92df96831 - fd9a67651baac6b0f40c46fb2ced4907 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 19:42:36 (1788738156) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:42:41 (1788738161) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:42:43 (1788738163) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043815 s, 23.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116483 s, 90.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 19:42:46 (1788738166) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0287978 s, 72.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.106115 s, 39.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0506453 s, 41.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 19:42:50 (1788738170) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 19:42:51 (1788738171) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:42:52 (1788738172) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111828 s, 93.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 19:42:55 (1788738175) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:42:56 (1788738176) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:42:59 (1788738179) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.119151 s, 88.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:43:19 (1788738199) llite.lustre-ffff9a7b462b2800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0933976 s, 112 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff9a7b43e48000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:43:24 (1788738204) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:43:37 (1788738217) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:43:50 (1788738230) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 19:43:51 (1788738231) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.073771 s, 42.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0668772 s, 47.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd6983c65 (nosync,stale) Mirror 2: 0xd6983c65 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3b854353f16454062f4547fffa31f95b - mirror 1 checksum: 9825d35af36059e48c06fc2e1d65a148 - mirror 2 checksum: 3b854353f16454062f4547fffa31f95b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T19:43:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T19:43:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3b854353f16454062f4547fffa31f95b - mirror 1 checksum: 9825d35af36059e48c06fc2e1d65a148 - mirror 2 checksum: 9825d35af36059e48c06fc2e1d65a148 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T19:43:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T19:43:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:43:55 (1788738235) Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 19:43:58 (1788738238) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 19:44:12 (1788738252) targets are mounted 19:44:12 (1788738252) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126408 s, 83.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (23s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:44:18 (1788738258) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:44:19 (1788738259) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:44:20 (1788738260) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00276087 s, 5.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:44:23 (1788738263) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:44:24 (1788738264) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00157397 s, 2.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0177551 s, 231 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:44:26 (1788738266) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:44:32 (1788738272) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:44:51 (1788738291) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:45:09 (1788738309) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:45:27 (1788738327) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 19:45:29 (1788738329) mirror create pid 54245 mirror split pid 54247 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54245 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54247 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 19:46:31 (1788738391) Seed set to 8392 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1788738392.228074 mapwrite 0x02ba24 thru 0x02ce85 (0x1462 bytes) 000004 1788738392.249368 read 0x027e91 thru 0x02ce85 (0x4ff5 bytes) truncating to largest ever: 0xf1f6 000005 1788738392.251584 trunc from 0x02ce86 to 0x00f1f5 (0x1dc90 bytes) 000006 1788738392.255511 mapread 0x007eed thru 0x00f1f5 (0x7309 bytes) 000008 1788738392.258782 punch from 0x008d16 to 0x00f1f5 (0x64e0 bytes) 000012 1788738392.261380 mapwrite 0x036013 thru 0x03ff0f (0x9efd bytes) 000014 1788738392.282250 mapwrite 0x0060a4 thru 0x010185 (0xa0e2 bytes) 000016 1788738392.299272 read 0x03bdd7 thru 0x03ff0f (0x4139 bytes) 000018 1788738392.300152 read 0x014b3d thru 0x01ac01 (0x60c5 bytes) 000020 1788738392.303775 trunc from 0x03ff10 to 0x00996c (0x365a3 bytes) 000021 1788738392.307909 write 0x02208f thru 0x0300ee (0xe060 bytes) 000023 1788738392.309161 mapread 0x007e7c thru 0x008a49 (0xbce bytes) 000025 1788738392.311790 MIRROR_EXTEND to 2 mirrors 000026 1788738392.509892 write 0x02ad55 thru 0x0397f2 (0xea9e bytes) 000027 1788738392.528014 mapread 0x00c4df thru 0x00f832 (0x3354 bytes) 000028 1788738392.532963 write 0x034127 thru 0x03a3ea (0x62c4 bytes) 000029 1788738392.534005 mapwrite 0x0393f0 thru 0x03ffff (0x6c10 bytes) 000030 1788738392.557581 mapwrite 0x01fd02 thru 0x02e855 (0xeb54 bytes) 000032 1788738392.575805 write 0x0286be thru 0x0294c2 (0xe05 bytes) 000034 1788738392.577109 trunc from 0x040000 to 0x00d53d (0x32ac2 bytes) truncating to largest ever: 0x2c517 000035 1788738392.582057 trunc from 0x00d53e to 0x02c516 (0x1efd9 bytes) 000036 1788738392.586011 read 0x021253 thru 0x02a2dd (0x908b bytes) 000037 1788738392.589747 MIRROR_RESYNC 2 mirrors 000038 1788738392.760232 read 0x00840b thru 0x00f563 (0x7159 bytes) 000039 1788738392.762667 mapread 0x0294ba thru 0x02c516 (0x305d bytes) 000041 1788738392.767756 trunc from 0x02c517 to 0x00070f (0x2be07 bytes) 000042 1788738392.793511 punch from 0x00035d to 0x00070f (0x3b3 bytes) 000044 1788738392.795470 trunc from 0x000710 to 0x000e00 (0x6f1 bytes) 000045 1788738392.800743 punch from 0x000b03 to 0x000e00 (0x2fe bytes) 000046 1788738392.803581 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe01): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe01): Mirror 2: 0x8b8a5012 (stale) CRC-32 checksum value for chunk [0, 0xe01): Mirror 1: 0x8b8a5012 Mirror 3: 0x8b8a5012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe01: skipped 000047 1788738393.027362 MIRROR_RESYNC 3 mirrors 000049 1788738393.182243 write 0x0062b2 thru 0x012eb6 (0xcc05 bytes) 000050 1788738393.200903 read 0x0021a0 thru 0x00570e (0x356f bytes) 000051 1788738393.203116 mapwrite 0x003b27 thru 0x00962b (0x5b05 bytes) 000052 1788738393.218059 trunc from 0x012eb7 to 0x026568 (0x136b2 bytes) 000054 1788738393.228404 write 0x00de99 thru 0x0119c6 (0x3b2e bytes) 000057 1788738393.229749 read 0x009a81 thru 0x010e3a (0x73ba bytes) 000059 1788738393.230515 mapread 0x023107 thru 0x026568 (0x3462 bytes) 000060 1788738393.233598 read 0x022c78 thru 0x026568 (0x38f1 bytes) 000061 1788738393.233994 mapread 0x01b36e thru 0x022601 (0x7294 bytes) 000062 1788738393.234607 read 0x023689 thru 0x026568 (0x2ee0 bytes) 000064 1788738393.235299 mapread 0x00c97e thru 0x0130b1 (0x6734 bytes) 000065 1788738393.235885 mapwrite 0x013dfe thru 0x022d84 (0xef87 bytes) 000066 1788738393.250560 mapread 0x01bd59 thru 0x026568 (0xa810 bytes) 000067 1788738393.252710 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26569): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26569): Mirror 2: 0x98695ce4 (stale) Mirror 3: 0x98695ce4 (stale) CRC-32 checksum value for chunk [0, 0x26569): Mirror 1: 0x9a667ed3 Mirror 4: 0x9a667ed3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26569: skipped 000068 1788738393.499057 punch from 0x002026 to 0x00dddc (0xbdb7 bytes) 000069 1788738393.517590 mapwrite 0x003627 thru 0x00cb33 (0x950d bytes) 000070 1788738393.535092 mapwrite 0x00cc9f thru 0x01be40 (0xf1a2 bytes) truncating to largest ever: 0x3a3ce 000071 1788738393.553085 trunc from 0x026569 to 0x03a3cd (0x13e65 bytes) 000072 1788738393.562556 punch from 0x00feba to 0x01217e (0x22c5 bytes) 000073 1788738393.564095 write 0x016682 thru 0x020652 (0x9fd1 bytes) 000074 1788738393.564861 punch from 0x027c5f to 0x02aa24 (0x2dc6 bytes) 000075 1788738393.568402 punch from 0x0201f9 to 0x02dcc4 (0xdacc bytes) 000076 1788738393.569895 write 0x00ad75 thru 0x016349 (0xb5d5 bytes) 000077 1788738393.570876 read 0x0131c8 thru 0x01a1fd (0x7036 bytes) 000078 1788738393.571249 read 0x031add thru 0x0360f0 (0x4614 bytes) 000079 1788738393.574011 punch from 0x03333b to 0x03a3cd (0x7093 bytes) 000081 1788738393.577534 read 0x007952 thru 0x015c96 (0xe345 bytes) 000082 1788738393.577890 read 0x000d58 thru 0x0076bc (0x6965 bytes) 000084 1788738393.578660 mapwrite 0x01c2ed thru 0x02bc2c (0xf940 bytes) 000085 1788738393.594070 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3ce): Mirror 2: 0x44c432b5 (stale) Mirror 4: 0xfa8fa2ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3ce): Mirror 2: 0x44c432b5 (stale) Mirror 4: 0xfa8fa2ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3ce: skipped 000086 1788738393.679042 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3ce): Mirror 4: 0xfa8fa2ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3ce): Mirror 4: 0xfa8fa2ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3ce: skipped 000087 1788738393.760575 write 0x01e6f5 thru 0x02d596 (0xeea2 bytes) 000089 1788738393.764811 read 0x00945e thru 0x00ceb0 (0x3a53 bytes) 000092 1788738393.769358 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) 000093 1788738393.808851 mapwrite 0x012178 thru 0x01aaac (0x8935 bytes) 000094 1788738393.827369 write 0x038e1f thru 0x03cd3d (0x3f1f bytes) 000095 1788738393.829538 mapwrite 0x03e8a4 thru 0x03ffff (0x175c bytes) 000097 1788738393.848871 mapwrite 0x03eb63 thru 0x03f751 (0xbef bytes) 000098 1788738393.861144 punch from 0x000589 to 0x00aeb2 (0xa92a bytes) 000099 1788738393.863040 punch from 0x01104f to 0x0157b5 (0x4767 bytes) 000101 1788738393.865201 write 0x02028d thru 0x0289ad (0x8721 bytes) 000102 1788738393.869023 mapwrite 0x027939 thru 0x02e5b4 (0x6c7c bytes) 000103 1788738393.886737 mapread 0x037f48 thru 0x03bf33 (0x3fec bytes) 000104 1788738393.887501 punch from 0x0160ad to 0x0170a7 (0xffb bytes) 000105 1788738393.889386 read 0x01e3fa thru 0x0284ae (0xa0b5 bytes) 000106 1788738393.891345 trunc from 0x040000 to 0x02d300 (0x12cff bytes) 000107 1788738393.896138 read 0x018579 thru 0x020079 (0x7b01 bytes) 000108 1788738393.899355 write 0x00fc06 thru 0x012231 (0x262c bytes) 000110 1788738393.900506 trunc from 0x02d301 to 0x038a43 (0xb743 bytes) 000111 1788738393.905146 mapwrite 0x008e2d thru 0x0180b2 (0xf286 bytes) 000113 1788738393.922820 read 0x02e1ff thru 0x0318e1 (0x36e3 bytes) 000114 1788738393.925803 write 0x007aab thru 0x01113d (0x9693 bytes) 000115 1788738393.926647 write 0x01e11e thru 0x026eef (0x8dd2 bytes) 000118 1788738393.927860 write 0x0051a4 thru 0x0148ed (0xf74a bytes) 000120 1788738393.930951 MIRROR_EXTEND to 2 mirrors 000121 1788738394.166443 punch from 0x00b2d7 to 0x01354d (0x8277 bytes) 000122 1788738394.185912 write 0x0067a4 thru 0x014837 (0xe094 bytes) 000123 1788738394.189724 punch from 0x017036 to 0x023d9a (0xcd65 bytes) 000124 1788738394.193757 mapwrite 0x00db4e thru 0x00ee55 (0x1308 bytes) 000125 1788738394.207826 trunc from 0x038a44 to 0x007d3f (0x30d04 bytes) 000127 1788738394.214352 trunc from 0x007d40 to 0x02f447 (0x27708 bytes) 000128 1788738394.218850 punch from 0x008c07 to 0x00fbe4 (0x6fde bytes) 000133 1788738394.222262 write 0x00ac32 thru 0x00b86d (0xc3c bytes) 000134 1788738394.225751 trunc from 0x02f448 to 0x01dc89 (0x117be bytes) 000135 1788738394.231958 MIRROR_RESYNC 2 mirrors 000137 1788738394.452866 read 0x018b2d thru 0x01dc89 (0x515d bytes) 000138 1788738394.455247 write 0x025cd0 thru 0x02c289 (0x65ba bytes) 000139 1788738394.470092 mapread 0x010ba4 thru 0x01f8f9 (0xed56 bytes) 000140 1788738394.478247 punch from 0x002b33 to 0x00a9d8 (0x7ea6 bytes) 000142 1788738394.482969 mapread 0x00dae7 thru 0x00fa6a (0x1f84 bytes) 000143 1788738394.483667 mapwrite 0x01beba thru 0x02a6dd (0xe824 bytes) 000145 1788738394.502315 mapread 0x01253e thru 0x015fad (0x3a70 bytes) 000147 1788738394.504919 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c28a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c28a): Mirror 2: 0x66730005 (stale) CRC-32 checksum value for chunk [0, 0x2c28a): Mirror 1: 0xdb01883 Mirror 3: 0xdb01883 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c28a: skipped 000148 1788738394.766332 punch from 0x0112ab to 0x01e94d (0xd6a3 bytes) 000149 1788738394.785518 read 0x023eb1 thru 0x028c50 (0x4da0 bytes) 000150 1788738394.788075 punch from 0x01c029 to 0x01ce46 (0xe1e bytes) 000154 1788738394.794174 read 0x01927b thru 0x01fb6d (0x68f3 bytes) 000156 1788738394.798098 read 0x00e2a5 thru 0x01daa4 (0xf800 bytes) 000157 1788738394.798695 read 0x009934 thru 0x015062 (0xb72f bytes) 000158 1788738394.799252 trunc from 0x02c28a to 0x0233c3 (0x8ec6 bytes) 000159 1788738394.809710 read 0x0067bc thru 0x00d16d (0x69b2 bytes) 000160 1788738394.810220 read 0x0022df thru 0x00d2d2 (0xaff4 bytes) 000161 1788738394.810678 punch from 0x002df9 to 0x00d301 (0xa509 bytes) 000162 1788738394.813904 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x233c4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233c4): Mirror 2: 0x44ea234c (stale) Mirror 3: 0x9f52e247 (stale) CRC-32 checksum value for chunk [0, 0x233c4): Mirror 1: 0x2e61052 Mirror 4: 0x2e61052 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233c4: skipped 000163 1788738395.074376 MIRROR_RESYNC 4 mirrors 000166 1788738395.341047 trunc from 0x0233c4 to 0x00eec6 (0x144fd bytes) 000168 1788738395.367924 trunc from 0x00eec7 to 0x01de30 (0xef6a bytes) 000169 1788738395.373857 punch from 0x00bb97 to 0x015720 (0x9b8a bytes) 000170 1788738395.375529 write 0x011d7d thru 0x0211f2 (0xf476 bytes) 000171 1788738395.378044 trunc from 0x0211f3 to 0x0109ae (0x10844 bytes) 000173 1788738395.383057 punch from 0x00ffa2 to 0x0109ae (0xa0d bytes) 000176 1788738395.385576 punch from 0x00f1e9 to 0x0109ae (0x17c6 bytes) 000177 1788738395.388720 MIRROR_RESYNC 4 mirrors 000178 1788738395.669311 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) 000179 1788738395.775694 mapread 0x0075c1 thru 0x00a21e (0x2c5e bytes) 000181 1788738395.781746 write 0x02c99b thru 0x03bf25 (0xf58b bytes) 000183 1788738395.801438 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bf26): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf26): Mirror 2: 0xcedbdfe8 (stale) Mirror 4: 0xcedbdfe8 (stale) CRC-32 checksum value for chunk [0, 0x3bf26): Mirror 1: 0x33adef62 Mirror 5: 0x33adef62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf26: skipped 000184 1788738396.134227 mapwrite 0x0156b3 thru 0x021677 (0xbfc5 bytes) 000185 1788738396.196980 trunc from 0x03bf26 to 0x01f8c0 (0x1c665 bytes) 000186 1788738396.204677 punch from 0x012c24 to 0x01347c (0x859 bytes) 000188 1788738396.207366 mapwrite 0x02f45e thru 0x03897f (0x9522 bytes) 000189 1788738396.236643 mapread 0x020fc2 thru 0x02293b (0x197a bytes) 000193 1788738396.243135 trunc from 0x038980 to 0x010bda (0x27da5 bytes) 000194 1788738396.250110 trunc from 0x010bdb to 0x027d59 (0x1717f bytes) 000195 1788738396.255133 mapread 0x022119 thru 0x025a60 (0x3948 bytes) 000197 1788738396.259117 mapwrite 0x00e747 thru 0x018431 (0x9ceb bytes) 000198 1788738396.280234 punch from 0x0239dc to 0x026785 (0x2daa bytes) 000200 1788738396.283083 punch from 0x00df4f to 0x01ba5a (0xdb0c bytes) 000202 1788738396.285813 mapwrite 0x0388a6 thru 0x03ffff (0x775a bytes) 000203 1788738396.309783 write 0x02175a thru 0x02c44d (0xacf4 bytes) 000206 1788738396.313630 mapread 0x01981f thru 0x025402 (0xbbe4 bytes) 000208 1788738396.317238 trunc from 0x040000 to 0x025362 (0x1ac9d bytes) truncating to largest ever: 0x3d097 000209 1788738396.323599 trunc from 0x025363 to 0x03d096 (0x17d34 bytes) 000211 1788738396.330109 mapread 0x01f6a9 thru 0x023fe6 (0x493e bytes) 000212 1788738396.330997 read 0x0044ed thru 0x012969 (0xe47d bytes) 000214 1788738396.335060 punch from 0x02272e to 0x02895f (0x6232 bytes) 000218 1788738396.341435 trunc from 0x03d097 to 0x026eb1 (0x161e5 bytes) 000219 1788738396.346916 read 0x01e23d thru 0x01eeb4 (0xc78 bytes) 000220 1788738396.347428 write 0x00b3ee thru 0x018a8a (0xd69d bytes) 000222 1788738396.348678 mapread 0x00afc7 thru 0x00f308 (0x4342 bytes) 000223 1788738396.349505 punch from 0x021f3a to 0x026eb1 (0x4f78 bytes) 000224 1788738396.355047 mapread 0x00009f thru 0x009a61 (0x99c3 bytes) 000225 1788738396.359124 write 0x01f552 thru 0x02b85c (0xc30b bytes) 000226 1788738396.362656 MIRROR_RESYNC 4 mirrors 000227 1788738396.727876 write 0x01df31 thru 0x02a279 (0xc349 bytes) 000228 1788738396.747634 mapwrite 0x0120fa thru 0x018a63 (0x696a bytes) 000229 1788738396.769640 mapwrite 0x0131ef thru 0x018532 (0x5344 bytes) 000230 1788738396.784980 mapread 0x027684 thru 0x02a1b7 (0x2b34 bytes) 000231 1788738396.785822 mapread 0x00087b thru 0x009f0d (0x9693 bytes) 000233 1788738396.790444 write 0x01f69c thru 0x0227a6 (0x310b bytes) 000235 1788738396.791833 mapwrite 0x01ae9f thru 0x01b72a (0x88c bytes) 000236 1788738396.806851 mapwrite 0x021ea6 thru 0x030d8b (0xeee6 bytes) 000237 1788738396.842224 mapwrite 0x0004a0 thru 0x00f3b6 (0xef17 bytes) 000239 1788738396.857495 read 0x023cca thru 0x026dbe (0x30f5 bytes) 000240 1788738396.859409 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30d8c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d8c): Mirror 2: 0xf351f40f (stale) Mirror 4: 0xf351f40f (stale) Mirror 5: 0xf351f40f (stale) CRC-32 checksum value for chunk [0, 0x30d8c): Mirror 1: 0x43ae79e2 Mirror 6: 0x43ae79e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d8c: skipped 000242 1788738397.089176 write 0x01bbda thru 0x0216f6 (0x5b1d bytes) 000245 1788738397.111063 mapwrite 0x01951f thru 0x01e2e8 (0x4dca bytes) 000246 1788738397.126701 mapread 0x020da5 thru 0x0257dc (0x4a38 bytes) 000247 1788738397.131112 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30d8c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d8c): Mirror 2: 0xf351f40f (stale) Mirror 4: 0xf351f40f (stale) Mirror 5: 0xf351f40f (stale) Mirror 6: 0x43ae79e2 (stale) CRC-32 checksum value for chunk [0, 0x30d8c): Mirror 1: 0x59ffba11 Mirror 7: 0x59ffba11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d8c: skipped 000248 1788738397.342598 read 0x01c458 thru 0x01c4af (0x58 bytes) 000250 1788738397.347269 MIRROR_RESYNC 6 mirrors 000251 1788738397.588901 read 0x00b7a6 thru 0x0124c5 (0x6d20 bytes) 000253 1788738397.591340 punch from 0x00a74c to 0x0183e0 (0xdc95 bytes) 000254 1788738397.609140 trunc from 0x030d8c to 0x03c79c (0xba11 bytes) 000255 1788738397.618674 punch from 0x030444 to 0x036210 (0x5dcd bytes) 000256 1788738397.620390 write 0x00c8ba thru 0x017489 (0xabd0 bytes) 000258 1788738397.624143 trunc from 0x03c79d to 0x01f904 (0x1ce98 bytes) 000259 1788738397.629841 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f905): Mirror 2: 0x4a9d11d4 (stale) Mirror 4: 0x4a9d11d4 (stale) Mirror 5: 0x4a9d11d4 (stale) Mirror 7: 0x4a9d11d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f905): Mirror 2: 0x4a9d11d4 (stale) Mirror 4: 0x4a9d11d4 (stale) Mirror 5: 0x4a9d11d4 (stale) Mirror 7: 0x4a9d11d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f905: skipped 000260 1788738397.729927 read 0x01d3af thru 0x01f904 (0x2556 bytes) 000261 1788738397.731751 mapread 0x00ca95 thru 0x00f341 (0x28ad bytes) 000262 1788738397.737699 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f905): Mirror 4: 0x4a9d11d4 (stale) Mirror 5: 0x4a9d11d4 (stale) Mirror 7: 0x4a9d11d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f905): Mirror 4: 0x4a9d11d4 (stale) Mirror 5: 0x4a9d11d4 (stale) Mirror 7: 0x4a9d11d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f905: skipped 000263 1788738397.822668 trunc from 0x01f905 to 0x01577e (0xa186 bytes) 000264 1788738397.826808 write 0x00d6fd thru 0x012af8 (0x53fc bytes) 000265 1788738397.829902 read 0x011f09 thru 0x01577e (0x3876 bytes) 000266 1788738397.831677 write 0x030088 thru 0x0333eb (0x3364 bytes) 000267 1788738397.832553 read 0x021105 thru 0x02d91f (0xc81b bytes) 000268 1788738397.836588 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x333ec): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333ec): Mirror 4: 0x327d1f9d (stale) Mirror 5: 0x327d1f9d (stale) Mirror 7: 0x327d1f9d (stale) CRC-32 checksum value for chunk [0, 0x333ec): Mirror 1: 0xe325858c Mirror 8: 0xe325858c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333ec: skipped truncating to largest ever: 0x3d2f3 000269 1788738398.116032 trunc from 0x0333ec to 0x03d2f2 (0x9f07 bytes) 000270 1788738398.142727 read 0x027a90 thru 0x02fd1d (0x828e bytes) 000272 1788738398.145891 trunc from 0x03d2f3 to 0x02c5d7 (0x10d1b bytes) 000273 1788738398.151341 mapread 0x0035c3 thru 0x00b455 (0x7e93 bytes) 000274 1788738398.157053 mapwrite 0x00db62 thru 0x016892 (0x8d31 bytes) 000275 1788738398.176682 write 0x02d634 thru 0x032ad0 (0x549d bytes) 000276 1788738398.177781 mapwrite 0x0092fd thru 0x00c7be (0x34c2 bytes) 000277 1788738398.191288 mapwrite 0x03bd68 thru 0x03ffff (0x4298 bytes) 000278 1788738398.216522 write 0x02b8e6 thru 0x035769 (0x9e84 bytes) 000280 1788738398.219876 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe74f06fa (stale) Mirror 5: 0xe74f06fa (stale) Mirror 7: 0xe74f06fa (stale) Mirror 8: 0xe439c7d4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa96d2c9 Mirror 9: 0xa96d2c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000282 1788738398.522101 punch from 0x000966 to 0x002106 (0x17a1 bytes) 000286 1788738398.546557 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe74f06fa (stale) Mirror 7: 0xe74f06fa (stale) Mirror 8: 0xe439c7d4 (stale) Mirror 9: 0xa96d2c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe74f06fa (stale) Mirror 7: 0xe74f06fa (stale) Mirror 8: 0xe439c7d4 (stale) Mirror 9: 0xa96d2c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000287 1788738398.647059 mapread 0x004c55 thru 0x00c398 (0x7744 bytes) 000288 1788738398.653243 trunc from 0x040000 to 0x0253d9 (0x1ac26 bytes) 000289 1788738398.662499 write 0x010103 thru 0x010614 (0x512 bytes) 000290 1788738398.663214 punch from 0x016b49 to 0x0187f1 (0x1ca9 bytes) 000293 1788738398.666546 write 0x033070 thru 0x035843 (0x27d4 bytes) truncating to largest ever: 0x3f96c 000295 1788738398.667389 trunc from 0x035844 to 0x03f96b (0xa128 bytes) 000297 1788738398.672093 write 0x001390 thru 0x00846f (0x70e0 bytes) 000298 1788738398.673114 trunc from 0x03f96c to 0x00c073 (0x338f8 bytes) 000299 1788738398.678175 punch from 0x004906 to 0x00c073 (0x776e bytes) 000300 1788738398.681767 punch from 0x004c67 to 0x0083e5 (0x377f bytes) 000305 1788738398.684946 trunc from 0x00c074 to 0x0114c4 (0x5451 bytes) 000306 1788738398.689622 read 0x003979 thru 0x010f08 (0xd590 bytes) 000307 1788738398.692166 mapwrite 0x02e27a thru 0x0384af (0xa236 bytes) 000309 1788738398.715512 punch from 0x00fd47 to 0x016329 (0x65e3 bytes) 000310 1788738398.717525 punch from 0x00f2eb to 0x01b974 (0xc68a bytes) 000311 1788738398.720230 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x384b0): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x384b0): Mirror 5: 0xabd7683f (stale) Mirror 7: 0xabd7683f (stale) Mirror 8: 0xe67f643c (stale) Mirror 9: 0x39185294 (stale) CRC-32 checksum value for chunk [0, 0x384b0): Mirror 1: 0x47c0a3bd Mirror 10: 0x47c0a3bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x384b0: skipped 000312 1788738398.999057 punch from 0x02c0ea to 0x02db95 (0x1aac bytes) 000313 1788738399.017174 trunc from 0x0384b0 to 0x03eb3f (0x6690 bytes) 000315 1788738399.026668 trunc from 0x03eb40 to 0x012552 (0x2c5ed bytes) 000317 1788738399.031605 mapread 0x000b1a thru 0x002708 (0x1bef bytes) 000319 1788738399.036463 punch from 0x005267 to 0x005a33 (0x7cd bytes) 000320 1788738399.038079 mapread 0x008d86 thru 0x00e9f4 (0x5c6f bytes) 000321 1788738399.041063 read 0x010060 thru 0x012552 (0x24f3 bytes) 000322 1788738399.042915 write 0x01091a thru 0x0207a2 (0xfe89 bytes) 000323 1788738399.043998 write 0x018b79 thru 0x024828 (0xbcb0 bytes) 000324 1788738399.045042 read 0x0114fd thru 0x01b3fe (0x9f02 bytes) 000325 1788738399.045405 write 0x006089 thru 0x009cad (0x3c25 bytes) 000327 1788738399.046378 read 0x018710 thru 0x024828 (0xc119 bytes) 000328 1788738399.048161 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24829): Mirror 7: 0x4ee62787 (stale) Mirror 8: 0x7074df43 (stale) Mirror 9: 0xac6a573f (stale) Mirror 10: 0x5e55e6cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24829): Mirror 7: 0x4ee62787 (stale) Mirror 8: 0x7074df43 (stale) Mirror 9: 0xac6a573f (stale) Mirror 10: 0x5e55e6cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24829: skipped 000329 1788738399.144075 mapread 0x00dd3f thru 0x01ad34 (0xcff6 bytes) 000330 1788738399.152892 mapread 0x00fd21 thru 0x01d249 (0xd529 bytes) 000331 1788738399.153774 write 0x0166a0 thru 0x01dd1c (0x767d bytes) 000332 1788738399.155649 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24829): Mirror 8: 0x7074df43 (stale) Mirror 9: 0xac6a573f (stale) Mirror 10: 0x5e55e6cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24829): Mirror 8: 0x7074df43 (stale) Mirror 9: 0xac6a573f (stale) Mirror 10: 0x5e55e6cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24829: skipped 000333 1788738399.254898 MIRROR_RESYNC 4 mirrors 000334 1788738399.529935 trunc from 0x024829 to 0x0190fe (0xb72a bytes) 000335 1788738399.554273 mapread 0x001529 thru 0x00c804 (0xb2dc bytes) 000337 1788738399.560907 punch from 0x00873e to 0x00a404 (0x1cc7 bytes) 000338 1788738399.562666 mapread 0x011c60 thru 0x013a2a (0x1dcb bytes) 000339 1788738399.566856 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x190ff): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x190ff): Mirror 8: 0x9e5d364d (stale) Mirror 9: 0x9e5d364d (stale) Mirror 10: 0x9e5d364d (stale) CRC-32 checksum value for chunk [0, 0x190ff): Mirror 1: 0xb3594b73 Mirror 11: 0xb3594b73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x190ff: skipped 000340 1788738399.816236 punch from 0x00c042 to 0x0190fe (0xd0bd bytes) 000341 1788738399.833485 trunc from 0x0190ff to 0x0095b5 (0xfb49 bytes) 000342 1788738399.842443 mapwrite 0x00a66f thru 0x00d9e0 (0x3372 bytes) 000343 1788738399.862649 mapread 0x000a08 thru 0x00d9e0 (0xcfd9 bytes) 000344 1788738399.865292 trunc from 0x00d9e1 to 0x037d2c (0x2a34c bytes) 000346 1788738399.869769 trunc from 0x037d2d to 0x00e52c (0x29800 bytes) 000348 1788738399.874042 mapwrite 0x01ad61 thru 0x01e3d9 (0x3679 bytes) 000349 1788738399.894136 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1e3da): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e3da): Mirror 8: 0xd7736576 (stale) Mirror 9: 0xd7736576 (stale) Mirror 10: 0xd7736576 (stale) Mirror 11: 0xd159d19a (stale) CRC-32 checksum value for chunk [0, 0x1e3da): Mirror 1: 0x4d3a01c0 Mirror 12: 0x4d3a01c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e3da: skipped 000351 1788738400.147399 mapwrite 0x03c7f6 thru 0x03ffff (0x380a bytes) 000353 1788738400.184249 trunc from 0x040000 to 0x00691e (0x396e1 bytes) 000355 1788738400.190135 mapwrite 0x00b1ee thru 0x013b25 (0x8938 bytes) 000356 1788738400.209749 punch from 0x005166 to 0x00bdf2 (0x6c8d bytes) 000357 1788738400.211945 punch from 0x00da28 to 0x013b25 (0x60fe bytes) 000358 1788738400.213854 mapread 0x0002d5 thru 0x001429 (0x1155 bytes) 000359 1788738400.216878 mapwrite 0x03bb8b thru 0x03ffff (0x4475 bytes) 000360 1788738400.237163 write 0x029479 thru 0x035438 (0xbfc0 bytes) 000363 1788738400.240196 mapread 0x01d94e thru 0x02414d (0x6800 bytes) 000367 1788738400.244223 punch from 0x00dddf to 0x017eac (0xa0ce bytes) 000372 1788738400.250959 MIRROR_RESYNC 6 mirrors 000373 1788738400.779185 write 0x0317a7 thru 0x0383f1 (0x6c4b bytes) 000376 1788738400.798042 trunc from 0x040000 to 0x021715 (0x1e8ea bytes) 000377 1788738400.807645 punch from 0x015e4d to 0x021715 (0xb8c9 bytes) 000378 1788738400.809134 read 0x016a9a thru 0x020664 (0x9bcb bytes) 000379 1788738400.813014 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x21716): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21716): Mirror 8: 0xed6a740d (stale) Mirror 9: 0xed6a740d (stale) Mirror 10: 0xed6a740d (stale) Mirror 11: 0xed6a740d (stale) Mirror 12: 0xed6a740d (stale) CRC-32 checksum value for chunk [0, 0x21716): Mirror 1: 0xed6a740d Mirror 13: 0xed6a740d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21716: skipped 000380 1788738401.099560 MIRROR_RESYNC 7 mirrors 000381 1788738401.509342 trunc from 0x021716 to 0x012616 (0xf0ff bytes) 000382 1788738401.535023 MIRROR_RESYNC 7 mirrors 000383 1788738402.008991 read 0x00c3f2 thru 0x012546 (0x6155 bytes) 000384 1788738402.011699 mapread 0x00655d thru 0x0095e6 (0x308a bytes) 000385 1788738402.016651 mapread 0x00080d thru 0x0054f9 (0x4ced bytes) 000386 1788738402.019783 MIRROR_RESYNC 7 mirrors 000387 1788738402.158990 trunc from 0x012617 to 0x038caa (0x26694 bytes) 000388 1788738402.180151 MIRROR_RESYNC 7 mirrors 000389 1788738402.655750 read 0x013505 thru 0x0157cf (0x22cb bytes) 000390 1788738402.657615 mapwrite 0x02fd08 thru 0x02fde9 (0xe2 bytes) 000392 1788738402.703208 punch from 0x00d3a6 to 0x0116f5 (0x4350 bytes) 000393 1788738402.705222 mapread 0x0250b8 thru 0x0280a6 (0x2fef bytes) 000394 1788738402.709516 mapwrite 0x02da4d thru 0x035323 (0x78d7 bytes) 000395 1788738402.728650 trunc from 0x038cab to 0x019b7e (0x1f12c bytes) 000396 1788738402.735435 punch from 0x016be9 to 0x019b7e (0x2f96 bytes) 000397 1788738402.737467 mapread 0x00dde9 thru 0x011310 (0x3528 bytes) 000398 1788738402.741628 mapread 0x008c13 thru 0x00c12e (0x351c bytes) 000399 1788738402.742359 write 0x02c23e thru 0x03860c (0xc3cf bytes) 000400 1788738402.743305 mapwrite 0x01a557 thru 0x01a575 (0x1f bytes) 000401 1788738402.761800 mapwrite 0x03799e thru 0x03ffff (0x8662 bytes) 000404 1788738402.788465 punch from 0x0223d6 to 0x02bbaa (0x97d5 bytes) 000405 1788738402.790630 mapread 0x00760b thru 0x00dcc4 (0x66ba bytes) 000407 1788738402.791978 mapwrite 0x014a9a thru 0x01d869 (0x8dd0 bytes) 000409 1788738402.807931 mapwrite 0x004953 thru 0x005bb0 (0x125e bytes) 000410 1788738402.824911 write 0x0265e4 thru 0x03625e (0xfc7b bytes) 000411 1788738402.828045 trunc from 0x040000 to 0x00895b (0x376a4 bytes) 000412 1788738402.833244 read 0x004e65 thru 0x00895b (0x3af7 bytes) 000413 1788738402.833828 mapread 0x005b84 thru 0x00895b (0x2dd8 bytes) 000414 1788738402.834629 trunc from 0x00895c to 0x01bac8 (0x1316d bytes) 000418 1788738402.841335 mapread 0x0016dc thru 0x00f63f (0xdf64 bytes) 000419 1788738402.845386 mapread 0x011dd6 thru 0x01bac8 (0x9cf3 bytes) 000420 1788738402.847971 MIRROR_RESYNC 7 mirrors 000421 1788738403.227622 mapread 0x0191a7 thru 0x01bac8 (0x2922 bytes) 000422 1788738403.232934 write 0x0316b8 thru 0x034140 (0x2a89 bytes) 000424 1788738403.249450 mapread 0x01a8cf thru 0x0247ac (0x9ede bytes) 000429 1788738403.256727 read 0x019aed thru 0x01ece4 (0x51f8 bytes) 000430 1788738403.257154 read 0x02a426 thru 0x02c9df (0x25ba bytes) 000432 1788738403.262354 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34141): Mirror 8: 0xcd714ff4 (stale) Mirror 9: 0xcd714ff4 (stale) Mirror 10: 0xcd714ff4 (stale) Mirror 12: 0xcd714ff4 (stale) Mirror 13: 0xcd714ff4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34141): Mirror 8: 0xcd714ff4 (stale) Mirror 9: 0xcd714ff4 (stale) Mirror 10: 0xcd714ff4 (stale) Mirror 12: 0xcd714ff4 (stale) Mirror 13: 0xcd714ff4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34141: skipped 000434 1788738403.381030 read 0x018ad9 thru 0x01a35c (0x1884 bytes) 000437 1788738403.384063 mapwrite 0x034fd9 thru 0x03ac30 (0x5c58 bytes) 000438 1788738403.410438 write 0x01451c thru 0x016b15 (0x25fa bytes) 000439 1788738403.415200 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac31): Mirror 8: 0x789e674c (stale) Mirror 9: 0x789e674c (stale) Mirror 10: 0x789e674c (stale) Mirror 13: 0x789e674c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac31): Mirror 8: 0x789e674c (stale) Mirror 9: 0x789e674c (stale) Mirror 10: 0x789e674c (stale) Mirror 13: 0x789e674c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac31: skipped 000441 1788738403.519960 punch from 0x037ddd to 0x03ac30 (0x2e54 bytes) 000443 1788738403.521853 read 0x02c38b thru 0x02f762 (0x33d8 bytes) 000444 1788738403.524001 mapread 0x018911 thru 0x01dbf2 (0x52e2 bytes) 000446 1788738403.529555 write 0x00a84d thru 0x0199bc (0xf170 bytes) 000447 1788738403.531850 trunc from 0x03ac31 to 0x021a9c (0x19194 bytes) 000448 1788738403.536216 write 0x01c341 thru 0x02945e (0xd11e bytes) 000449 1788738403.537396 mapwrite 0x039d2b thru 0x03ffff (0x62d5 bytes) 000452 1788738403.561229 trunc from 0x040000 to 0x03334d (0xccb2 bytes) 000453 1788738403.566021 mapread 0x0113fb thru 0x01d428 (0xc02e bytes) 000454 1788738403.566910 trunc from 0x03334e to 0x00ea53 (0x248fa bytes) 000456 1788738403.573355 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xea54): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea54): Mirror 8: 0x50e20dd2 (stale) Mirror 9: 0x50e20dd2 (stale) Mirror 10: 0x50e20dd2 (stale) Mirror 13: 0x50e20dd2 (stale) CRC-32 checksum value for chunk [0, 0xea54): Mirror 1: 0xa70ea736 Mirror 14: 0xa70ea736 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea54: skipped 000457 1788738403.831884 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xea54): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea54): Mirror 8: 0x50e20dd2 (stale) Mirror 9: 0x50e20dd2 (stale) Mirror 10: 0x50e20dd2 (stale) Mirror 13: 0x50e20dd2 (stale) CRC-32 checksum value for chunk [0, 0xea54): Mirror 1: 0xa70ea736 Mirror 14: 0xa70ea736 Mirror 15: 0xa70ea736 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea54: skipped 000458 1788738404.058281 read 0x00147d thru 0x0058b7 (0x443b bytes) 000459 1788738404.060703 write 0x001271 thru 0x00a1a5 (0x8f35 bytes) 000461 1788738404.083237 read 0x009586 thru 0x00ea53 (0x54ce bytes) 000462 1788738404.085598 write 0x03dd42 thru 0x03ffff (0x22be bytes) 000463 1788738404.087681 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb89db879 (stale) Mirror 9: 0xb89db879 (stale) Mirror 10: 0xb89db879 (stale) Mirror 13: 0xb89db879 (stale) Mirror 14: 0xe19f1cff (stale) Mirror 15: 0xe19f1cff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd2a2efa6 Mirror 16: 0xd2a2efa6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000464 1788738404.372969 mapwrite 0x038eba thru 0x03c67f (0x37c6 bytes) 000465 1788738404.430774 read 0x00e4e5 thru 0x019601 (0xb11d bytes) 000466 1788738404.433564 read 0x018c8f thru 0x02530d (0xc67f bytes) 000467 1788738404.436656 read 0x01b98c thru 0x020149 (0x47be bytes) 000468 1788738404.437199 mapread 0x00abe5 thru 0x00b720 (0xb3c bytes) 000470 1788738404.438380 punch from 0x00bbd7 to 0x0177d9 (0xbc03 bytes) 000471 1788738404.440400 punch from 0x01c928 to 0x024a2d (0x8106 bytes) 000472 1788738404.441907 mapread 0x001bdc thru 0x00dfa1 (0xc3c6 bytes) 000473 1788738404.445110 punch from 0x01a9e9 to 0x01e659 (0x3c71 bytes) 000476 1788738404.447482 read 0x00aa02 thru 0x014360 (0x995f bytes) 000477 1788738404.447870 write 0x0254df thru 0x02ab9c (0x56be bytes) 000479 1788738404.449049 write 0x00c97c thru 0x01b228 (0xe8ad bytes) 000481 1788738404.451808 mapread 0x023b20 thru 0x0253d6 (0x18b7 bytes) 000482 1788738404.452492 read 0x005567 thru 0x009fcb (0x4a65 bytes) 000483 1788738404.452885 punch from 0x00ef05 to 0x01e492 (0xf58e bytes) 000484 1788738404.458193 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb89db879 (stale) Mirror 9: 0xb89db879 (stale) Mirror 10: 0xb89db879 (stale) Mirror 13: 0xb89db879 (stale) Mirror 14: 0xe19f1cff (stale) Mirror 15: 0xe19f1cff (stale) Mirror 16: 0xd2a2efa6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x61659958 Mirror 17: 0x61659958 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000485 1788738404.752634 mapwrite 0x0163b9 thru 0x01857c (0x21c4 bytes) 000486 1788738404.810964 write 0x026e17 thru 0x03424f (0xd439 bytes) 000487 1788738404.814437 punch from 0x039880 to 0x03ffff (0x6780 bytes) 000488 1788738404.818136 punch from 0x03f42e to 0x03ffff (0xbd2 bytes) 000489 1788738404.819540 mapwrite 0x01a18f thru 0x022bf7 (0x8a69 bytes) 000490 1788738404.834470 mapwrite 0x00b6a5 thru 0x01adaf (0xf70b bytes) 000491 1788738404.852793 read 0x02ec7d thru 0x03addc (0xc160 bytes) 000492 1788738404.855604 trunc from 0x040000 to 0x02b937 (0x146c8 bytes) 000495 1788738404.866133 trunc from 0x02b938 to 0x02e146 (0x280f bytes) 000496 1788738404.870297 punch from 0x0280fd to 0x028a00 (0x904 bytes) 000498 1788738404.872236 write 0x00aca2 thru 0x00f138 (0x4497 bytes) 000499 1788738404.872994 punch from 0x00e13a to 0x0150f9 (0x6fc0 bytes) 000500 1788738404.876917 read 0x02b5b8 thru 0x02b77f (0x1c8 bytes) 000501 1788738404.877264 punch from 0x00ca70 to 0x01788e (0xae1f bytes) 000503 1788738404.879520 trunc from 0x02e147 to 0x032b0c (0x49c6 bytes) 000504 1788738404.883866 mapwrite 0x03034d thru 0x03682e (0x64e2 bytes) 000507 1788738404.904270 write 0x0078a3 thru 0x015744 (0xdea2 bytes) 000508 1788738404.905217 mapread 0x0303c3 thru 0x03274e (0x238c bytes) 000509 1788738404.905845 write 0x0348ae thru 0x03f8a9 (0xaffc bytes) 000511 1788738404.906934 read 0x004b05 thru 0x00af2c (0x6428 bytes) 000512 1788738404.907261 trunc from 0x03f8aa to 0x03e43a (0x146f bytes) 000513 1788738404.911782 trunc from 0x03e43b to 0x0054a9 (0x38f91 bytes) 000514 1788738404.916130 write 0x03bc97 thru 0x03ffff (0x4369 bytes) 000515 1788738404.916854 punch from 0x02189e to 0x02ea5a (0xd1bd bytes) 000516 1788738404.921470 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb89db879 (stale) Mirror 10: 0xb89db879 (stale) Mirror 13: 0xb89db879 (stale) Mirror 14: 0xe19f1cff (stale) Mirror 15: 0xe19f1cff (stale) Mirror 16: 0xd2a2efa6 (stale) Mirror 17: 0x61659958 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb89db879 (stale) Mirror 10: 0xb89db879 (stale) Mirror 13: 0xb89db879 (stale) Mirror 14: 0xe19f1cff (stale) Mirror 15: 0xe19f1cff (stale) Mirror 16: 0xd2a2efa6 (stale) Mirror 17: 0x61659958 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000517 1788738405.050903 mapwrite 0x00fbb6 thru 0x018448 (0x8893 bytes) 000519 1788738405.071301 read 0x01925a thru 0x01fd8f (0x6b36 bytes) 000520 1788738405.075896 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb89db879 (stale) Mirror 10: 0xb89db879 (stale) Mirror 13: 0xb89db879 (stale) Mirror 14: 0xe19f1cff (stale) Mirror 16: 0xd2a2efa6 (stale) Mirror 17: 0x61659958 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb89db879 (stale) Mirror 10: 0xb89db879 (stale) Mirror 13: 0xb89db879 (stale) Mirror 14: 0xe19f1cff (stale) Mirror 16: 0xd2a2efa6 (stale) Mirror 17: 0x61659958 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000523 1788738405.185911 read 0x021bd3 thru 0x025f61 (0x438f bytes) 000524 1788738405.188535 punch from 0x029a4d to 0x034317 (0xa8cb bytes) 000526 1788738405.190356 read 0x02d770 thru 0x03d032 (0xf8c3 bytes) 000531 1788738405.196848 MIRROR_RESYNC 7 mirrors 000532 1788738405.821353 trunc from 0x040000 to 0x0306cb (0xf934 bytes) 000534 1788738405.850028 read 0x02e3d4 thru 0x0306cb (0x22f8 bytes) 000536 1788738405.856446 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x306cc): Mirror 9: 0xc5dbe5e7 (stale) Mirror 10: 0xc5dbe5e7 (stale) Mirror 13: 0xc5dbe5e7 (stale) Mirror 16: 0xc5dbe5e7 (stale) Mirror 17: 0xc5dbe5e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x306cc): Mirror 9: 0xc5dbe5e7 (stale) Mirror 10: 0xc5dbe5e7 (stale) Mirror 13: 0xc5dbe5e7 (stale) Mirror 16: 0xc5dbe5e7 (stale) Mirror 17: 0xc5dbe5e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x306cc: skipped 000538 1788738405.972858 read 0x00927e thru 0x00eb74 (0x58f7 bytes) 000539 1788738405.976505 MIRROR_RESYNC 6 mirrors 000540 1788738406.418206 write 0x0210f4 thru 0x027ef7 (0x6e04 bytes) 000541 1788738406.436263 MIRROR_RESYNC 6 mirrors 000542 1788738406.899095 trunc from 0x0306cc to 0x016057 (0x1a674 bytes) 000544 1788738406.918565 mapread 0x000409 thru 0x00c93a (0xc532 bytes) 000545 1788738406.925317 write 0x00bd0d thru 0x00cb60 (0xe54 bytes) 000546 1788738406.925971 mapwrite 0x0182fc thru 0x02465a (0xc35f bytes) 000547 1788738406.949409 write 0x00eefd thru 0x0175f8 (0x86fc bytes) 000549 1788738406.951020 punch from 0x00bd1f to 0x01064e (0x4930 bytes) 000551 1788738406.955154 mapwrite 0x009b33 thru 0x01791d (0xddeb bytes) 000552 1788738406.971954 write 0x017f4a thru 0x01956f (0x1626 bytes) 000553 1788738406.974009 MIRROR_RESYNC 6 mirrors 000554 1788738407.326458 write 0x001e51 thru 0x010c50 (0xee00 bytes) 000555 1788738407.344686 mapwrite 0x022fdc thru 0x031570 (0xe595 bytes) 000556 1788738407.371491 mapread 0x00e5a9 thru 0x01c4c9 (0xdf21 bytes) 000557 1788738407.376817 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31571): Mirror 10: 0x17363f35 (stale) Mirror 13: 0x17363f35 (stale) Mirror 16: 0x17363f35 (stale) Mirror 17: 0x17363f35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31571): Mirror 10: 0x17363f35 (stale) Mirror 13: 0x17363f35 (stale) Mirror 16: 0x17363f35 (stale) Mirror 17: 0x17363f35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31571: skipped 000560 1788738407.469679 trunc from 0x031571 to 0x02b4df (0x6091 bytes) 000561 1788738407.473569 read 0x000caa thru 0x003187 (0x24de bytes) 000562 1788738407.475318 mapread 0x00761c thru 0x00b3a1 (0x3d86 bytes) 000563 1788738407.481308 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b4e0): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b4e0): Mirror 10: 0xce3462a8 (stale) Mirror 13: 0xce3462a8 (stale) Mirror 16: 0xce3462a8 (stale) Mirror 17: 0xce3462a8 (stale) CRC-32 checksum value for chunk [0, 0x2b4e0): Mirror 1: 0xf86d2b63 Mirror 18: 0xf86d2b63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b4e0: skipped 000564 1788738407.700142 write 0x02ff9d thru 0x035c9f (0x5d03 bytes) 000565 1788738407.717241 punch from 0x00524d to 0x00ccd7 (0x7a8b bytes) 000567 1788738407.721178 mapread 0x012eee thru 0x02154b (0xe65e bytes) 000570 1788738407.729019 punch from 0x03398b to 0x035c9f (0x2315 bytes) 000571 1788738407.730609 punch from 0x001995 to 0x00f165 (0xd7d1 bytes) 000573 1788738407.732366 mapread 0x01d57c thru 0x020522 (0x2fa7 bytes) 000574 1788738407.732893 mapread 0x00bfe7 thru 0x01411b (0x8135 bytes) 000576 1788738407.736094 write 0x02f790 thru 0x03729e (0x7b0f bytes) 000577 1788738407.737119 read 0x006475 thru 0x0104c3 (0xa04f bytes) 000578 1788738407.737562 punch from 0x029bdf to 0x02fd7d (0x619f bytes) 000579 1788738407.741008 write 0x005792 thru 0x013212 (0xda81 bytes) 000580 1788738407.741742 mapread 0x00044a thru 0x00ceb5 (0xca6c bytes) 000582 1788738407.744269 mapread 0x0261e7 thru 0x02dc8d (0x7aa7 bytes) 000584 1788738407.747111 write 0x00c3f6 thru 0x00e608 (0x2213 bytes) 000585 1788738407.747811 mapread 0x00a0ab thru 0x00ca49 (0x299f bytes) 000586 1788738407.748266 punch from 0x00740c to 0x01178b (0xa380 bytes) 000590 1788738407.752998 trunc from 0x03729f to 0x01740e (0x1fe90 bytes) 000591 1788738407.762752 punch from 0x01293e to 0x01740e (0x4ad1 bytes) 000592 1788738407.764141 trunc from 0x01740f to 0x02136c (0x9f5e bytes) 000593 1788738407.768836 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2136d): Mirror 10: 0x621b12ff (stale) Mirror 13: 0x621b12ff (stale) Mirror 16: 0x621b12ff (stale) Mirror 17: 0x621b12ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2136d): Mirror 10: 0x621b12ff (stale) Mirror 13: 0x621b12ff (stale) Mirror 16: 0x621b12ff (stale) Mirror 17: 0x621b12ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2136d: skipped 000594 1788738407.856280 punch from 0x0112dc to 0x01a42f (0x9154 bytes) 000595 1788738407.857980 trunc from 0x02136d to 0x03b1cd (0x19e61 bytes) 000597 1788738407.863105 MIRROR_RESYNC 5 mirrors 000602 1788738408.285649 trunc from 0x03b1ce to 0x00908d (0x32140 bytes) 000603 1788738408.301023 trunc from 0x00908e to 0x00ab39 (0x1aac bytes) 000605 1788738408.305398 mapread 0x004d51 thru 0x00ab39 (0x5de9 bytes) 000606 1788738408.309959 trunc from 0x00ab3a to 0x01efaa (0x14471 bytes) 000607 1788738408.313720 mapread 0x007218 thru 0x0088d7 (0x16c0 bytes) 000610 1788738408.315011 read 0x00634f thru 0x009246 (0x2ef8 bytes) 000611 1788738408.315454 mapwrite 0x03aacb thru 0x03ffff (0x5535 bytes) 000612 1788738408.332092 read 0x0232dd thru 0x02cff4 (0x9d18 bytes) 000613 1788738408.335331 punch from 0x02959e to 0x036b0a (0xd56d bytes) 000615 1788738408.337377 mapwrite 0x030cd3 thru 0x036453 (0x5781 bytes) 000616 1788738408.356805 mapread 0x003710 thru 0x008ef0 (0x57e1 bytes) 000617 1788738408.357351 write 0x03659a thru 0x039226 (0x2c8d bytes) 000618 1788738408.358014 mapwrite 0x031246 thru 0x034949 (0x3704 bytes) 000619 1788738408.371732 punch from 0x0279a7 to 0x032a32 (0xb08c bytes) 000620 1788738408.373715 write 0x02ef25 thru 0x0352f9 (0x63d5 bytes) 000621 1788738408.375756 mapwrite 0x0013da thru 0x005553 (0x417a bytes) 000622 1788738408.388689 write 0x01ede7 thru 0x025471 (0x668b bytes) 000624 1788738408.390020 trunc from 0x040000 to 0x0262a0 (0x19d5f bytes) 000625 1788738408.394346 trunc from 0x0262a1 to 0x000238 (0x26068 bytes) 000626 1788738408.398347 mapwrite 0x03ff78 thru 0x03ffff (0x88 bytes) 000628 1788738408.415222 mapread 0x025a71 thru 0x02a1bf (0x474f bytes) 000629 1788738408.422522 punch from 0x007a9b to 0x00ae40 (0x33a6 bytes) 000630 1788738408.424276 trunc from 0x040000 to 0x006ef5 (0x3910a bytes) 000631 1788738408.429181 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x6ef6): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6ef6): Mirror 10: 0x85cc106a (stale) Mirror 13: 0x85cc106a (stale) Mirror 16: 0x85cc106a (stale) Mirror 17: 0x85cc106a (stale) CRC-32 checksum value for chunk [0, 0x6ef6): Mirror 1: 0xa4d5deab Mirror 18: 0xa4d5deab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6ef6: skipped 000632 1788738408.657044 punch from 0x004ec3 to 0x006ef5 (0x2033 bytes) 000633 1788738408.675811 mapwrite 0x014441 thru 0x024120 (0xfce0 bytes) 000635 1788738408.709047 write 0x014e7e thru 0x0212ac (0xc42f bytes) 000636 1788738408.710257 punch from 0x009357 to 0x00ac05 (0x18af bytes) 000637 1788738408.715457 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24121): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24121): Mirror 10: 0xa7191efc (stale) Mirror 13: 0xa7191efc (stale) Mirror 16: 0xa7191efc (stale) Mirror 17: 0xa7191efc (stale) Mirror 18: 0xc94cbb83 (stale) CRC-32 checksum value for chunk [0, 0x24121): Mirror 1: 0xa64adf26 Mirror 19: 0xa64adf26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24121: skipped 000639 1788738408.974130 mapwrite 0x021453 thru 0x02b50a (0xa0b8 bytes) 000640 1788738409.015915 trunc from 0x02b50b to 0x02bddb (0x8d1 bytes) 000642 1788738409.022144 MIRROR_RESYNC 7 mirrors 000643 1788738409.392225 trunc from 0x02bddc to 0x035d4d (0x9f72 bytes) 000644 1788738409.410189 MIRROR_RESYNC 7 mirrors 000645 1788738409.804281 mapwrite 0x02ce3b thru 0x030990 (0x3b56 bytes) 000650 1788738409.852005 punch from 0x025e7a to 0x02b3c7 (0x554e bytes) 000651 1788738409.854001 read 0x0148ef thru 0x0174e0 (0x2bf2 bytes) 000653 1788738409.856504 trunc from 0x035d4e to 0x01d6c3 (0x1868a bytes) 000656 1788738409.863798 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d6c4): Mirror 10: 0xd3965c73 (stale) Mirror 13: 0xd3965c73 (stale) Mirror 16: 0xd3965c73 (stale) Mirror 17: 0xd3965c73 (stale) Mirror 19: 0xd3965c73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d6c4): Mirror 10: 0xd3965c73 (stale) Mirror 13: 0xd3965c73 (stale) Mirror 16: 0xd3965c73 (stale) Mirror 17: 0xd3965c73 (stale) Mirror 19: 0xd3965c73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d6c4: skipped 000657 1788738409.960547 mapread 0x0098c4 thru 0x0158e2 (0xc01f bytes) 000658 1788738409.967339 punch from 0x0010dc to 0x00ac21 (0x9b46 bytes) 000659 1788738409.968736 mapwrite 0x0353a1 thru 0x03e9fc (0x965c bytes) 000661 1788738409.989771 read 0x02be1a thru 0x0359f4 (0x9bdb bytes) 000663 1788738409.991975 write 0x00f0ec thru 0x018cbe (0x9bd3 bytes) 000664 1788738409.993031 mapread 0x00a0ce thru 0x010dfd (0x6d30 bytes) 000666 1788738409.996231 punch from 0x019331 to 0x0290ec (0xfdbc bytes) 000668 1788738410.000387 mapread 0x030db5 thru 0x03a74f (0x999b bytes) 000669 1788738410.001067 punch from 0x027bf7 to 0x0303a8 (0x87b2 bytes) 000670 1788738410.003768 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e9fd): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9fd): Mirror 10: 0x141ad33b (stale) Mirror 13: 0x141ad33b (stale) Mirror 16: 0x141ad33b (stale) Mirror 17: 0x141ad33b (stale) Mirror 19: 0x141ad33b (stale) CRC-32 checksum value for chunk [0, 0x3e9fd): Mirror 1: 0x74de0b60 Mirror 20: 0x74de0b60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9fd: skipped 000671 1788738410.303494 write 0x03bd07 thru 0x03ffff (0x42f9 bytes) 000672 1788738410.326041 read 0x01c77a thru 0x02b4ba (0xed41 bytes) 000674 1788738410.329238 mapwrite 0x018072 thru 0x01b4b7 (0x3446 bytes) 000675 1788738410.348153 write 0x028c0e thru 0x037c75 (0xf068 bytes) 000676 1788738410.350823 mapwrite 0x02cdc8 thru 0x036344 (0x957d bytes) 000678 1788738410.364736 punch from 0x00ad40 to 0x014066 (0x9327 bytes) 000680 1788738410.367218 read 0x02c155 thru 0x032780 (0x662c bytes) 000682 1788738410.368090 read 0x012520 thru 0x012d53 (0x834 bytes) 000683 1788738410.373457 MIRROR_RESYNC 7 mirrors 000686 1788738410.856217 trunc from 0x040000 to 0x030cbb (0xf344 bytes) 000687 1788738410.885382 MIRROR_RESYNC 7 mirrors 000688 1788738411.384462 write 0x0211ea thru 0x02ba2b (0xa842 bytes) 000690 1788738411.407168 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30cbc): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cbc): Mirror 10: 0x96868d9e (stale) Mirror 13: 0x96868d9e (stale) Mirror 16: 0x96868d9e (stale) Mirror 17: 0x96868d9e (stale) Mirror 19: 0x96868d9e (stale) Mirror 20: 0x96868d9e (stale) CRC-32 checksum value for chunk [0, 0x30cbc): Mirror 1: 0xf6ada266 Mirror 21: 0xf6ada266 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cbc: skipped 000691 1788738411.697381 read 0x02f616 thru 0x030cbb (0x16a6 bytes) 000692 1788738411.700760 write 0x01fc99 thru 0x0293c8 (0x9730 bytes) 000693 1788738411.722470 trunc from 0x030cbc to 0x036908 (0x5c4d bytes) 000694 1788738411.727863 write 0x025fa1 thru 0x030a13 (0xaa73 bytes) 000695 1788738411.730156 read 0x0072b4 thru 0x00c2fb (0x5048 bytes) 000697 1788738411.734875 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36909): Mirror 10: 0x30344457 (stale) Mirror 13: 0x30344457 (stale) Mirror 16: 0x30344457 (stale) Mirror 17: 0x30344457 (stale) Mirror 20: 0x30344457 (stale) Mirror 21: 0x170ed30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36909): Mirror 10: 0x30344457 (stale) Mirror 13: 0x30344457 (stale) Mirror 16: 0x30344457 (stale) Mirror 17: 0x30344457 (stale) Mirror 20: 0x30344457 (stale) Mirror 21: 0x170ed30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36909: skipped 000699 1788738411.858499 mapwrite 0x01c610 thru 0x020c36 (0x4627 bytes) 000700 1788738411.878233 write 0x03ddb2 thru 0x03ffff (0x224e bytes) 000702 1788738411.879303 write 0x0024cd thru 0x00b0ca (0x8bfe bytes) 000703 1788738411.882700 mapwrite 0x031faf thru 0x03c2d2 (0xa324 bytes) 000704 1788738411.901336 write 0x03e747 thru 0x03ffff (0x18b9 bytes) 000705 1788738411.902214 punch from 0x02f37e to 0x0383cf (0x9052 bytes) 000707 1788738411.905947 write 0x024695 thru 0x028239 (0x3ba5 bytes) 000708 1788738411.909365 MIRROR_RESYNC 7 mirrors 000710 1788738412.410266 read 0x007f48 thru 0x016846 (0xe8ff bytes) 000711 1788738412.413856 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) 000713 1788738412.512981 mapread 0x00ab09 thru 0x01a1ec (0xf6e4 bytes) 000714 1788738412.519779 trunc from 0x040000 to 0x02f81a (0x107e5 bytes) 000715 1788738412.538240 read 0x00c4e1 thru 0x011e90 (0x59b0 bytes) 000716 1788738412.540824 write 0x02876b thru 0x028ff3 (0x889 bytes) 000717 1788738412.542386 trunc from 0x02f81b to 0x00e675 (0x211a5 bytes) 000719 1788738412.548003 mapread 0x003948 thru 0x00a799 (0x6e52 bytes) 000720 1788738412.552202 MIRROR_RESYNC 6 mirrors 000721 1788738412.839582 read 0x0000a3 thru 0x0005d8 (0x536 bytes) 000722 1788738412.841096 mapread 0x00877a thru 0x00bbd4 (0x345b bytes) 000724 1788738412.845958 punch from 0x00cc49 to 0x00e675 (0x1a2d bytes) 000725 1788738412.862702 write 0x02aab2 thru 0x02b3f5 (0x944 bytes) 000726 1788738412.863621 punch from 0x01039d to 0x014e39 (0x4a9d bytes) 000728 1788738412.866902 trunc from 0x02b3f6 to 0x0319e3 (0x65ee bytes) 000729 1788738412.876921 write 0x0025e5 thru 0x011634 (0xf050 bytes) 000730 1788738412.880263 trunc from 0x0319e4 to 0x000bf2 (0x30df1 bytes) 000731 1788738412.884633 mapread 0x0009ac thru 0x000bf2 (0x247 bytes) 000732 1788738412.886898 punch from 0x0001cb to 0x000bf2 (0xa28 bytes) 000733 1788738412.888522 mapread 0x000791 thru 0x000bf2 (0x462 bytes) 000734 1788738412.890543 mapwrite 0x02c2ef thru 0x02c412 (0x124 bytes) 000735 1788738412.909254 mapread 0x021505 thru 0x02bbb6 (0xa6b2 bytes) 000737 1788738412.912843 mapread 0x003e5c thru 0x007336 (0x34db bytes) 000738 1788738412.915658 trunc from 0x02c413 to 0x034f90 (0x8b7e bytes) 000740 1788738412.919931 mapread 0x02a314 thru 0x034f90 (0xac7d bytes) 000741 1788738412.922591 mapwrite 0x01dfec thru 0x028a3d (0xaa52 bytes) 000742 1788738412.935534 punch from 0x02ccaf to 0x034f90 (0x82e2 bytes) 000743 1788738412.937397 mapread 0x00ffea thru 0x0183f6 (0x840d bytes) 000744 1788738412.939959 mapread 0x020f51 thru 0x026947 (0x59f7 bytes) 000745 1788738412.940407 write 0x007923 thru 0x00ea46 (0x7124 bytes) 000746 1788738412.941033 mapread 0x00444d thru 0x004aaf (0x663 bytes) 000747 1788738412.941602 read 0x027399 thru 0x02fd1c (0x8984 bytes) 000749 1788738412.944091 write 0x01d73c thru 0x02763b (0x9f00 bytes) 000750 1788738412.944718 read 0x02eb8c thru 0x030a2e (0x1ea3 bytes) 000752 1788738412.945229 mapwrite 0x011f4e thru 0x01536d (0x3420 bytes) 000754 1788738412.958682 mapread 0x02a21e thru 0x034f90 (0xad73 bytes) 000758 1788738412.960173 punch from 0x022721 to 0x02f143 (0xca23 bytes) 000759 1788738412.962202 mapwrite 0x0316e4 thru 0x037bc2 (0x64df bytes) 000760 1788738412.981510 mapwrite 0x02d6b3 thru 0x039897 (0xc1e5 bytes) 000761 1788738413.001844 punch from 0x019477 to 0x01adb5 (0x193f bytes) 000763 1788738413.005516 MIRROR_RESYNC 6 mirrors 000764 1788738413.528086 write 0x00265d thru 0x006d60 (0x4704 bytes) 000765 1788738413.544870 read 0x01ae8a thru 0x023b6b (0x8ce2 bytes) 000766 1788738413.547319 mapread 0x02d8f0 thru 0x036ba2 (0x92b3 bytes) 000767 1788738413.552995 trunc from 0x039898 to 0x01ff64 (0x19933 bytes) 000768 1788738413.559131 mapread 0x018dfa thru 0x01ff64 (0x716b bytes) 000769 1788738413.562371 write 0x0112fe thru 0x01cfb9 (0xbcbc bytes) 000770 1788738413.563127 mapwrite 0x033940 thru 0x036677 (0x2d38 bytes) 000771 1788738413.585721 punch from 0x02fe75 to 0x036677 (0x6803 bytes) 000772 1788738413.587544 mapwrite 0x01068e thru 0x01e568 (0xdedb bytes) 000773 1788738413.607001 write 0x02b6ed thru 0x02bc28 (0x53c bytes) 000774 1788738413.610287 write 0x034119 thru 0x03b45d (0x7345 bytes) 000775 1788738413.613088 mapwrite 0x03890d thru 0x03ffff (0x76f3 bytes) 000776 1788738413.634067 trunc from 0x040000 to 0x03623b (0x9dc4 bytes) 000780 1788738413.640537 read 0x016f19 thru 0x019fe4 (0x30cc bytes) 000781 1788738413.640956 trunc from 0x03623c to 0x0007ee (0x35a4d bytes) 000782 1788738413.646189 read 0x00060c thru 0x0007ee (0x1e3 bytes) 000783 1788738413.648324 mapread 0x000393 thru 0x0007ee (0x45c bytes) 000784 1788738413.649124 mapwrite 0x02c388 thru 0x030e24 (0x4a9d bytes) 000786 1788738413.672687 read 0x026671 thru 0x03093a (0xa2ca bytes) 000789 1788738413.674170 read 0x02147a thru 0x027511 (0x6098 bytes) 000794 1788738413.678365 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30e25): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e25): Mirror 10: 0x72256d09 (stale) Mirror 13: 0x72256d09 (stale) Mirror 16: 0x72256d09 (stale) Mirror 17: 0x72256d09 (stale) Mirror 21: 0x72256d09 (stale) CRC-32 checksum value for chunk [0, 0x30e25): Mirror 1: 0xdd37b6ba Mirror 22: 0xdd37b6ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e25: skipped 000796 1788738413.939002 write 0x02d282 thru 0x03b726 (0xe4a5 bytes) 000797 1788738413.960952 mapwrite 0x009ddf thru 0x0117f5 (0x7a17 bytes) 000798 1788738413.985095 write 0x032e96 thru 0x038525 (0x5690 bytes) 000799 1788738413.985974 trunc from 0x03b727 to 0x02b8f7 (0xfe2f bytes) 000800 1788738413.996936 read 0x023e97 thru 0x02b8f7 (0x7a61 bytes) 000801 1788738414.000361 trunc from 0x02b8f8 to 0x014418 (0x174df bytes) 000802 1788738414.004857 write 0x02a0e4 thru 0x02b8cd (0x17ea bytes) 000803 1788738414.007391 MIRROR_RESYNC 7 mirrors 000805 1788738414.536542 read 0x0133ae thru 0x018c15 (0x5868 bytes) 000806 1788738414.539379 write 0x0281f7 thru 0x037e82 (0xfc8c bytes) 000810 1788738414.560743 write 0x027e02 thru 0x034a53 (0xcc52 bytes) 000812 1788738414.563848 mapwrite 0x004cc5 thru 0x00a3b5 (0x56f1 bytes) 000815 1788738414.587129 mapwrite 0x01914b thru 0x023461 (0xa317 bytes) 000816 1788738414.606002 punch from 0x026c9a to 0x02d855 (0x6bbc bytes) 000818 1788738414.608512 mapread 0x01c95f thru 0x020250 (0x38f2 bytes) 000820 1788738414.609234 punch from 0x02a817 to 0x02c1c6 (0x19b0 bytes) 000821 1788738414.610763 mapread 0x00b043 thru 0x00ef93 (0x3f51 bytes) 000822 1788738414.611428 write 0x022614 thru 0x028b99 (0x6586 bytes) 000825 1788738414.614020 write 0x004769 thru 0x006c78 (0x2510 bytes) 000827 1788738414.614870 mapread 0x01d0b7 thru 0x01e577 (0x14c1 bytes) 000828 1788738414.616902 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37e83): Mirror 10: 0xf0f15da7 (stale) Mirror 13: 0xf0f15da7 (stale) Mirror 16: 0xf0f15da7 (stale) Mirror 17: 0xf0f15da7 (stale) Mirror 21: 0xf0f15da7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e83): Mirror 10: 0xf0f15da7 (stale) Mirror 13: 0xf0f15da7 (stale) Mirror 16: 0xf0f15da7 (stale) Mirror 17: 0xf0f15da7 (stale) Mirror 21: 0xf0f15da7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e83: skipped 000829 1788738414.745638 MIRROR_RESYNC 6 mirrors 000830 1788738415.203989 mapwrite 0x033907 thru 0x03ffff (0xc6f9 bytes) 000831 1788738415.245066 punch from 0x01e14f to 0x02083a (0x26ec bytes) 000832 1788738415.246869 trunc from 0x040000 to 0x015fb2 (0x2a04d bytes) 000833 1788738415.253961 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x15fb3): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15fb3): Mirror 10: 0x6c6722cd (stale) Mirror 13: 0x6c6722cd (stale) Mirror 16: 0x6c6722cd (stale) Mirror 17: 0x6c6722cd (stale) Mirror 21: 0x6c6722cd (stale) CRC-32 checksum value for chunk [0, 0x15fb3): Mirror 1: 0x6c6722cd Mirror 22: 0x6c6722cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15fb3: skipped 000834 1788738415.565389 punch from 0x009a7f to 0x015c67 (0xc1e9 bytes) 000835 1788738415.586263 trunc from 0x015fb3 to 0x007130 (0xee82 bytes) 000838 1788738415.592943 trunc from 0x007131 to 0x033a12 (0x2c8e2 bytes) 000840 1788738415.599667 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a13): Mirror 13: 0x5b576dab (stale) Mirror 16: 0x5b576dab (stale) Mirror 17: 0x5b576dab (stale) Mirror 21: 0x5b576dab (stale) Mirror 22: 0x1777e005 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a13): Mirror 13: 0x5b576dab (stale) Mirror 16: 0x5b576dab (stale) Mirror 17: 0x5b576dab (stale) Mirror 21: 0x5b576dab (stale) Mirror 22: 0x1777e005 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a13: skipped 000841 1788738415.702195 punch from 0x02d988 to 0x033a12 (0x608b bytes) 000842 1788738415.703816 mapread 0x016253 thru 0x024aeb (0xe899 bytes) 000843 1788738415.711688 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a13): Mirror 13: 0x5b576dab (stale) Mirror 16: 0x5b576dab (stale) Mirror 17: 0x5b576dab (stale) Mirror 22: 0x1777e005 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a13): Mirror 13: 0x5b576dab (stale) Mirror 16: 0x5b576dab (stale) Mirror 17: 0x5b576dab (stale) Mirror 22: 0x1777e005 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a13: skipped 000844 1788738415.812269 write 0x006356 thru 0x00ca94 (0x673f bytes) 000845 1788738415.815766 mapwrite 0x01e841 thru 0x027f57 (0x9717 bytes) 000846 1788738415.836389 mapread 0x01bab0 thru 0x020d47 (0x5298 bytes) 000849 1788738415.838143 punch from 0x0112d2 to 0x012e28 (0x1b57 bytes) 000850 1788738415.840350 mapread 0x032595 thru 0x033a12 (0x147e bytes) 000851 1788738415.844204 write 0x025b7e thru 0x032950 (0xcdd3 bytes) 000852 1788738415.846408 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a13): Mirror 16: 0x5b576dab (stale) Mirror 17: 0x5b576dab (stale) Mirror 22: 0x1777e005 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a13): Mirror 16: 0x5b576dab (stale) Mirror 17: 0x5b576dab (stale) Mirror 22: 0x1777e005 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a13: skipped 000853 1788738415.942257 trunc from 0x033a13 to 0x000ba3 (0x32e6f bytes) 000855 1788738415.947057 mapread 0x0002b2 thru 0x000ba3 (0x8f2 bytes) 000857 1788738415.949666 mapread 0x000629 thru 0x000ba3 (0x57b bytes) 000858 1788738415.950220 write 0x02876e thru 0x036a44 (0xe2d7 bytes) 000860 1788738415.951562 trunc from 0x036a45 to 0x025291 (0x117b3 bytes) 000863 1788738415.956616 mapwrite 0x01282a thru 0x01a4eb (0x7cc2 bytes) 000864 1788738415.975088 mapwrite 0x00d423 thru 0x01832c (0xaf0a bytes) 000865 1788738415.992370 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25292): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25292): Mirror 16: 0xf646416b (stale) Mirror 17: 0xf646416b (stale) Mirror 22: 0xb7c89a72 (stale) CRC-32 checksum value for chunk [0, 0x25292): Mirror 1: 0xe6734c08 Mirror 23: 0xe6734c08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25292: skipped 000866 1788738416.306025 write 0x00d132 thru 0x0110ef (0x3fbe bytes) 000867 1788738416.329689 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25292): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25292): Mirror 16: 0xf646416b (stale) Mirror 17: 0xf646416b (stale) Mirror 22: 0xb7c89a72 (stale) Mirror 23: 0xe6734c08 (stale) CRC-32 checksum value for chunk [0, 0x25292): Mirror 1: 0x4f3c0d25 Mirror 24: 0x4f3c0d25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25292: skipped 000869 1788738416.634298 punch from 0x00924f to 0x009d7a (0xb2c bytes) 000870 1788738416.654665 trunc from 0x025292 to 0x0069ce (0x1e8c3 bytes) 000871 1788738416.659771 trunc from 0x0069cf to 0x00895d (0x1f8f bytes) 000872 1788738416.663677 punch from 0x001a8d to 0x005bf7 (0x416b bytes) 000873 1788738416.665105 mapread 0x005582 thru 0x00895d (0x33dc bytes) 000874 1788738416.670102 write 0x03cbc7 thru 0x03ffff (0x3439 bytes) 000877 1788738416.672505 MIRROR_RESYNC 6 mirrors 000878 1788738417.038219 MIRROR_EXTEND to 7 mirrors 000879 1788738417.260093 read 0x0194d0 thru 0x024aea (0xb61b bytes) 000880 1788738417.263098 mapread 0x00f747 thru 0x017849 (0x8103 bytes) 000881 1788738417.268893 read 0x006146 thru 0x014db8 (0xec73 bytes) 000882 1788738417.271815 mapwrite 0x03d045 thru 0x03ffff (0x2fbb bytes) 000883 1788738417.320828 mapread 0x00a471 thru 0x01931f (0xeeaf bytes) 000887 1788738417.342022 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xaea85b5a (stale) Mirror 17: 0xaea85b5a (stale) Mirror 22: 0xaea85b5a (stale) Mirror 24: 0xaea85b5a (stale) Mirror 25: 0xaea85b5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xaea85b5a (stale) Mirror 17: 0xaea85b5a (stale) Mirror 22: 0xaea85b5a (stale) Mirror 24: 0xaea85b5a (stale) Mirror 25: 0xaea85b5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000889 1788738417.450007 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xaea85b5a (stale) Mirror 17: 0xaea85b5a (stale) Mirror 22: 0xaea85b5a (stale) Mirror 24: 0xaea85b5a (stale) Mirror 25: 0xaea85b5a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x321ced20 Mirror 26: 0x321ced20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000891 1788738417.722336 trunc from 0x040000 to 0x004ceb (0x3b314 bytes) 000892 1788738417.743003 read 0x00181e thru 0x004ceb (0x34ce bytes) 000893 1788738417.745303 trunc from 0x004cec to 0x020517 (0x1b82c bytes) 000894 1788738417.750084 punch from 0x0136ac to 0x017707 (0x405c bytes) 000895 1788738417.751452 write 0x00e412 thru 0x0109ff (0x25ee bytes) 000896 1788738417.754348 write 0x018e4a thru 0x024570 (0xb727 bytes) 000898 1788738417.757864 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24571): Mirror 17: 0x197445b (stale) Mirror 22: 0x197445b (stale) Mirror 24: 0x197445b (stale) Mirror 25: 0x197445b (stale) Mirror 26: 0x197445b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24571): Mirror 17: 0x197445b (stale) Mirror 22: 0x197445b (stale) Mirror 24: 0x197445b (stale) Mirror 25: 0x197445b (stale) Mirror 26: 0x197445b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24571: skipped 000899 1788738417.870132 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x678:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24571): Mirror 17: 0x197445b (stale) Mirror 22: 0x197445b (stale) Mirror 24: 0x197445b (stale) Mirror 26: 0x197445b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24571): Mirror 17: 0x197445b (stale) Mirror 22: 0x197445b (stale) Mirror 24: 0x197445b (stale) Mirror 26: 0x197445b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24571: skipped 000902 1788738417.964441 punch from 0x00009e to 0x008d27 (0x8c8a bytes) 000903 1788738417.965825 trunc from 0x024571 to 0x00562d (0x1ef43 bytes) 000904 1788738417.971020 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x562e): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x562e): Mirror 17: 0x3dcdf97e (stale) Mirror 22: 0x3dcdf97e (stale) Mirror 24: 0x3dcdf97e (stale) Mirror 26: 0x3dcdf97e (stale) CRC-32 checksum value for chunk [0, 0x562e): Mirror 1: 0x3dcdf97e Mirror 27: 0x3dcdf97e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x562e: skipped 000911 1788738418.255477 write 0x03d803 thru 0x03ffff (0x27fd bytes) 000912 1788738418.272753 write 0x005da1 thru 0x00962f (0x388f bytes) 000913 1788738418.275390 trunc from 0x040000 to 0x002e91 (0x3d16e bytes) 000914 1788738418.280373 punch from 0x00090d to 0x002e91 (0x2585 bytes) 000916 1788738418.282147 mapwrite 0x0019f0 thru 0x00df0c (0xc51d bytes) 000917 1788738418.305859 punch from 0x0070e6 to 0x00df0c (0x6e27 bytes) 000918 1788738418.307572 punch from 0x005c87 to 0x00d9cd (0x7d47 bytes) 000921 1788738418.309583 write 0x026cf8 thru 0x027b47 (0xe50 bytes) 000922 1788738418.311305 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x27b48): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b48): Mirror 17: 0x2d28f0b9 (stale) Mirror 22: 0x2d28f0b9 (stale) Mirror 24: 0x2d28f0b9 (stale) Mirror 26: 0x2d28f0b9 (stale) Mirror 27: 0x2d28f0b9 (stale) CRC-32 checksum value for chunk [0, 0x27b48): Mirror 1: 0xcae7311d Mirror 28: 0xcae7311d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b48: skipped 000923 1788738418.578617 mapwrite 0x028102 thru 0x032493 (0xa392 bytes) 000924 1788738418.620676 write 0x006dde thru 0x014044 (0xd267 bytes) 000926 1788738418.623803 read 0x02b203 thru 0x0300a4 (0x4ea2 bytes) 000929 1788738418.626000 MIRROR_RESYNC 7 mirrors 000930 1788738419.016115 mapwrite 0x001e15 thru 0x010ee9 (0xf0d5 bytes) 000931 1788738419.066588 trunc from 0x032494 to 0x011c4c (0x20847 bytes) 000932 1788738419.075318 mapwrite 0x03dca8 thru 0x03ffff (0x2358 bytes) 000933 1788738419.092679 write 0x011499 thru 0x014696 (0x31fe bytes) 000934 1788738419.095600 mapwrite 0x010536 thru 0x01ae89 (0xa954 bytes) 000935 1788738419.110914 punch from 0x03ed5e to 0x03ffff (0x12a2 bytes) 000936 1788738419.112488 mapwrite 0x016da7 thru 0x01a7a0 (0x39fa bytes) 000937 1788738419.124651 write 0x01d66b thru 0x021120 (0x3ab6 bytes) 000940 1788738419.127783 write 0x02ea37 thru 0x03779a (0x8d64 bytes) 000942 1788738419.130609 trunc from 0x040000 to 0x020eaa (0x1f155 bytes) 000943 1788738419.134993 punch from 0x002027 to 0x00b2a6 (0x9280 bytes) 000944 1788738419.137964 write 0x010d79 thru 0x01706c (0x62f4 bytes) 000945 1788738419.138818 write 0x03eafd thru 0x03ffff (0x1503 bytes) 000947 1788738419.139616 punch from 0x02d21b to 0x039d5d (0xcb43 bytes) 000948 1788738419.143057 punch from 0x016de0 to 0x022ca6 (0xbec7 bytes) 000949 1788738419.145714 MIRROR_RESYNC 7 mirrors 000950 1788738419.642122 write 0x01044c thru 0x01eb04 (0xe6b9 bytes) 000951 1788738419.659441 mapwrite 0x0157f0 thru 0x020193 (0xa9a4 bytes) 000952 1788738419.675917 punch from 0x01a11b to 0x020ae4 (0x69ca bytes) 000953 1788738419.677790 read 0x024783 thru 0x0287d3 (0x4051 bytes) 000956 1788738419.682637 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xbebbb96e (stale) Mirror 22: 0xbebbb96e (stale) Mirror 24: 0xbebbb96e (stale) Mirror 26: 0xbebbb96e (stale) Mirror 27: 0xbebbb96e (stale) Mirror 28: 0xbebbb96e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe28a2e3 Mirror 29: 0xe28a2e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000959 1788738419.996802 mapread 0x00727f thru 0x00a758 (0x34da bytes) 000960 1788738420.004983 read 0x03aacd thru 0x03e0d5 (0x3609 bytes) 000961 1788738420.008885 read 0x0331ce thru 0x03539c (0x21cf bytes) 000962 1788738420.013376 write 0x03b0c7 thru 0x03ffff (0x4f39 bytes) 000963 1788738420.033117 read 0x036530 thru 0x0368a7 (0x378 bytes) 000965 1788738420.037144 read 0x03b4fa thru 0x03ffff (0x4b06 bytes) 000966 1788738420.037727 trunc from 0x040000 to 0x003892 (0x3c76d bytes) 000967 1788738420.048485 mapwrite 0x00e14a thru 0x016981 (0x8838 bytes) 000968 1788738420.070717 read 0x000841 thru 0x00a84c (0xa00c bytes) 000969 1788738420.071273 mapwrite 0x03d2d9 thru 0x03ffff (0x2d27 bytes) 000970 1788738420.089369 mapread 0x03248b thru 0x03ffff (0xdb75 bytes) 000971 1788738420.091158 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xbebbb96e (stale) Mirror 22: 0xbebbb96e (stale) Mirror 24: 0xbebbb96e (stale) Mirror 26: 0xbebbb96e (stale) Mirror 27: 0xbebbb96e (stale) Mirror 28: 0xbebbb96e (stale) Mirror 29: 0xe28a2e3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3f251d4c Mirror 30: 0x3f251d4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000972 1788738420.367780 trunc from 0x040000 to 0x03ad1a (0x52e5 bytes) 000973 1788738420.392941 trunc from 0x03ad1b to 0x00e519 (0x2c801 bytes) truncating to largest ever: 0x3fad3 000974 1788738420.397887 trunc from 0x00e51a to 0x03fad2 (0x315b9 bytes) 000975 1788738420.402179 mapread 0x002476 thru 0x00b4e1 (0x906c bytes) 000977 1788738420.408043 write 0x017a10 thru 0x01f3e1 (0x79d2 bytes) 000978 1788738420.411110 mapwrite 0x00a25b thru 0x016844 (0xc5ea bytes) 000979 1788738420.427970 write 0x020eeb thru 0x02a589 (0x969f bytes) 000980 1788738420.431383 read 0x037534 thru 0x03fad2 (0x859f bytes) 000981 1788738420.433798 mapwrite 0x017895 thru 0x018123 (0x88f bytes) 000983 1788738420.447111 read 0x007dd8 thru 0x014379 (0xc5a2 bytes) 000984 1788738420.447737 write 0x03de82 thru 0x03ffff (0x217e bytes) 000986 1788738420.449029 write 0x00a16c thru 0x01513c (0xafd1 bytes) 000988 1788738420.452246 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xbebbb96e (stale) Mirror 22: 0xbebbb96e (stale) Mirror 24: 0xbebbb96e (stale) Mirror 26: 0xbebbb96e (stale) Mirror 27: 0xbebbb96e (stale) Mirror 28: 0xbebbb96e (stale) Mirror 29: 0xe28a2e3 (stale) Mirror 30: 0x3f251d4c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5f33ab85 Mirror 31: 0x5f33ab85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1788738420.765903 punch from 0x03c33f to 0x03ffff (0x3cc1 bytes) 000995 1788738420.787681 punch from 0x03c986 to 0x03ffff (0x367a bytes) 000996 1788738420.789456 write 0x027593 thru 0x036b93 (0xf601 bytes) 000997 1788738420.794912 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xbebbb96e (stale) Mirror 22: 0xbebbb96e (stale) Mirror 24: 0xbebbb96e (stale) Mirror 26: 0xbebbb96e (stale) Mirror 27: 0xbebbb96e (stale) Mirror 28: 0xbebbb96e (stale) Mirror 29: 0xe28a2e3 (stale) Mirror 30: 0x3f251d4c (stale) Mirror 31: 0x5f33ab85 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x36520afa Mirror 32: 0x36520afa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped All operations completed A-OK! PASS 70a (31s) == sanity-flr test 71: check mirror extend parent fid ==== 19:47:02 (1788738422) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 19:47:03 (1788738423) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa10:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0740719 s, 28.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa10:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa11:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471669 s, 22.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa10:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa11:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa10:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa11:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa12:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 19:47:06 (1788738426) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3125120 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6376736 .. done Extending file size to 8116800 .. Extending file size to 3276736 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6364160 .. Extending file size to 8254304 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9174752 .. Extending file size to 8363360 .. Extending file size to 8849088 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9368192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60887 60888 60890 60892 60893 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:48:12 (1788738492) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to Extending file size to 8629760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3529824 .. Extending file size to 3958592 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5256288 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6582976 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8684480 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8806208 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8856160 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9101408 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8999680 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9200544 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64659 64660 64662 64665 64666 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:49:17 (1788738557) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3840 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3840 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0381513 s, 55.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0678379 s, 30.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9ebc1a2a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:49:23 (1788738563) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.021897 s, 95.8 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 19:49:24 (1788738564) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6c7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa23:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa23:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa24:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xa27:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:49:26 (1788738566) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423088 s, 49.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa25:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa28:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa29:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:49:29 (1788738569) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0338394 s, 62.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa26:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa27:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa26:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa27:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2b:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:49:31 (1788738571) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0450541 s, 46.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa28:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa28:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2d:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:49:55 (1788738595) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220154 s, 47.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa29:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa29:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2a:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:50:42 (1788738642) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0437396 s, 47.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:51:27 (1788738687) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Creating new pool test_204f oleg148-server: Pool lustre.test_204f created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0447668 s, 46.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa31:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa31:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg148-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (59s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:52:26 (1788738746) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:52:28 (1788738748) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:52:31 (1788738751) oleg148-server: Pool lustre.test_206 created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg148-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:52:44 (1788738764) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275964 s, 38.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138465 s, 75.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf8fe4d40 Mirror 3: 0xf8fe4d40 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:52:48 (1788738768) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:52:49 (1788738769) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:52:50 (1788738770) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0193286 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1270000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x970000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x970000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x970000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1700000 Read this (4 bytes): BBBB 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:53:06 (1788738786) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00289748 s, 2.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:53:08 (1788738788) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125 s, 83.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6ec lmm_fid: [0x200000404:0x6ec:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6ec lmm_fid: [0x200000404:0x6ec:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835692 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 19:53:08.000000000 -0400 Modify: 2026-09-06 19:53:08.000000000 -0400 Change: 2026-09-06 19:53:09.000000000 -0400 Birth: 2026-09-06 19:53:08.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:53:10 (1788738790) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:53:11 (1788738791) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.101673 s, 403 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0127794 s, 321 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 6 19:53 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6ee:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 6 19:53 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:53:14 (1788738794) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:53:16 (1788738796) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 857 sec ============ 19:53:19 (1788738799) === sanity-flr: start cleanup 19:53:19 (1788738799) === === sanity-flr: finish cleanup 19:53:21 (1788738801) ===