-----============= acceptance-small: sanity-flr ============----- Sat Sep 12 21:48:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg412-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:48:26 (1789264106) === oleg412-client.virtnet: executing check_config_client /mnt/lustre oleg412-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg412-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff995b58b41000.idle_timeout=debug osc.lustre-OST0001-osc-ffff995b58b41000.idle_timeout=debug disable quota as required oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:48:51 (1789264131) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8195] == sanity-flr test 0a: lfs mirror create with -N option == 21:48:52 (1789264132) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:49:00 (1789264140) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:49:02 (1789264142) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:49:04 (1789264144) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:49:18 (1789264158) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:49:20 (1789264160) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 21:49:22 (1789264162) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff995b58b41000.stats=clear osc.lustre-OST0001-osc-ffff995b58b41000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:49:26 (1789264166) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:49:32 (1789264172) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:49:36 (1789264176) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:49:38 (1789264178) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:49:42 (1789264182) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 21:49:47 (1789264187) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:49:51 (1789264191) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:49:55 (1789264195) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:50:00 (1789264200) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:50:06 (1789264206) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:50:09 (1789264209) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.869736 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802967 s, 13.1 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:50:15 (1789264215) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.134043 s, 62.6 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:50:19 (1789264219) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:50:24 (1789264224) Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (39s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:51:03 (1789264263) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.215227 s, 34.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (24s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:51:27 (1789264287) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (42s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:52:09 (1789264329) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50833 s, 34.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 12 21:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.440s user 0m0.000s sys 0m0.185s Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff995b58b41000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.494s user 0m0.000s sys 0m0.188s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.439s user 0m0.001s sys 0m0.163s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg412-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff995b58b41000.ost_server_uuid in FULL state after 0 sec PASS 33b (46s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:52:55 (1789264375) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:52:57 (1789264377) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:52:58 (1789264378) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 21:53:00 (1789264380) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 21:53:03 (1789264383) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.257107 s, 48.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:53:09 (1789264389) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209533 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195405 s, 53.7 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:53:13 (1789264393) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:53:17 (1789264397) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000950307 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000820998 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0017139 s, 597 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104711 s, 978 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 21:53:22 (1789264402) create test files with size 15731453 .. 079dc9f6360e5b56c0e51f3b4caae178 - d1629aa5b86f94203c2824095f7f2fa4 - af0346b3f2031a676befb03d6739997f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 21:53:33 (1789264413) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:53:40 (1789264420) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:53:43 (1789264423) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381998 s, 27.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159908 s, 65.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 21:53:48 (1789264428) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0446054 s, 47.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.193085 s, 21.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0839805 s, 25.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 21:53:55 (1789264435) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 21:53:57 (1789264437) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:53:58 (1789264438) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173699 s, 60.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 21:54:02 (1789264442) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.155582 s, 20.2 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:54:06 (1789264446) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:54:11 (1789264451) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.171328 s, 61.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:54:23 (1789264463) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff995b45121800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.147621 s, 71.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff995b617c4800.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:54:30 (1789264470) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (17s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:54:47 (1789264487) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (18s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:55:05 (1789264505) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 21:55:06 (1789264506) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.127173 s, 24.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0727071 s, 43.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xddc91c7c (nosync,stale) Mirror 2: 0xddc91c7c (nosync,stale) ** verify mirror 2 doesn't change original checksum: e970090d9f0d9c0408f8224bcd4944a1 - mirror 1 checksum: c23a71304b777a719de94a31a614181d - mirror 2 checksum: e970090d9f0d9c0408f8224bcd4944a1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T21:55:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T21:55:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e970090d9f0d9c0408f8224bcd4944a1 - mirror 1 checksum: c23a71304b777a719de94a31a614181d - mirror 2 checksum: c23a71304b777a719de94a31a614181d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T21:55:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T21:55:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:55:11 (1789264511) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190055 s, 55.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 314368.. 315391: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:55:14 (1789264514) Failing mds1 on oleg412-server Stopping /mnt/lustre-mds1 (opts:) on oleg412-server 21:55:22 (1789264522) shut down facet: mds1 facet_host: oleg412-server facet_failover_host: oleg412-server Failover mds1 to oleg412-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 21:55:39 (1789264539) targets are mounted 21:55:39 (1789264539) facet_failover done oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157372 s, 66.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 50A (35s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:55:49 (1789264549) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0241301 s, 849 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0119352 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:55:54 (1789264554) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0301358 s, 680 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00577073 s, 3.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:55:58 (1789264558) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00508956 s, 3.2 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:56:01 (1789264561) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0245063 s, 836 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:56:05 (1789264565) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00569386 s, 719 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0235804 s, 174 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:56:08 (1789264568) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (13s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:56:21 (1789264581) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:56:43 (1789264603) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:57:03 (1789264623) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:57:22 (1789264642) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 21:57:25 (1789264645) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57452 mirror split pid 57453 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57452 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57453 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 21:58:29 (1789264709) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4710 000001 1789264710.446771 MIRROR_EXTEND to 2 mirrors skipping zero length zero range skipping zero length zero range skipping zero length zero range truncating to largest ever: 0x1b4dc 000006 1789264710.706908 trunc from 00000000 to 0x01b4db (0x1b4dc bytes) 000007 1789264710.770964 read 0x006e6d thru 0x008543 (0x16d7 bytes) 000008 1789264710.780917 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000009 1789264710.878842 mapwrite 0x03a9d6 thru 0x03f783 (0x4dae bytes) 000010 1789264710.934082 write 0x0362cd thru 0x03cb89 (0x68bd bytes) 000011 1789264710.935117 mapwrite 0x01ccd9 thru 0x02249b (0x57c3 bytes) 000012 1789264710.961035 write 0x000045 thru 0x00319a (0x3156 bytes) 000014 1789264710.968305 trunc from 0x015615 to 0x019968 (0x4354 bytes) 000015 1789264710.972328 trunc from 0x03f784 to 0x00b549 (0x3423a bytes) fallocating to largest ever: 0x3931e 000016 1789264710.984262 fallocate 0x030bec thru 0x03931d (0x8732 bytes) 000017 1789264710.987718 read 0x0048ec thru 0x00768b (0x2da0 bytes) 000019 1789264710.993622 trunc from 0x00b54a to 0x02c02a (0x20ae1 bytes) 000020 1789264711.008606 punch from 0x00d324 to 0x01478f (0x746c bytes) 000021 1789264711.013579 mapread 0x005c89 thru 0x00869d (0x2a15 bytes) 000023 1789264711.022316 trunc from 0x02c02b to 0x00dbb2 (0x1e478 bytes) 000024 1789264711.033267 punch from 0x0068b6 to 0x006ce2 (0x42d bytes) 000026 1789264711.037494 write 0x036804 thru 0x03bb50 (0x534d bytes) 000027 1789264711.038694 mapread 0x02a0ec thru 0x02bb6b (0x1a80 bytes) 000028 1789264711.045105 write 0x03d32a thru 0x03ffff (0x2cd6 bytes) 000030 1789264711.046578 punch from 0x01c865 to 0x01edac (0x2548 bytes) 000031 1789264711.050498 write 0x03f300 thru 0x03ffff (0xd00 bytes) 000032 1789264711.051036 trunc from 0x040000 to 0x01ac34 (0x253cb bytes) 000033 1789264711.065541 write 0x010216 thru 0x012aa2 (0x288d bytes) 000035 1789264711.079163 mapread 0x000b30 thru 0x00bc9d (0xb16e bytes) 000036 1789264711.087601 write 0x00bb90 thru 0x019516 (0xd987 bytes) 000038 1789264711.094367 trunc from 0x0170bd to 0x01724e (0x192 bytes) 000039 1789264711.108842 read 0x00d2fc thru 0x016a20 (0x9725 bytes) 000041 1789264711.111512 write 0x027a5b thru 0x031696 (0x9c3c bytes) 000042 1789264711.113334 mapwrite 0x0031af thru 0x00e83f (0xb691 bytes) 000043 1789264711.128068 trunc from 0x031697 to 0x02b01b (0x667b bytes) 000044 1789264711.138918 punch from 0x0022d6 to 0x00484c (0x2577 bytes) 000045 1789264711.145987 trunc from 0x000498 to 0x006f32 (0x6a9b bytes) 000047 1789264711.151796 trunc from 0x02b01c to 0x010f7e (0x1a09d bytes) 000048 1789264711.169202 trunc from 0x010f7f to 0x01aeb4 (0x9f36 bytes) 000049 1789264711.180878 mapread 0x012a35 thru 0x012ad1 (0x9d bytes) 000050 1789264711.196962 mapread 0x000d61 thru 0x001b61 (0xe01 bytes) 000051 1789264711.197813 mapwrite 0x00ba55 thru 0x0148ed (0x8e99 bytes) 000052 1789264711.225078 trunc from 0x01aeb5 to 0x01654d (0x4967 bytes) 000053 1789264711.238646 read 0x011cb1 thru 0x014626 (0x2976 bytes) 000054 1789264711.239807 fallocate 0x022925 thru 0x022ba1 (0x27d bytes) 000055 1789264711.245529 mapwrite 0x00e53f thru 0x010b00 (0x25c2 bytes) 000056 1789264711.254773 trunc from 0x00e3ff to 0x00f8e0 (0x14e2 bytes) 000057 1789264711.259008 trunc from 0x0101d5 to 0x01305f (0x2e8b bytes) fallocating to largest ever: 0x40000 000058 1789264711.265170 fallocate 0x03d963 thru 0x03ffff (0x269d bytes) 000059 1789264711.269379 mapread 0x01d4d4 thru 0x022ba1 (0x56ce bytes) 000060 1789264711.280287 mapread 0x000690 thru 0x008d8d (0x86fe bytes) 000061 1789264711.281305 trunc from 0x022ba2 to 0x0037c0 (0x1f3e1 bytes) 000062 1789264711.295615 mapwrite 0x00a6ee thru 0x012326 (0x7c39 bytes) 000063 1789264711.364025 write 0x008634 thru 0x01472e (0xc0fb bytes) 000064 1789264711.374422 mapread 0x003dee thru 0x00d923 (0x9b36 bytes) 000065 1789264711.394308 MIRROR_EXTEND to 2 mirrors 000066 1789264711.783741 trunc from 0x01472f to 0x00ff1a (0x4814 bytes) 000068 1789264711.833424 trunc from 0x00ff1b to 0x010646 (0x72c bytes) 000069 1789264711.848138 trunc from 0x00099d to 0x002700 (0x1d64 bytes) 000071 1789264711.852763 fallocate 0x0108d2 thru 0x010a2d (0x15c bytes) 000072 1789264711.857716 trunc from 0x010647 to 0x006c74 (0x99d2 bytes) 000073 1789264711.872991 write 0x00bb46 thru 0x01376a (0x7c25 bytes) 000074 1789264711.874361 fallocate 0x01c998 thru 0x025803 (0x8e6c bytes) 000075 1789264711.879910 write 0x029626 thru 0x031061 (0x7a3c bytes) 000076 1789264711.882705 punch from 0x00f905 to 0x016462 (0x6b5e bytes) 000077 1789264711.893468 mapread 0x027327 thru 0x031061 (0x9d3b bytes) 000078 1789264711.905269 read 0x01fbac thru 0x024b26 (0x4f7b bytes) 000079 1789264711.912577 write 0x0280f5 thru 0x02f525 (0x7431 bytes) 000080 1789264711.913659 read 0x006d05 thru 0x009544 (0x2840 bytes) 000081 1789264711.920414 read 0x020d2f thru 0x024f79 (0x424b bytes) 000082 1789264711.921042 punch from 0x013185 to 0x018a0b (0x5887 bytes) 000083 1789264711.925136 mapread 0x0232c3 thru 0x02776a (0x44a8 bytes) 000084 1789264711.926098 mapread 0x0115a6 thru 0x021536 (0xff91 bytes) 000085 1789264711.933318 write 0x034367 thru 0x035d0d (0x19a7 bytes) 000086 1789264711.934628 read 0x027b3c thru 0x03031e (0x87e3 bytes) 000087 1789264711.935243 trunc from 0x027e9b to 0x02aa41 (0x2ba7 bytes) 000088 1789264711.946767 trunc from 0x035d0e to 0x02e31e (0x79ef bytes) 000089 1789264711.958661 MIRROR_RESYNC 2 mirrors 000090 1789264712.326417 punch from 0x007d5f to 0x0148da (0xcb7c bytes) 000091 1789264712.347993 read 0x01f2e2 thru 0x022200 (0x2f1f bytes) 000092 1789264712.352700 read 0x009f30 thru 0x00d861 (0x3932 bytes) 000093 1789264712.360090 trunc from 0x022f8a to 0x025fe9 (0x3060 bytes) 000094 1789264712.363666 trunc from 0x02ba1f to 0x02e31e (0x2900 bytes) 000095 1789264712.368468 trunc from 0x003caf to 0x00f275 (0xb5c7 bytes) 000096 1789264712.371931 punch from 0x026393 to 0x02ca64 (0x66d2 bytes) 000098 1789264712.378106 mapread 0x027145 thru 0x029836 (0x26f2 bytes) 000100 1789264712.392157 read 0x02c3fd thru 0x02e31e (0x1f22 bytes) 000101 1789264712.402203 read 0x01621a thru 0x019121 (0x2f08 bytes) 000102 1789264712.406055 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000103 1789264712.510128 trunc from 0x02e31f to 0x03ef34 (0x10c16 bytes) 000104 1789264712.526923 mapwrite 0x0153a1 thru 0x018dfa (0x3a5a bytes) 000105 1789264712.554084 mapread 0x0241d7 thru 0x033f73 (0xfd9d bytes) 000106 1789264712.569159 write 0x03416d thru 0x037d44 (0x3bd8 bytes) 000107 1789264712.575267 write 0x01e6a2 thru 0x01f23e (0xb9d bytes) 000108 1789264712.576067 punch from 0x00a7d0 to 0x00ba1e (0x124f bytes) 000111 1789264712.582685 trunc from 0x03ef35 to 0x0230f0 (0x1be44 bytes) 000112 1789264712.598010 punch from 0x013324 to 0x0207f1 (0xd4ce bytes) 000113 1789264712.611017 fallocate 0x00ac1e thru 0x011a63 (0x6e46 bytes) 000114 1789264712.615965 read 0x0214f4 thru 0x0230f0 (0x1bfd bytes) 000115 1789264712.616616 trunc from 0x01cf7a to 0x0230f0 (0x6177 bytes) 000116 1789264712.621517 mapread 0x00de53 thru 0x01b336 (0xd4e4 bytes) 000117 1789264712.638637 write 0x00bec9 thru 0x01927f (0xd3b7 bytes) 000118 1789264712.640048 mapread 0x002895 thru 0x00593d (0x30a9 bytes) 000119 1789264712.646674 read 0x022d8f thru 0x0230f0 (0x362 bytes) 000120 1789264712.653898 mapread 0x006cc9 thru 0x014c97 (0xdfcf bytes) 000121 1789264712.654928 mapwrite 0x019906 thru 0x01b233 (0x192e bytes) 000122 1789264712.667639 read 0x01fbff thru 0x0230f0 (0x34f2 bytes) 000123 1789264712.668303 mapread 0x00d481 thru 0x012b33 (0x56b3 bytes) 000125 1789264712.669928 mapread 0x011444 thru 0x015463 (0x4020 bytes) 000126 1789264712.670582 mapread 0x0092c4 thru 0x009535 (0x272 bytes) 000127 1789264712.671406 write 0x029552 thru 0x02e82e (0x52dd bytes) 000130 1789264712.673770 trunc from 0x02e82f to 0x02c4bd (0x2371 bytes) 000131 1789264712.685864 fallocate 0x032a3c thru 0x03fc37 (0xd1fc bytes) 000133 1789264712.690488 trunc from 0x02c4be to 0x018b24 (0x13999 bytes) 000134 1789264712.706160 trunc from 0x018b25 to 0x0014a4 (0x17680 bytes) 000135 1789264712.718246 mapwrite 0x038ffb thru 0x03fb32 (0x6b38 bytes) 000137 1789264712.749554 trunc from 0x02b37f to 0x035c6f (0xa8f1 bytes) 000138 1789264712.757396 trunc from 0x000fef to 0x004120 (0x3132 bytes) 000139 1789264712.765077 trunc from 0x02b509 to 0x039faf (0xeaa7 bytes) 000140 1789264712.770783 trunc from 0x00a62d to 0x012546 (0x7f1a bytes) 000142 1789264712.776640 read 0x01f8e6 thru 0x021ce1 (0x23fc bytes) 000143 1789264712.777306 punch from 0x03a202 to 0x03fb32 (0x5931 bytes) 000146 1789264712.787598 read 0x003a16 thru 0x00d900 (0x9eeb bytes) 000147 1789264712.798092 mapwrite 0x017177 thru 0x026afe (0xf988 bytes) 000148 1789264712.819061 MIRROR_EXTEND to 2 mirrors 000150 1789264713.233501 mapread 0x010f20 thru 0x01da3e (0xcb1f bytes) 000151 1789264713.267623 MIRROR_RESYNC 2 mirrors 000152 1789264713.457709 MIRROR_EXTEND to 3 mirrors 000153 1789264713.865123 mapread 0x023ce4 thru 0x02cf7b (0x9298 bytes) 000154 1789264713.884827 mapwrite 0x01592a thru 0x01ce30 (0x7507 bytes) 000155 1789264713.966687 mapread 0x037487 thru 0x03fb32 (0x86ac bytes) 000156 1789264713.973121 read 0x02de4e thru 0x03434d (0x6500 bytes) 000157 1789264713.979103 read 0x012799 thru 0x0176ad (0x4f15 bytes) 000158 1789264713.979949 mapwrite 0x003be9 thru 0x00f845 (0xbc5d bytes) 000159 1789264714.011130 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 2: 0x8510b2d0 (stale) Mirror 3: 0x8510b2d0 (stale) CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 1: 0xf88a1183 Mirror 4: 0xf88a1183 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb33: skipped 000160 1789264714.560882 read 0x02d9fe thru 0x03113f (0x3742 bytes) 000161 1789264714.571049 mapread 0x011474 thru 0x01b661 (0xa1ee bytes) 000162 1789264714.598563 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 2: 0x8510b2d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 2: 0x8510b2d0 (stale) CRC-32 checksum value for chunk [0, 0x3fb33): Mirror 1: 0xf88a1183 Mirror 4: 0xf88a1183 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb33: skipped 000163 1789264714.999629 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000164 1789264715.230466 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000165 1789264715.324335 trunc from 0x02bf3b to 0x038d44 (0xce0a bytes) 000166 1789264715.333757 fallocate 0x0287ac thru 0x038021 (0xf876 bytes) 000167 1789264715.337532 mapread 0x0385e5 thru 0x03fb32 (0x754e bytes) 000169 1789264715.353086 mapread 0x01d1bd thru 0x02294c (0x5790 bytes) 000170 1789264715.360974 read 0x02e9b5 thru 0x0304ad (0x1af9 bytes) 000171 1789264715.365002 write 0x00f81a thru 0x018e1a (0x9601 bytes) 000172 1789264715.378340 punch from 0x01449c to 0x01b07a (0x6bdf bytes) 000173 1789264715.389221 read 0x0325a6 thru 0x03f88e (0xd2e9 bytes) 000174 1789264715.398706 mapwrite 0x013571 thru 0x01c415 (0x8ea5 bytes) 000175 1789264715.413148 write 0x03a88f thru 0x03af4e (0x6c0 bytes) 000176 1789264715.413991 punch from 0x011669 to 0x013c14 (0x25ac bytes) 000178 1789264715.418959 mapwrite 0x01bfa9 thru 0x01cb92 (0xbea bytes) 000179 1789264715.431783 mapread 0x0341d0 thru 0x03fb32 (0xb963 bytes) 000181 1789264715.433316 mapwrite 0x018fc4 thru 0x01dcc6 (0x4d03 bytes) 000182 1789264715.449446 punch from 0x03b9ff to 0x03fb32 (0x4134 bytes) 000183 1789264715.455299 write 0x01ba42 thru 0x026e7f (0xb43e bytes) 000184 1789264715.457232 mapwrite 0x01df13 thru 0x022c66 (0x4d54 bytes) 000185 1789264715.472032 read 0x02bfda thru 0x02efd6 (0x2ffd bytes) 000187 1789264715.474695 fallocate 0x036120 thru 0x03b33f (0x5220 bytes) 000188 1789264715.478660 fallocate 0x0156e3 thru 0x01adec (0x570a bytes) 000189 1789264715.483364 read 0x00ac32 thru 0x013515 (0x88e4 bytes) 000190 1789264715.491712 fallocate 0x00a22e thru 0x00d098 (0x2e6b bytes) 000191 1789264715.496815 punch from 0x03343f to 0x03ef30 (0xbaf2 bytes) 000193 1789264715.500830 read 0x00ac29 thru 0x0107a6 (0x5b7e bytes) 000194 1789264715.501414 read 0x028dbd thru 0x02f234 (0x6478 bytes) 000196 1789264715.502423 read 0x01af0e thru 0x02aec4 (0xffb7 bytes) 000198 1789264715.508227 write 0x014fab thru 0x01b335 (0x638b bytes) 000199 1789264715.510365 write 0x02add1 thru 0x037110 (0xc340 bytes) 000201 1789264715.516862 mapwrite 0x01ef2d thru 0x02ee6a (0xff3e bytes) 000202 1789264715.536123 read 0x016eb0 thru 0x0221c8 (0xb319 bytes) 000204 1789264715.538351 trunc from 0x03fb33 to 0x00272f (0x3d403 bytes) 000205 1789264715.551403 mapread 0x001d6f thru 0x00272f (0x9c1 bytes) 000206 1789264715.552288 mapread 0x000034 thru 0x00272f (0x26fc bytes) 000207 1789264715.554100 write 0x02285b thru 0x026660 (0x3e06 bytes) 000208 1789264715.555444 trunc from 0x0153e2 to 0x017fe4 (0x2c03 bytes) 000209 1789264715.560451 punch from 0x009921 to 0x00bc3b (0x231b bytes) 000210 1789264715.564410 fallocate 0x02f3fb thru 0x03145d (0x2063 bytes) 000211 1789264715.570027 trunc from 0x026661 to 0x01e577 (0x80e9 bytes) 000212 1789264715.585059 trunc from 0x01e578 to 0x02df7c (0xfa05 bytes) 000213 1789264715.597559 mapread 0x009c98 thru 0x012bb8 (0x8f21 bytes) 000214 1789264715.607099 mapwrite 0x03bd2b thru 0x03ffff (0x42d5 bytes) 000215 1789264715.653143 trunc from 0x020f8f to 0x024336 (0x33a8 bytes) 000217 1789264715.657703 punch from 0x03837b to 0x03ffff (0x7c85 bytes) 000218 1789264715.661722 punch from 0x0024e7 to 0x01033d (0xde57 bytes) 000219 1789264715.665884 trunc from 0x001680 to 0x009d12 (0x8693 bytes) 000220 1789264715.670281 read 0x019066 thru 0x0246af (0xb64a bytes) 000221 1789264715.678141 punch from 0x0121b0 to 0x01e454 (0xc2a5 bytes) 000222 1789264715.684126 read 0x02c268 thru 0x03bb32 (0xf8cb bytes) 000223 1789264715.684987 mapwrite 0x027eec thru 0x02b679 (0x378e bytes) 000224 1789264715.700543 write 0x01a406 thru 0x0224ef (0x80ea bytes) 000225 1789264715.708796 write 0x02c179 thru 0x038739 (0xc5c1 bytes) 000227 1789264715.710494 read 0x03701c thru 0x037052 (0x37 bytes) 000228 1789264715.710889 write 0x032f2f thru 0x03618e (0x3260 bytes) 000229 1789264715.711940 trunc from 0x013cea to 0x014b52 (0xe69 bytes) 000230 1789264715.715368 mapread 0x01134a thru 0x01f005 (0xdcbc bytes) 000231 1789264715.719436 trunc from 0x002699 to 0x00edb5 (0xc71d bytes) 000232 1789264715.723782 mapread 0x00a7b9 thru 0x014ce6 (0xa52e bytes) 000233 1789264715.743083 fallocate 0x00a283 thru 0x00fbd2 (0x5950 bytes) 000235 1789264715.747427 read 0x00c878 thru 0x00eb69 (0x22f2 bytes) 000237 1789264715.748681 trunc from 0x016f66 to 0x01ce74 (0x5f0f bytes) 000239 1789264715.760114 trunc from 0x040000 to 0x01be22 (0x241dd bytes) 000241 1789264715.771277 trunc from 0x01a0be to 0x01be22 (0x1d65 bytes) 000242 1789264715.779416 MIRROR_EXTEND to 2 mirrors 000243 1789264716.092512 write 0x00d4e8 thru 0x0151e9 (0x7d02 bytes) 000244 1789264716.116562 mapwrite 0x02fcdc thru 0x032c05 (0x2f2a bytes) 000245 1789264716.176044 trunc from 0x01b8b7 to 0x028b50 (0xd29a bytes) 000246 1789264716.181168 trunc from 0x01eda7 to 0x027cb4 (0x8f0e bytes) 000247 1789264716.190377 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32c06): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c06): Mirror 2: 0xf0d707b2 (stale) CRC-32 checksum value for chunk [0, 0x32c06): Mirror 1: 0xdf6e9a97 Mirror 3: 0xdf6e9a97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c06: skipped 000248 1789264716.648899 mapwrite 0x009334 thru 0x00f7ac (0x6479 bytes) 000250 1789264716.729954 read 0x029107 thru 0x032c05 (0x9aff bytes) 000251 1789264716.738246 trunc from 0x0239ed to 0x02cae4 (0x90f8 bytes) 000252 1789264716.743105 mapwrite 0x02e183 thru 0x02f8e5 (0x1763 bytes) 000253 1789264716.754144 trunc from 0x007fc2 to 0x00cc0b (0x4c4a bytes) 000254 1789264716.765416 MIRROR_RESYNC 3 mirrors 000255 1789264717.225151 fallocate 0x028bb0 thru 0x02acb8 (0x2109 bytes) 000256 1789264717.244254 trunc from 0x01fd83 to 0x02911d (0x939b bytes) 000257 1789264717.247479 trunc from 0x000ee8 to 0x010ab9 (0xfbd2 bytes) 000258 1789264717.251164 fallocate 0x00ab5c thru 0x018867 (0xdd0c bytes) 000259 1789264717.255106 trunc from 0x032c06 to 0x032a76 (0x18f bytes) 000260 1789264717.271433 write 0x03faa0 thru 0x03ffff (0x560 bytes) 000261 1789264717.272592 punch from 0x00c933 to 0x013312 (0x69e0 bytes) 000262 1789264717.276892 punch from 0x01d90e to 0x02790c (0x9fff bytes) 000263 1789264717.281363 write 0x03cc39 thru 0x03ffff (0x33c7 bytes) 000264 1789264717.286612 trunc from 0x040000 to 0x018f17 (0x270e8 bytes) 000265 1789264717.295444 mapwrite 0x023cf5 thru 0x024063 (0x36f bytes) 000266 1789264717.324175 punch from 0x0142fa to 0x01f93b (0xb642 bytes) 000267 1789264717.334345 MIRROR_RESYNC 3 mirrors 000268 1789264717.717532 mapread 0x016d4d thru 0x024063 (0xd317 bytes) 000269 1789264717.732037 write 0x002105 thru 0x009406 (0x7302 bytes) 000270 1789264717.754406 fallocate 0x01c78a thru 0x02916d (0xc9e4 bytes) 000271 1789264717.758167 fallocate 0x02b09a thru 0x033dba (0x8d21 bytes) 000272 1789264717.762917 mapwrite 0x00fa01 thru 0x01172b (0x1d2b bytes) 000273 1789264717.782056 write 0x00a5e3 thru 0x016406 (0xbe24 bytes) 000274 1789264717.783392 punch from 0x01a87d to 0x02011b (0x589f bytes) 000275 1789264717.786296 trunc from 0x0171ea to 0x022961 (0xb778 bytes) 000276 1789264717.790062 punch from 0x01cca5 to 0x024063 (0x73bf bytes) 000277 1789264717.793861 trunc from 0x024064 to 0x01c0cf (0x7f94 bytes) 000278 1789264717.816940 punch from 0x00673c to 0x00e047 (0x790c bytes) 000279 1789264717.827344 punch from 0x00ad53 to 0x018aac (0xdd5a bytes) 000280 1789264717.830548 trunc from 0x01c0d0 to 0x02e2d4 (0x12205 bytes) 000281 1789264717.843779 write 0x03638c thru 0x03ffff (0x9c74 bytes) 000282 1789264717.845884 trunc from 0x0277b0 to 0x03084d (0x909e bytes) 000286 1789264717.850808 fallocate 0x025432 thru 0x02558e (0x15d bytes) 000287 1789264717.855675 trunc from 0x0100ff to 0x0150b2 (0x4fb4 bytes) 000288 1789264717.859966 punch from 0x001ab0 to 0x00c51c (0xaa6d bytes) 000289 1789264717.865137 punch from 0x0180b3 to 0x018482 (0x3d0 bytes) 000290 1789264717.870590 read 0x03de10 thru 0x03ffff (0x21f0 bytes) 000291 1789264717.871169 read 0x015f83 thru 0x01c5ec (0x666a bytes) 000292 1789264717.881882 mapwrite 0x0370b1 thru 0x03a6ad (0x35fd bytes) 000293 1789264717.900790 trunc from 0x01e556 to 0x0219d8 (0x3483 bytes) 000294 1789264717.905043 trunc from 0x023737 to 0x02dbb3 (0xa47d bytes) 000295 1789264717.909854 read 0x0232be thru 0x02b096 (0x7dd9 bytes) 000296 1789264717.918866 trunc from 0x040000 to 0x012996 (0x2d669 bytes) 000298 1789264717.931526 write 0x00cc79 thru 0x01b1ac (0xe534 bytes) 000299 1789264717.933756 mapread 0x00f901 thru 0x019857 (0x9f57 bytes) 000300 1789264717.934872 mapread 0x00a0bd thru 0x011df7 (0x7d3b bytes) 000301 1789264717.936019 write 0x032b1d thru 0x03ffff (0xd4e3 bytes) 000303 1789264717.938330 mapread 0x02a806 thru 0x03a78a (0xff85 bytes) 000304 1789264717.968090 write 0x002d43 thru 0x0086b6 (0x5974 bytes) 000305 1789264717.969661 fallocate 0x014f7c thru 0x017a43 (0x2ac8 bytes) 000306 1789264717.974851 fallocate 0x004c13 thru 0x007893 (0x2c81 bytes) 000307 1789264717.979986 trunc from 0x03a321 to 0x03f16e (0x4e4e bytes) 000309 1789264717.998296 MIRROR_RESYNC 3 mirrors 000311 1789264718.541807 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000312 1789264718.723288 mapread 0x0072a2 thru 0x0114f0 (0xa24f bytes) 000313 1789264718.739363 read 0x01a9b1 thru 0x02907c (0xe6cc bytes) 000314 1789264718.749944 read 0x00ed49 thru 0x012aa8 (0x3d60 bytes) 000315 1789264718.756979 punch from 0x0090a7 to 0x010383 (0x72dd bytes) 000316 1789264718.779104 trunc from 0x040000 to 0x032d10 (0xd2ef bytes) 000317 1789264718.799383 read 0x0090d3 thru 0x009bf3 (0xb21 bytes) 000318 1789264718.808890 fallocate 0x02a503 thru 0x02aa8f (0x58d bytes) 000319 1789264718.813274 mapread 0x021f23 thru 0x02c382 (0xa460 bytes) 000320 1789264718.815156 mapwrite 0x021ee1 thru 0x031258 (0xf378 bytes) 000321 1789264718.833100 mapread 0x00b003 thru 0x00dcdf (0x2cdd bytes) 000322 1789264718.834019 write 0x00f702 thru 0x01f44f (0xfd4e bytes) 000323 1789264718.835596 mapread 0x0232cd thru 0x02b0a2 (0x7dd6 bytes) 000324 1789264718.839174 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32d11): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d11): Mirror 2: 0x91be7ca7 (stale) CRC-32 checksum value for chunk [0, 0x32d11): Mirror 1: 0xf34feab4 Mirror 3: 0xf34feab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d11: skipped 000325 1789264719.322614 write 0x003606 thru 0x00a5e6 (0x6fe1 bytes) 000326 1789264719.349767 punch from 0x026748 to 0x02f492 (0x8d4b bytes) 000327 1789264719.353999 trunc from 0x032d11 to 0x0339ac (0xc9c bytes) 000328 1789264719.366885 mapwrite 0x01219d thru 0x01bab6 (0x991a bytes) 000329 1789264719.395843 fallocate 0x011a77 thru 0x0164ec (0x4a76 bytes) 000330 1789264719.399982 trunc from 0x013298 to 0x0213fc (0xe165 bytes) 000331 1789264719.404613 fallocate 0x01ecfd thru 0x024fda (0x62de bytes) 000332 1789264719.409496 mapwrite 0x0085c3 thru 0x00f0c9 (0x6b07 bytes) 000333 1789264719.433343 mapwrite 0x01e1b8 thru 0x02bb86 (0xd9cf bytes) 000334 1789264719.459899 mapwrite 0x003776 thru 0x00fa9a (0xc325 bytes) 000335 1789264719.479825 punch from 0x02fab4 to 0x0339ac (0x3ef9 bytes) 000336 1789264719.483658 trunc from 0x00a39b to 0x00e98c (0x45f2 bytes) 000337 1789264719.488559 read 0x030336 thru 0x0339ac (0x3677 bytes) 000338 1789264719.496793 mapwrite 0x01bb68 thru 0x022c25 (0x70be bytes) 000339 1789264719.511137 mapwrite 0x01575d thru 0x01a047 (0x48eb bytes) 000340 1789264719.528840 fallocate 0x009416 thru 0x01100c (0x7bf7 bytes) 000341 1789264719.532944 punch from 0x008674 to 0x00aec4 (0x2851 bytes) 000342 1789264719.536994 punch from 0x00309e to 0x00594e (0x28b1 bytes) 000343 1789264719.541377 mapwrite 0x00d366 thru 0x014fdf (0x7c7a bytes) 000344 1789264719.561838 punch from 0x02637e to 0x0339ac (0xd62f bytes) 000345 1789264719.567886 trunc from 0x001199 to 0x00ea05 (0xd86d bytes) 000346 1789264719.572274 trunc from 0x0339ad to 0x00a014 (0x29998 bytes) 000347 1789264719.582758 mapread 0x00544c thru 0x00a014 (0x4bc9 bytes) 000348 1789264719.588843 fallocate 0x028097 thru 0x03053a (0x84a4 bytes) 000349 1789264719.592452 mapwrite 0x032e8f thru 0x0384df (0x5651 bytes) 000351 1789264719.623087 mapwrite 0x0229d9 thru 0x02ab92 (0x81ba bytes) 000352 1789264719.643117 punch from 0x02edb1 to 0x0384df (0x972f bytes) 000354 1789264719.647064 punch from 0x02a793 to 0x0343a7 (0x9c15 bytes) 000355 1789264719.655607 MIRROR_RESYNC 3 mirrors 000356 1789264720.072780 write 0x0176b1 thru 0x0224da (0xae2a bytes) 000357 1789264720.103385 punch from 0x0133ce to 0x01dd97 (0xa9ca bytes) 000358 1789264720.117856 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x384e0): Mirror 3: 0x40f8524d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x384e0): Mirror 3: 0x40f8524d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x384e0: skipped 000359 1789264720.282006 mapread 0x02b8d3 thru 0x032ac7 (0x71f5 bytes) 000360 1789264720.299785 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000361 1789264720.398494 write 0x01acec thru 0x01fa94 (0x4da9 bytes) 000362 1789264720.411575 punch from 0x0316e4 to 0x0321ce (0xaeb bytes) 000363 1789264720.415652 fallocate 0x020182 thru 0x027e70 (0x7cef bytes) 000364 1789264720.419051 trunc from 0x003377 to 0x00c1e1 (0x8e6b bytes) 000365 1789264720.423355 mapwrite 0x029c41 thru 0x039c22 (0xffe2 bytes) 000366 1789264720.485826 trunc from 0x039c23 to 0x0240c1 (0x15b61 bytes) 000367 1789264720.502715 MIRROR_EXTEND to 2 mirrors 000368 1789264720.907661 fallocate 0x03a549 thru 0x03ace8 (0x7a0 bytes) 000369 1789264720.928117 mapwrite 0x03cd13 thru 0x03ffff (0x32ed bytes) 000370 1789264720.972142 trunc from 0x020121 to 0x02c90c (0xc7ec bytes) 000371 1789264720.976483 mapread 0x00ddc6 thru 0x013fef (0x622a bytes) 000372 1789264720.982793 mapwrite 0x0314e4 thru 0x03ffff (0xeb1c bytes) 000373 1789264721.010660 fallocate 0x00e524 thru 0x01e127 (0xfc04 bytes) 000374 1789264721.015251 punch from 0x028e20 to 0x032920 (0x9b01 bytes) 000375 1789264721.020736 fallocate 0x027bc4 thru 0x036521 (0xe95e bytes) 000376 1789264721.026175 mapread 0x01cc46 thru 0x0289c2 (0xbd7d bytes) 000377 1789264721.039175 write 0x03ec4a thru 0x03ffff (0x13b6 bytes) 000378 1789264721.040272 mapread 0x03ddfe thru 0x03ffff (0x2202 bytes) 000379 1789264721.040950 mapwrite 0x020999 thru 0x027936 (0x6f9e bytes) 000380 1789264721.060855 mapread 0x02a642 thru 0x02b824 (0x11e3 bytes) 000381 1789264721.061888 punch from 0x014a2d to 0x019401 (0x49d5 bytes) 000382 1789264721.067971 mapread 0x017287 thru 0x01b6c0 (0x443a bytes) 000383 1789264721.074734 read 0x02742c thru 0x030d8b (0x9960 bytes) 000384 1789264721.084160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000385 1789264721.180879 mapread 0x02caa3 thru 0x02fa7c (0x2fda bytes) 000386 1789264721.197844 punch from 0x0078f8 to 0x00d505 (0x5c0e bytes) 000387 1789264721.202036 punch from 0x00fdcf to 0x01d80e (0xda40 bytes) 000388 1789264721.206925 punch from 0x016859 to 0x02417c (0xd924 bytes) 000389 1789264721.212225 mapread 0x00860a thru 0x00c682 (0x4079 bytes) 000390 1789264721.222955 write 0x015fb5 thru 0x022793 (0xc7df bytes) 000392 1789264721.235133 read 0x000ff9 thru 0x00fce0 (0xece8 bytes) 000393 1789264721.240869 trunc from 0x040000 to 0x035300 (0xacff bytes) 000394 1789264721.255010 mapwrite 0x03c5b0 thru 0x03ffff (0x3a50 bytes) 000395 1789264721.288623 mapwrite 0x027177 thru 0x02d41c (0x62a6 bytes) 000396 1789264721.305412 trunc from 0x040000 to 0x00fb4b (0x304b4 bytes) 000397 1789264721.324244 trunc from 0x00013b to 0x0018f8 (0x17be bytes) 000398 1789264721.328829 trunc from 0x00fb4c to 0x00bb79 (0x3fd2 bytes) 000399 1789264721.339306 mapwrite 0x019895 thru 0x028a30 (0xf19c bytes) 000400 1789264721.383109 trunc from 0x028a31 to 0x032b47 (0xa117 bytes) 000401 1789264721.394775 write 0x013772 thru 0x01fbbe (0xc44d bytes) 000402 1789264721.396226 read 0x003ab6 thru 0x00e3fc (0xa947 bytes) 000403 1789264721.404218 fallocate 0x0006c4 thru 0x00da6c (0xd3a9 bytes) 000404 1789264721.412792 trunc from 0x032b48 to 0x01c030 (0x16b17 bytes) 000405 1789264721.429283 write 0x0009bc thru 0x00fc66 (0xf2ab bytes) 000406 1789264721.430946 mapread 0x015d0d thru 0x01c030 (0x6324 bytes) 000407 1789264721.431805 write 0x029ffe thru 0x036c64 (0xcc67 bytes) 000408 1789264721.433310 trunc from 0x036c65 to 0x01be42 (0x1ae22 bytes) 000409 1789264721.444169 punch from 0x00999e to 0x0134b8 (0x9b1b bytes) 000410 1789264721.459030 punch from 0x01ac63 to 0x01be42 (0x11e0 bytes) 000411 1789264721.463021 mapread 0x00e63d thru 0x01a452 (0xbe16 bytes) 000413 1789264721.489890 punch from 0x0188d4 to 0x01be42 (0x356f bytes) 000414 1789264721.493934 read 0x0166a8 thru 0x01be42 (0x579b bytes) 000415 1789264721.499922 punch from 0x013af0 to 0x01be42 (0x8353 bytes) 000416 1789264721.504796 punch from 0x009ab3 to 0x00e4ed (0x4a3b bytes) 000418 1789264721.509834 trunc from 0x014db6 to 0x019267 (0x44b2 bytes) 000419 1789264721.515604 mapread 0x014e6d thru 0x01be42 (0x6fd6 bytes) 000420 1789264721.551555 write 0x006aed thru 0x013040 (0xc554 bytes) 000421 1789264721.558108 punch from 0x00c6c7 to 0x016768 (0xa0a2 bytes) 000422 1789264721.571977 mapread 0x0133e6 thru 0x01be42 (0x8a5d bytes) 000423 1789264721.592813 MIRROR_EXTEND to 2 mirrors 000424 1789264721.975130 trunc from 0x01be43 to 0x020d2b (0x4ee9 bytes) 000425 1789264722.016200 read 0x0033ab thru 0x00fefb (0xcb51 bytes) 000427 1789264722.025732 read 0x008a3d thru 0x017e72 (0xf436 bytes) 000428 1789264722.035060 trunc from 0x0008d7 to 0x004543 (0x3c6d bytes) 000429 1789264722.038636 write 0x021279 thru 0x02ef33 (0xdcbb bytes) 000431 1789264722.040780 mapread 0x00bc15 thru 0x016c66 (0xb052 bytes) 000432 1789264722.041852 punch from 0x01b0ea to 0x01cae5 (0x19fc bytes) 000433 1789264722.045147 mapread 0x000697 thru 0x00821d (0x7b87 bytes) 000434 1789264722.063997 mapread 0x00db3e thru 0x012fd6 (0x5499 bytes) 000435 1789264722.064803 write 0x00e11e thru 0x018004 (0x9ee7 bytes) 000436 1789264722.065926 read 0x00fc41 thru 0x019102 (0x94c2 bytes) 000437 1789264722.066640 fallocate 0x025022 thru 0x02ddb5 (0x8d94 bytes) 000438 1789264722.070418 punch from 0x00bb43 to 0x013457 (0x7915 bytes) 000439 1789264722.082581 punch from 0x00e1ab to 0x01852e (0xa384 bytes) 000440 1789264722.087907 write 0x024a8e thru 0x028245 (0x37b8 bytes) 000441 1789264722.088865 fallocate 0x03a6c9 thru 0x03d9c5 (0x32fd bytes) 000442 1789264722.092706 trunc from 0x02ef34 to 0x033a41 (0x4b0e bytes) 000443 1789264722.109737 write 0x000bab thru 0x00e875 (0xdccb bytes) 000444 1789264722.114505 write 0x0000ab thru 0x00f828 (0xf77e bytes) 000445 1789264722.125121 write 0x0369fd thru 0x03ffff (0x9603 bytes) 000446 1789264722.126408 trunc from 0x040000 to 0x02c625 (0x139da bytes) 000447 1789264722.137800 trunc from 0x02c626 to 0x00fe53 (0x1c7d2 bytes) 000448 1789264722.155658 write 0x01650e thru 0x01975d (0x3250 bytes) 000449 1789264722.157534 mapread 0x006c73 thru 0x015281 (0xe60f bytes) 000450 1789264722.180788 punch from 0x00e1ce to 0x00fcfa (0x1b2d bytes) 000451 1789264722.192803 trunc from 0x01975e to 0x03a8a5 (0x21148 bytes) 000453 1789264722.203285 write 0x01d735 thru 0x01dd38 (0x604 bytes) 000454 1789264722.213491 read 0x004acc thru 0x00d976 (0x8eab bytes) 000455 1789264722.214285 read 0x01ffaf thru 0x02101c (0x106e bytes) 000456 1789264722.225482 write 0x00704d thru 0x00e0d9 (0x708d bytes) 000457 1789264722.226787 trunc from 0x03a8a6 to 0x02a53e (0x10367 bytes) 000459 1789264722.246741 MIRROR_RESYNC 2 mirrors 000460 1789264722.682114 mapread 0x0283cf thru 0x02a53e (0x2170 bytes) 000461 1789264722.700829 trunc from 0x02a53f to 0x006f99 (0x235a5 bytes) 000462 1789264722.755489 punch from 0x002215 to 0x0038f6 (0x16e2 bytes) 000463 1789264722.760798 mapwrite 0x006f9e thru 0x00b158 (0x41bb bytes) 000464 1789264722.813332 mapread 0x007d0d thru 0x00b158 (0x344c bytes) 000465 1789264722.814366 trunc from 0x003f13 to 0x00b158 (0x7246 bytes) 000466 1789264722.818383 mapread 0x007f6b thru 0x008b11 (0xba7 bytes) 000467 1789264722.823910 write 0x013070 thru 0x015a73 (0x2a04 bytes) 000468 1789264722.828853 MIRROR_RESYNC 2 mirrors 000469 1789264723.223967 punch from 0x0010d1 to 0x00125f (0x18f bytes) 000470 1789264723.245957 mapwrite 0x03ecf3 thru 0x03ffff (0x130d bytes) 000472 1789264723.280167 mapwrite 0x00d1c6 thru 0x011276 (0x40b1 bytes) 000473 1789264723.304798 punch from 0x02ad1a to 0x03a642 (0xf929 bytes) 000475 1789264723.309072 mapwrite 0x023b0e thru 0x02b8bf (0x7db2 bytes) 000476 1789264723.330170 trunc from 0x021af5 to 0x029cdc (0x81e8 bytes) 000477 1789264723.337089 read 0x00b2e4 thru 0x00f44d (0x416a bytes) 000478 1789264723.342505 MIRROR_RESYNC 2 mirrors 000479 1789264723.808435 fallocate 0x00c9ea thru 0x012a1f (0x6036 bytes) 000480 1789264723.828459 fallocate 0x016f75 thru 0x01fd6b (0x8df7 bytes) 000481 1789264723.833359 punch from 0x03c0ab to 0x03ffff (0x3f55 bytes) 000482 1789264723.838523 punch from 0x031ab0 to 0x036534 (0x4a85 bytes) 000483 1789264723.842742 punch from 0x013bc0 to 0x01895d (0x4d9e bytes) 000485 1789264723.847528 trunc from 0x03040e to 0x032235 (0x1e28 bytes) 000486 1789264723.852717 write 0x0332ab thru 0x03dcf6 (0xaa4c bytes) 000487 1789264723.861453 read 0x02ae9e thru 0x02dde2 (0x2f45 bytes) 000489 1789264723.868724 read 0x0039b7 thru 0x01396e (0xffb8 bytes) 000490 1789264723.880717 mapwrite 0x00737e thru 0x0163c0 (0xf043 bytes) 000491 1789264723.897708 fallocate 0x0168ec thru 0x0186b3 (0x1dc8 bytes) 000492 1789264723.901295 mapwrite 0x036baf thru 0x03ffff (0x9451 bytes) 000493 1789264723.916394 punch from 0x03e068 to 0x03ffff (0x1f98 bytes) 000494 1789264723.923039 punch from 0x03cd60 to 0x03d4bd (0x75e bytes) 000495 1789264723.931778 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4735e45a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7c3349b1 Mirror 3: 0x7c3349b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000496 1789264724.493588 punch from 0x0369ce to 0x03ffff (0x9632 bytes) 000497 1789264724.518224 write 0x02f96a thru 0x035b5d (0x61f4 bytes) 000498 1789264724.532480 mapwrite 0x00e5e4 thru 0x0151bc (0x6bd9 bytes) 000499 1789264724.572113 MIRROR_RESYNC 3 mirrors 000500 1789264725.222997 trunc from 0x0260cd to 0x02d734 (0x7668 bytes) 000501 1789264725.255818 trunc from 0x040000 to 0x0329f0 (0xd60f bytes) 000502 1789264725.272334 trunc from 0x0329f1 to 0x0099de (0x29012 bytes) 000503 1789264725.285104 punch from 0x002fb5 to 0x008865 (0x58b1 bytes) 000504 1789264725.290651 trunc from 0x00910e to 0x0099de (0x8d1 bytes) 000505 1789264725.295507 mapread 0x001508 thru 0x0099de (0x84d7 bytes) 000506 1789264725.311634 read 0x007346 thru 0x0099de (0x2699 bytes) 000507 1789264725.313599 punch from 0x0037f9 to 0x0099de (0x61e6 bytes) 000508 1789264725.318638 punch from 0x0093f3 to 0x0099de (0x5ec bytes) 000509 1789264725.322989 fallocate 0x00e704 thru 0x0142f8 (0x5bf5 bytes) 000510 1789264725.326991 trunc from 0x0142f9 to 0x01388e (0xa6a bytes) 000511 1789264725.341909 fallocate 0x03ee0d thru 0x03ffff (0x11f3 bytes) 000513 1789264725.348175 trunc from 0x040000 to 0x0280c8 (0x17f37 bytes) 000514 1789264725.363806 trunc from 0x00b5bc to 0x014fc4 (0x9a09 bytes) 000515 1789264725.376155 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x280c9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x280c9): Mirror 2: 0x81db8b42 (stale) Mirror 3: 0x81db8b42 (stale) CRC-32 checksum value for chunk [0, 0x280c9): Mirror 1: 0x963a7d27 Mirror 4: 0x963a7d27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x280c9: skipped 000516 1789264725.914384 mapread 0x02704b thru 0x0280c8 (0x107e bytes) 000517 1789264725.935498 trunc from 0x00f73f to 0x01d8c3 (0xe185 bytes) 000518 1789264725.958666 fallocate 0x000c32 thru 0x009b65 (0x8f34 bytes) 000519 1789264725.965871 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x280c9): Mirror 2: 0x81db8b42 (stale) Mirror 4: 0x963a7d27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x280c9): Mirror 2: 0x81db8b42 (stale) Mirror 4: 0x963a7d27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x280c9: skipped 000520 1789264726.150850 trunc from 0x003b08 to 0x005701 (0x1bfa bytes) 000521 1789264726.154991 fallocate 0x03bdc4 thru 0x03ffff (0x423c bytes) 000522 1789264726.159994 write 0x01cfc5 thru 0x0225a2 (0x55de bytes) 000523 1789264726.171112 punch from 0x01c642 to 0x0280c8 (0xba87 bytes) 000524 1789264726.182382 fallocate 0x03e428 thru 0x03ffff (0x1bd8 bytes) 000525 1789264726.185811 mapwrite 0x03f80a thru 0x03ffff (0x7f6 bytes) 000526 1789264726.218388 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2472e9e1 (stale) Mirror 4: 0x9f412528 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb547c91b Mirror 5: 0xb547c91b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000527 1789264726.749552 trunc from 0x005086 to 0x00ca3d (0x79b8 bytes) 000528 1789264726.776734 MIRROR_RESYNC 4 mirrors 000529 1789264727.321345 mapwrite 0x002c44 thru 0x00d4d1 (0xa88e bytes) 000530 1789264727.384080 read 0x007d13 thru 0x015cc8 (0xdfb6 bytes) 000531 1789264727.389577 mapwrite 0x00d710 thru 0x0109eb (0x32dc bytes) 000532 1789264727.404586 read 0x03e0b1 thru 0x03ffff (0x1f4f bytes) 000533 1789264727.411122 read 0x019758 thru 0x023041 (0x98ea bytes) 000534 1789264727.414540 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb547c91b (stale) Mirror 4: 0xb547c91b (stale) Mirror 5: 0xb547c91b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xebcbd935 Mirror 6: 0xebcbd935 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000535 1789264727.947325 mapread 0x01f1fc thru 0x01f82c (0x631 bytes) 000536 1789264727.951166 trunc from 0x040000 to 0x02527d (0x1ad82 bytes) 000537 1789264728.001589 MIRROR_RESYNC 5 mirrors 000538 1789264728.611346 MIRROR_RESYNC 5 mirrors 000539 1789264728.871736 read 0x00fc75 thru 0x0167ce (0x6b5a bytes) 000540 1789264728.877326 mapread 0x00c317 thru 0x0175e6 (0xb2d0 bytes) 000541 1789264728.896372 trunc from 0x00b007 to 0x0165ca (0xb5c4 bytes) 000542 1789264728.928355 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2527e): Mirror 2: 0xeedd9bdc (stale) Mirror 5: 0xeedd9bdc (stale) Mirror 6: 0xeedd9bdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2527e): Mirror 2: 0xeedd9bdc (stale) Mirror 5: 0xeedd9bdc (stale) Mirror 6: 0xeedd9bdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2527e: skipped 000543 1789264729.148899 fallocate 0x03ea18 thru 0x03ffff (0x15e8 bytes) 000544 1789264729.154075 mapread 0x01a2a1 thru 0x02527d (0xafdd bytes) 000545 1789264729.181595 read 0x018de8 thru 0x0216f4 (0x890d bytes) 000546 1789264729.182160 read 0x01dcac thru 0x02527d (0x75d2 bytes) 000547 1789264729.182627 write 0x039062 thru 0x03ffff (0x6f9e bytes) 000548 1789264729.186936 trunc from 0x00005b to 0x001258 (0x11fe bytes) 000549 1789264729.191379 fallocate 0x0087e9 thru 0x012d89 (0xa5a1 bytes) 000550 1789264729.198911 MIRROR_RESYNC 4 mirrors 000551 1789264729.778797 mapwrite 0x001118 thru 0x00fbee (0xead7 bytes) 000553 1789264729.855666 fallocate 0x038f77 thru 0x03ffff (0x7089 bytes) 000554 1789264729.860617 mapwrite 0x01476d thru 0x01b241 (0x6ad5 bytes) 000555 1789264729.888016 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa77186df (stale) Mirror 5: 0xa77186df (stale) Mirror 6: 0xa77186df (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd1bd88a Mirror 7: 0xfd1bd88a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000556 1789264730.343556 mapread 0x00f3b7 thru 0x01455d (0x51a7 bytes) 000558 1789264730.357539 mapread 0x03d712 thru 0x03ffff (0x28ee bytes) 000560 1789264730.363974 mapread 0x036920 thru 0x03f718 (0x8df9 bytes) 000561 1789264730.364899 read 0x02abe0 thru 0x032cdc (0x80fd bytes) 000564 1789264730.375756 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa77186df (stale) Mirror 5: 0xa77186df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa77186df (stale) Mirror 5: 0xa77186df (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd1bd88a Mirror 7: 0xfd1bd88a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000565 1789264730.729369 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa77186df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa77186df (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd1bd88a Mirror 7: 0xfd1bd88a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000566 1789264731.034307 mapwrite 0x030764 thru 0x0372a6 (0x6b43 bytes) 000567 1789264731.100611 mapread 0x02b0f5 thru 0x02da9f (0x29ab bytes) 000568 1789264731.101790 mapread 0x001055 thru 0x009fe9 (0x8f95 bytes) 000569 1789264731.109039 write 0x00f380 thru 0x01d023 (0xdca4 bytes) 000571 1789264731.118876 MIRROR_RESYNC 3 mirrors 000572 1789264731.592050 trunc from 0x00a6d4 to 0x014799 (0xa0c6 bytes) 000573 1789264731.613233 read 0x03d159 thru 0x03f916 (0x27be bytes) 000574 1789264731.616747 punch from 0x01711c to 0x02488c (0xd771 bytes) 000576 1789264731.621102 punch from 0x01234d to 0x02057e (0xe232 bytes) 000577 1789264731.624504 mapread 0x007a8e thru 0x0081a0 (0x713 bytes) 000578 1789264731.631616 write 0x02cf03 thru 0x032a7d (0x5b7b bytes) 000580 1789264731.639212 fallocate 0x00476f thru 0x01354a (0xeddc bytes) 000581 1789264731.642756 mapread 0x0395bd thru 0x03ffff (0x6a43 bytes) 000583 1789264731.649703 mapwrite 0x032bf5 thru 0x03ffff (0xd40b bytes) 000584 1789264731.664966 punch from 0x03431f to 0x03ffff (0xbce1 bytes) 000585 1789264731.669614 write 0x03916b thru 0x03ffff (0x6e95 bytes) 000586 1789264731.674483 fallocate 0x000255 thru 0x008835 (0x85e1 bytes) 000587 1789264731.678980 trunc from 0x040000 to 0x00ca35 (0x335ca bytes) 000590 1789264731.705126 trunc from 0x00ca36 to 0x00b330 (0x1705 bytes) 000591 1789264731.715118 mapwrite 0x024a0c thru 0x02c9f5 (0x7fea bytes) 000592 1789264731.746707 trunc from 0x02c9f6 to 0x011b1a (0x1aedb bytes) 000593 1789264731.760927 mapread 0x006a57 thru 0x011b1a (0xb0c4 bytes) 000596 1789264731.775003 write 0x03f3c2 thru 0x03ffff (0xc3e bytes) 000597 1789264731.776013 read 0x0022cf thru 0x00a5ea (0x831c bytes) 000598 1789264731.776613 trunc from 0x040000 to 0x0167a4 (0x2985b bytes) 000599 1789264731.791773 MIRROR_RESYNC 3 mirrors 000601 1789264732.133712 trunc from 0x0167a5 to 0x03eada (0x28336 bytes) 000602 1789264732.172842 mapread 0x037625 thru 0x03cf01 (0x58dd bytes) 000603 1789264732.188694 trunc from 0x03eadb to 0x025281 (0x19859 bytes) 000604 1789264732.203065 trunc from 0x02011c to 0x0234ba (0x339f bytes) 000605 1789264732.207618 mapread 0x00bb44 thru 0x01b4be (0xf97b bytes) 000606 1789264732.229511 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25282): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25282): Mirror 5: 0x9ff901f7 (stale) Mirror 7: 0x9ff901f7 (stale) CRC-32 checksum value for chunk [0, 0x25282): Mirror 1: 0x9ff901f7 Mirror 8: 0x9ff901f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25282: skipped 000607 1789264732.593819 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25282): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25282): Mirror 5: 0x9ff901f7 (stale) Mirror 7: 0x9ff901f7 (stale) CRC-32 checksum value for chunk [0, 0x25282): Mirror 1: 0x9ff901f7 Mirror 8: 0x9ff901f7 Mirror 9: 0x9ff901f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25282: skipped 000608 1789264733.026807 trunc from 0x025282 to 0x03ae62 (0x15be1 bytes) 000610 1789264733.068602 read 0x01f620 thru 0x029375 (0x9d56 bytes) 000612 1789264733.074717 punch from 0x0202c7 to 0x023e4c (0x3b86 bytes) 000613 1789264733.077816 punch from 0x02a787 to 0x032ebc (0x8736 bytes) 000614 1789264733.081650 punch from 0x03adf6 to 0x03ae62 (0x6d bytes) 000615 1789264733.084403 read 0x0298de thru 0x02e12b (0x484e bytes) 000616 1789264733.091778 trunc from 0x03ae63 to 0x022ad3 (0x1838f bytes) 000617 1789264733.107758 mapwrite 0x0295ae thru 0x032ffb (0x9a4e bytes) 000618 1789264733.164386 punch from 0x01a051 to 0x01d6c3 (0x3673 bytes) 000619 1789264733.167954 trunc from 0x020444 to 0x022b9e (0x275b bytes) 000620 1789264733.172287 mapread 0x01345d thru 0x013ba8 (0x74c bytes) 000621 1789264733.173028 read 0x0217f7 thru 0x02924c (0x7a56 bytes) 000622 1789264733.179887 fallocate 0x026504 thru 0x030f89 (0xaa86 bytes) 000623 1789264733.183790 trunc from 0x032ffc to 0x039fee (0x6ff3 bytes) 000624 1789264733.196907 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39fef): Mirror 5: 0xd94de0d6 (stale) Mirror 7: 0xd94de0d6 (stale) Mirror 8: 0xd94de0d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39fef): Mirror 5: 0xd94de0d6 (stale) Mirror 7: 0xd94de0d6 (stale) Mirror 8: 0xd94de0d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39fef: skipped 000625 1789264733.398731 trunc from 0x00b6c3 to 0x017ff1 (0xc92f bytes) 000627 1789264733.404296 write 0x007714 thru 0x00fc29 (0x8516 bytes) 000628 1789264733.421345 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39fef): Mirror 7: 0xd94de0d6 (stale) Mirror 8: 0xd94de0d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39fef): Mirror 7: 0xd94de0d6 (stale) Mirror 8: 0xd94de0d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39fef: skipped 000629 1789264733.599295 trunc from 0x033bb7 to 0x039fee (0x6438 bytes) 000631 1789264733.607441 mapwrite 0x02599a thru 0x03349d (0xdb04 bytes) 000632 1789264733.641714 write 0x037a13 thru 0x03ffff (0x85ed bytes) 000633 1789264733.652581 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x351a43b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x351a43b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000634 1789264733.821490 trunc from 0x040000 to 0x00e668 (0x31997 bytes) 000635 1789264733.840083 write 0x01177b thru 0x015407 (0x3c8d bytes) 000636 1789264733.841372 mapwrite 0x01fece thru 0x0238ac (0x39df bytes) 000637 1789264733.884031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000639 1789264733.970791 trunc from 0x0238ad to 0x00c12e (0x1777e bytes) 000641 1789264733.984214 write 0x005ce9 thru 0x010b4d (0xae65 bytes) 000642 1789264733.989329 trunc from 0x010b4e to 0x027b55 (0x17008 bytes) 000643 1789264733.998671 trunc from 0x027b56 to 0x03a5e0 (0x12a8b bytes) 000644 1789264734.011897 punch from 0x0166b7 to 0x01e10d (0x7a57 bytes) 000645 1789264734.016427 mapwrite 0x00798f thru 0x008ed6 (0x1548 bytes) 000646 1789264734.030687 write 0x01fd24 thru 0x022964 (0x2c41 bytes) 000647 1789264734.040160 punch from 0x0321a1 to 0x034d6d (0x2bcd bytes) 000648 1789264734.043278 write 0x02d6ca thru 0x03b2e6 (0xdc1d bytes) 000649 1789264734.048298 mapread 0x01f740 thru 0x02e120 (0xe9e1 bytes) 000650 1789264734.062465 trunc from 0x03b2e7 to 0x006b1c (0x347ca bytes) 000651 1789264734.074864 write 0x01143e thru 0x0195fb (0x81be bytes) 000652 1789264734.076012 fallocate 0x001707 thru 0x006865 (0x515f bytes) 000653 1789264734.079631 write 0x037dd9 thru 0x03b260 (0x3488 bytes) 000654 1789264734.080751 trunc from 0x03b261 to 0x027be3 (0x1367d bytes) 000655 1789264734.094143 write 0x022df4 thru 0x02f4d3 (0xc6e0 bytes) 000656 1789264734.098830 mapread 0x004ce7 thru 0x007825 (0x2b3f bytes) 000657 1789264734.104581 punch from 0x019685 to 0x01c5e4 (0x2f60 bytes) 000659 1789264734.117324 mapwrite 0x00e345 thru 0x01b050 (0xcd0c bytes) 000660 1789264734.135248 mapread 0x0201f1 thru 0x02f4d3 (0xf2e3 bytes) 000661 1789264734.139171 MIRROR_EXTEND to 2 mirrors 000662 1789264734.531516 MIRROR_RESYNC 2 mirrors 000663 1789264734.711039 mapwrite 0x016e8b thru 0x01add5 (0x3f4b bytes) 000664 1789264734.797663 fallocate 0x0163e2 thru 0x023e35 (0xda54 bytes) 000665 1789264734.804048 mapread 0x017654 thru 0x018ec9 (0x1876 bytes) 000666 1789264734.804843 read 0x018ed4 thru 0x0224ac (0x95d9 bytes) 000668 1789264734.809930 mapread 0x0031b6 thru 0x004400 (0x124b bytes) 000669 1789264734.827608 MIRROR_RESYNC 2 mirrors 000670 1789264735.262086 read 0x01383a thru 0x0207c4 (0xcf8b bytes) 000672 1789264735.273451 MIRROR_RESYNC 2 mirrors 000673 1789264735.481044 mapread 0x0135e2 thru 0x01f425 (0xbe44 bytes) 000674 1789264735.481997 punch from 0x00cce0 to 0x01841f (0xb740 bytes) 000675 1789264735.510259 MIRROR_RESYNC 2 mirrors 000676 1789264735.879082 trunc from 0x02f4d4 to 0x01a97c (0x14b57 bytes) 000677 1789264735.906578 read 0x00be17 thru 0x010e67 (0x5051 bytes) 000678 1789264735.914085 mapwrite 0x034c4b thru 0x03de72 (0x9228 bytes) 000679 1789264735.970436 write 0x030120 thru 0x030834 (0x715 bytes) 000680 1789264735.971767 trunc from 0x03de73 to 0x01c183 (0x21cef bytes) 000682 1789264735.983880 read 0x013f09 thru 0x017375 (0x346d bytes) 000683 1789264735.989738 read 0x017a30 thru 0x01c183 (0x4754 bytes) 000684 1789264735.990310 mapwrite 0x01e8b9 thru 0x0258f9 (0x7041 bytes) 000685 1789264736.022527 mapread 0x00d450 thru 0x01113d (0x3cee bytes) 000686 1789264736.026807 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000687 1789264736.129134 trunc from 0x0258fa to 0x007543 (0x1e3b6 bytes) 000688 1789264736.142590 trunc from 0x007544 to 0x03c6b7 (0x35174 bytes) 000689 1789264736.152563 mapwrite 0x0103e3 thru 0x0139ac (0x35ca bytes) 000690 1789264736.180135 punch from 0x016d2d to 0x01c3ac (0x5680 bytes) 000691 1789264736.185806 trunc from 0x00201e to 0x007aa8 (0x5a8b bytes) 000692 1789264736.191352 trunc from 0x03c6b8 to 0x00383e (0x38e79 bytes) 000694 1789264736.203693 mapread 0x000e48 thru 0x00205f (0x1218 bytes) 000696 1789264736.208446 write 0x01286c thru 0x01639b (0x3b30 bytes) 000697 1789264736.209553 trunc from 0x006bbf to 0x0110cc (0xa50e bytes) 000698 1789264736.213617 mapwrite 0x0148ff thru 0x01a975 (0x6077 bytes) 000699 1789264736.247915 punch from 0x00d211 to 0x019239 (0xc029 bytes) 000701 1789264736.252353 read 0x00228d thru 0x008e9d (0x6c11 bytes) 000702 1789264736.258773 trunc from 0x00470a to 0x00c254 (0x7b4b bytes) 000703 1789264736.263138 write 0x005fc9 thru 0x00aaf7 (0x4b2f bytes) 000704 1789264736.271760 mapread 0x017a01 thru 0x019f81 (0x2581 bytes) 000705 1789264736.278847 trunc from 0x01a976 to 0x01111b (0x985a bytes) 000706 1789264736.290318 read 0x001ec6 thru 0x007273 (0x53ae bytes) 000707 1789264736.296522 write 0x038691 thru 0x03fde7 (0x7757 bytes) 000708 1789264736.298539 mapwrite 0x0360b9 thru 0x03aa78 (0x49c0 bytes) 000709 1789264736.321366 fallocate 0x01ea4f thru 0x02392c (0x4ede bytes) 000710 1789264736.327218 fallocate 0x007699 thru 0x0101f1 (0x8b59 bytes) 000711 1789264736.332246 mapwrite 0x027282 thru 0x0301d9 (0x8f58 bytes) 000713 1789264736.350207 punch from 0x017267 to 0x01fe77 (0x8c11 bytes) 000714 1789264736.359851 MIRROR_EXTEND to 2 mirrors 000715 1789264736.753979 fallocate 0x011ec3 thru 0x013030 (0x116e bytes) 000716 1789264736.780553 read 0x002c05 thru 0x00c11e (0x951a bytes) 000717 1789264736.789417 fallocate 0x0336ee thru 0x03b920 (0x8233 bytes) 000718 1789264736.795774 fallocate 0x009aea thru 0x012bd9 (0x90f0 bytes) 000719 1789264736.800242 fallocate 0x01ac23 thru 0x02027a (0x5658 bytes) 000720 1789264736.803989 trunc from 0x03fde8 to 0x020256 (0x1fb91 bytes) 000721 1789264736.842295 read 0x00a541 thru 0x00b2c7 (0xd87 bytes) 000722 1789264736.842929 punch from 0x00282b to 0x00be33 (0x9609 bytes) 000723 1789264736.847370 write 0x0266a5 thru 0x0282c8 (0x1c24 bytes) 000724 1789264736.848399 trunc from 0x0282c9 to 0x02b1a0 (0x2ed8 bytes) 000725 1789264736.856912 fallocate 0x01dbe2 thru 0x023ec4 (0x62e3 bytes) 000726 1789264736.860595 write 0x030a6e thru 0x03a242 (0x97d5 bytes) 000727 1789264736.862649 mapwrite 0x03307b thru 0x03e9c9 (0xb94f bytes) 000729 1789264736.898069 punch from 0x006c49 to 0x00b702 (0x4aba bytes) 000730 1789264736.902522 read 0x004a79 thru 0x010d7a (0xc302 bytes) 000731 1789264736.910261 read 0x02ab77 thru 0x03636a (0xb7f4 bytes) 000732 1789264736.912225 punch from 0x03165b to 0x035c3d (0x45e3 bytes) 000733 1789264736.918285 mapread 0x002c05 thru 0x00ff34 (0xd330 bytes) 000734 1789264736.919271 trunc from 0x033e5a to 0x03abb7 (0x6d5e bytes) 000735 1789264736.924548 fallocate 0x01d5da thru 0x026f57 (0x997e bytes) 000736 1789264736.928408 punch from 0x023222 to 0x02b1ab (0x7f8a bytes) 000737 1789264736.932386 punch from 0x01fc2f to 0x02f7bd (0xfb8f bytes) 000739 1789264736.938252 fallocate 0x00d8bc thru 0x01b0bf (0xd804 bytes) 000740 1789264736.942498 fallocate 0x02c3a5 thru 0x035d9f (0x99fb bytes) 000741 1789264736.947415 mapwrite 0x0032a5 thru 0x00f65b (0xc3b7 bytes) 000742 1789264736.964442 trunc from 0x03e9ca to 0x00cb26 (0x31ea3 bytes) 000743 1789264736.979293 mapwrite 0x019c2e thru 0x028ec9 (0xf29c bytes) 000744 1789264737.024247 trunc from 0x028eca to 0x02713b (0x1d8e bytes) 000745 1789264737.039911 mapwrite 0x00217b thru 0x00d573 (0xb3f9 bytes) 000746 1789264737.061671 punch from 0x023a95 to 0x02713b (0x36a7 bytes) 000747 1789264737.066576 write 0x02227f thru 0x031f27 (0xfca9 bytes) 000749 1789264737.069991 mapwrite 0x02f22e thru 0x03e11a (0xeeed bytes) 000750 1789264737.118876 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000751 1789264737.219000 fallocate 0x02ab57 thru 0x030914 (0x5dbe bytes) 000753 1789264737.224430 trunc from 0x01a848 to 0x01e85b (0x4014 bytes) 000754 1789264737.229015 fallocate 0x000d28 thru 0x005451 (0x472a bytes) 000756 1789264737.234161 trunc from 0x03e11b to 0x02ec10 (0xf50a bytes) 000757 1789264737.251659 punch from 0x02788c to 0x02ec10 (0x7385 bytes) 000758 1789264737.256558 fallocate 0x02ce80 thru 0x034e2b (0x7fac bytes) 000760 1789264737.262460 write 0x009ddb thru 0x00a893 (0xab9 bytes) 000761 1789264737.271375 read 0x01d988 thru 0x02bc4f (0xe2c8 bytes) 000763 1789264737.280061 punch from 0x014847 to 0x01b2d4 (0x6a8e bytes) 000764 1789264737.283657 mapread 0x007e34 thru 0x00a2bd (0x248a bytes) 000765 1789264737.290236 mapread 0x012382 thru 0x021521 (0xf1a0 bytes) 000766 1789264737.300070 write 0x01d49b thru 0x02ae0c (0xd972 bytes) 000767 1789264737.301701 mapwrite 0x002eeb thru 0x00d725 (0xa83b bytes) 000768 1789264737.321024 read 0x008a79 thru 0x0121d1 (0x9759 bytes) 000769 1789264737.321707 trunc from 0x00c81f to 0x01889f (0xc081 bytes) 000770 1789264737.327748 trunc from 0x034a0c to 0x034e2b (0x420 bytes) 000771 1789264737.332716 write 0x005980 thru 0x012ffe (0xd67f bytes) 000772 1789264737.340314 read 0x01a124 thru 0x0211e2 (0x70bf bytes) 000773 1789264737.346894 MIRROR_EXTEND to 2 mirrors 000774 1789264737.721852 MIRROR_EXTEND to 3 mirrors 000775 1789264738.082506 punch from 0x017a64 to 0x02222f (0xa7cc bytes) 000776 1789264738.112317 trunc from 0x001793 to 0x008bf3 (0x7461 bytes) 000777 1789264738.118057 fallocate 0x03152b thru 0x03ffff (0xead5 bytes) 000778 1789264738.123556 mapread 0x032669 thru 0x03b07b (0x8a13 bytes) 000780 1789264738.135911 mapread 0x027c94 thru 0x036db9 (0xf126 bytes) 000781 1789264738.147909 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xea8f18d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xea8f18d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1789264738.309702 trunc from 0x040000 to 0x02d7ab (0x12854 bytes) 000784 1789264738.339868 trunc from 0x00531c to 0x014ed0 (0xfbb5 bytes) 000785 1789264738.343627 write 0x01c44e thru 0x027989 (0xb53c bytes) 000786 1789264738.351034 fallocate 0x0029a2 thru 0x00dcd9 (0xb338 bytes) 000787 1789264738.355042 trunc from 0x02d7ac to 0x032e94 (0x56e9 bytes) 000788 1789264738.367717 read 0x02356e thru 0x0243bb (0xe4e bytes) 000789 1789264738.368472 mapread 0x031c15 thru 0x032e94 (0x1280 bytes) 000790 1789264738.380637 mapwrite 0x005b86 thru 0x0158ac (0xfd27 bytes) 000791 1789264738.413047 mapread 0x02e470 thru 0x032e94 (0x4a25 bytes) 000792 1789264738.421493 mapread 0x0238be thru 0x032184 (0xe8c7 bytes) 000793 1789264738.422523 mapread 0x024fc8 thru 0x02d95c (0x8995 bytes) 000795 1789264738.423861 write 0x02f68b thru 0x032159 (0x2acf bytes) 000796 1789264738.425054 write 0x035905 thru 0x03ffff (0xa6fb bytes) 000797 1789264738.426540 read 0x014dd3 thru 0x01f157 (0xa385 bytes) 000798 1789264738.432804 mapwrite 0x002716 thru 0x012285 (0xfb70 bytes) 000799 1789264738.450630 trunc from 0x040000 to 0x01beb6 (0x24149 bytes) 000800 1789264738.465810 mapwrite 0x03fbb1 thru 0x03ffff (0x44f bytes) 000801 1789264738.499586 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) 000803 1789264738.588815 punch from 0x03e2fd to 0x03ffff (0x1d03 bytes) 000804 1789264738.596124 mapread 0x02beca thru 0x037d79 (0xbeb0 bytes) 000807 1789264738.633898 MIRROR_EXTEND to 2 mirrors 000808 1789264738.982911 punch from 0x0134d9 to 0x018631 (0x5159 bytes) 000809 1789264739.005640 write 0x03f4d5 thru 0x03ffff (0xb2b bytes) 000810 1789264739.013172 write 0x018cdb thru 0x01fee5 (0x720b bytes) 000811 1789264739.019550 punch from 0x00d8af to 0x017118 (0x986a bytes) 000812 1789264739.023706 punch from 0x017533 to 0x020e11 (0x98df bytes) 000813 1789264739.034337 trunc from 0x00c63e to 0x0179de (0xb3a1 bytes) 000814 1789264739.041510 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x713b1c4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc27a7f5 Mirror 3: 0xc27a7f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000815 1789264739.483858 read 0x036760 thru 0x03de93 (0x7734 bytes) 000816 1789264739.492051 trunc from 0x040000 to 0x006c8b (0x39374 bytes) 000817 1789264739.521480 MIRROR_RESYNC 3 mirrors 000818 1789264739.837818 punch from 0x004b6c to 0x006c8b (0x2120 bytes) 000819 1789264739.860988 trunc from 0x006c8c to 0x01ea8b (0x17e00 bytes) 000820 1789264739.874535 fallocate 0x03ab74 thru 0x03b349 (0x7d6 bytes) 000821 1789264739.879773 write 0x00ba52 thru 0x00d7ec (0x1d9b bytes) 000822 1789264739.888721 trunc from 0x01ea8c to 0x006cb4 (0x17dd7 bytes) 000823 1789264739.902134 punch from 0x00202f to 0x003277 (0x1249 bytes) 000824 1789264739.906578 punch from 0x003402 to 0x006cb4 (0x38b3 bytes) 000827 1789264739.911910 write 0x02060f thru 0x029933 (0x9325 bytes) 000828 1789264739.913850 fallocate 0x00b6ee thru 0x016eb2 (0xb7c5 bytes) 000829 1789264739.918166 mapwrite 0x025efa thru 0x03478c (0xe893 bytes) 000830 1789264739.965262 fallocate 0x01af3a thru 0x029760 (0xe827 bytes) 000832 1789264739.969589 mapwrite 0x0333fe thru 0x03b18c (0x7d8f bytes) 000833 1789264740.004435 write 0x004204 thru 0x005d50 (0x1b4d bytes) 000834 1789264740.014629 trunc from 0x00fe1a to 0x010912 (0xaf9 bytes) 000835 1789264740.018755 trunc from 0x02bb65 to 0x02ed15 (0x31b1 bytes) 000836 1789264740.022557 write 0x030110 thru 0x038459 (0x834a bytes) 000837 1789264740.023641 mapwrite 0x02d1fa thru 0x03c9f9 (0xf800 bytes) 000838 1789264740.057496 mapwrite 0x007b1b thru 0x017470 (0xf956 bytes) 000839 1789264740.084376 read 0x03bcda thru 0x03c9f9 (0xd20 bytes) 000840 1789264740.084891 trunc from 0x0284b4 to 0x02c890 (0x43dd bytes) 000841 1789264740.088658 trunc from 0x0347d7 to 0x03c2dd (0x7b07 bytes) 000842 1789264740.092532 fallocate 0x0112c4 thru 0x018584 (0x72c1 bytes) 000843 1789264740.097047 mapread 0x036866 thru 0x03adc9 (0x4564 bytes) 000844 1789264740.102764 fallocate 0x037ec3 thru 0x03c8bd (0x49fb bytes) 000845 1789264740.112444 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c9fa): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9fa): Mirror 2: 0x83c44bb3 (stale) Mirror 3: 0x83c44bb3 (stale) CRC-32 checksum value for chunk [0, 0x3c9fa): Mirror 1: 0x35cd83fb Mirror 4: 0x35cd83fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9fa: skipped 000846 1789264740.519870 mapwrite 0x02a5b5 thru 0x0338d0 (0x931c bytes) 000848 1789264740.595993 MIRROR_RESYNC 4 mirrors 000849 1789264741.166701 read 0x01abae thru 0x01ecfc (0x414f bytes) 000850 1789264741.172695 write 0x03834e thru 0x03e5e3 (0x6296 bytes) 000851 1789264741.195289 mapwrite 0x0081e8 thru 0x013323 (0xb13c bytes) 000852 1789264741.227089 fallocate 0x032f42 thru 0x03e51a (0xb5d9 bytes) 000853 1789264741.233140 trunc from 0x00e974 to 0x013605 (0x4c92 bytes) 000854 1789264741.238490 trunc from 0x03e5e4 to 0x03149c (0xd147 bytes) 000855 1789264741.256579 trunc from 0x03149d to 0x02b7a2 (0x5cfa bytes) 000856 1789264741.268954 read 0x00fac1 thru 0x018c63 (0x91a3 bytes) 000857 1789264741.278740 read 0x028dd0 thru 0x02b7a2 (0x29d3 bytes) 000858 1789264741.279461 trunc from 0x02b7a3 to 0x008d36 (0x22a6c bytes) 000859 1789264741.296702 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8d37): Mirror 3: 0x2b6d87c3 (stale) Mirror 4: 0x2b6d87c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8d37): Mirror 3: 0x2b6d87c3 (stale) Mirror 4: 0x2b6d87c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8d37: skipped 000861 1789264741.487215 mapwrite 0x03161d thru 0x03220b (0xbef bytes) 000862 1789264741.515705 mapread 0x004d21 thru 0x0123ee (0xd6ce bytes) 000863 1789264741.526995 read 0x020411 thru 0x029cf3 (0x98e3 bytes) 000864 1789264741.534435 trunc from 0x03220c to 0x027742 (0xaac9 bytes) 000865 1789264741.547746 trunc from 0x0235c1 to 0x027742 (0x4182 bytes) 000866 1789264741.550763 mapread 0x022622 thru 0x027742 (0x5121 bytes) 000867 1789264741.557767 mapread 0x0175ef thru 0x0246fc (0xd10e bytes) 000868 1789264741.562832 mapread 0x006c96 thru 0x015d99 (0xf104 bytes) 000869 1789264741.564041 trunc from 0x027743 to 0x01417f (0x135c3 bytes) 000870 1789264741.574015 trunc from 0x014180 to 0x02a022 (0x15ea3 bytes) 000871 1789264741.586762 punch from 0x023af7 to 0x02a022 (0x652c bytes) 000872 1789264741.590761 write 0x00c319 thru 0x0166ae (0xa396 bytes) 000873 1789264741.596127 fallocate 0x00fa2d thru 0x017ca9 (0x827d bytes) 000874 1789264741.600527 fallocate 0x00d4ef thru 0x00f43d (0x1f4f bytes) 000875 1789264741.604945 mapwrite 0x02e846 thru 0x036b11 (0x82cc bytes) 000876 1789264741.639355 write 0x003ea6 thru 0x00a274 (0x63cf bytes) 000877 1789264741.641056 trunc from 0x0280ea to 0x028b77 (0xa8e bytes) 000879 1789264741.645792 trunc from 0x036b12 to 0x03c44d (0x593c bytes) 000880 1789264741.658660 MIRROR_RESYNC 3 mirrors 000881 1789264742.127551 punch from 0x013547 to 0x018ec1 (0x597b bytes) 000883 1789264742.155318 trunc from 0x014bd0 to 0x02249e (0xd8cf bytes) 000884 1789264742.158398 mapwrite 0x00b97f thru 0x019ecd (0xe54f bytes) 000886 1789264742.191459 punch from 0x028b92 to 0x02bedc (0x334b bytes) 000887 1789264742.195806 mapread 0x010212 thru 0x01695c (0x674b bytes) 000888 1789264742.196823 mapwrite 0x02ed0b thru 0x031c35 (0x2f2b bytes) 000889 1789264742.214734 MIRROR_RESYNC 3 mirrors 000890 1789264742.582362 mapwrite 0x01ffdb thru 0x028f7c (0x8fa2 bytes) 000891 1789264742.658310 mapwrite 0x034a37 thru 0x03ffff (0xb5c9 bytes) 000892 1789264742.712158 trunc from 0x010aba to 0x0187da (0x7d21 bytes) 000893 1789264742.717092 write 0x00d9be thru 0x01c99c (0xefdf bytes) 000894 1789264742.722684 write 0x02055d thru 0x026d5f (0x6803 bytes) 000895 1789264742.723815 trunc from 0x040000 to 0x02b586 (0x14a79 bytes) 000897 1789264742.732125 write 0x029d86 thru 0x02aaf6 (0xd71 bytes) 000898 1789264742.742432 fallocate 0x029192 thru 0x02ccc1 (0x3b30 bytes) 000899 1789264742.746558 write 0x02ea0b thru 0x03d7b4 (0xedaa bytes) 000900 1789264742.748174 punch from 0x0023b3 to 0x00d875 (0xb4c3 bytes) 000901 1789264742.760621 mapread 0x0350b1 thru 0x03b078 (0x5fc8 bytes) 000903 1789264742.761837 write 0x02107d thru 0x02baad (0xaa31 bytes) 000904 1789264742.766502 read 0x02d194 thru 0x036437 (0x92a4 bytes) 000906 1789264742.770165 mapwrite 0x0334ca thru 0x03f932 (0xc469 bytes) 000907 1789264742.804940 fallocate 0x011f80 thru 0x01cb04 (0xab85 bytes) 000908 1789264742.808806 write 0x01b246 thru 0x01e733 (0x34ee bytes) 000909 1789264742.809765 fallocate 0x014d34 thru 0x0233c6 (0xe693 bytes) 000910 1789264742.813342 write 0x010e98 thru 0x01f633 (0xe79c bytes) 000911 1789264742.814809 mapwrite 0x01c2dd thru 0x021469 (0x518d bytes) 000912 1789264742.831036 mapwrite 0x005a59 thru 0x00a571 (0x4b19 bytes) 000913 1789264742.852936 MIRROR_RESYNC 3 mirrors 000914 1789264743.345614 mapwrite 0x0022b1 thru 0x010df7 (0xeb47 bytes) 000915 1789264743.413594 mapwrite 0x024f99 thru 0x030811 (0xb879 bytes) 000916 1789264743.441477 trunc from 0x03f933 to 0x037ae4 (0x7e4e bytes) 000917 1789264743.457474 punch from 0x0315bd to 0x037ae4 (0x6528 bytes) 000918 1789264743.462792 MIRROR_RESYNC 3 mirrors 000919 1789264743.825994 trunc from 0x037ae5 to 0x023b41 (0x13fa3 bytes) 000920 1789264743.855117 read 0x00fc3c thru 0x011a0b (0x1dd0 bytes) 000921 1789264743.859412 mapread 0x01d28d thru 0x023b41 (0x68b5 bytes) 000922 1789264743.873489 mapwrite 0x033a36 thru 0x033be0 (0x1ab bytes) 000924 1789264743.910384 fallocate 0x0028ae thru 0x00515f (0x28b2 bytes) 000925 1789264743.916210 write 0x01291f thru 0x015d01 (0x33e3 bytes) 000926 1789264743.922659 write 0x03833a thru 0x03a945 (0x260c bytes) 000927 1789264743.923916 punch from 0x01e005 to 0x01fe83 (0x1e7f bytes) 000929 1789264743.927505 write 0x0352ca thru 0x038038 (0x2d6f bytes) 000931 1789264743.932105 mapread 0x0119a9 thru 0x01b9d7 (0xa02f bytes) 000932 1789264743.933047 read 0x0044f5 thru 0x013425 (0xef31 bytes) 000934 1789264743.939806 trunc from 0x03a946 to 0x0066c9 (0x3427c bytes) 000935 1789264743.953783 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x66ca): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x66ca): Mirror 3: 0x86af4134 (stale) Mirror 4: 0x86af4134 (stale) CRC-32 checksum value for chunk [0, 0x66ca): Mirror 1: 0x86af4134 Mirror 5: 0x86af4134 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x66ca: skipped 000936 1789264744.336472 mapwrite 0x035307 thru 0x036d74 (0x1a6e bytes) 000937 1789264744.393039 fallocate 0x0210ba thru 0x02a287 (0x91ce bytes) 000938 1789264744.396330 trunc from 0x031b5a to 0x036d74 (0x521b bytes) 000939 1789264744.401228 read 0x00eba6 thru 0x01018e (0x15e9 bytes) 000940 1789264744.405751 punch from 0x0021e4 to 0x01103a (0xee57 bytes) 000941 1789264744.409632 write 0x036a9c thru 0x03a020 (0x3585 bytes) 000943 1789264744.413711 read 0x012b7d thru 0x01322e (0x6b2 bytes) 000944 1789264744.419981 mapread 0x005243 thru 0x00ccc7 (0x7a85 bytes) 000945 1789264744.420951 mapwrite 0x02ad60 thru 0x03aad8 (0xfd79 bytes) 000946 1789264744.446104 mapwrite 0x00241b thru 0x0042e1 (0x1ec7 bytes) 000947 1789264744.457152 punch from 0x0213b2 to 0x023841 (0x2490 bytes) 000948 1789264744.460680 write 0x022934 thru 0x02a548 (0x7c15 bytes) 000950 1789264744.465575 trunc from 0x03aad9 to 0x011f4d (0x28b8b bytes) 000951 1789264744.478284 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x11f4e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11f4e): Mirror 3: 0x82ecb4d9 (stale) Mirror 4: 0x82ecb4d9 (stale) Mirror 5: 0xbd73b90f (stale) CRC-32 checksum value for chunk [0, 0x11f4e): Mirror 1: 0x438104c6 Mirror 6: 0x438104c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11f4e: skipped 000952 1789264744.829980 read 0x005213 thru 0x0057ce (0x5bc bytes) 000953 1789264744.832937 fallocate 0x005719 thru 0x0148c1 (0xf1a9 bytes) 000956 1789264744.861953 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11f4e): Mirror 3: 0x82ecb4d9 (stale) Mirror 4: 0x82ecb4d9 (stale) Mirror 6: 0x438104c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11f4e): Mirror 3: 0x82ecb4d9 (stale) Mirror 4: 0x82ecb4d9 (stale) Mirror 6: 0x438104c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11f4e: skipped 000957 1789264745.039796 trunc from 0x009de4 to 0x00faac (0x5cc9 bytes) 000961 1789264745.044550 trunc from 0x011f4e to 0x003ad3 (0xe47a bytes) 000962 1789264745.070805 trunc from 0x001821 to 0x003ad3 (0x22b3 bytes) 000963 1789264745.073514 trunc from 0x003ad4 to 0x033476 (0x2f9a3 bytes) 000964 1789264745.081396 punch from 0x005cb6 to 0x00b3fe (0x5749 bytes) 000965 1789264745.084784 mapwrite 0x0233a9 thru 0x02deaf (0xab07 bytes) 000966 1789264745.111168 punch from 0x013bea to 0x020ea7 (0xd2be bytes) 000967 1789264745.115094 fallocate 0x03efea thru 0x03ffff (0x1016 bytes) 000968 1789264745.121049 mapwrite 0x00fb4c thru 0x01fa91 (0xff46 bytes) 000970 1789264745.148765 mapread 0x01e026 thru 0x027151 (0x912c bytes) 000971 1789264745.149781 punch from 0x0147e7 to 0x01728d (0x2aa7 bytes) 000972 1789264745.152974 trunc from 0x02e260 to 0x0363c1 (0x8162 bytes) 000973 1789264745.155889 mapwrite 0x003c15 thru 0x00a949 (0x6d35 bytes) 000974 1789264745.181474 MIRROR_RESYNC 4 mirrors 000975 1789264745.900655 mapwrite 0x0001f7 thru 0x007be8 (0x79f2 bytes) 000976 1789264745.968487 mapread 0x0002d5 thru 0x005e11 (0x5b3d bytes) 000978 1789264745.972717 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x353:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc3622188 (stale) Mirror 6: 0xc3622188 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc3622188 (stale) Mirror 6: 0xc3622188 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000979 1789264746.148987 mapwrite 0x014fe1 thru 0x018a74 (0x3a94 bytes) 000980 1789264746.173961 mapread 0x001370 thru 0x002abc (0x174d bytes) 000981 1789264746.185092 MIRROR_RESYNC 3 mirrors 000982 1789264746.639489 fallocate 0x00a2d5 thru 0x012121 (0x7e4d bytes) 000984 1789264746.659879 punch from 0x021090 to 0x023d18 (0x2c89 bytes) 000985 1789264746.664157 trunc from 0x03ae23 to 0x03ffff (0x51dd bytes) 000986 1789264746.667686 trunc from 0x020cb6 to 0x025750 (0x4a9b bytes) 000987 1789264746.671097 fallocate 0x01cf69 thru 0x022bda (0x5c72 bytes) 000988 1789264746.674286 write 0x018f91 thru 0x01d6db (0x474b bytes) 000989 1789264746.680482 mapread 0x0346fb thru 0x03f622 (0xaf28 bytes) 000990 1789264746.695615 write 0x03d5a7 thru 0x03ffff (0x2a59 bytes) 000991 1789264746.696704 punch from 0x0246f2 to 0x025b92 (0x14a1 bytes) 000992 1789264746.700804 mapread 0x00f1d7 thru 0x015fe2 (0x6e0c bytes) 000993 1789264746.707352 read 0x033bae thru 0x03b1c4 (0x7617 bytes) 000995 1789264746.708506 fallocate 0x0356ad thru 0x03ffff (0xa953 bytes) 000996 1789264746.711579 mapwrite 0x03ff56 thru 0x03ffff (0xaa bytes) 000997 1789264746.722964 mapread 0x02cb18 thru 0x02f33d (0x2826 bytes) 000998 1789264746.729973 read 0x01687e thru 0x021a2e (0xb1b1 bytes) 000999 1789264746.736468 trunc from 0x01cfc3 to 0x01d9e4 (0xa22 bytes) 001000 1789264746.739660 trunc from 0x02884e to 0x0358d7 (0xd08a bytes) All operations completed A-OK! PASS 70a (39s) == sanity-flr test 71: check mirror extend parent fid ==== 21:59:08 (1789264748) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x389:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 72: check append on FLR file ========== 21:59:13 (1789264753) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0968125 s, 21.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077714 s, 13.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 21:59:17 (1789264757) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1188640 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6252000 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 4251136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 4400384 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7666560 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6533728 .. Extending file size to 7219648 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7924832 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8791872 .. Extending file size to 7454144 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8862464 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8977088 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8958304 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9146816 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62334 62335 62336 62339 62341 failed 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) 192.168.204.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:00:26 (1789264826) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4232384 .. Extending file size to 1144992 .. failed Extending file size to 4207424 .. Extending file size to 7731200 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4853952 .. Extending file size to 7204448 .. Extending file size to 7917056 .. Extending file size to 8911488 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8144800 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8890528 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8961376 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8964544 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9202592 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 64727 64728 64730 64733 64734 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) 192.168.204.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:01:32 (1789264892) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2396 1268908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24760 3582260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18936 3588084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43696 7170344 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2396 1268908 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24760 3582260 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18936 3588084 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43696 7170344 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0538803 s, 38.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0850617 s, 24.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xea5b3bb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:01:41 (1789264901) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0485518 s, 43.2 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 22:01:44 (1789264904) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:01:47 (1789264907) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0475624 s, 44.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4ab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:01:51 (1789264911) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0500494 s, 41.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:01:54 (1789264914) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0637512 s, 32.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4ae:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:02:20 (1789264940) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170684 s, 6.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b1:0x0] } PASS 204d (50s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:03:10 (1789264990) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0873359 s, 24.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:04:03 (1789265043) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg412-server: Pool lustre.test_204f created Adding targets to pool oleg412-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg412-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.068895 s, 30.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg412-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg412-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg412-server: Pool lustre.test_204f destroyed PASS 204f (89s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:05:32 (1789265132) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:05:40 (1789265140) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:05:49 (1789265149) oleg412-server: Pool lustre.test_206 created Adding targets to pool oleg412-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg412-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg412-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg412-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg412-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:06:18 (1789265178) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103798 s, 10.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572349 s, 18.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x31977fbd Mirror 3: 0x31977fbd PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:06:30 (1789265190) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:06:34 (1789265194) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:06:37 (1789265197) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0782132 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ad0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ad0000 Read this (4 bytes): BBBB 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:07:14 (1789265234) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0154555 s, 530 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:07:21 (1789265241) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100068 s, 10.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-12 22:07:23.000000000 -0400 Modify: 2026-09-12 22:07:23.000000000 -0400 Change: 2026-09-12 22:07:25.000000000 -0400 Birth: 2026-09-12 22:07:23.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:07:31 (1789265251) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365689 s, 28.7 MB/s oleg412-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg412-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg412-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (30s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:08:01 (1789265281) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.320331 s, 128 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0319256 s, 128 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 12 22:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 12 22:08 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:08:12 (1789265292) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:08:20 (1789265300) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1213 sec =========== 22:08:29 (1789265309) === sanity-flr: start cleanup 22:08:31 (1789265311) === === sanity-flr: finish cleanup 22:08:41 (1789265321) ===