-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 04:26:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:27:15 (1788510435) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b45066a4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b45066a4000.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:27:59 (1788510479) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8249] == sanity-flr test 0a: lfs mirror create with -N option == 04:28:02 (1788510482) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:28:22 (1788510502) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:28:26 (1788510506) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:28:30 (1788510510) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:29:02 (1788510542) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:29:06 (1788510546) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 04:29:09 (1788510549) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b45066a4000.stats=clear osc.lustre-OST0001-osc-ffff9b45066a4000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:29:18 (1788510558) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:29:28 (1788510568) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:29:38 (1788510578) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:29:42 (1788510582) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:29:50 (1788510590) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 04:29:59 (1788510599) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:30:08 (1788510608) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:30:15 (1788510615) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:30:22 (1788510622) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:30:34 (1788510634) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:30:41 (1788510641) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.326733 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724225 s, 14.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:30:49 (1788510649) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.367596 s, 42.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:30:58 (1788510658) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:31:08 (1788510668) Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:32:21 (1788510741) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.63769 s, 18.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:33:07 (1788510787) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:34:34 (1788510874) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.5614 s, 20.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 04:34 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.963s user 0m0.001s sys 0m0.191s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b45066a4000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.251s user 0m0.000s sys 0m0.203s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.057s user 0m0.000s sys 0m0.166s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b45066a4000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:36:11 (1788510971) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:36:15 (1788510975) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:36:18 (1788510978) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:36:21 (1788510981) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 04:36:29 (1788510989) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.566045 s, 22.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:36:42 (1788511002) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262083 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022889 s, 45.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:36:49 (1788511009) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:36:57 (1788511017) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010914 s, 938 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129889 s, 788 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125722 s, 814 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00174533 s, 587 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 04:37:06 (1788511026) create test files with size 15754315 .. 309bc65bfc9120a40eb0062723d85a36 - a973e5eb62a0f984fea4779a77ac1b1d - bf52565fd13e4e5060b32f714b7e8ef2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 04:37:29 (1788511049) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:37:45 (1788511065) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:37:54 (1788511074) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150232 s, 7.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526151 s, 19.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 04:38:07 (1788511087) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132815 s, 15.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.410029 s, 10.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153188 s, 13.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 04:38:26 (1788511106) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 04:38:30 (1788511110) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:38:33 (1788511113) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230913 s, 45.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 04:38:41 (1788511121) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.211444 s, 14.9 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:38:52 (1788511132) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:39:00 (1788511140) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.41094 s, 25.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:39:12 (1788511152) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b45061b0000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.433703 s, 24.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b45029ba800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:39:27 (1788511167) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:40:03 (1788511203) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:40:40 (1788511240) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 04:40:43 (1788511243) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.296673 s, 10.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.118762 s, 26.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x532ff386 (nosync,stale) Mirror 2: 0x532ff386 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5c0dd4751f413f8c848ff9a43fdf418e - mirror 1 checksum: 1ddfa0f8c0e39681070b7ee5be7095c9 - mirror 2 checksum: 5c0dd4751f413f8c848ff9a43fdf418e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T04:40:45-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T04:40:45-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5c0dd4751f413f8c848ff9a43fdf418e - mirror 1 checksum: 1ddfa0f8c0e39681070b7ee5be7095c9 - mirror 2 checksum: 1ddfa0f8c0e39681070b7ee5be7095c9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T04:40:45-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T04:40:45-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:40:53 (1788511253) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619789 s, 16.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:41:01 (1788511261) Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 04:41:13 (1788511273) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 04:41:38 (1788511298) targets are mounted 04:41:38 (1788511298) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392702 s, 26.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:41:54 (1788511314) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.059111 s, 346 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0154534 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:42:04 (1788511324) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0544036 s, 376 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0187637 s, 1.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:42:13 (1788511333) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0058682 s, 2.8 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:42:22 (1788511342) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0796977 s, 257 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:42:31 (1788511351) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00356066 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0378324 s, 108 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:42:38 (1788511358) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:42:57 (1788511377) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:43:24 (1788511404) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:43:48 (1788511428) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:44:14 (1788511454) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 04:44:22 (1788511462) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57579 mirror split pid 57580 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57579 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57580 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 04:45:29 (1788511529) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 1531 fallocating to largest ever: 0x1317b 000001 1788511532.121137 fallocate 0x00ce8a thru 0x01317a (0x62f1 bytes) 000002 1788511532.132192 trunc from 0x009a42 to 0x00dc84 (0x4243 bytes) 000003 1788511532.139780 mapwrite 0x02451d thru 0x02f5f6 (0xb0da bytes) fallocating to largest ever: 0x2ad0f 000004 1788511532.215197 fallocate 0x0222d2 thru 0x02ad0e (0x8a3d bytes) 000006 1788511532.226596 punch from 0x009346 to 0x015d2e (0xc9e9 bytes) 000007 1788511532.231563 mapread 0x027c3a thru 0x02f5f6 (0x79bd bytes) 000009 1788511532.234678 trunc from 0x02f5f7 to 0x01e8a0 (0x10d56 bytes) 000011 1788511532.251679 trunc from 0x01e8a1 to 0x00bed7 (0x129c9 bytes) 000012 1788511532.268025 mapwrite 0x03793f thru 0x03fd76 (0x8438 bytes) 000013 1788511532.309439 read 0x005e94 thru 0x0145fd (0xe76a bytes) 000015 1788511532.319053 fallocate 0x01c9c5 thru 0x028381 (0xb9bd bytes) 000016 1788511532.323241 read 0x03c0be thru 0x03fd76 (0x3cb9 bytes) 000017 1788511532.324380 trunc from 0x03fd77 to 0x0055ea (0x3a78c bytes) 000018 1788511532.341920 trunc from 0x000042 to 0x003efb (0x3eba bytes) 000019 1788511532.346557 write 0x0148ec thru 0x015f40 (0x1655 bytes) 000020 1788511532.347821 punch from 0x00ae4f to 0x015f40 (0xb0f2 bytes) 000021 1788511532.356662 mapread 0x014801 thru 0x015f40 (0x1740 bytes) fallocating to largest ever: 0x36f8c 000022 1788511532.374235 fallocate 0x02dd32 thru 0x036f8b (0x925a bytes) 000023 1788511532.379049 read 0x00fccb thru 0x016d67 (0x709d bytes) 000024 1788511532.387156 write 0x01a4ad thru 0x026345 (0xbe99 bytes) 000025 1788511532.389039 trunc from 0x036f8c to 0x0190ec (0x1de9f bytes) 000026 1788511532.403895 trunc from 0x00ffbb to 0x0190ec (0x9132 bytes) 000027 1788511532.408380 write 0x013f9a thru 0x01fedc (0xbf43 bytes) 000028 1788511532.413433 write 0x037962 thru 0x03e405 (0x6aa4 bytes) 000029 1788511532.415572 fallocate 0x012d0e thru 0x016eae (0x41a1 bytes) 000030 1788511532.420481 write 0x01bedf thru 0x02b074 (0xf196 bytes) 000031 1788511532.425923 punch from 0x02a2fa to 0x032410 (0x8117 bytes) 000033 1788511532.450192 mapwrite 0x00aaeb thru 0x00ec41 (0x4157 bytes) 000034 1788511532.465730 mapread 0x0366d9 thru 0x03e405 (0x7d2d bytes) 000036 1788511532.473946 write 0x0031dc thru 0x00f1d8 (0xbffd bytes) 000038 1788511532.480730 trunc from 0x017ba8 to 0x01a105 (0x255e bytes) 000039 1788511532.486658 fallocate 0x034be7 thru 0x035826 (0xc40 bytes) 000040 1788511532.492407 read 0x031cd8 thru 0x0331cb (0x14f4 bytes) 000041 1788511532.505233 write 0x01af85 thru 0x02a0c6 (0xf142 bytes) 000042 1788511532.508004 mapread 0x02d119 thru 0x039265 (0xc14d bytes) 000043 1788511532.510967 trunc from 0x00659a to 0x006cac (0x713 bytes) 000046 1788511532.530010 trunc from 0x03cc5b to 0x03e405 (0x17ab bytes) 000047 1788511532.535539 read 0x00cf85 thru 0x01b62f (0xe6ab bytes) 000048 1788511532.536318 trunc from 0x03e406 to 0x010e69 (0x2d59c bytes) 000049 1788511532.558135 MIRROR_EXTEND to 2 mirrors 000050 1788511532.993810 punch from 0x006141 to 0x00bd0d (0x5bcd bytes) 000051 1788511533.047413 mapwrite 0x004146 thru 0x011a1e (0xd8d9 bytes) 000052 1788511533.241952 trunc from 0x011a1f to 0x029754 (0x17d36 bytes) 000053 1788511533.261414 write 0x02ad69 thru 0x02dd06 (0x2f9e bytes) 000054 1788511533.262788 punch from 0x01a29f to 0x01dcdf (0x3a41 bytes) 000056 1788511533.281389 trunc from 0x02350e to 0x0256b2 (0x21a5 bytes) 000057 1788511533.287559 mapread 0x0232c3 thru 0x025dd5 (0x2b13 bytes) 000058 1788511533.309882 fallocate 0x014cc1 thru 0x01dd78 (0x90b8 bytes) fallocating to largest ever: 0x40000 000059 1788511533.315782 fallocate 0x038304 thru 0x03ffff (0x7cfc bytes) 000060 1788511533.319975 write 0x02412c thru 0x027eb9 (0x3d8e bytes) 000061 1788511533.321369 trunc from 0x03e755 to 0x03ffff (0x18ab bytes) 000062 1788511533.324549 read 0x033d6d thru 0x037e20 (0x40b4 bytes) 000063 1788511533.342674 write 0x03c2f5 thru 0x03ffff (0x3d0b bytes) 000064 1788511533.349529 trunc from 0x040000 to 0x02416a (0x1be95 bytes) 000066 1788511533.374431 fallocate 0x036684 thru 0x03f282 (0x8bff bytes) 000067 1788511533.382631 trunc from 0x02af9b to 0x037941 (0xc9a7 bytes) 000068 1788511533.389414 trunc from 0x02cee5 to 0x038f79 (0xc095 bytes) 000069 1788511533.397359 read 0x008b85 thru 0x00e326 (0x57a2 bytes) 000070 1788511533.402621 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f283): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f283): Mirror 2: 0xb3ec4507 (stale) CRC-32 checksum value for chunk [0, 0x3f283): Mirror 1: 0x93c40d8 Mirror 3: 0x93c40d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f283: skipped 000071 1788511534.145126 trunc from 0x01fda5 to 0x02eb17 (0xed73 bytes) 000072 1788511534.186499 mapread 0x02df78 thru 0x02fbfa (0x1c83 bytes) 000073 1788511534.204896 mapwrite 0x00fa3d thru 0x015658 (0x5c1c bytes) 000074 1788511534.231971 trunc from 0x0319c1 to 0x03eb88 (0xd1c8 bytes) 000076 1788511534.242201 MIRROR_RESYNC 3 mirrors 000077 1788511534.895344 mapread 0x016602 thru 0x01eac9 (0x84c8 bytes) 000079 1788511534.913671 fallocate 0x02cf5a thru 0x03b275 (0xe31c bytes) 000080 1788511534.947194 mapread 0x03b1cd thru 0x03f282 (0x40b6 bytes) 000081 1788511534.960034 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f283): Mirror 3: 0xaa31f57a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f283): Mirror 3: 0xaa31f57a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f283: skipped 000082 1788511535.201822 mapread 0x01420f thru 0x01423b (0x2d bytes) 000083 1788511535.207612 mapwrite 0x03f30f thru 0x03ffff (0xcf1 bytes) 000084 1788511535.257451 fallocate 0x0353aa thru 0x03ffff (0xac56 bytes) 000085 1788511535.262494 trunc from 0x040000 to 0x00353f (0x3cac0 bytes) 000086 1788511535.284555 read 0x000426 thru 0x00353f (0x311a bytes) 000087 1788511535.293185 mapwrite 0x00ecb2 thru 0x012950 (0x3c9f bytes) 000088 1788511535.353494 MIRROR_RESYNC 2 mirrors 000089 1788511535.786531 mapread 0x01117b thru 0x012950 (0x17d6 bytes) 000090 1788511535.809598 punch from 0x00ee25 to 0x012950 (0x3b2c bytes) 000091 1788511535.859879 read 0x0053e4 thru 0x008662 (0x327f bytes) 000092 1788511535.866345 write 0x022ae0 thru 0x022d5e (0x27f bytes) 000093 1788511535.867355 write 0x004072 thru 0x00f679 (0xb608 bytes) 000094 1788511535.877639 mapread 0x015607 thru 0x020efe (0xb8f8 bytes) 000095 1788511535.890871 fallocate 0x00a847 thru 0x00a9a0 (0x15a bytes) 000096 1788511535.895587 punch from 0x00ff87 to 0x01a78a (0xa804 bytes) 000097 1788511535.914539 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000098 1788511536.045830 MIRROR_EXTEND to 2 mirrors 000099 1788511536.607979 punch from 0x01d3e3 to 0x01d5f7 (0x215 bytes) 000100 1788511536.651823 mapwrite 0x00c6d7 thru 0x010f50 (0x487a bytes) 000101 1788511536.682754 write 0x03c240 thru 0x03ffff (0x3dc0 bytes) 000102 1788511536.691555 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4bfbcebf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe741509a Mirror 3: 0xe741509a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000103 1788511537.498619 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4bfbcebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4bfbcebf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000105 1788511537.827606 write 0x036b5f thru 0x03b8c7 (0x4d69 bytes) 000106 1788511537.875771 mapread 0x02b7ed thru 0x038b38 (0xd34c bytes) 000107 1788511537.905603 punch from 0x000675 to 0x00f4cc (0xee58 bytes) 000108 1788511537.923639 trunc from 0x040000 to 0x03f39a (0xc65 bytes) 000109 1788511537.950668 write 0x0171db thru 0x021da7 (0xabcd bytes) 000110 1788511537.963535 read 0x00a707 thru 0x018ba6 (0xe4a0 bytes) 000111 1788511537.982571 MIRROR_RESYNC 2 mirrors 000113 1788511538.666038 trunc from 0x03f39b to 0x021982 (0x1da18 bytes) 000114 1788511538.710197 write 0x03d85f thru 0x03ffff (0x27a1 bytes) 000115 1788511538.711929 mapread 0x03283b thru 0x036cb8 (0x447e bytes) 000116 1788511538.726754 punch from 0x0009c1 to 0x00560d (0x4c4d bytes) 000118 1788511538.731533 mapread 0x03c144 thru 0x03ffff (0x3ebc bytes) 000119 1788511538.732591 fallocate 0x0279d4 thru 0x03468c (0xccb9 bytes) 000120 1788511538.736790 trunc from 0x02816f to 0x02d678 (0x550a bytes) 000121 1788511538.741022 mapwrite 0x00a44a thru 0x00cdc6 (0x297d bytes) 000122 1788511538.762515 read 0x0153c7 thru 0x01ce16 (0x7a50 bytes) 000123 1788511538.770519 trunc from 0x016597 to 0x01ae75 (0x48df bytes) 000124 1788511538.775868 punch from 0x031cab to 0x0344a7 (0x27fd bytes) 000125 1788511538.793101 read 0x010b60 thru 0x0190ea (0x858b bytes) 000126 1788511538.802425 read 0x03ed1a thru 0x03ffff (0x12e6 bytes) 000128 1788511538.803638 trunc from 0x040000 to 0x03ec7f (0x1380 bytes) 000129 1788511538.824214 mapwrite 0x02c887 thru 0x03736e (0xaae8 bytes) 000130 1788511538.843370 trunc from 0x03ec80 to 0x027b74 (0x1710b bytes) 000131 1788511538.862825 MIRROR_RESYNC 2 mirrors 000132 1788511539.438547 mapwrite 0x032922 thru 0x033cfd (0x13dc bytes) 000133 1788511539.477570 punch from 0x017fef to 0x021281 (0x9293 bytes) 000135 1788511539.480754 punch from 0x019c8d to 0x01ccca (0x303e bytes) 000136 1788511539.483008 punch from 0x0310b5 to 0x033abb (0x2a07 bytes) 000137 1788511539.486116 mapwrite 0x031307 thru 0x03ec38 (0xd932 bytes) 000138 1788511539.533464 fallocate 0x003bfb thru 0x01334a (0xf750 bytes) 000139 1788511539.546004 read 0x010450 thru 0x013566 (0x3117 bytes) 000140 1788511539.553554 mapwrite 0x01fff8 thru 0x023e23 (0x3e2c bytes) 000141 1788511539.583413 mapread 0x0248d8 thru 0x034462 (0xfb8b bytes) 000142 1788511539.588802 read 0x005272 thru 0x011f54 (0xcce3 bytes) 000143 1788511539.600821 mapwrite 0x039d7f thru 0x03ffff (0x6281 bytes) 000144 1788511539.659225 read 0x01e032 thru 0x020c9e (0x2c6d bytes) 000145 1788511539.660658 mapwrite 0x034429 thru 0x03ffff (0xbbd7 bytes) 000146 1788511539.680611 read 0x002a4e thru 0x00de71 (0xb424 bytes) 000147 1788511539.681912 trunc from 0x03b264 to 0x03d3be (0x215b bytes) 000148 1788511539.688117 write 0x01aa17 thru 0x01ddef (0x33d9 bytes) 000149 1788511539.690471 trunc from 0x002e8d to 0x00e59b (0xb70f bytes) 000151 1788511539.695981 write 0x02b7a3 thru 0x039aa3 (0xe301 bytes) 000152 1788511539.702550 MIRROR_RESYNC 2 mirrors 000153 1788511540.331519 mapread 0x0304d0 thru 0x031074 (0xba5 bytes) 000154 1788511540.342938 punch from 0x02f2f5 to 0x039926 (0xa632 bytes) 000155 1788511540.403554 trunc from 0x037673 to 0x03ad70 (0x36fe bytes) 000156 1788511540.409636 write 0x015062 thru 0x019a0a (0x49a9 bytes) 000158 1788511540.421166 read 0x03f967 thru 0x03ffff (0x699 bytes) 000159 1788511540.427034 write 0x02bd5f thru 0x032ac4 (0x6d66 bytes) 000160 1788511540.440839 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd86f092d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x35b50a0e Mirror 3: 0x35b50a0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000161 1788511541.416942 punch from 0x02bde6 to 0x0354b8 (0x96d3 bytes) 000162 1788511541.454196 mapread 0x00dbe1 thru 0x01db06 (0xff26 bytes) 000163 1788511541.484767 mapread 0x0233a3 thru 0x02f774 (0xc3d2 bytes) 000164 1788511541.504557 read 0x01867f thru 0x02399f (0xb321 bytes) 000165 1788511541.505284 mapwrite 0x030d5f thru 0x03e095 (0xd337 bytes) 000166 1788511541.553853 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x35b50a0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x35b50a0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000167 1788511542.023161 trunc from 0x040000 to 0x031ece (0xe131 bytes) 000169 1788511542.108808 trunc from 0x031ecf to 0x031d55 (0x179 bytes) 000170 1788511542.127797 write 0x01d55a thru 0x0216d2 (0x4179 bytes) 000171 1788511542.145425 MIRROR_RESYNC 2 mirrors 000172 1788511543.210182 trunc from 0x00bb70 to 0x0153a3 (0x9834 bytes) 000173 1788511543.247485 fallocate 0x009bf6 thru 0x013bf7 (0xa002 bytes) 000176 1788511543.255083 trunc from 0x00e41b to 0x014377 (0x5f5d bytes) 000177 1788511543.260860 read 0x0025b7 thru 0x007663 (0x50ad bytes) 000178 1788511543.273961 trunc from 0x02fc4c to 0x0319a6 (0x1d5b bytes) 000179 1788511543.280064 mapread 0x01285e thru 0x01f4d9 (0xcc7c bytes) 000180 1788511543.305017 punch from 0x02b6a7 to 0x031d55 (0x66af bytes) 000181 1788511543.310135 mapwrite 0x0121d0 thru 0x014ecf (0x2d00 bytes) 000182 1788511543.322893 mapread 0x0213ef thru 0x02b337 (0x9f49 bytes) 000183 1788511543.338485 MIRROR_RESYNC 2 mirrors 000184 1788511544.032209 mapwrite 0x0280e6 thru 0x02ed73 (0x6c8e bytes) 000185 1788511544.098082 punch from 0x000008 to 0x00fb01 (0xfafa bytes) 000186 1788511544.109665 MIRROR_RESYNC 2 mirrors 000187 1788511545.157571 write 0x03ab8c thru 0x03ffff (0x5474 bytes) 000188 1788511545.244122 write 0x019539 thru 0x01e49c (0x4f64 bytes) 000189 1788511545.263416 mapwrite 0x01a4e6 thru 0x02292f (0x844a bytes) 000190 1788511545.356198 trunc from 0x012fbb to 0x019dd8 (0x6e1e bytes) 000191 1788511545.368675 read 0x0247a6 thru 0x0343c3 (0xfc1e bytes) 000192 1788511545.377721 mapread 0x026250 thru 0x02bf8e (0x5d3f bytes) 000193 1788511545.384696 MIRROR_RESYNC 2 mirrors 000194 1788511546.603317 MIRROR_EXTEND to 3 mirrors 000195 1788511548.079836 read 0x0149d9 thru 0x01df31 (0x9559 bytes) 000196 1788511548.089916 fallocate 0x038038 thru 0x0396bc (0x1685 bytes) 000197 1788511548.148566 trunc from 0x040000 to 0x0001b9 (0x3fe46 bytes) 000198 1788511548.188093 trunc from 0x0001ba to 0x03b88a (0x3b6d1 bytes) 000199 1788511548.220726 trunc from 0x005601 to 0x01416a (0xeb6a bytes) 000200 1788511548.228018 punch from 0x0060bb to 0x00a1d4 (0x411a bytes) 000201 1788511548.242542 write 0x030814 thru 0x03fcc2 (0xf4af bytes) 000202 1788511548.254631 punch from 0x034242 to 0x036a87 (0x2846 bytes) 000203 1788511548.273912 mapwrite 0x00bff3 thru 0x018279 (0xc287 bytes) 000204 1788511548.364157 mapread 0x002430 thru 0x011005 (0xebd6 bytes) 000205 1788511548.378748 read 0x01f8d6 thru 0x02a1a0 (0xa8cb bytes) 000206 1788511548.395207 trunc from 0x025419 to 0x02b456 (0x603e bytes) 000207 1788511548.401347 mapwrite 0x00c312 thru 0x00f5f7 (0x32e6 bytes) 000208 1788511548.432004 punch from 0x0279d2 to 0x03482a (0xce59 bytes) 000209 1788511548.443643 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fcc3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fcc3): Mirror 3: 0x77c02376 (stale) Mirror 4: 0x77c02376 (stale) CRC-32 checksum value for chunk [0, 0x3fcc3): Mirror 1: 0xb9287e63 Mirror 5: 0xb9287e63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fcc3: skipped 000211 1788511549.757922 mapwrite 0x00b9e6 thru 0x011702 (0x5d1d bytes) 000212 1788511549.872691 mapread 0x00f98c thru 0x017e45 (0x84ba bytes) 000213 1788511549.882018 write 0x002fa1 thru 0x011e7d (0xeedd bytes) 000214 1788511549.894102 write 0x020a8a thru 0x02898b (0x7f02 bytes) 000216 1788511549.914344 mapread 0x00cd47 thru 0x017017 (0xa2d1 bytes) 000217 1788511549.915349 read 0x02cedf thru 0x033548 (0x666a bytes) 000218 1788511549.927727 read 0x03e0d0 thru 0x03fcc2 (0x1bf3 bytes) 000219 1788511549.940452 trunc from 0x03fcc3 to 0x008674 (0x3764e bytes) 000220 1788511550.011356 write 0x02ccb0 thru 0x02ddb6 (0x1107 bytes) 000221 1788511550.013376 punch from 0x011dda to 0x017bad (0x5dd4 bytes) 000222 1788511550.017937 mapread 0x00d5bc thru 0x014c14 (0x7659 bytes) 000223 1788511550.110365 MIRROR_RESYNC 4 mirrors 000224 1788511551.099224 MIRROR_EXTEND to 5 mirrors 000225 1788511552.173792 mapread 0x014ab5 thru 0x01a449 (0x5995 bytes) 000226 1788511552.191041 read 0x011079 thru 0x01ddb3 (0xcd3b bytes) 000227 1788511552.196362 MIRROR_RESYNC 5 mirrors 000228 1788511552.569147 MIRROR_EXTEND to 6 mirrors 000229 1788511553.273764 MIRROR_EXTEND to 7 mirrors 000230 1788511554.102379 trunc from 0x011a70 to 0x020730 (0xecc1 bytes) 000231 1788511554.148407 mapread 0x0095ce thru 0x014f02 (0xb935 bytes) 000232 1788511554.182104 punch from 0x00f9cc to 0x019d9a (0xa3cf bytes) 000233 1788511554.187634 mapread 0x001c8e thru 0x008a99 (0x6e0c bytes) 000234 1788511554.195004 punch from 0x010d8d to 0x0128ff (0x1b73 bytes) 000235 1788511554.198872 read 0x024210 thru 0x029c6f (0x5a60 bytes) 000236 1788511554.213399 mapwrite 0x01a122 thru 0x01fb3a (0x5a19 bytes) 000237 1788511554.238812 mapwrite 0x037d90 thru 0x03ffff (0x8270 bytes) 000238 1788511554.304677 read 0x00fa9b thru 0x017a71 (0x7fd7 bytes) 000239 1788511554.320092 MIRROR_RESYNC 7 mirrors 000240 1788511557.136088 trunc from 0x040000 to 0x01629b (0x29d64 bytes) 000241 1788511557.192719 mapread 0x00cc6b thru 0x010c21 (0x3fb7 bytes) 000242 1788511557.221466 trunc from 0x01629c to 0x025442 (0xf1a7 bytes) 000243 1788511557.242452 read 0x021555 thru 0x025442 (0x3eee bytes) 000244 1788511557.250521 mapread 0x00eeaa thru 0x00f400 (0x557 bytes) 000245 1788511557.252180 fallocate 0x0090b2 thru 0x00bdba (0x2d09 bytes) 000246 1788511557.258508 read 0x016d7a thru 0x01fb7c (0x8e03 bytes) 000248 1788511557.269838 read 0x000b85 thru 0x001570 (0x9ec bytes) 000249 1788511557.270567 trunc from 0x025443 to 0x03690f (0x114cd bytes) 000250 1788511557.289916 trunc from 0x036910 to 0x03355b (0x33b4 bytes) 000251 1788511557.318738 write 0x018c9e thru 0x01aa6c (0x1dcf bytes) 000253 1788511557.320927 trunc from 0x0154ca to 0x01fee9 (0xaa20 bytes) 000255 1788511557.345826 fallocate 0x025045 thru 0x0347b6 (0xf772 bytes) 000257 1788511557.357565 mapwrite 0x026454 thru 0x032933 (0xc4e0 bytes) 000258 1788511557.399885 write 0x03bb89 thru 0x03ecd9 (0x3151 bytes) 000259 1788511557.401522 read 0x01bd5e thru 0x02a19c (0xe43f bytes) 000260 1788511557.402278 punch from 0x00fc67 to 0x01d80c (0xdba6 bytes) 000261 1788511557.407577 punch from 0x02ecc3 to 0x02fcc7 (0x1005 bytes) 000262 1788511557.414872 read 0x011f2c thru 0x015c01 (0x3cd6 bytes) 000264 1788511557.448789 fallocate 0x021c24 thru 0x02795c (0x5d39 bytes) 000265 1788511557.458156 trunc from 0x01310c to 0x0224a7 (0xf39c bytes) 000266 1788511557.469894 fallocate 0x015598 thru 0x016dcb (0x1834 bytes) 000267 1788511557.476880 read 0x007214 thru 0x010e10 (0x9bfd bytes) 000268 1788511557.479192 fallocate 0x0122da thru 0x019f43 (0x7c6a bytes) 000269 1788511557.485297 read 0x00aa73 thru 0x018527 (0xdab5 bytes) 000270 1788511557.516888 write 0x01c0a1 thru 0x01c5ea (0x54a bytes) 000271 1788511557.525339 fallocate 0x0234bd thru 0x0242d0 (0xe14 bytes) 000272 1788511557.552846 write 0x017445 thru 0x0262b6 (0xee72 bytes) 000273 1788511557.554399 write 0x036557 thru 0x03a9f0 (0x449a bytes) 000274 1788511557.564770 mapwrite 0x016ab8 thru 0x025ad0 (0xf019 bytes) 000275 1788511557.613487 MIRROR_RESYNC 7 mirrors 000277 1788511559.754164 mapread 0x002767 thru 0x00dcb8 (0xb552 bytes) 000278 1788511559.799225 fallocate 0x0347d7 thru 0x03d834 (0x905e bytes) 000279 1788511559.858574 read 0x01ec6c thru 0x026b08 (0x7e9d bytes) 000280 1788511559.878865 read 0x015576 thru 0x024629 (0xf0b4 bytes) 000281 1788511559.888810 trunc from 0x01fef5 to 0x0275d9 (0x76e5 bytes) 000282 1788511559.897521 mapread 0x009675 thru 0x00ceeb (0x3877 bytes) 000283 1788511559.898401 write 0x02f60b thru 0x02fecf (0x8c5 bytes) 000284 1788511559.899288 write 0x00b168 thru 0x00df84 (0x2e1d bytes) 000285 1788511559.909550 punch from 0x02d2e6 to 0x03276d (0x5488 bytes) 000287 1788511559.943597 MIRROR_RESYNC 7 mirrors 000288 1788511564.630603 mapwrite 0x0211ff thru 0x0267ce (0x55d0 bytes) 000290 1788511564.823326 trunc from 0x00ae13 to 0x0151b0 (0xa39e bytes) 000291 1788511564.863905 trunc from 0x03ecda to 0x0060a6 (0x38c33 bytes) 000292 1788511564.921349 punch from 0x000959 to 0x003cc4 (0x336c bytes) 000294 1788511564.960685 read 0x004b3d thru 0x0060a6 (0x156a bytes) 000295 1788511564.973764 trunc from 0x0018af to 0x0060a6 (0x47f8 bytes) 000297 1788511564.980666 trunc from 0x001abc to 0x0060a6 (0x45eb bytes) 000299 1788511564.991794 fallocate 0x01ac2f thru 0x0225e4 (0x79b6 bytes) 000300 1788511564.999955 mapread 0x0000b7 thru 0x0060a6 (0x5ff0 bytes) 000302 1788511565.014378 mapwrite 0x01f8dc thru 0x0234de (0x3c03 bytes) 000303 1788511565.138174 punch from 0x015eaf to 0x022e32 (0xcf84 bytes) 000304 1788511565.155296 trunc from 0x00ebdf to 0x0190f7 (0xa519 bytes) 000305 1788511565.177627 mapwrite 0x02e392 thru 0x03cc0e (0xe87d bytes) 000306 1788511565.329188 mapread 0x018bda thru 0x02877f (0xfba6 bytes) 000307 1788511565.397415 trunc from 0x007253 to 0x010fbc (0x9d6a bytes) 000308 1788511565.405193 fallocate 0x03e241 thru 0x03ffff (0x1dbf bytes) 000309 1788511565.419985 read 0x03baae thru 0x03ffff (0x4552 bytes) 000310 1788511565.444199 mapwrite 0x026967 thru 0x02dcfa (0x7394 bytes) 000311 1788511565.474110 read 0x0157fe thru 0x01f12a (0x992d bytes) 000312 1788511565.474783 read 0x033282 thru 0x035c13 (0x2992 bytes) 000313 1788511565.484829 MIRROR_RESYNC 7 mirrors 000314 1788511569.196647 write 0x00f90a thru 0x0191e9 (0x98e0 bytes) 000315 1788511569.297202 trunc from 0x0157e7 to 0x020e61 (0xb67b bytes) 000316 1788511569.329517 mapread 0x03316e thru 0x03b217 (0x80aa bytes) 000317 1788511569.375096 trunc from 0x040000 to 0x0021d1 (0x3de2e bytes) 000318 1788511569.440239 trunc from 0x0021d2 to 0x030148 (0x2df77 bytes) 000319 1788511569.476198 punch from 0x02dd6c to 0x030148 (0x23dd bytes) 000320 1788511569.484413 read 0x00ab13 thru 0x00da4b (0x2f39 bytes) 000321 1788511569.494069 trunc from 0x030149 to 0x0323e3 (0x229b bytes) 000323 1788511569.521187 write 0x00e7e4 thru 0x015461 (0x6c7e bytes) 000324 1788511569.547108 mapwrite 0x01a2b2 thru 0x022b6d (0x88bc bytes) 000325 1788511569.592124 trunc from 0x0323e4 to 0x01ca94 (0x1594f bytes) 000326 1788511569.631942 trunc from 0x01ca95 to 0x033c71 (0x171dd bytes) 000327 1788511569.651841 mapwrite 0x017e1d thru 0x020d79 (0x8f5d bytes) 000328 1788511569.682109 MIRROR_RESYNC 7 mirrors 000329 1788511572.564211 fallocate 0x007f50 thru 0x00fe5c (0x7f0d bytes) 000330 1788511572.630721 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c72): Mirror 3: 0x9f887d2f (stale) Mirror 4: 0x9f887d2f (stale) Mirror 6: 0x9f887d2f (stale) Mirror 7: 0x9f887d2f (stale) Mirror 8: 0x9f887d2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c72): Mirror 3: 0x9f887d2f (stale) Mirror 4: 0x9f887d2f (stale) Mirror 6: 0x9f887d2f (stale) Mirror 7: 0x9f887d2f (stale) Mirror 8: 0x9f887d2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c72: skipped 000331 1788511573.371947 mapwrite 0x02515b thru 0x033f6b (0xee11 bytes) 000332 1788511573.444551 mapread 0x028047 thru 0x02eefb (0x6eb5 bytes) 000333 1788511573.445493 punch from 0x01ed73 to 0x02a762 (0xb9f0 bytes) 000334 1788511573.450659 read 0x01798a thru 0x01e695 (0x6d0c bytes) 000336 1788511573.466015 write 0x036cdb thru 0x03ffff (0x9325 bytes) 000337 1788511573.469175 punch from 0x036e01 to 0x03edb0 (0x7fb0 bytes) 000338 1788511573.484756 fallocate 0x02f315 thru 0x0307d3 (0x14bf bytes) 000339 1788511573.489019 mapread 0x03e392 thru 0x03ffff (0x1c6e bytes) 000340 1788511573.500678 mapwrite 0x0375f0 thru 0x03a33d (0x2d4e bytes) 000341 1788511573.518050 trunc from 0x00d1ee to 0x018f13 (0xbd26 bytes) 000342 1788511573.524726 mapwrite 0x023dcc thru 0x02e1e2 (0xa417 bytes) 000343 1788511573.571154 mapwrite 0x022dc0 thru 0x02db7f (0xadc0 bytes) 000344 1788511573.598192 fallocate 0x026bc2 thru 0x031736 (0xab75 bytes) 000345 1788511573.614942 punch from 0x00ad6a to 0x0108cb (0x5b62 bytes) 000346 1788511573.627079 read 0x021750 thru 0x02d278 (0xbb29 bytes) 000347 1788511573.630011 mapwrite 0x022f8f thru 0x023c46 (0xcb8 bytes) 000348 1788511573.646600 mapread 0x0026bc thru 0x00ee28 (0xc76d bytes) 000349 1788511573.666854 read 0x0272f1 thru 0x02db34 (0x6844 bytes) 000350 1788511573.667483 trunc from 0x03a3bc to 0x03ffff (0x5c44 bytes) 000351 1788511573.677703 trunc from 0x040000 to 0x0140ef (0x2bf10 bytes) 000352 1788511573.693133 write 0x001475 thru 0x01047f (0xf00b bytes) 000354 1788511573.699240 mapwrite 0x02e12a thru 0x03df95 (0xfe6c bytes) 000355 1788511573.760861 fallocate 0x0364b6 thru 0x03d4f0 (0x703b bytes) 000356 1788511573.767167 trunc from 0x03df96 to 0x00e456 (0x2fb3f bytes) 000357 1788511573.796907 fallocate 0x01435d thru 0x020f5e (0xcc02 bytes) 000358 1788511573.803027 trunc from 0x0008f0 to 0x004c39 (0x434a bytes) 000359 1788511573.809910 mapread 0x00ac00 thru 0x00e456 (0x3857 bytes) 000360 1788511573.813024 mapread 0x005ea4 thru 0x00e456 (0x85b3 bytes) 000361 1788511573.815596 mapwrite 0x03190e thru 0x03a182 (0x8875 bytes) 000362 1788511573.929213 fallocate 0x00f3da thru 0x0137a0 (0x43c7 bytes) 000363 1788511573.934861 trunc from 0x021bf5 to 0x028e00 (0x720c bytes) 000364 1788511573.940240 write 0x0315fc thru 0x03f0ab (0xdab0 bytes) 000365 1788511573.941972 trunc from 0x022436 to 0x02771a (0x52e5 bytes) 000366 1788511573.945918 trunc from 0x004832 to 0x00e6af (0x9e7e bytes) 000367 1788511573.950781 punch from 0x0062e3 to 0x014364 (0xe082 bytes) 000369 1788511573.959157 MIRROR_RESYNC 6 mirrors 000370 1788511575.982885 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000371 1788511576.558850 trunc from 0x008477 to 0x013ee5 (0xba6f bytes) 000372 1788511576.639427 write 0x004738 thru 0x013292 (0xeb5b bytes) 000373 1788511576.668166 read 0x01c5da thru 0x01f82f (0x3256 bytes) 000374 1788511576.688756 read 0x00c99a thru 0x010587 (0x3bee bytes) 000375 1788511576.690705 mapread 0x019eff thru 0x01ee29 (0x4f2b bytes) 000377 1788511576.731457 punch from 0x0302dd to 0x0337f3 (0x3517 bytes) 000378 1788511576.745593 fallocate 0x0172ce thru 0x01c51b (0x524e bytes) 000379 1788511576.753338 trunc from 0x009ba9 to 0x013135 (0x958d bytes) 000380 1788511576.777420 trunc from 0x0366ff to 0x0377bf (0x10c1 bytes) 000381 1788511576.793991 MIRROR_RESYNC 5 mirrors 000383 1788511578.873937 mapread 0x035b6a thru 0x03f0ab (0x9542 bytes) 000384 1788511578.905026 fallocate 0x03268b thru 0x03d750 (0xb0c6 bytes) 000385 1788511578.956141 punch from 0x01ac05 to 0x020c0f (0x600b bytes) 000386 1788511578.963077 punch from 0x0274c3 to 0x034bbf (0xd6fd bytes) 000387 1788511578.969884 punch from 0x02d22f to 0x03af5a (0xdd2c bytes) 000388 1788511578.979713 MIRROR_RESYNC 5 mirrors 000389 1788511580.389021 read 0x03e1f3 thru 0x03f0ab (0xeb9 bytes) 000390 1788511580.397363 write 0x0135e6 thru 0x0175b7 (0x3fd2 bytes) 000391 1788511580.538905 trunc from 0x03f0ac to 0x00a1da (0x34ed1 bytes) 000394 1788511580.630709 MIRROR_RESYNC 5 mirrors 000395 1788511582.035624 mapread 0x004a31 thru 0x00a1da (0x57aa bytes) 000396 1788511582.087829 trunc from 0x00a1db to 0x02206b (0x17e91 bytes) 000397 1788511582.174576 trunc from 0x02206c to 0x038329 (0x162be bytes) 000398 1788511582.215149 fallocate 0x004326 thru 0x007bee (0x38c9 bytes) 000399 1788511582.224089 mapwrite 0x02307c thru 0x031663 (0xe5e8 bytes) 000400 1788511582.306083 trunc from 0x03832a to 0x00fffd (0x2832c bytes) 000401 1788511582.343235 mapread 0x00991e thru 0x00fffd (0x66e0 bytes) 000402 1788511582.367304 mapread 0x006399 thru 0x0098e9 (0x3551 bytes) 000403 1788511582.370337 mapread 0x00c8d8 thru 0x00fffd (0x3726 bytes) 000404 1788511582.377971 write 0x02f661 thru 0x030d9b (0x173b bytes) 000405 1788511582.381110 write 0x00ed65 thru 0x01d192 (0xe42e bytes) 000406 1788511582.394331 trunc from 0x0136f8 to 0x0190a6 (0x59af bytes) 000407 1788511582.452568 MIRROR_RESYNC 5 mirrors 000408 1788511584.559808 trunc from 0x030d9c to 0x03f419 (0xe67e bytes) 000409 1788511584.636604 mapread 0x004964 thru 0x004d54 (0x3f1 bytes) 000410 1788511584.642069 mapwrite 0x01ebb9 thru 0x02b59a (0xc9e2 bytes) 000411 1788511584.717044 mapread 0x03c3a6 thru 0x03f419 (0x3074 bytes) 000412 1788511584.786393 trunc from 0x01dfc4 to 0x01f228 (0x1265 bytes) 000413 1788511584.806605 fallocate 0x002a5d thru 0x00ebb8 (0xc15c bytes) 000414 1788511584.815447 mapwrite 0x00c0bb thru 0x019d0e (0xdc54 bytes) 000415 1788511584.966733 trunc from 0x03f41a to 0x037335 (0x80e4 bytes) 000417 1788511585.038580 read 0x016099 thru 0x025ba9 (0xfb11 bytes) 000418 1788511585.057394 trunc from 0x037336 to 0x005cc6 (0x3166f bytes) 000419 1788511585.085717 mapwrite 0x01caf1 thru 0x0297bb (0xcccb bytes) 000420 1788511585.187013 mapwrite 0x029da8 thru 0x02fbf4 (0x5e4d bytes) 000421 1788511585.233991 mapread 0x008c7d thru 0x00a939 (0x1cbd bytes) 000422 1788511585.249317 fallocate 0x02a459 thru 0x02c5ca (0x2172 bytes) 000424 1788511585.259227 read 0x0175fb thru 0x01eb28 (0x752e bytes) 000425 1788511585.259857 fallocate 0x0055b4 thru 0x00f89f (0xa2ec bytes) 000426 1788511585.264772 read 0x0061d5 thru 0x00cae5 (0x6911 bytes) 000427 1788511585.269858 trunc from 0x02e540 to 0x02fbf4 (0x16b5 bytes) 000428 1788511585.286344 trunc from 0x02fbf5 to 0x014720 (0x1b4d4 bytes) 000429 1788511585.332588 write 0x02f445 thru 0x036602 (0x71be bytes) 000430 1788511585.336207 trunc from 0x009eff to 0x014442 (0xa544 bytes) 000431 1788511585.342850 trunc from 0x0075d0 to 0x00debf (0x68f0 bytes) 000432 1788511585.356452 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36603): Mirror 3: 0x8e1a4312 (stale) Mirror 4: 0x8e1a4312 (stale) Mirror 8: 0x8e1a4312 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36603): Mirror 3: 0x8e1a4312 (stale) Mirror 4: 0x8e1a4312 (stale) Mirror 8: 0x8e1a4312 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36603: skipped 000434 1788511585.998136 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36603): Mirror 4: 0x8e1a4312 (stale) Mirror 8: 0x8e1a4312 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36603): Mirror 4: 0x8e1a4312 (stale) Mirror 8: 0x8e1a4312 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36603: skipped 000435 1788511586.673210 trunc from 0x036603 to 0x00f9ee (0x26c14 bytes) 000436 1788511586.700227 read 0x00e277 thru 0x00f9ee (0x1778 bytes) 000437 1788511586.707605 write 0x03a3d1 thru 0x03d03f (0x2c6f bytes) 000438 1788511586.712373 read 0x01a383 thru 0x02663a (0xc2b8 bytes) 000439 1788511586.724448 write 0x02e1c6 thru 0x035cc7 (0x7b02 bytes) 000440 1788511586.725742 mapwrite 0x00d9a1 thru 0x00ff88 (0x25e8 bytes) 000441 1788511586.749780 mapread 0x0250cb thru 0x032ec1 (0xddf7 bytes) 000442 1788511586.751080 trunc from 0x03d040 to 0x0145ee (0x28a51 bytes) 000443 1788511586.765396 mapread 0x003d6c thru 0x013b6d (0xfe02 bytes) 000444 1788511586.767536 trunc from 0x0103c3 to 0x010981 (0x5bf bytes) 000445 1788511586.772406 read 0x007326 thru 0x0121b1 (0xae8c bytes) 000446 1788511586.778415 trunc from 0x01146b to 0x0145ee (0x3184 bytes) 000447 1788511586.783673 fallocate 0x022088 thru 0x029019 (0x6f92 bytes) 000448 1788511586.788766 punch from 0x00e44b to 0x012742 (0x42f8 bytes) 000449 1788511586.793729 write 0x02952c thru 0x0296bc (0x191 bytes) 000450 1788511586.795401 punch from 0x01da64 to 0x027944 (0x9ee1 bytes) 000451 1788511586.799474 punch from 0x028c62 to 0x0296bc (0xa5b bytes) 000452 1788511586.808454 fallocate 0x00454e thru 0x013374 (0xee27 bytes) 000453 1788511586.813438 write 0x03bccc thru 0x03ffff (0x4334 bytes) 000455 1788511586.815160 trunc from 0x040000 to 0x005ca4 (0x3a35b bytes) 000456 1788511586.837440 read 0x0020cc thru 0x005ca4 (0x3bd9 bytes) 000457 1788511586.838647 trunc from 0x0026bd to 0x005ca4 (0x35e8 bytes) 000458 1788511586.842966 write 0x01ffde thru 0x0236fa (0x371d bytes) 000459 1788511586.845876 fallocate 0x01efd3 thru 0x027007 (0x8035 bytes) 000460 1788511586.850742 mapwrite 0x02b971 thru 0x02bead (0x53d bytes) 000461 1788511586.902135 mapread 0x003d4f thru 0x00ed8c (0xb03e bytes) 000462 1788511586.903618 mapwrite 0x01c88b thru 0x021732 (0x4ea8 bytes) 000463 1788511586.919168 trunc from 0x02beae to 0x0301f4 (0x4347 bytes) 000464 1788511586.936683 mapread 0x00e7d4 thru 0x011a0b (0x3238 bytes) 000465 1788511586.937634 mapread 0x0038c2 thru 0x00e8ca (0xb009 bytes) 000466 1788511586.938482 read 0x01c3fa thru 0x0259e9 (0x95f0 bytes) 000467 1788511586.939037 fallocate 0x000310 thru 0x002cb6 (0x29a7 bytes) 000468 1788511586.948301 MIRROR_RESYNC 3 mirrors 000469 1788511588.353467 mapwrite 0x039b37 thru 0x03ffff (0x64c9 bytes) 000470 1788511588.487670 trunc from 0x02d6a9 to 0x038d34 (0xb68c bytes) 000471 1788511588.504894 write 0x032106 thru 0x03aa85 (0x8980 bytes) 000473 1788511588.512456 trunc from 0x040000 to 0x01b0fc (0x24f03 bytes) 000474 1788511588.543400 read 0x016980 thru 0x01b0fc (0x477d bytes) 000475 1788511588.555369 trunc from 0x014946 to 0x014979 (0x34 bytes) 000476 1788511588.562074 punch from 0x01970d to 0x01b0fc (0x19f0 bytes) 000477 1788511588.568519 read 0x0004fd thru 0x00ad14 (0xa818 bytes) 000478 1788511588.586030 punch from 0x00a4e5 to 0x01a3af (0xfecb bytes) 000479 1788511588.594687 read 0x00bb02 thru 0x0189de (0xcedd bytes) 000480 1788511588.606636 trunc from 0x01b0fd to 0x02027b (0x517f bytes) 000481 1788511588.622652 read 0x00cea5 thru 0x01218e (0x52ea bytes) 000482 1788511588.623401 write 0x03638e thru 0x03b10e (0x4d81 bytes) 000483 1788511588.625734 fallocate 0x020690 thru 0x020cce (0x63f bytes) 000484 1788511588.631448 punch from 0x01415a to 0x01f76b (0xb612 bytes) 000485 1788511588.636189 trunc from 0x011cf0 to 0x0155c0 (0x38d1 bytes) 000486 1788511588.639813 punch from 0x0183a9 to 0x0249ed (0xc645 bytes) 000487 1788511588.644539 mapwrite 0x02ce2e thru 0x02f4ef (0x26c2 bytes) 000488 1788511588.670344 write 0x016096 thru 0x01b8bf (0x582a bytes) 000490 1788511588.672665 trunc from 0x03b10f to 0x002df7 (0x38317 bytes) 000491 1788511588.689056 read 0x002179 thru 0x002df7 (0xc7f bytes) 000492 1788511588.689695 mapwrite 0x023e59 thru 0x02d0e1 (0x9289 bytes) 000493 1788511588.734390 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d0e2): Mirror 4: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0e2): Mirror 4: 0xb9dc993f (stale) Mirror 8: 0xb9dc993f (stale) CRC-32 checksum value for chunk [0, 0x2d0e2): Mirror 1: 0x563d4d8 Mirror 9: 0x563d4d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0e2: skipped 000494 1788511589.290634 write 0x01fdd7 thru 0x02ec38 (0xee62 bytes) 000495 1788511589.334798 trunc from 0x02ec39 to 0x03bd7d (0xd145 bytes) 000496 1788511589.368035 punch from 0x02b6b6 to 0x034895 (0x91e0 bytes) 000497 1788511589.385669 punch from 0x0066f0 to 0x00ad27 (0x4638 bytes) 000498 1788511589.390904 mapwrite 0x01e056 thru 0x023fb3 (0x5f5e bytes) 000499 1788511589.411144 write 0x030963 thru 0x035460 (0x4afe bytes) 000500 1788511589.421102 mapread 0x029e3a thru 0x036ca9 (0xce70 bytes) 000502 1788511589.440296 write 0x02d8e0 thru 0x038190 (0xa8b1 bytes) 000505 1788511589.444427 trunc from 0x03bd7e to 0x002b50 (0x3922d bytes) 000506 1788511589.457742 trunc from 0x002782 to 0x002b50 (0x3cf bytes) 000507 1788511589.460720 mapwrite 0x034e8a thru 0x03f82d (0xa9a4 bytes) 000508 1788511589.514082 trunc from 0x014931 to 0x0212d7 (0xc9a7 bytes) 000509 1788511589.522505 write 0x039b52 thru 0x03bce4 (0x2193 bytes) 000510 1788511589.526961 trunc from 0x03f82e to 0x029820 (0x1600d bytes) 000511 1788511589.554864 fallocate 0x00a9e2 thru 0x01a948 (0xff67 bytes) 000512 1788511589.562269 trunc from 0x01e503 to 0x027cc0 (0x97be bytes) 000513 1788511589.568390 trunc from 0x029821 to 0x030f71 (0x7751 bytes) 000514 1788511589.585595 mapwrite 0x00c57f thru 0x01b223 (0xeca5 bytes) 000515 1788511589.634573 punch from 0x00015c to 0x00d632 (0xd4d7 bytes) 000516 1788511589.646966 MIRROR_RESYNC 4 mirrors 000517 1788511590.821427 fallocate 0x039e2c thru 0x03e2a8 (0x447d bytes) 000518 1788511590.875026 mapread 0x020cac thru 0x0217c9 (0xb1e bytes) 000519 1788511590.884574 punch from 0x002438 to 0x00c128 (0x9cf1 bytes) 000521 1788511590.889877 trunc from 0x030f72 to 0x03462f (0x36be bytes) 000522 1788511590.905538 fallocate 0x02677f thru 0x02fe53 (0x96d5 bytes) 000523 1788511590.914408 write 0x0007af thru 0x00fd2e (0xf580 bytes) 000524 1788511590.935036 fallocate 0x022b46 thru 0x02671d (0x3bd8 bytes) 000525 1788511590.944785 mapread 0x021128 thru 0x02abe4 (0x9abd bytes) 000526 1788511590.969539 mapread 0x02afdc thru 0x03462f (0x9654 bytes) 000527 1788511591.000963 punch from 0x02733d to 0x02f907 (0x85cb bytes) 000528 1788511591.014133 mapread 0x0091d1 thru 0x013825 (0xa655 bytes) 000529 1788511591.036584 write 0x02295c thru 0x02b1cb (0x8870 bytes) 000530 1788511591.062521 fallocate 0x02bae3 thru 0x0371e3 (0xb701 bytes) 000531 1788511591.074816 punch from 0x01fdef to 0x021abb (0x1ccd bytes) 000532 1788511591.089855 punch from 0x0209c7 to 0x021a48 (0x1082 bytes) 000533 1788511591.099451 mapread 0x0118b5 thru 0x01e859 (0xcfa5 bytes) 000535 1788511591.117314 write 0x014f7d thru 0x015ac3 (0xb47 bytes) 000536 1788511591.118212 write 0x036efe thru 0x03ffff (0x9102 bytes) 000537 1788511591.130686 punch from 0x01105b to 0x01f2d7 (0xe27d bytes) 000538 1788511591.175703 punch from 0x003b28 to 0x0111e6 (0xd6bf bytes) 000540 1788511591.199763 mapwrite 0x035643 thru 0x03ffff (0xa9bd bytes) 000542 1788511591.293197 write 0x03f5a7 thru 0x03ffff (0xa59 bytes) 000543 1788511591.294223 trunc from 0x013b7a to 0x017975 (0x3dfc bytes) 000544 1788511591.305082 write 0x006977 thru 0x00e555 (0x7bdf bytes) 000545 1788511591.317379 trunc from 0x040000 to 0x010ab6 (0x2f549 bytes) 000546 1788511591.364940 trunc from 0x00c643 to 0x010ab6 (0x4474 bytes) 000547 1788511591.404697 fallocate 0x00e664 thru 0x01a391 (0xbd2e bytes) 000548 1788511591.415978 trunc from 0x003015 to 0x00bf31 (0x8f1d bytes) 000549 1788511591.424297 mapwrite 0x02a4f8 thru 0x0300ba (0x5bc3 bytes) 000550 1788511591.521055 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x300bb): Mirror 4: 0x694886ea (stale) Mirror 8: 0x694886ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x300bb): Mirror 4: 0x694886ea (stale) Mirror 8: 0x694886ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x300bb: skipped 000551 1788511591.962276 read 0x029744 thru 0x02b776 (0x2033 bytes) 000553 1788511591.991115 punch from 0x01aa75 to 0x029457 (0xe9e3 bytes) 000554 1788511592.000626 trunc from 0x0300bb to 0x00c4cd (0x23bed bytes) 000555 1788511592.026531 fallocate 0x0186b3 thru 0x01bc3a (0x3588 bytes) 000556 1788511592.038652 mapread 0x009430 thru 0x014c64 (0xb835 bytes) 000557 1788511592.105196 trunc from 0x000b82 to 0x00a7c1 (0x9c40 bytes) 000558 1788511592.122621 punch from 0x00dbbc to 0x012d79 (0x51be bytes) 000559 1788511592.143749 mapwrite 0x02ec09 thru 0x031795 (0x2b8d bytes) 000560 1788511592.271506 mapread 0x01af43 thru 0x0214b1 (0x656f bytes) 000562 1788511592.316947 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31796): Mirror 8: 0x311f3b34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31796): Mirror 8: 0x311f3b34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31796: skipped 000563 1788511592.696491 write 0x03f8a4 thru 0x03ffff (0x75c bytes) 000564 1788511592.704535 mapread 0x037ecd thru 0x039787 (0x18bb bytes) 000565 1788511592.719569 mapread 0x034fbf thru 0x03ffff (0xb041 bytes) 000566 1788511592.721104 write 0x03454e thru 0x03ffff (0xbab2 bytes) 000567 1788511592.723090 mapread 0x03d970 thru 0x03ffff (0x2690 bytes) 000568 1788511592.724085 mapread 0x02d821 thru 0x03b4c0 (0xdca0 bytes) 000569 1788511592.733402 trunc from 0x002d02 to 0x005439 (0x2738 bytes) 000570 1788511592.737354 trunc from 0x040000 to 0x005774 (0x3a88b bytes) 000571 1788511592.756922 MIRROR_RESYNC 2 mirrors 000572 1788511593.166306 trunc from 0x005775 to 0x0386a6 (0x32f32 bytes) 000573 1788511593.202310 fallocate 0x001d6e thru 0x0099dd (0x7c70 bytes) 000575 1788511593.208659 punch from 0x020714 to 0x026151 (0x5a3e bytes) 000576 1788511593.212598 trunc from 0x0386a7 to 0x00a468 (0x2e23e bytes) 000578 1788511593.231089 trunc from 0x003694 to 0x008d19 (0x5686 bytes) 000579 1788511593.234982 trunc from 0x000fa8 to 0x005b13 (0x4b6c bytes) 000580 1788511593.239662 fallocate 0x02cbbb thru 0x0355cb (0x8a11 bytes) 000581 1788511593.243870 mapread 0x007cd6 thru 0x00a468 (0x2793 bytes) 000582 1788511593.263823 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xa469): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa469): Mirror 8: 0xefbf7541 (stale) CRC-32 checksum value for chunk [0, 0xa469): Mirror 1: 0xefbf7541 Mirror 9: 0xefbf7541 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa469: skipped 000583 1788511593.958613 fallocate 0x017713 thru 0x01ea64 (0x7352 bytes) 000586 1788511594.003223 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ea65): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ea65): Mirror 8: 0x59d11566 (stale) Mirror 9: 0x59d11566 (stale) CRC-32 checksum value for chunk [0, 0x1ea65): Mirror 1: 0x59d11566 Mirror 10: 0x59d11566 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ea65: skipped 000587 1788511594.724560 mapwrite 0x0268a7 thru 0x0294d7 (0x2c31 bytes) 000588 1788511594.840952 fallocate 0x01d200 thru 0x028bc6 (0xb9c7 bytes) 000589 1788511594.851184 punch from 0x02713d to 0x0294d7 (0x239b bytes) 000590 1788511594.856315 trunc from 0x018635 to 0x01a267 (0x1c33 bytes) 000591 1788511594.865835 mapread 0x01cc62 thru 0x020854 (0x3bf3 bytes) 000593 1788511594.883570 trunc from 0x0294d8 to 0x02856f (0xf68 bytes) 000594 1788511594.901198 trunc from 0x00396b to 0x011d57 (0xe3ed bytes) 000595 1788511594.915174 fallocate 0x03a086 thru 0x03ffff (0x5f7a bytes) 000596 1788511594.928110 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28570): Mirror 9: 0x67b2af3e (stale) Mirror 10: 0x67b2af3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28570): Mirror 9: 0x67b2af3e (stale) Mirror 10: 0x67b2af3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28570: skipped 000597 1788511595.558015 trunc from 0x01830f to 0x01da5b (0x574d bytes) 000598 1788511595.565317 trunc from 0x028570 to 0x000bba (0x279b5 bytes) 000599 1788511595.608092 trunc from 0x000bbb to 0x0065cc (0x5a12 bytes) 000600 1788511595.628456 mapread 0x00572f thru 0x0065cc (0xe9e bytes) 000601 1788511595.642897 fallocate 0x02abd3 thru 0x02eec5 (0x42f3 bytes) 000602 1788511595.648433 trunc from 0x02eec6 to 0x026346 (0x8b7f bytes) 000604 1788511595.668931 trunc from 0x026347 to 0x03b3cb (0x15085 bytes) 000605 1788511595.691937 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b3cc): Mirror 10: 0xdd5a3727 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3cc): Mirror 10: 0xdd5a3727 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3cc: skipped 000606 1788511595.979112 read 0x019cb9 thru 0x0235e7 (0x992f bytes) 000607 1788511595.987211 write 0x03eb7d thru 0x03ffff (0x1483 bytes) 000608 1788511595.995634 mapread 0x008d02 thru 0x01584a (0xcb49 bytes) 000609 1788511596.022892 read 0x01009c thru 0x01c146 (0xc0ab bytes) 000610 1788511596.023542 trunc from 0x040000 to 0x035964 (0xa69b bytes) 000611 1788511596.046447 trunc from 0x035965 to 0x026119 (0xf84b bytes) 000614 1788511596.075830 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2611a): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2611a): Mirror 10: 0x5862bc24 (stale) CRC-32 checksum value for chunk [0, 0x2611a): Mirror 1: 0x5862bc24 Mirror 11: 0x5862bc24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2611a: skipped 000615 1788511597.469870 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000616 1788511598.049050 read 0x02273c thru 0x026119 (0x39de bytes) 000617 1788511598.063005 mapwrite 0x00ac72 thru 0x00be88 (0x1217 bytes) 000619 1788511598.235458 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2611a): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2611a): Mirror 11: 0x5862bc24 (stale) CRC-32 checksum value for chunk [0, 0x2611a): Mirror 1: 0x7d62ee95 Mirror 12: 0x7d62ee95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2611a: skipped 000620 1788511599.477910 punch from 0x0016ac to 0x0050ae (0x3a03 bytes) 000621 1788511599.543839 trunc from 0x02611a to 0x025a0a (0x70f bytes) 000622 1788511599.575233 fallocate 0x0222e4 thru 0x029bb2 (0x78cf bytes) 000623 1788511599.591077 mapread 0x009fda thru 0x016d39 (0xcd60 bytes) 000624 1788511599.643386 read 0x013767 thru 0x01d87d (0xa117 bytes) 000625 1788511599.667490 MIRROR_RESYNC 3 mirrors 000626 1788511600.937658 trunc from 0x025a0b to 0x0379cb (0x11fc1 bytes) 000628 1788511601.020679 read 0x0185fc thru 0x02167c (0x9081 bytes) 000629 1788511601.031237 mapwrite 0x0295b1 thru 0x02f9b0 (0x6400 bytes) 000630 1788511601.114311 trunc from 0x00489f to 0x012965 (0xe0c7 bytes) 000631 1788511601.119993 mapwrite 0x029fdd thru 0x035f3e (0xbf62 bytes) 000632 1788511601.155885 trunc from 0x0379cc to 0x0385a7 (0xbdc bytes) 000633 1788511601.186010 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x385a8): Mirror 12: 0x74344c74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385a8): Mirror 12: 0x74344c74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385a8: skipped 000634 1788511601.389934 fallocate 0x03afb7 thru 0x03ffff (0x5049 bytes) 000635 1788511601.396507 punch from 0x00a493 to 0x014bff (0xa76d bytes) 000636 1788511601.400936 punch from 0x03a201 to 0x03ffff (0x5dff bytes) 000637 1788511601.407710 trunc from 0x03cdc9 to 0x03ffff (0x3237 bytes) 000638 1788511601.415399 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xeab68a17 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6e6c304f Mirror 13: 0x6e6c304f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000639 1788511602.288092 trunc from 0x028ff4 to 0x02ba40 (0x2a4d bytes) 000640 1788511602.374499 write 0x01cfa9 thru 0x0261cc (0x9224 bytes) 000641 1788511602.421669 trunc from 0x040000 to 0x020984 (0x1f67b bytes) 000642 1788511602.522679 read 0x00efd4 thru 0x010757 (0x1784 bytes) 000643 1788511602.558004 MIRROR_RESYNC 3 mirrors 000644 1788511603.698816 trunc from 0x013ee9 to 0x0204c8 (0xc5e0 bytes) 000645 1788511603.799997 trunc from 0x020985 to 0x016027 (0xa95d bytes) 000646 1788511603.872635 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16028): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16028): Mirror 12: 0xec183971 (stale) Mirror 13: 0xec183971 (stale) CRC-32 checksum value for chunk [0, 0x16028): Mirror 1: 0xec183971 Mirror 14: 0xec183971 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16028: skipped 000647 1788511605.006824 MIRROR_RESYNC 4 mirrors 000648 1788511605.769714 mapread 0x01450f thru 0x016027 (0x1b19 bytes) 000650 1788511605.803143 read 0x00eb85 thru 0x013967 (0x4de3 bytes) 000651 1788511605.808802 punch from 0x00d536 to 0x016027 (0x8af2 bytes) 000652 1788511605.859218 fallocate 0x021d2f thru 0x02aa0e (0x8ce0 bytes) 000653 1788511605.876126 MIRROR_RESYNC 4 mirrors 000654 1788511606.869683 write 0x036e28 thru 0x0395ae (0x2787 bytes) 000655 1788511606.939806 punch from 0x0090e6 to 0x013851 (0xa76c bytes) 000656 1788511606.949230 read 0x015a54 thru 0x023f0e (0xe4bb bytes) 000657 1788511606.977763 write 0x00c6c9 thru 0x01987e (0xd1b6 bytes) 000658 1788511606.995304 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x395af): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395af): Mirror 12: 0xdfd61f6 (stale) Mirror 13: 0xdfd61f6 (stale) Mirror 14: 0xdfd61f6 (stale) CRC-32 checksum value for chunk [0, 0x395af): Mirror 1: 0x8b80b2b4 Mirror 15: 0x8b80b2b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395af: skipped 000659 1788511608.389666 mapwrite 0x035237 thru 0x03e941 (0x970b bytes) 000660 1788511608.573446 mapwrite 0x00b211 thru 0x012b17 (0x7907 bytes) 000661 1788511608.621759 trunc from 0x01ee62 to 0x0251f7 (0x6396 bytes) 000662 1788511608.635502 mapread 0x005805 thru 0x011f73 (0xc76f bytes) 000663 1788511608.642104 punch from 0x0311a5 to 0x038762 (0x75be bytes) 000664 1788511608.656007 mapwrite 0x03ee50 thru 0x03ffff (0x11b0 bytes) 000665 1788511608.722885 trunc from 0x022a13 to 0x02d9be (0xafac bytes) 000667 1788511608.734299 MIRROR_RESYNC 5 mirrors 000669 1788511610.095925 trunc from 0x035f45 to 0x03ffff (0xa0bb bytes) 000670 1788511610.139081 trunc from 0x040000 to 0x039bd7 (0x6428 bytes) 000671 1788511610.158970 trunc from 0x039bd8 to 0x014fe6 (0x24bf1 bytes) 000673 1788511610.177888 trunc from 0x00912d to 0x0103c5 (0x7299 bytes) 000674 1788511610.183605 read 0x00327b thru 0x0086f8 (0x547e bytes) 000675 1788511610.198320 write 0x03e317 thru 0x03ffff (0x1ce9 bytes) 000676 1788511610.200425 fallocate 0x03c71c thru 0x03ffff (0x38e4 bytes) 000677 1788511610.205582 punch from 0x02eecd to 0x03c393 (0xd4c7 bytes) 000678 1788511610.209973 trunc from 0x040000 to 0x01feab (0x20154 bytes) 000679 1788511610.225931 fallocate 0x035b77 thru 0x0372bc (0x1746 bytes) 000680 1788511610.231208 mapread 0x018d83 thru 0x01bb2b (0x2da9 bytes) 000681 1788511610.251545 mapread 0x02f8db thru 0x0346dd (0x4e03 bytes) 000682 1788511610.264956 fallocate 0x02d6b7 thru 0x02e9a9 (0x12f3 bytes) 000683 1788511610.272588 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x372bd): Mirror 13: 0xcb4ec9fe (stale) Mirror 14: 0xcb4ec9fe (stale) Mirror 15: 0xcb4ec9fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x372bd): Mirror 13: 0xcb4ec9fe (stale) Mirror 14: 0xcb4ec9fe (stale) Mirror 15: 0xcb4ec9fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x372bd: skipped 000684 1788511610.558794 mapread 0x0253d5 thru 0x0317f8 (0xc424 bytes) 000685 1788511610.596906 read 0x01bb5e thru 0x01e6b2 (0x2b55 bytes) 000686 1788511610.602318 mapwrite 0x01d150 thru 0x02a32b (0xd1dc bytes) 000687 1788511610.629219 trunc from 0x0372bd to 0x0301f6 (0x70c6 bytes) 000688 1788511610.674974 mapwrite 0x015220 thru 0x021b03 (0xc8e4 bytes) 000690 1788511610.741632 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301f7): Mirror 13: 0x99ee5acf (stale) Mirror 14: 0x99ee5acf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301f7): Mirror 13: 0x99ee5acf (stale) Mirror 14: 0x99ee5acf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301f7: skipped 000691 1788511611.080193 trunc from 0x018d53 to 0x021ecb (0x9179 bytes) 000692 1788511611.085213 trunc from 0x0301f7 to 0x03ee84 (0xec8e bytes) 000694 1788511611.120587 trunc from 0x01c3ca to 0x022960 (0x6597 bytes) 000695 1788511611.132653 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ee85): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee85): Mirror 13: 0xac0bb481 (stale) Mirror 14: 0xac0bb481 (stale) CRC-32 checksum value for chunk [0, 0x3ee85): Mirror 1: 0xc6639617 Mirror 15: 0xc6639617 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee85: skipped 000696 1788511612.057393 mapread 0x03051d thru 0x039e68 (0x994c bytes) 000697 1788511612.097286 punch from 0x00e9bf to 0x01b4b8 (0xcafa bytes) 000698 1788511612.161325 fallocate 0x023da7 thru 0x025779 (0x19d3 bytes) 000699 1788511612.175332 MIRROR_RESYNC 4 mirrors 000703 1788511613.230848 fallocate 0x021f59 thru 0x027c40 (0x5ce8 bytes) 000704 1788511613.276798 mapwrite 0x01205b thru 0x0163bd (0x4363 bytes) 000705 1788511613.323142 trunc from 0x03ee85 to 0x01dc6d (0x21217 bytes) 000706 1788511613.348113 mapread 0x001aff thru 0x008a71 (0x6f73 bytes) 000708 1788511613.361621 punch from 0x00fdd3 to 0x0121e0 (0x240e bytes) 000709 1788511613.368865 trunc from 0x001bc2 to 0x005cdd (0x411c bytes) 000711 1788511613.375620 mapread 0x00e41d thru 0x01a91f (0xc503 bytes) 000712 1788511613.382903 punch from 0x0073fe to 0x00c8ce (0x54d1 bytes) 000713 1788511613.389071 write 0x019376 thru 0x025c6c (0xc8f7 bytes) 000714 1788511613.392283 read 0x010e99 thru 0x01d142 (0xc2aa bytes) 000715 1788511613.394921 fallocate 0x0116d3 thru 0x019c8c (0x85ba bytes) 000716 1788511613.401205 write 0x02d660 thru 0x0360b7 (0x8a58 bytes) 000717 1788511613.402511 trunc from 0x00f980 to 0x012dba (0x343b bytes) 000719 1788511613.412036 fallocate 0x02701d thru 0x031652 (0xa636 bytes) 000722 1788511613.425944 mapread 0x01defd thru 0x028a84 (0xab88 bytes) 000723 1788511613.435806 mapwrite 0x033fd5 thru 0x034374 (0x3a0 bytes) 000724 1788511613.462072 write 0x00bab7 thru 0x016e4f (0xb399 bytes) 000725 1788511613.470977 write 0x03a71e thru 0x03ffff (0x58e2 bytes) 000726 1788511613.475486 fallocate 0x036c67 thru 0x03ffff (0x9399 bytes) 000727 1788511613.480836 punch from 0x0391f8 to 0x03ffff (0x6e08 bytes) 000728 1788511613.506815 write 0x02182e thru 0x02b376 (0x9b49 bytes) 000729 1788511613.513139 mapwrite 0x015425 thru 0x01ce79 (0x7a55 bytes) 000730 1788511613.545091 punch from 0x03e9ff to 0x03ffff (0x1601 bytes) 000731 1788511613.551726 write 0x0289c8 thru 0x02ce5e (0x4497 bytes) 000732 1788511613.554874 write 0x02c546 thru 0x03458f (0x804a bytes) 000733 1788511613.557393 trunc from 0x040000 to 0x00b9bb (0x34644 bytes) 000734 1788511613.594040 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xb9bc): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb9bc): Mirror 13: 0x569314f3 (stale) Mirror 14: 0x569314f3 (stale) Mirror 15: 0x569314f3 (stale) CRC-32 checksum value for chunk [0, 0xb9bc): Mirror 1: 0x569314f3 Mirror 16: 0x569314f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb9bc: skipped 000736 1788511614.944970 trunc from 0x00b9bc to 0x016c21 (0xb266 bytes) 000737 1788511615.085583 fallocate 0x02b246 thru 0x03611f (0xaeda bytes) 000738 1788511615.111461 read 0x002902 thru 0x00d704 (0xae03 bytes) 000739 1788511615.142365 trunc from 0x00a833 to 0x00e992 (0x4160 bytes) 000740 1788511615.166769 fallocate 0x02caf5 thru 0x03a525 (0xda31 bytes) 000741 1788511615.184888 punch from 0x0075f6 to 0x008377 (0xd82 bytes) 000742 1788511615.192356 punch from 0x001160 to 0x009076 (0x7f17 bytes) 000743 1788511615.197338 trunc from 0x0264bf to 0x031682 (0xb1c4 bytes) 000744 1788511615.206352 trunc from 0x03a2b5 to 0x03a525 (0x271 bytes) 000745 1788511615.219036 punch from 0x0165ae to 0x018ae0 (0x2533 bytes) 000746 1788511615.229559 read 0x02c4de thru 0x03a525 (0xe048 bytes) 000747 1788511615.251824 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a526): Mirror 13: 0x8b2230e8 (stale) Mirror 14: 0x8b2230e8 (stale) Mirror 15: 0x8b2230e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a526): Mirror 13: 0x8b2230e8 (stale) Mirror 14: 0x8b2230e8 (stale) Mirror 15: 0x8b2230e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a526: skipped 000748 1788511615.642314 mapread 0x0189e2 thru 0x028740 (0xfd5f bytes) 000749 1788511615.670936 trunc from 0x02f4d6 to 0x035ba6 (0x66d1 bytes) 000750 1788511615.675482 trunc from 0x004ac9 to 0x0138ce (0xee06 bytes) 000751 1788511615.680605 write 0x017034 thru 0x01cba7 (0x5b74 bytes) 000752 1788511615.686229 write 0x022d7a thru 0x025d20 (0x2fa7 bytes) 000753 1788511615.688030 read 0x02a188 thru 0x02e0fe (0x3f77 bytes) 000754 1788511615.693269 mapwrite 0x01d0e0 thru 0x028fdf (0xbf00 bytes) 000755 1788511615.711492 read 0x002c40 thru 0x004ec8 (0x2289 bytes) 000756 1788511615.720494 punch from 0x00accb to 0x011d50 (0x7086 bytes) 000757 1788511615.724938 trunc from 0x03a526 to 0x00738b (0x3319a bytes) 000758 1788511615.741136 trunc from 0x00738c to 0x004259 (0x3132 bytes) 000759 1788511615.756612 fallocate 0x02ceb5 thru 0x03a802 (0xd94e bytes) 000760 1788511615.765764 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a803): Mirror 13: 0xcd147e2e (stale) Mirror 15: 0xcd147e2e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a803): Mirror 13: 0xcd147e2e (stale) Mirror 15: 0xcd147e2e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a803: skipped 000761 1788511616.043037 write 0x00125a thru 0x00c33b (0xb0e2 bytes) 000762 1788511616.059133 fallocate 0x0163f1 thru 0x0253b7 (0xefc7 bytes) 000763 1788511616.065651 mapread 0x00ec4a thru 0x01d2b2 (0xe669 bytes) 000764 1788511616.100291 trunc from 0x035cc0 to 0x03a802 (0x4b43 bytes) 000765 1788511616.105894 trunc from 0x03a803 to 0x00c764 (0x2e09e bytes) 000766 1788511616.120224 mapwrite 0x02502e thru 0x0279fc (0x29cf bytes) 000767 1788511616.167141 trunc from 0x0279fd to 0x039e23 (0x12427 bytes) 000768 1788511616.184377 fallocate 0x031561 thru 0x03172d (0x1cd bytes) 000769 1788511616.191360 trunc from 0x0360d5 to 0x039e23 (0x3d4f bytes) 000770 1788511616.195864 mapwrite 0x01969d thru 0x02016c (0x6ad0 bytes) 000771 1788511616.222573 trunc from 0x01ef24 to 0x02c93c (0xda19 bytes) 000772 1788511616.227691 fallocate 0x023efd thru 0x02d46c (0x9570 bytes) 000773 1788511616.235523 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e24): Mirror 15: 0xf85819b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e24): Mirror 15: 0xf85819b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e24: skipped 000774 1788511616.466702 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000775 1788511616.571671 mapwrite 0x037478 thru 0x03a6ad (0x3236 bytes) 000776 1788511616.626591 fallocate 0x011ccc thru 0x012469 (0x79e bytes) 000777 1788511616.633774 mapread 0x0161a3 thru 0x016a85 (0x8e3 bytes) 000779 1788511616.647286 trunc from 0x032107 to 0x0364dd (0x43d7 bytes) 000780 1788511616.654882 punch from 0x019859 to 0x022bda (0x9382 bytes) 000781 1788511616.658967 trunc from 0x0260cc to 0x0333ac (0xd2e1 bytes) 000782 1788511616.667875 MIRROR_EXTEND to 2 mirrors 000783 1788511617.161943 trunc from 0x03a6ae to 0x0038c2 (0x36deb bytes) 000784 1788511617.219087 trunc from 0x001d0e to 0x0038c2 (0x1bb5 bytes) 000785 1788511617.226893 trunc from 0x0038c3 to 0x00aed3 (0x7611 bytes) 000786 1788511617.245947 mapwrite 0x0382b3 thru 0x03ffff (0x7d4d bytes) 000787 1788511617.310949 trunc from 0x040000 to 0x02d1dd (0x12e22 bytes) 000788 1788511617.332088 mapwrite 0x019e18 thru 0x027b29 (0xdd12 bytes) 000789 1788511617.380627 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d1de): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1de): Mirror 2: 0x14a06391 (stale) CRC-32 checksum value for chunk [0, 0x2d1de): Mirror 1: 0x150b7e31 Mirror 3: 0x150b7e31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1de: skipped 000790 1788511618.274661 write 0x0287ef thru 0x030850 (0x8062 bytes) 000791 1788511618.354917 read 0x01525b thru 0x018e07 (0x3bad bytes) 000792 1788511618.364241 read 0x02b090 thru 0x02d048 (0x1fb9 bytes) 000793 1788511618.370669 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30851): Mirror 2: 0xe268145d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30851): Mirror 2: 0xe268145d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30851: skipped 000794 1788511618.639418 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000795 1788511618.761987 mapwrite 0x03f207 thru 0x03ffff (0xdf9 bytes) 000796 1788511618.821107 mapwrite 0x024c9c thru 0x02ae38 (0x619d bytes) 000797 1788511618.854544 trunc from 0x00cff1 to 0x014603 (0x7613 bytes) 000798 1788511618.858925 fallocate 0x0170ba thru 0x024ea7 (0xddee bytes) 000800 1788511618.863855 read 0x010b4d thru 0x0180ce (0x7582 bytes) 000801 1788511618.873929 fallocate 0x03acc0 thru 0x03e193 (0x34d4 bytes) 000802 1788511618.878679 punch from 0x00e0e9 to 0x01070a (0x2622 bytes) 000803 1788511618.883103 mapread 0x03bd1c thru 0x03ffff (0x42e4 bytes) 000804 1788511618.894775 trunc from 0x040000 to 0x034e21 (0xb1de bytes) 000806 1788511618.906802 read 0x020ba7 thru 0x029f46 (0x93a0 bytes) 000808 1788511618.908191 fallocate 0x01dc25 thru 0x023496 (0x5872 bytes) 000809 1788511618.912795 fallocate 0x01ce2d thru 0x01d902 (0xad6 bytes) 000810 1788511618.917189 write 0x02bbd1 thru 0x034db3 (0x91e3 bytes) 000811 1788511618.918775 write 0x0200db thru 0x02e46a (0xe390 bytes) 000812 1788511618.920895 read 0x01a71d thru 0x02759e (0xce82 bytes) 000814 1788511618.929822 mapread 0x01ea6b thru 0x02d236 (0xe7cc bytes) 000817 1788511618.933177 punch from 0x033689 to 0x034e21 (0x1799 bytes) 000819 1788511618.948955 write 0x03b48b thru 0x03ffff (0x4b75 bytes) 000820 1788511618.950065 write 0x009e6e thru 0x018f22 (0xf0b5 bytes) 000821 1788511618.952035 read 0x034fbd thru 0x038d7b (0x3dbf bytes) 000822 1788511618.958061 punch from 0x02cd4a to 0x03a03d (0xd2f4 bytes) 000824 1788511618.963465 write 0x015042 thru 0x023be8 (0xeba7 bytes) 000826 1788511618.966912 read 0x0391d9 thru 0x03ffff (0x6e27 bytes) 000827 1788511618.975050 write 0x00f7fd thru 0x0141a6 (0x49aa bytes) 000828 1788511618.977201 mapwrite 0x03f357 thru 0x03ffff (0xca9 bytes) 000829 1788511619.008116 read 0x00ecc0 thru 0x01280f (0x3b50 bytes) 000830 1788511619.009180 fallocate 0x021fa2 thru 0x0244de (0x253d bytes) 000831 1788511619.018924 mapwrite 0x003493 thru 0x00c510 (0x907e bytes) 000833 1788511619.045978 fallocate 0x00fd46 thru 0x01e75f (0xea1a bytes) 000834 1788511619.051967 read 0x02dcf3 thru 0x03c772 (0xea80 bytes) 000835 1788511619.052644 read 0x01bfae thru 0x01edd0 (0x2e23 bytes) 000836 1788511619.053553 trunc from 0x036690 to 0x03928a (0x2bfb bytes) 000837 1788511619.058192 fallocate 0x004df6 thru 0x00c70e (0x7919 bytes) 000838 1788511619.065935 trunc from 0x040000 to 0x0377bb (0x8844 bytes) 000839 1788511619.081509 mapread 0x012a55 thru 0x020f50 (0xe4fc bytes) 000840 1788511619.083142 read 0x01f450 thru 0x02a88d (0xb43e bytes) 000841 1788511619.084461 fallocate 0x032ba2 thru 0x03c2b8 (0x9717 bytes) 000843 1788511619.091041 mapwrite 0x01bab2 thru 0x01c293 (0x7e2 bytes) 000844 1788511619.106548 punch from 0x0105c1 to 0x0105d5 (0x15 bytes) 000845 1788511619.111907 read 0x014e79 thru 0x022d9f (0xdf27 bytes) 000846 1788511619.112643 fallocate 0x011244 thru 0x0141d0 (0x2f8d bytes) 000847 1788511619.117465 mapwrite 0x00ae59 thru 0x013f1e (0x90c6 bytes) 000848 1788511619.144203 read 0x00b731 thru 0x01a4bc (0xed8c bytes) 000849 1788511619.144917 read 0x014d7c thru 0x01aa7e (0x5d03 bytes) 000850 1788511619.147729 mapwrite 0x0312ae thru 0x03e20e (0xcf61 bytes) 000851 1788511619.201794 write 0x0194dd thru 0x01cc0e (0x3732 bytes) 000852 1788511619.203672 mapread 0x0346f1 thru 0x03e20e (0x9b1e bytes) 000853 1788511619.209188 punch from 0x01e521 to 0x0217ec (0x32cc bytes) 000854 1788511619.216534 mapread 0x025983 thru 0x02e8f2 (0x8f70 bytes) 000855 1788511619.218265 write 0x030ff3 thru 0x032fe7 (0x1ff5 bytes) 000856 1788511619.221872 fallocate 0x007f95 thru 0x00bb44 (0x3bb0 bytes) 000857 1788511619.226226 trunc from 0x03e20f to 0x00eed6 (0x2f338 bytes) 000858 1788511619.264037 punch from 0x009376 to 0x00eed6 (0x5b61 bytes) 000859 1788511619.279469 trunc from 0x00eed7 to 0x03e6d9 (0x2f803 bytes) 000860 1788511619.304785 trunc from 0x03e6da to 0x00a0b8 (0x34621 bytes) 000861 1788511619.319095 trunc from 0x003e17 to 0x00a0b8 (0x62a2 bytes) 000862 1788511619.323840 mapwrite 0x03fd2b thru 0x03ffff (0x2d5 bytes) 000863 1788511619.356010 punch from 0x019b95 to 0x02157a (0x79e6 bytes) 000865 1788511619.361780 read 0x019c37 thru 0x01f837 (0x5c01 bytes) 000866 1788511619.373948 read 0x0030e3 thru 0x010158 (0xd076 bytes) 000867 1788511619.374773 mapwrite 0x0011af thru 0x005cc8 (0x4b1a bytes) 000868 1788511619.399449 punch from 0x03c652 to 0x03ffff (0x39ae bytes) 000869 1788511619.404002 mapwrite 0x0329ab thru 0x03ffff (0xd655 bytes) 000870 1788511619.432232 read 0x025255 thru 0x02c93e (0x76ea bytes) 000871 1788511619.433053 write 0x004a14 thru 0x009c4c (0x5239 bytes) 000872 1788511619.435048 fallocate 0x008518 thru 0x0138df (0xb3c8 bytes) 000873 1788511619.441884 trunc from 0x0264eb to 0x033bfa (0xd710 bytes) 000874 1788511619.448024 punch from 0x00d4f3 to 0x00dd02 (0x810 bytes) 000875 1788511619.452467 read 0x006699 thru 0x010653 (0x9fbb bytes) 000876 1788511619.468005 trunc from 0x040000 to 0x0202b9 (0x1fd46 bytes) 000877 1788511619.493038 trunc from 0x0202ba to 0x0278f2 (0x7639 bytes) 000878 1788511619.505538 mapwrite 0x01a993 thru 0x01ddf8 (0x3466 bytes) 000880 1788511619.523116 fallocate 0x02d22a thru 0x0375d8 (0xa3af bytes) 000881 1788511619.529232 mapwrite 0x02fb15 thru 0x0347df (0x4ccb bytes) 000884 1788511619.575475 write 0x007e3a thru 0x01210c (0xa2d3 bytes) 000885 1788511619.576771 trunc from 0x021b02 to 0x02ea72 (0xcf71 bytes) 000886 1788511619.582696 fallocate 0x022e3e thru 0x02f38f (0xc552 bytes) 000887 1788511619.588102 trunc from 0x0375d9 to 0x035f2d (0x16ab bytes) 000888 1788511619.618729 mapread 0x013cb2 thru 0x01dd01 (0xa050 bytes) 000890 1788511619.621028 fallocate 0x01aa91 thru 0x022cc3 (0x8233 bytes) 000891 1788511619.627222 mapwrite 0x00b34e thru 0x01812a (0xcddd bytes) 000892 1788511619.648437 punch from 0x020da4 to 0x02d7d6 (0xca33 bytes) 000893 1788511619.653738 write 0x000d05 thru 0x005836 (0x4b32 bytes) 000894 1788511619.655065 fallocate 0x002fe3 thru 0x009f08 (0x6f26 bytes) 000897 1788511619.665491 punch from 0x034674 to 0x035f2d (0x18ba bytes) 000898 1788511619.671806 read 0x02f045 thru 0x035f2d (0x6ee9 bytes) 000900 1788511619.691791 trunc from 0x0106b8 to 0x019a2f (0x9378 bytes) 000901 1788511619.706925 trunc from 0x034bb9 to 0x035f2d (0x1375 bytes) 000903 1788511619.719193 MIRROR_EXTEND to 2 mirrors 000904 1788511620.540904 write 0x02ab69 thru 0x02c1b7 (0x164f bytes) 000905 1788511620.601810 read 0x013ffc thru 0x01f2be (0xb2c3 bytes) 000906 1788511620.612507 trunc from 0x01577d to 0x01fbe9 (0xa46d bytes) 000907 1788511620.618073 mapwrite 0x003e11 thru 0x00b6d1 (0x78c1 bytes) 000909 1788511620.664466 trunc from 0x025960 to 0x03178f (0xbe30 bytes) 000910 1788511620.669634 write 0x024905 thru 0x024b6d (0x269 bytes) 000911 1788511620.675753 mapread 0x00a6c7 thru 0x015ca4 (0xb5de bytes) 000913 1788511620.686185 trunc from 0x02cd8c to 0x035f2d (0x91a2 bytes) 000914 1788511620.695809 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35f2e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f2e): Mirror 2: 0xa7d9ab75 (stale) CRC-32 checksum value for chunk [0, 0x35f2e): Mirror 1: 0xe94bfffb Mirror 3: 0xe94bfffb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f2e: skipped 000915 1788511621.846218 mapwrite 0x031283 thru 0x0345d1 (0x334f bytes) 000916 1788511622.015240 trunc from 0x035f2e to 0x027865 (0xe6c8 bytes) 000917 1788511622.075388 mapread 0x01ffe1 thru 0x027865 (0x7885 bytes) 000918 1788511622.121225 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27866): Mirror 3: 0x8f4e8042 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27866): Mirror 3: 0x8f4e8042 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27866: skipped 000919 1788511622.541627 punch from 0x004eab to 0x006827 (0x197d bytes) 000920 1788511622.546576 trunc from 0x027866 to 0x03151b (0x9cb6 bytes) 000921 1788511622.575393 punch from 0x024151 to 0x02f08f (0xaf3f bytes) 000922 1788511622.580187 write 0x00f9fa thru 0x019a7e (0xa085 bytes) 000923 1788511622.594394 punch from 0x01933a to 0x023750 (0xa417 bytes) 000924 1788511622.612448 fallocate 0x0126b2 thru 0x0189ea (0x6339 bytes) 000925 1788511622.619453 punch from 0x020111 to 0x02a18d (0xa07d bytes) 000926 1788511622.627037 trunc from 0x007b35 to 0x016f3e (0xf40a bytes) 000928 1788511622.634068 read 0x001b6d thru 0x005992 (0x3e26 bytes) 000929 1788511622.642458 read 0x01b14c thru 0x023132 (0x7fe7 bytes) 000930 1788511622.674446 mapwrite 0x035210 thru 0x03f388 (0xa179 bytes) 000931 1788511622.734828 write 0x02cd52 thru 0x034be2 (0x7e91 bytes) 000932 1788511622.738168 write 0x038355 thru 0x03ffff (0x7cab bytes) 000933 1788511622.740852 write 0x015ca1 thru 0x02025e (0xa5be bytes) 000934 1788511622.744018 punch from 0x029919 to 0x0354eb (0xbbd3 bytes) 000935 1788511622.782927 fallocate 0x02531d thru 0x02c55c (0x7240 bytes) 000936 1788511622.791965 read 0x00efa6 thru 0x01c3e8 (0xd443 bytes) 000937 1788511622.797185 trunc from 0x024f93 to 0x032d67 (0xddd5 bytes) 000938 1788511622.809129 punch from 0x013064 to 0x021a54 (0xe9f1 bytes) 000939 1788511622.823023 write 0x028c82 thru 0x029f45 (0x12c4 bytes) 000940 1788511622.846140 write 0x0161df thru 0x01a26e (0x4090 bytes) 000941 1788511622.847632 read 0x019367 thru 0x02080c (0x74a6 bytes) 000942 1788511622.849195 trunc from 0x01845d to 0x0221bf (0x9d63 bytes) 000943 1788511622.877519 punch from 0x024569 to 0x03391e (0xf3b6 bytes) 000944 1788511622.885111 read 0x01153f thru 0x01c032 (0xaaf4 bytes) 000946 1788511622.907433 read 0x02d3c2 thru 0x030e58 (0x3a97 bytes) 000947 1788511622.912434 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcf5d6bc9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb7f2c93 Mirror 4: 0xbb7f2c93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000948 1788511623.903303 mapwrite 0x00649e thru 0x01560f (0xf172 bytes) 000951 1788511624.022748 mapwrite 0x018446 thru 0x027694 (0xf24f bytes) 000952 1788511624.065993 mapwrite 0x01592f thru 0x020a96 (0xb168 bytes) 000953 1788511624.082702 mapread 0x009f3d thru 0x018e11 (0xeed5 bytes) 000954 1788511624.083923 mapwrite 0x01b296 thru 0x0205d8 (0x5343 bytes) 000955 1788511624.098816 mapread 0x03c198 thru 0x03ffff (0x3e68 bytes) 000956 1788511624.109306 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcf5d6bc9 (stale) Mirror 4: 0xbb7f2c93 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4eb6a0bd Mirror 5: 0x4eb6a0bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000957 1788511624.728764 read 0x01030e thru 0x0172fd (0x6ff0 bytes) 000958 1788511624.738882 MIRROR_RESYNC 4 mirrors 000959 1788511625.397723 write 0x02df8a thru 0x039aa6 (0xbb1d bytes) 000960 1788511625.446578 mapwrite 0x022200 thru 0x02c801 (0xa602 bytes) 000961 1788511625.485058 mapread 0x02565a thru 0x026418 (0xdbf bytes) 000962 1788511625.486634 mapread 0x025833 thru 0x027a78 (0x2246 bytes) 000963 1788511625.487501 mapwrite 0x031e3e thru 0x037465 (0x5628 bytes) 000964 1788511625.505434 read 0x008154 thru 0x00b7ee (0x369b bytes) 000965 1788511625.510384 mapread 0x035809 thru 0x03d087 (0x787f bytes) 000966 1788511625.521372 write 0x010daa thru 0x012c35 (0x1e8c bytes) 000968 1788511625.531924 mapread 0x01ad00 thru 0x025d84 (0xb085 bytes) 000969 1788511625.540829 mapread 0x023792 thru 0x0266c3 (0x2f32 bytes) 000970 1788511625.542198 trunc from 0x02c7d7 to 0x037a99 (0xb2c3 bytes) 000971 1788511625.547896 write 0x00783d thru 0x00facb (0x828f bytes) 000972 1788511625.559713 read 0x01727a thru 0x01d429 (0x61b0 bytes) 000973 1788511625.561147 punch from 0x01bdc2 to 0x025d4a (0x9f89 bytes) 000974 1788511625.565613 mapwrite 0x02551a thru 0x02f264 (0x9d4b bytes) 000975 1788511625.593798 punch from 0x017f6a to 0x01fe26 (0x7ebd bytes) 000976 1788511625.602957 punch from 0x02951b to 0x036416 (0xcefc bytes) 000977 1788511625.608537 trunc from 0x040000 to 0x01c5a6 (0x23a59 bytes) 000978 1788511625.630791 trunc from 0x01c5a7 to 0x02eb79 (0x125d3 bytes) 000979 1788511625.649658 fallocate 0x031d6e thru 0x03ffff (0xe292 bytes) 000980 1788511625.654186 fallocate 0x00d7de thru 0x00ddf9 (0x61c bytes) 000982 1788511625.662945 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4eb6a0bd (stale) Mirror 4: 0x4eb6a0bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4eb6a0bd (stale) Mirror 4: 0x4eb6a0bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000983 1788511625.943984 fallocate 0x017b8d thru 0x01d76e (0x5be2 bytes) 000984 1788511625.951558 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4eb6a0bd (stale) Mirror 4: 0x4eb6a0bd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4125cc0 Mirror 5: 0x4125cc0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000985 1788511626.920045 mapread 0x03e29b thru 0x03ffff (0x1d65 bytes) 000988 1788511626.932311 mapread 0x02591b thru 0x027be6 (0x22cc bytes) 000989 1788511626.944846 MIRROR_RESYNC 4 mirrors 000990 1788511627.913526 mapread 0x01e9f8 thru 0x0215c9 (0x2bd2 bytes) 000991 1788511627.941156 write 0x03606d thru 0x03ffff (0x9f93 bytes) 000992 1788511627.993764 punch from 0x003ba0 to 0x012c33 (0xf094 bytes) 000993 1788511627.998544 punch from 0x0058c4 to 0x00e988 (0x90c5 bytes) 000994 1788511628.004304 fallocate 0x03ac3c thru 0x03cfe7 (0x23ac bytes) 000995 1788511628.009234 write 0x01b2a6 thru 0x02adbd (0xfb18 bytes) 000996 1788511628.061640 trunc from 0x040000 to 0x0135ef (0x2ca10 bytes) 000997 1788511628.121812 trunc from 0x0135f0 to 0x03b2e5 (0x27cf6 bytes) 000998 1788511628.158556 mapwrite 0x0184ad thru 0x01f840 (0x7394 bytes) 000999 1788511628.222619 fallocate 0x014263 thru 0x01af33 (0x6cd1 bytes) 001000 1788511628.227672 mapwrite 0x026de7 thru 0x02d817 (0x6a31 bytes) All operations completed A-OK! PASS 70a (106s) == sanity-flr test 71: check mirror extend parent fid ==== 04:47:15 (1788511635) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1e5:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 04:47:29 (1788511649) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191013 s, 11.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137333 s, 7.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 04:47:38 (1788511658) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7793792 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6659840 .. failed Extending file size to 7100288 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7417376 .. Extending file size to 8305152 .. Extending file size to 8516864 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8566048 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8646656 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8847136 .. Waiting 61548 61549 61550 61552 61555 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:49:00 (1788511740) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7018656 .. Extending file size to 6733984 .. done Extending file size to 7679680 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7999808 .. Extending file size to 8075104 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8130144 .. Extending file size to 7858784 .. Extending file size to 8050592 .. Extending file size to 8326464 .. Extending file size to 9126016 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8146624 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8556768 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63095 63096 63097 63098 63101 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:50:19 (1788511819) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1268892 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1269180 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17776 3589244 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25740 3581280 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43516 7170524 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1268892 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2124 1269180 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17776 3589244 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25740 3581280 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43516 7170524 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0879633 s, 23.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250488 s, 8.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3c11688e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:50:40 (1788511840) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0975945 s, 21.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 04:50:48 (1788511848) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x98:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:50:56 (1788511856) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119204 s, 17.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x27e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x27f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:51:05 (1788511865) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113608 s, 18.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:51:14 (1788511874) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133209 s, 15.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:51:49 (1788511909) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10942 s, 9.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x281:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x280:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x281:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x280:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27f:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:52:44 (1788511964) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0588528 s, 35.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:53:32 (1788512012) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg148-server: Warning, pool lustre.test_204f not found Adding targets to pool oleg148-server: Warning, OST lustre-OST0000_UUID not found in pool lustre.test_204f oleg148-server: Warning, OST lustre-OST0001_UUID not found in pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148824 s, 14.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg148-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:54:53 (1788512093) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:55:02 (1788512102) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:55:11 (1788512111) oleg148-server: Warning, pool lustre.test_206 not found Adding targets to pool oleg148-server: Warning, OST lustre-OST0000_UUID not found in pool lustre.test_206 oleg148-server: Warning, OST lustre-OST0001_UUID not found in pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg148-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' Updated after 2s: want 'foo' got 'foo' PASS 206 (34s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:55:44 (1788512144) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546242 s, 19.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685382 s, 15.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf5ff0a8 Mirror 3: 0xf5ff0a8 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:55:57 (1788512157) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:56:02 (1788512162) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:56:06 (1788512166) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.035568 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1af0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ca0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xa40000 Read this (4 bytes): BBBB 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:56:47 (1788512207) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0423276 s, 194 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:56:57 (1788512217) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124462 s, 8.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb9 lmm_fid: [0x200000405:0xb9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb9 lmm_fid: [0x200000405:0xb9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611321 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 04:57:00.000000000 -0400 Modify: 2026-09-04 04:57:00.000000000 -0400 Change: 2026-09-04 04:57:02.000000000 -0400 Birth: 2026-09-04 04:56:59.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:57:09 (1788512229) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285448 s, 36.7 MB/s oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: O/280000401/d27/15b: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:57:35 (1788512255) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.220167 s, 186 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0169977 s, 241 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 04:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbe:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 04:57 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:57:44 (1788512264) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:57:52 (1788512272) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1864 sec =========== 04:58:01 (1788512281) === sanity-flr: start cleanup 04:58:02 (1788512282) === === sanity-flr: finish cleanup 04:58:10 (1788512290) ===