-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 15:29:31 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg135-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:29:52 (1788636592) === oleg135-client.virtnet: executing check_config_client /mnt/lustre oleg135-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg135-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8bf488796000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bf488796000.idle_timeout=debug disable quota as required oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:30:31 (1788636631) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8205] == sanity-flr test 0a: lfs mirror create with -N option == 15:30:34 (1788636634) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:30:53 (1788636653) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:30:56 (1788636656) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:31:00 (1788636660) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:31:25 (1788636685) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:31:29 (1788636689) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 15:31:32 (1788636692) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8bf488796000.stats=clear osc.lustre-OST0001-osc-ffff8bf488796000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:31:39 (1788636699) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:31:50 (1788636710) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:31:59 (1788636719) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:32:03 (1788636723) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:32:11 (1788636731) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 15:32:19 (1788636739) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:32:28 (1788636748) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:32:37 (1788636757) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:32:48 (1788636768) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (11s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:33:03 (1788636783) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:33:12 (1788636792) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.66932 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118667 s, 8.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:33:23 (1788636803) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.11512 s, 17.9 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:33:35 (1788636815) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:33:48 (1788636828) Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg135-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (87s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:35:15 (1788636915) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.45224 s, 18.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:36:02 (1788636962) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg135-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:37:30 (1788637050) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.33057 s, 22.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 15:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.339s user 0m0.000s sys 0m0.140s Stopping /mnt/lustre-ost1 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bf488796000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.826s user 0m0.000s sys 0m0.161s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg135-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg135-server oleg135-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.311s user 0m0.000s sys 0m0.169s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg135-server: oleg135-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg135-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bf488796000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:39:06 (1788637146) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:39:10 (1788637150) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:39:14 (1788637154) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 15:39:18 (1788637158) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 15:39:25 (1788637165) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.397148 s, 31.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:39:37 (1788637177) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456381 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306297 s, 34.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:39:45 (1788637185) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:39:54 (1788637194) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101776 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125614 s, 815 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00332841 s, 308 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000969048 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 15:40:05 (1788637205) create test files with size 15736559 .. 13d2091a75da75397a548d47ed68836e - e4d40f19eba4b3a3e60067b6ba1eee74 - 3989660046effff5bd1a0a541e443595 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 15:40:27 (1788637227) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:40:39 (1788637239) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:40:46 (1788637246) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108014 s, 9.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383874 s, 27.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 15:40:54 (1788637254) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111288 s, 18.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.31357 s, 13.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147759 s, 14.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 15:41:10 (1788637270) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 15:41:14 (1788637274) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:41:19 (1788637279) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370416 s, 28.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 15:41:29 (1788637289) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.217296 s, 14.5 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:41:39 (1788637299) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:41:50 (1788637310) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.244745 s, 42.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:42:04 (1788637324) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8bf485187000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.339241 s, 30.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8bf483e86000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:42:22 (1788637342) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:43:07 (1788637387) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (47s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:43:54 (1788637434) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:43:59 (1788637439) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.307921 s, 10.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.174205 s, 18.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd859308b (nosync,stale) Mirror 2: 0xd859308b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9062e45c03fee283bdfe05a0beffab1e - mirror 1 checksum: 88bf43c67bc866602d9d55269f82a224 - mirror 2 checksum: 9062e45c03fee283bdfe05a0beffab1e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T15:44:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T15:44:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9062e45c03fee283bdfe05a0beffab1e - mirror 1 checksum: 88bf43c67bc866602d9d55269f82a224 - mirror 2 checksum: 88bf43c67bc866602d9d55269f82a224 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T15:44:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T15:44:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:44:11 (1788637451) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497144 s, 21.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:44:20 (1788637460) Failing mds1 on oleg135-server Stopping /mnt/lustre-mds1 (opts:) on oleg135-server 15:44:33 (1788637473) shut down facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server Failover mds1 to oleg135-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 15:44:59 (1788637499) targets are mounted 15:44:59 (1788637499) facet_failover done oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042267 s, 24.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:45:16 (1788637516) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.03379 s, 606 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0319123 s, 642 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:45:26 (1788637526) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0642032 s, 319 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0073389 s, 2.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:45:35 (1788637535) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00738946 s, 2.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:45:43 (1788637543) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0395714 s, 518 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:45:51 (1788637551) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0107642 s, 381 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0346332 s, 118 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:45:58 (1788637558) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:46:27 (1788637587) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:47:00 (1788637620) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:47:24 (1788637644) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:47:49 (1788637669) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 15:47:56 (1788637676) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57564 mirror split pid 57565 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57564 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57565 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 15:49:05 (1788637745) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7747 skipping zero size read 000003 1788637747.416328 MIRROR_EXTEND to 2 mirrors 000004 1788637747.754670 MIRROR_RESYNC 2 mirrors fallocating to largest ever: 0x18320 000005 1788637747.880950 fallocate 0x014b4e thru 0x01831f (0x37d2 bytes) 000006 1788637747.911763 read 0x0132d0 thru 0x01831f (0x5050 bytes) 000007 1788637747.926072 mapwrite 0x023c24 thru 0x02f98b (0xbd68 bytes) 000008 1788637748.057026 write 0x02ae70 thru 0x0349ca (0x9b5b bytes) 000009 1788637748.074341 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x349cb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349cb): Mirror 2: 0xa2373b3a (stale) CRC-32 checksum value for chunk [0, 0x349cb): Mirror 1: 0x45d08e6c Mirror 3: 0x45d08e6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349cb: skipped 000010 1788637749.162970 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x349cb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349cb): Mirror 2: 0xa2373b3a (stale) CRC-32 checksum value for chunk [0, 0x349cb): Mirror 1: 0x45d08e6c Mirror 3: 0x45d08e6c Mirror 4: 0x45d08e6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349cb: skipped 000011 1788637750.068725 trunc from 0x02e04d to 0x0349ca (0x697e bytes) 000012 1788637750.114585 write 0x01597b thru 0x024481 (0xeb07 bytes) fallocating to largest ever: 0x347ca 000013 1788637750.134480 fallocate 0x027a99 thru 0x0347c9 (0xcd31 bytes) 000014 1788637750.148169 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x349cb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349cb): Mirror 2: 0xa2373b3a (stale) Mirror 3: 0x45d08e6c (stale) Mirror 4: 0x45d08e6c (stale) CRC-32 checksum value for chunk [0, 0x349cb): Mirror 1: 0xa681d7af Mirror 5: 0xa681d7af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349cb: skipped 000015 1788637751.196450 write 0x03597c thru 0x03ffff (0xa684 bytes) 000016 1788637751.242635 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x2f6a147 (stale) Mirror 4: 0x2f6a147 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x2f6a147 (stale) Mirror 4: 0x2f6a147 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000017 1788637751.620270 trunc from 0x01816c to 0x0231b9 (0xb04e bytes) 000018 1788637751.625980 trunc from 0x040000 to 0x02018c (0x1fe73 bytes) 000019 1788637751.669578 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2018d): Mirror 3: 0x83f3da19 (stale) Mirror 4: 0x83f3da19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2018d): Mirror 3: 0x83f3da19 (stale) Mirror 4: 0x83f3da19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2018d: skipped 000020 1788637752.198217 trunc from 0x02018d to 0x007c8d (0x184ff bytes) 000021 1788637752.262650 trunc from 0x007c8e to 0x02affa (0x2336d bytes) 000022 1788637752.279850 punch from 0x01870b to 0x01aa66 (0x235c bytes) 000023 1788637752.285031 mapwrite 0x004cb0 thru 0x005db0 (0x1101 bytes) 000024 1788637752.309742 write 0x00baf2 thru 0x01ac73 (0xf182 bytes) 000025 1788637752.324930 punch from 0x01f93e to 0x0232ff (0x39c2 bytes) 000026 1788637752.331307 read 0x0001ae thru 0x009ad5 (0x9928 bytes) 000027 1788637752.346839 punch from 0x027c46 to 0x02affa (0x33b5 bytes) 000028 1788637752.352590 trunc from 0x02affb to 0x01219a (0x18e60 bytes) fallocating to largest ever: 0x40000 000030 1788637752.374140 fallocate 0x03a489 thru 0x03ffff (0x5b77 bytes) 000031 1788637752.390503 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2f6a147 (stale) Mirror 4: 0x2f6a147 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x729f7ed8 Mirror 5: 0x729f7ed8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000032 1788637753.480328 mapwrite 0x01da87 thru 0x01e254 (0x7ce bytes) 000033 1788637753.706253 trunc from 0x028768 to 0x02deba (0x5753 bytes) 000035 1788637753.714223 trunc from 0x03c023 to 0x03ffff (0x3fdd bytes) 000036 1788637753.723896 mapwrite 0x01b1c3 thru 0x0235d7 (0x8415 bytes) 000037 1788637753.754834 read 0x00b3b8 thru 0x0154f1 (0xa13a bytes) 000038 1788637753.766239 read 0x039b8a thru 0x03c468 (0x28df bytes) 000039 1788637753.790873 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2f6a147 (stale) Mirror 4: 0x2f6a147 (stale) Mirror 5: 0x729f7ed8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf9da8a9 Mirror 6: 0xaf9da8a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000040 1788637755.147520 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2f6a147 (stale) Mirror 5: 0x729f7ed8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2f6a147 (stale) Mirror 5: 0x729f7ed8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf9da8a9 Mirror 6: 0xaf9da8a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000041 1788637756.008648 fallocate 0x01ce86 thru 0x02931a (0xc495 bytes) 000042 1788637756.090717 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x729f7ed8 (stale) Mirror 6: 0xaf9da8a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x729f7ed8 (stale) Mirror 6: 0xaf9da8a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000044 1788637756.479007 write 0x0108ed thru 0x01f3e7 (0xeafb bytes) 000045 1788637756.498636 read 0x0003da thru 0x003f55 (0x3b7c bytes) 000046 1788637756.505240 trunc from 0x01b3ea to 0x01d6ab (0x22c2 bytes) 000047 1788637756.520819 mapread 0x00a14a thru 0x00d7b8 (0x366f bytes) 000048 1788637756.539261 fallocate 0x007f7f thru 0x017019 (0xf09b bytes) 000049 1788637756.548694 MIRROR_RESYNC 3 mirrors 000050 1788637758.100009 mapread 0x039d2d thru 0x03ffff (0x62d3 bytes) 000051 1788637758.146623 read 0x0062b9 thru 0x00b2e3 (0x502b bytes) 000052 1788637758.165286 trunc from 0x040000 to 0x03467b (0xb984 bytes) 000053 1788637758.228729 write 0x011446 thru 0x0129da (0x1595 bytes) 000054 1788637758.247494 fallocate 0x02b6a6 thru 0x02f4f7 (0x3e52 bytes) 000055 1788637758.268587 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3467c): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3467c): Mirror 5: 0xe0719e57 (stale) Mirror 6: 0xe0719e57 (stale) CRC-32 checksum value for chunk [0, 0x3467c): Mirror 1: 0x26dd80ac Mirror 7: 0x26dd80ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3467c: skipped 000056 1788637759.532778 write 0x000fcd thru 0x00bd5a (0xad8e bytes) 000057 1788637759.658178 mapwrite 0x03f1c4 thru 0x03ffff (0xe3c bytes) 000058 1788637759.765167 mapread 0x03267f thru 0x03ffff (0xd981 bytes) 000059 1788637759.818116 punch from 0x030366 to 0x03bead (0xbb48 bytes) 000061 1788637759.827158 mapread 0x01c919 thru 0x023a75 (0x715d bytes) 000062 1788637759.856051 read 0x03c1ea thru 0x03ffff (0x3e16 bytes) 000063 1788637759.858000 fallocate 0x02e3b0 thru 0x03dffb (0xfc4c bytes) 000064 1788637759.873292 trunc from 0x00ef37 to 0x00f0d0 (0x19a bytes) 000065 1788637759.883787 read 0x03be87 thru 0x03ffff (0x4179 bytes) 000066 1788637759.917191 read 0x0199d9 thru 0x02452f (0xab57 bytes) 000067 1788637759.924805 trunc from 0x02d2a9 to 0x02e94d (0x16a5 bytes) 000068 1788637759.951610 trunc from 0x040000 to 0x038908 (0x76f7 bytes) 000070 1788637759.986964 trunc from 0x01dbea to 0x024a92 (0x6ea9 bytes) 000071 1788637759.993301 trunc from 0x00ed7c to 0x011bed (0x2e72 bytes) 000072 1788637759.998747 write 0x0308c5 thru 0x035b8e (0x52ca bytes) 000073 1788637760.000023 trunc from 0x022f86 to 0x02e0c4 (0xb13f bytes) 000075 1788637760.006126 fallocate 0x0303a3 thru 0x03d3e7 (0xd045 bytes) 000076 1788637760.011308 fallocate 0x03df53 thru 0x03ffff (0x20ad bytes) 000077 1788637760.017186 read 0x0328dd thru 0x034bf5 (0x2319 bytes) 000078 1788637760.017854 trunc from 0x010106 to 0x01b8e5 (0xb7e0 bytes) 000079 1788637760.026184 trunc from 0x040000 to 0x0124fd (0x2db02 bytes) 000080 1788637760.053727 mapread 0x007750 thru 0x00e542 (0x6df3 bytes) 000081 1788637760.064832 trunc from 0x00dfab to 0x0124fd (0x4553 bytes) 000082 1788637760.072184 trunc from 0x0050ba to 0x00c961 (0x78a8 bytes) 000083 1788637760.079854 mapread 0x009bc4 thru 0x00cadd (0x2f1a bytes) 000084 1788637760.118910 mapwrite 0x01b616 thru 0x020129 (0x4b14 bytes) 000087 1788637760.214841 punch from 0x01cf3f to 0x020129 (0x31eb bytes) 000088 1788637760.228877 punch from 0x002f0b to 0x00400b (0x1101 bytes) 000089 1788637760.241700 punch from 0x0122d5 to 0x01b7b6 (0x94e2 bytes) 000090 1788637760.250661 fallocate 0x014727 thru 0x016513 (0x1ded bytes) 000091 1788637760.259232 trunc from 0x02012a to 0x02c2f4 (0xc1cb bytes) 000092 1788637760.278496 fallocate 0x03c3d2 thru 0x03ffff (0x3c2e bytes) 000093 1788637760.288730 trunc from 0x040000 to 0x03397f (0xc680 bytes) 000094 1788637760.307228 trunc from 0x0020f1 to 0x011896 (0xf7a6 bytes) 000095 1788637760.313315 trunc from 0x033980 to 0x02df95 (0x59ea bytes) 000096 1788637760.331219 punch from 0x028bb3 to 0x02b8b8 (0x2d06 bytes) 000097 1788637760.337507 write 0x00bb8c thru 0x018741 (0xcbb6 bytes) 000098 1788637760.357806 fallocate 0x0043aa thru 0x012ac9 (0xe720 bytes) 000099 1788637760.365733 write 0x0331b1 thru 0x03f1b7 (0xc007 bytes) 000100 1788637760.372330 trunc from 0x036d62 to 0x03b3cf (0x466e bytes) 000101 1788637760.395914 read 0x021edf thru 0x02e2e3 (0xc405 bytes) 000102 1788637760.408862 write 0x035eae thru 0x03ffff (0xa152 bytes) 000103 1788637760.410496 fallocate 0x03afdf thru 0x03ffff (0x5021 bytes) 000104 1788637760.415283 write 0x02ae02 thru 0x031ffb (0x71fa bytes) 000105 1788637760.416640 read 0x025728 thru 0x034f71 (0xf84a bytes) 000106 1788637760.420765 MIRROR_RESYNC 4 mirrors 000107 1788637761.874740 mapwrite 0x018632 thru 0x02844d (0xfe1c bytes) 000108 1788637762.013996 trunc from 0x040000 to 0x01175a (0x2e8a5 bytes) 000109 1788637762.046394 read 0x00de4f thru 0x01175a (0x390c bytes) 000110 1788637762.056527 fallocate 0x0138af thru 0x01450d (0xc5f bytes) 000111 1788637762.063064 mapread 0x0116c5 thru 0x01450d (0x2e49 bytes) 000112 1788637762.074104 mapwrite 0x00e3af thru 0x01ce97 (0xeae9 bytes) 000113 1788637762.154295 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ce98): Mirror 6: 0x5d9e0b6d (stale) Mirror 7: 0x5d9e0b6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ce98): Mirror 6: 0x5d9e0b6d (stale) Mirror 7: 0x5d9e0b6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ce98: skipped 000114 1788637762.503780 read 0x00990b thru 0x0108b7 (0x6fad bytes) 000115 1788637762.511314 write 0x02963a thru 0x03894d (0xf314 bytes) 000116 1788637762.520939 MIRROR_RESYNC 3 mirrors 000117 1788637763.236448 trunc from 0x028f63 to 0x035e89 (0xcf27 bytes) 000118 1788637763.288872 fallocate 0x02debf thru 0x0320dc (0x421e bytes) 000119 1788637763.296169 fallocate 0x002dec thru 0x00e8ec (0xbb01 bytes) 000120 1788637763.306103 MIRROR_RESYNC 3 mirrors 000122 1788637764.037853 write 0x006486 thru 0x00711d (0xc98 bytes) 000123 1788637764.081088 mapread 0x004833 thru 0x009c2f (0x53fd bytes) 000124 1788637764.111204 read 0x014717 thru 0x01b0aa (0x6994 bytes) 000125 1788637764.126231 MIRROR_RESYNC 3 mirrors 000126 1788637765.290309 punch from 0x00e6e2 to 0x018f8f (0xa8ae bytes) 000127 1788637765.342698 punch from 0x020e07 to 0x023ed9 (0x30d3 bytes) 000128 1788637765.358185 mapread 0x026804 thru 0x02996b (0x3168 bytes) 000129 1788637765.453725 mapread 0x00a073 thru 0x01244c (0x83da bytes) 000130 1788637765.519864 fallocate 0x0282ef thru 0x028be1 (0x8f3 bytes) 000131 1788637765.525778 fallocate 0x021a5e thru 0x0223b5 (0x958 bytes) 000132 1788637765.532098 punch from 0x00c288 to 0x00f223 (0x2f9c bytes) 000133 1788637765.538834 mapread 0x01edd1 thru 0x0221d8 (0x3408 bytes) 000134 1788637765.554703 mapwrite 0x00413a thru 0x012886 (0xe74d bytes) 000135 1788637765.621600 mapread 0x01a82f thru 0x025683 (0xae55 bytes) 000136 1788637765.622602 punch from 0x0375e4 to 0x03894d (0x136a bytes) 000138 1788637765.631332 write 0x017c56 thru 0x01eee6 (0x7291 bytes) 000139 1788637765.639236 MIRROR_RESYNC 3 mirrors 000140 1788637766.605562 punch from 0x01046c to 0x0106a9 (0x23e bytes) 000142 1788637766.671022 mapread 0x00721d thru 0x01258b (0xb36f bytes) 000143 1788637766.734256 write 0x03903d thru 0x03ffff (0x6fc3 bytes) 000144 1788637766.738195 write 0x0154ec thru 0x01a141 (0x4c56 bytes) 000146 1788637766.742171 trunc from 0x040000 to 0x015f9b (0x2a064 bytes) 000147 1788637766.774743 write 0x02f1c4 thru 0x0326db (0x3518 bytes) 000148 1788637766.775878 punch from 0x00c5e6 to 0x019222 (0xcc3d bytes) 000149 1788637766.804310 mapread 0x009776 thru 0x01671b (0xcfa6 bytes) 000150 1788637766.861300 mapwrite 0x00dbd1 thru 0x0115c7 (0x39f7 bytes) 000151 1788637766.923258 mapread 0x020193 thru 0x02d325 (0xd193 bytes) 000152 1788637766.963061 punch from 0x031cfe to 0x0326db (0x9de bytes) 000153 1788637766.968305 read 0x0260b2 thru 0x0270e1 (0x1030 bytes) 000154 1788637766.968879 read 0x013353 thru 0x018486 (0x5134 bytes) 000155 1788637766.969429 trunc from 0x01f765 to 0x020955 (0x11f1 bytes) 000156 1788637766.979082 mapread 0x0097be thru 0x0126af (0x8ef2 bytes) 000157 1788637766.980110 mapread 0x02f6d4 thru 0x0326db (0x3008 bytes) 000158 1788637767.004031 trunc from 0x01d2fd to 0x026418 (0x911c bytes) 000159 1788637767.013480 trunc from 0x030590 to 0x0326db (0x214c bytes) 000160 1788637767.023281 trunc from 0x02ccf6 to 0x0325e0 (0x58eb bytes) 000161 1788637767.030321 write 0x0028d0 thru 0x008514 (0x5c45 bytes) 000162 1788637767.031553 mapread 0x02bc41 thru 0x02d6ea (0x1aaa bytes) 000163 1788637767.054782 write 0x022d54 thru 0x02a9dc (0x7c89 bytes) 000165 1788637767.068300 mapwrite 0x024c01 thru 0x029a06 (0x4e06 bytes) 000166 1788637767.093703 read 0x012642 thru 0x01d0b8 (0xaa77 bytes) 000167 1788637767.105748 fallocate 0x00b75f thru 0x011216 (0x5ab8 bytes) 000168 1788637767.112192 write 0x001e2f thru 0x0109f3 (0xebc5 bytes) 000169 1788637767.113640 trunc from 0x0326dc to 0x03b9ec (0x9311 bytes) 000170 1788637767.132767 fallocate 0x033bac thru 0x035e7b (0x22d0 bytes) 000171 1788637767.146709 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b9ed): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9ed): Mirror 6: 0xfdf89866 (stale) Mirror 7: 0xfdf89866 (stale) CRC-32 checksum value for chunk [0, 0x3b9ed): Mirror 1: 0xe3c9918b Mirror 8: 0xe3c9918b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9ed: skipped 000172 1788637768.447262 mapread 0x010fb5 thru 0x01e019 (0xd065 bytes) 000173 1788637768.498068 trunc from 0x000b0a to 0x0090d7 (0x85ce bytes) 000174 1788637768.538436 punch from 0x0260f2 to 0x0329d8 (0xc8e7 bytes) 000175 1788637768.552186 trunc from 0x03b9ed to 0x00586f (0x3617d bytes) 000177 1788637768.607653 mapread 0x003b41 thru 0x00586f (0x1d2f bytes) 000178 1788637768.618067 read 0x000b3f thru 0x00586f (0x4d31 bytes) 000179 1788637768.623033 mapread 0x0004ea thru 0x002133 (0x1c4a bytes) 000180 1788637768.624411 write 0x035711 thru 0x03ffff (0xa8ef bytes) 000182 1788637768.627654 mapread 0x036dfe thru 0x03dedd (0x70e0 bytes) 000185 1788637768.631614 mapwrite 0x03cbcd thru 0x03ffff (0x3433 bytes) 000186 1788637768.719338 write 0x01a257 thru 0x0261f8 (0xbfa2 bytes) 000187 1788637768.737029 mapwrite 0x016ee3 thru 0x01d473 (0x6591 bytes) 000188 1788637768.777039 trunc from 0x040000 to 0x02126c (0x1ed93 bytes) 000189 1788637768.812079 read 0x01ef7e thru 0x020662 (0x16e5 bytes) 000190 1788637768.813985 trunc from 0x00e7f3 to 0x01caf3 (0xe301 bytes) 000191 1788637768.824030 trunc from 0x02126d to 0x005d6e (0x1b4fe bytes) 000192 1788637768.846594 read 0x001895 thru 0x005d6e (0x44da bytes) 000193 1788637768.848047 fallocate 0x026631 thru 0x02e572 (0x7f42 bytes) 000194 1788637768.854915 fallocate 0x03fd9b thru 0x03ffff (0x265 bytes) 000195 1788637768.860890 write 0x0287df thru 0x02f782 (0x6fa4 bytes) 000196 1788637768.893050 mapread 0x0240f5 thru 0x026992 (0x289e bytes) 000197 1788637768.893983 punch from 0x027ae0 to 0x027d49 (0x26a bytes) 000198 1788637768.899713 fallocate 0x0039c4 thru 0x0089f4 (0x5031 bytes) 000199 1788637768.905458 write 0x016ecc thru 0x01e244 (0x7379 bytes) 000200 1788637768.908566 punch from 0x01919f to 0x027d5a (0xebbc bytes) 000201 1788637768.929708 mapwrite 0x00ad0f thru 0x0132a9 (0x859b bytes) 000202 1788637768.950545 read 0x028806 thru 0x02f782 (0x6f7d bytes) 000203 1788637768.951406 read 0x029003 thru 0x02f782 (0x6780 bytes) 000204 1788637768.952742 punch from 0x02915e to 0x02b56a (0x240d bytes) 000205 1788637768.958318 trunc from 0x02f783 to 0x001076 (0x2e70c bytes) 000206 1788637768.985730 read 0x00023d thru 0x001076 (0xe3a bytes) 000207 1788637768.986525 read 0x000bb5 thru 0x001076 (0x4c2 bytes) 000208 1788637768.988258 fallocate 0x01f4e7 thru 0x024635 (0x514f bytes) 000209 1788637768.994293 trunc from 0x001077 to 0x005ac8 (0x4a52 bytes) 000210 1788637769.015671 mapwrite 0x02740b thru 0x03223e (0xae34 bytes) 000211 1788637769.089847 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3223f): Mirror 6: 0x6214fc41 (stale) Mirror 8: 0xd6e9c53a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3223f): Mirror 6: 0x6214fc41 (stale) Mirror 8: 0xd6e9c53a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3223f: skipped 000212 1788637769.651195 mapread 0x0126c6 thru 0x0135bc (0xef7 bytes) 000213 1788637769.675641 write 0x001750 thru 0x00d1b6 (0xba67 bytes) 000214 1788637769.724423 mapread 0x01b497 thru 0x026e7f (0xb9e9 bytes) 000215 1788637769.768538 write 0x02afcb thru 0x035189 (0xa1bf bytes) 000216 1788637769.769840 mapread 0x01e114 thru 0x0284d2 (0xa3bf bytes) 000217 1788637769.770753 punch from 0x01206c to 0x020fb4 (0xef49 bytes) 000218 1788637769.786705 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3518a): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3518a): Mirror 6: 0x8e862b0c (stale) Mirror 8: 0xfc2a7c1a (stale) CRC-32 checksum value for chunk [0, 0x3518a): Mirror 1: 0x2766301f Mirror 9: 0x2766301f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3518a: skipped 000219 1788637771.047291 trunc from 0x03518a to 0x031cab (0x34de bytes) 000220 1788637771.125414 mapwrite 0x036709 thru 0x03ffff (0x98f7 bytes) 000221 1788637771.202183 trunc from 0x024245 to 0x02561c (0x13d8 bytes) 000223 1788637771.210194 fallocate 0x01df0f thru 0x021399 (0x348b bytes) 000224 1788637771.225046 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9d190c8f (stale) Mirror 8: 0xd45251ef (stale) Mirror 9: 0x2ca0ebaa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9baa1abc Mirror 10: 0x9baa1abc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000225 1788637772.519109 punch from 0x038204 to 0x03ef3d (0x6d3a bytes) 000227 1788637772.565885 fallocate 0x01463d thru 0x01797f (0x3343 bytes) 000228 1788637772.575533 mapread 0x01b4e5 thru 0x028a72 (0xd58e bytes) 000229 1788637772.607825 fallocate 0x015210 thru 0x01fdd9 (0xabca bytes) 000230 1788637772.615494 trunc from 0x040000 to 0x01e683 (0x2197c bytes) 000231 1788637772.702221 punch from 0x010bdb to 0x016761 (0x5b87 bytes) 000233 1788637772.715660 write 0x007a52 thru 0x016c20 (0xf1cf bytes) 000234 1788637772.729880 fallocate 0x03f28d thru 0x03ffff (0xd73 bytes) 000235 1788637772.735131 trunc from 0x019eb7 to 0x021ecc (0x8016 bytes) 000236 1788637772.746112 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9d190c8f (stale) Mirror 8: 0xd45251ef (stale) Mirror 9: 0x2ca0ebaa (stale) Mirror 10: 0x9baa1abc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x191b25f6 Mirror 11: 0x191b25f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000237 1788637774.200871 mapwrite 0x032b51 thru 0x037167 (0x4617 bytes) 000238 1788637774.458971 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9d190c8f (stale) Mirror 8: 0xd45251ef (stale) Mirror 9: 0x2ca0ebaa (stale) Mirror 10: 0x9baa1abc (stale) Mirror 11: 0x191b25f6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40de5ec0 Mirror 12: 0x40de5ec0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000239 1788637776.349188 fallocate 0x006d2b thru 0x00cd7a (0x6050 bytes) 000241 1788637776.395077 write 0x004e59 thru 0x00e7f6 (0x999e bytes) 000242 1788637776.412630 mapread 0x003b72 thru 0x0058da (0x1d69 bytes) 000243 1788637776.420571 read 0x008b42 thru 0x0170f8 (0xe5b7 bytes) 000244 1788637776.445610 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9d190c8f (stale) Mirror 8: 0xd45251ef (stale) Mirror 9: 0x2ca0ebaa (stale) Mirror 10: 0x9baa1abc (stale) Mirror 11: 0x191b25f6 (stale) Mirror 12: 0x40de5ec0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xce80fba Mirror 13: 0xce80fba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000245 1788637777.645082 trunc from 0x027fe5 to 0x034f5e (0xcf7a bytes) 000246 1788637777.693170 mapread 0x0033f9 thru 0x00777a (0x4382 bytes) 000248 1788637777.715992 mapread 0x000dba thru 0x005afb (0x4d42 bytes) 000250 1788637777.724186 trunc from 0x0038a8 to 0x00f530 (0xbc89 bytes) 000251 1788637777.731844 trunc from 0x0033a2 to 0x0076e6 (0x4345 bytes) 000252 1788637777.742959 punch from 0x02d79b to 0x02fe69 (0x26cf bytes) 000253 1788637777.757929 trunc from 0x0069ba to 0x01368e (0xccd5 bytes) 000254 1788637777.774937 mapread 0x01d5db thru 0x02094a (0x3370 bytes) 000256 1788637777.795130 write 0x0213c6 thru 0x02cb71 (0xb7ac bytes) 000257 1788637777.816906 trunc from 0x040000 to 0x017da7 (0x28258 bytes) 000258 1788637777.922955 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x17da8): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17da8): Mirror 6: 0xcde3121e (stale) Mirror 8: 0x787b7412 (stale) Mirror 9: 0x7463f2ed (stale) Mirror 10: 0x7463f2ed (stale) Mirror 11: 0xa7678833 (stale) Mirror 12: 0xa7678833 (stale) Mirror 13: 0x6a79d03e (stale) CRC-32 checksum value for chunk [0, 0x17da8): Mirror 1: 0xc496fa51 Mirror 14: 0xc496fa51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17da8: skipped 000261 1788637780.062951 mapread 0x0155a0 thru 0x017da7 (0x2808 bytes) 000263 1788637780.162479 trunc from 0x017da8 to 0x010459 (0x794e bytes) 000264 1788637780.250882 trunc from 0x00edd3 to 0x010459 (0x1687 bytes) 000266 1788637780.258124 mapwrite 0x02fbc2 thru 0x03bafb (0xbf3a bytes) 000267 1788637780.335307 trunc from 0x00cb08 to 0x019d8c (0xd285 bytes) 000269 1788637780.342114 write 0x022bec thru 0x02ba23 (0x8e38 bytes) 000270 1788637780.351537 trunc from 0x039d40 to 0x03bafb (0x1dbc bytes) 000271 1788637780.358168 punch from 0x0184a5 to 0x023693 (0xb1ef bytes) 000272 1788637780.395982 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bafc): Mirror 6: 0x3b93d584 (stale) Mirror 8: 0x8567d459 (stale) Mirror 9: 0xbfe5a57d (stale) Mirror 10: 0xe796faf8 (stale) Mirror 11: 0xd42aed09 (stale) Mirror 13: 0xbb6b0633 (stale) Mirror 14: 0xbd2cb130 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bafc): Mirror 6: 0x3b93d584 (stale) Mirror 8: 0x8567d459 (stale) Mirror 9: 0xbfe5a57d (stale) Mirror 10: 0xe796faf8 (stale) Mirror 11: 0xd42aed09 (stale) Mirror 13: 0xbb6b0633 (stale) Mirror 14: 0xbd2cb130 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bafc: skipped 000273 1788637781.407939 read 0x008c11 thru 0x013ae3 (0xaed3 bytes) 000274 1788637781.471182 mapwrite 0x037a9a thru 0x03ae20 (0x3387 bytes) 000276 1788637781.568094 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3bafc): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bafc): Mirror 6: 0x3b93d584 (stale) Mirror 8: 0x8567d459 (stale) Mirror 9: 0xbfe5a57d (stale) Mirror 10: 0xe796faf8 (stale) Mirror 11: 0xd42aed09 (stale) Mirror 13: 0xbb6b0633 (stale) Mirror 14: 0xbd2cb130 (stale) CRC-32 checksum value for chunk [0, 0x3bafc): Mirror 1: 0xbe1a5b7 Mirror 15: 0xbe1a5b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bafc: skipped 000277 1788637782.904234 trunc from 0x03bafc to 0x0164b7 (0x25644 bytes) 000278 1788637782.968318 read 0x001944 thru 0x006509 (0x4bc6 bytes) 000282 1788637782.982463 punch from 0x00d768 to 0x0164b7 (0x8d50 bytes) 000283 1788637782.990717 punch from 0x004830 to 0x009795 (0x4f66 bytes) 000284 1788637783.002852 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x164b8): Mirror 6: 0x51319e01 (stale) Mirror 8: 0xa4a6ccff (stale) Mirror 9: 0xc7a9368c (stale) Mirror 10: 0xc7a9368c (stale) Mirror 13: 0x7fa0dfce (stale) Mirror 14: 0x3cff5f24 (stale) Mirror 15: 0x9e76ea98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x164b8): Mirror 6: 0x51319e01 (stale) Mirror 8: 0xa4a6ccff (stale) Mirror 9: 0xc7a9368c (stale) Mirror 10: 0xc7a9368c (stale) Mirror 13: 0x7fa0dfce (stale) Mirror 14: 0x3cff5f24 (stale) Mirror 15: 0x9e76ea98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x164b8: skipped 000285 1788637783.685890 trunc from 0x0164b8 to 0x03824a (0x21d93 bytes) 000286 1788637783.734027 write 0x004f3e thru 0x006e28 (0x1eeb bytes) 000287 1788637783.756041 trunc from 0x026adf to 0x02c0ce (0x55f0 bytes) 000288 1788637783.778192 mapwrite 0x013563 thru 0x01df8d (0xaa2b bytes) 000289 1788637783.890282 write 0x019078 thru 0x027cd5 (0xec5e bytes) 000290 1788637783.903873 punch from 0x013f50 to 0x020657 (0xc708 bytes) 000291 1788637783.958878 trunc from 0x0166be to 0x02610b (0xfa4e bytes) 000292 1788637783.982565 trunc from 0x03824b to 0x01fe32 (0x18418 bytes) 000293 1788637784.058855 MIRROR_RESYNC 8 mirrors 000294 1788637786.446173 MIRROR_RESYNC 8 mirrors 000295 1788637787.115166 fallocate 0x02d968 thru 0x033f45 (0x65de bytes) 000296 1788637787.184715 MIRROR_RESYNC 8 mirrors 000297 1788637788.963239 trunc from 0x01fe33 to 0x028e63 (0x9031 bytes) 000298 1788637789.014614 trunc from 0x028e64 to 0x01048a (0x189d9 bytes) 000299 1788637789.046595 fallocate 0x013781 thru 0x017739 (0x3fb9 bytes) 000300 1788637789.053554 mapread 0x01376d thru 0x013bcb (0x45f bytes) 000301 1788637789.060740 trunc from 0x01773a to 0x01d127 (0x59ee bytes) 000302 1788637789.080894 fallocate 0x023673 thru 0x02715a (0x3ae8 bytes) 000303 1788637789.087230 mapread 0x00607f thru 0x00cfc0 (0x6f42 bytes) 000304 1788637789.107753 read 0x019745 thru 0x02430d (0xabc9 bytes) 000305 1788637789.123249 trunc from 0x020b44 to 0x02715a (0x6617 bytes) 000306 1788637789.132090 trunc from 0x01f597 to 0x02715a (0x7bc4 bytes) 000307 1788637789.158929 fallocate 0x03fd05 thru 0x03ffff (0x2fb bytes) 000308 1788637789.182613 MIRROR_RESYNC 8 mirrors 000309 1788637790.678318 mapread 0x03b70b thru 0x03dea9 (0x279f bytes) 000310 1788637790.709140 mapread 0x0116a7 thru 0x01b3e9 (0x9d43 bytes) 000311 1788637790.734608 mapread 0x01f45c thru 0x025c51 (0x67f6 bytes) 000312 1788637790.758893 MIRROR_EXTEND to 9 mirrors 000313 1788637792.026775 punch from 0x02267b to 0x02df44 (0xb8ca bytes) 000314 1788637792.065159 read 0x029524 thru 0x030849 (0x7326 bytes) 000316 1788637792.082305 fallocate 0x0064ec thru 0x00e75b (0x8270 bytes) 000317 1788637792.093727 trunc from 0x0074c0 to 0x0125dc (0xb11d bytes) 000318 1788637792.106924 trunc from 0x01bc7a to 0x01e8f1 (0x2c78 bytes) 000319 1788637792.120720 mapwrite 0x028686 thru 0x02ae13 (0x278e bytes) 000320 1788637792.148807 write 0x03736c thru 0x03ffff (0x8c94 bytes) 000321 1788637792.169932 read 0x02d9e8 thru 0x030bd7 (0x31f0 bytes) 000322 1788637792.170783 read 0x03ec18 thru 0x03ffff (0x13e8 bytes) 000323 1788637792.176816 trunc from 0x02d420 to 0x0330b1 (0x5c92 bytes) 000324 1788637792.182914 mapwrite 0x02f7c7 thru 0x03771c (0x7f56 bytes) 000325 1788637792.260702 mapwrite 0x0392e7 thru 0x03ffff (0x6d19 bytes) 000326 1788637792.277308 mapwrite 0x001590 thru 0x006dc9 (0x583a bytes) 000327 1788637792.296899 trunc from 0x019a85 to 0x022f0f (0x948b bytes) 000328 1788637792.303223 trunc from 0x02a01e to 0x037ec9 (0xdeac bytes) 000329 1788637792.313188 read 0x0275e4 thru 0x02925a (0x1c77 bytes) 000330 1788637792.316017 fallocate 0x01a55d thru 0x01eb61 (0x4605 bytes) 000331 1788637792.328000 mapread 0x01ce22 thru 0x01f935 (0x2b14 bytes) 000332 1788637792.345015 read 0x026d79 thru 0x030ead (0xa135 bytes) 000333 1788637792.345906 write 0x03e5bf thru 0x03ffff (0x1a41 bytes) 000334 1788637792.348166 write 0x0045a5 thru 0x00aa2b (0x6487 bytes) 000335 1788637792.351098 mapread 0x03cb26 thru 0x03ffff (0x34da bytes) 000337 1788637792.353673 mapwrite 0x03bf7c thru 0x03ffff (0x4084 bytes) 000338 1788637792.392774 mapread 0x034197 thru 0x03cb2a (0x8994 bytes) 000339 1788637792.394315 trunc from 0x040000 to 0x006a65 (0x3959a bytes) 000340 1788637792.416117 mapread 0x005611 thru 0x006a65 (0x1455 bytes) 000342 1788637792.419449 punch from 0x0036bf to 0x006a65 (0x33a7 bytes) 000343 1788637792.424876 mapwrite 0x032710 thru 0x03be26 (0x9717 bytes) 000344 1788637792.503954 write 0x019edd thru 0x01a051 (0x175 bytes) 000345 1788637792.505997 read 0x007307 thru 0x00c21e (0x4f18 bytes) 000346 1788637792.507319 punch from 0x01f5a5 to 0x02c7b6 (0xd212 bytes) 000347 1788637792.513508 punch from 0x01ab61 to 0x0217d2 (0x6c72 bytes) 000348 1788637792.525219 mapread 0x008774 thru 0x015f45 (0xd7d2 bytes) 000349 1788637792.526680 read 0x02906d thru 0x02f0cf (0x6063 bytes) 000350 1788637792.541159 write 0x00d4cc thru 0x01221f (0x4d54 bytes) 000351 1788637792.546899 trunc from 0x02d33a to 0x03bbeb (0xe8b2 bytes) 000352 1788637792.563916 MIRROR_RESYNC 9 mirrors 000353 1788637795.590860 write 0x0114bd thru 0x011c16 (0x75a bytes) 000354 1788637795.657341 read 0x020d84 thru 0x023d09 (0x2f86 bytes) 000355 1788637795.679088 mapread 0x01cb11 thru 0x024f73 (0x8463 bytes) 000356 1788637795.727171 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3be27): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be27): Mirror 6: 0x305ceab8 (stale) Mirror 8: 0x305ceab8 (stale) Mirror 9: 0x305ceab8 (stale) Mirror 10: 0x305ceab8 (stale) Mirror 13: 0x305ceab8 (stale) Mirror 14: 0x305ceab8 (stale) Mirror 15: 0x305ceab8 (stale) Mirror 16: 0x305ceab8 (stale) CRC-32 checksum value for chunk [0, 0x3be27): Mirror 1: 0xe37eb114 Mirror 17: 0xe37eb114 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be27: skipped 000357 1788637797.488206 trunc from 0x03be27 to 0x0162df (0x25b47 bytes) 000358 1788637797.575264 mapread 0x00f57e thru 0x0162df (0x6d62 bytes) 000359 1788637797.638919 mapwrite 0x02bc5f thru 0x036d27 (0xb0c9 bytes) 000360 1788637797.750197 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36d28): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d28): Mirror 6: 0x43be7fed (stale) Mirror 8: 0x43be7fed (stale) Mirror 9: 0x43be7fed (stale) Mirror 10: 0x43be7fed (stale) Mirror 13: 0x43be7fed (stale) Mirror 14: 0x43be7fed (stale) Mirror 15: 0x43be7fed (stale) Mirror 16: 0x43be7fed (stale) Mirror 17: 0x4a5b341c (stale) CRC-32 checksum value for chunk [0, 0x36d28): Mirror 1: 0x3f241fa4 Mirror 18: 0x3f241fa4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d28: skipped 000361 1788637799.168542 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d28): Mirror 6: 0x43be7fed (stale) Mirror 8: 0x43be7fed (stale) Mirror 10: 0x43be7fed (stale) Mirror 13: 0x43be7fed (stale) Mirror 14: 0x43be7fed (stale) Mirror 15: 0x43be7fed (stale) Mirror 16: 0x43be7fed (stale) Mirror 17: 0x4a5b341c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d28): Mirror 6: 0x43be7fed (stale) Mirror 8: 0x43be7fed (stale) Mirror 10: 0x43be7fed (stale) Mirror 13: 0x43be7fed (stale) Mirror 14: 0x43be7fed (stale) Mirror 15: 0x43be7fed (stale) Mirror 16: 0x43be7fed (stale) Mirror 17: 0x4a5b341c (stale) CRC-32 checksum value for chunk [0, 0x36d28): Mirror 1: 0x3f241fa4 Mirror 18: 0x3f241fa4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d28: skipped 000362 1788637800.400326 trunc from 0x036d28 to 0x006f6c (0x2fdbb bytes) 000364 1788637800.481536 read 0x006435 thru 0x006f6c (0xb38 bytes) 000365 1788637800.488179 fallocate 0x006886 thru 0x00cccb (0x6446 bytes) 000366 1788637800.494705 mapwrite 0x0247ea thru 0x032bb3 (0xe3ca bytes) 000367 1788637800.579818 trunc from 0x032bb4 to 0x00ebac (0x24007 bytes) 000368 1788637800.603985 read 0x00471d thru 0x0092c8 (0x4bac bytes) 000369 1788637800.623038 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xebad): Mirror 6: 0xb3556261 (stale) Mirror 8: 0xb3556261 (stale) Mirror 10: 0xb3556261 (stale) Mirror 13: 0xb3556261 (stale) Mirror 14: 0xb3556261 (stale) Mirror 15: 0xb3556261 (stale) Mirror 16: 0xb3556261 (stale) Mirror 18: 0xb3556261 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebad): Mirror 6: 0xb3556261 (stale) Mirror 8: 0xb3556261 (stale) Mirror 10: 0xb3556261 (stale) Mirror 13: 0xb3556261 (stale) Mirror 14: 0xb3556261 (stale) Mirror 15: 0xb3556261 (stale) Mirror 16: 0xb3556261 (stale) Mirror 18: 0xb3556261 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebad: skipped 000370 1788637801.109115 trunc from 0x0027ed to 0x005df1 (0x3605 bytes) 000371 1788637801.117864 read 0x006c09 thru 0x00ebac (0x7fa4 bytes) 000372 1788637801.126303 punch from 0x00b9af to 0x00ebac (0x31fe bytes) 000373 1788637801.131659 mapread 0x00d383 thru 0x00ebac (0x182a bytes) 000374 1788637801.145459 trunc from 0x0041b8 to 0x00ebac (0xa9f5 bytes) 000375 1788637801.151297 trunc from 0x00d9e3 to 0x00ebac (0x11ca bytes) 000376 1788637801.160920 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xebad): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebad): Mirror 6: 0xb3556261 (stale) Mirror 8: 0xb3556261 (stale) Mirror 10: 0xb3556261 (stale) Mirror 13: 0xb3556261 (stale) Mirror 14: 0xb3556261 (stale) Mirror 15: 0xb3556261 (stale) Mirror 16: 0xb3556261 (stale) Mirror 18: 0xb3556261 (stale) CRC-32 checksum value for chunk [0, 0xebad): Mirror 1: 0xfa04e69e Mirror 19: 0xfa04e69e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebad: skipped 000377 1788637802.240016 fallocate 0x017d73 thru 0x01b330 (0x35be bytes) 000378 1788637802.341467 write 0x01f2e0 thru 0x028f6b (0x9c8c bytes) 000379 1788637802.355015 mapwrite 0x0343bf thru 0x03ffff (0xbc41 bytes) 000380 1788637802.532272 trunc from 0x040000 to 0x00f1c6 (0x30e39 bytes) 000381 1788637802.557034 mapwrite 0x00a4b3 thru 0x00bda8 (0x18f6 bytes) 000382 1788637802.609540 mapread 0x00dabb thru 0x00f1c6 (0x170c bytes) 000383 1788637802.610564 punch from 0x00af15 to 0x00f1c6 (0x42b2 bytes) 000384 1788637802.619721 trunc from 0x00f1c7 to 0x02e259 (0x1f093 bytes) 000385 1788637802.651637 write 0x00f867 thru 0x01792d (0x80c7 bytes) 000386 1788637802.674262 punch from 0x005140 to 0x014106 (0xefc7 bytes) 000387 1788637802.725494 write 0x01f317 thru 0x024a89 (0x5773 bytes) 000388 1788637802.740467 read 0x01bca0 thru 0x029e40 (0xe1a1 bytes) 000389 1788637802.763138 trunc from 0x02e25a to 0x03bf84 (0xdd2b bytes) 000390 1788637802.784646 read 0x01d7b3 thru 0x02c6ea (0xef38 bytes) 000391 1788637802.806848 read 0x0112a6 thru 0x012933 (0x168e bytes) 000392 1788637802.813201 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf85): Mirror 6: 0xe3942e93 (stale) Mirror 8: 0xe3942e93 (stale) Mirror 10: 0xe3942e93 (stale) Mirror 13: 0xe3942e93 (stale) Mirror 14: 0xe3942e93 (stale) Mirror 15: 0xe3942e93 (stale) Mirror 18: 0x85bfd19d (stale) Mirror 19: 0x34415e0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf85): Mirror 6: 0xe3942e93 (stale) Mirror 8: 0xe3942e93 (stale) Mirror 10: 0xe3942e93 (stale) Mirror 13: 0xe3942e93 (stale) Mirror 14: 0xe3942e93 (stale) Mirror 15: 0xe3942e93 (stale) Mirror 18: 0x85bfd19d (stale) Mirror 19: 0x34415e0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf85: skipped 000393 1788637803.799311 mapwrite 0x03308e thru 0x0353dc (0x234f bytes) 000394 1788637803.881699 trunc from 0x00fbdd to 0x01658a (0x69ae bytes) 000395 1788637803.905019 trunc from 0x03bf85 to 0x036587 (0x59fd bytes) 000396 1788637803.947021 trunc from 0x036588 to 0x002362 (0x34225 bytes) 000397 1788637804.004954 fallocate 0x039143 thru 0x03f0a2 (0x5f60 bytes) 000398 1788637804.033894 mapwrite 0x030e3b thru 0x033143 (0x2309 bytes) 000399 1788637804.077402 trunc from 0x019387 to 0x027adc (0xe756 bytes) 000400 1788637804.088277 mapread 0x011cb1 thru 0x012fea (0x133a bytes) 000401 1788637804.105040 read 0x0209da thru 0x02fd3e (0xf365 bytes) 000402 1788637804.128314 write 0x014f37 thru 0x015ca4 (0xd6e bytes) 000403 1788637804.129672 write 0x03f0d0 thru 0x03ffff (0xf30 bytes) 000404 1788637804.148041 mapread 0x0271da thru 0x028dd8 (0x1bff bytes) 000405 1788637804.150948 trunc from 0x040000 to 0x03757d (0x8a82 bytes) 000406 1788637804.204010 punch from 0x03544a to 0x03757d (0x2134 bytes) 000407 1788637804.214062 read 0x033bc3 thru 0x03757d (0x39bb bytes) 000408 1788637804.232337 write 0x0035d7 thru 0x00c3c7 (0x8df1 bytes) 000409 1788637804.237416 trunc from 0x03757e to 0x008a91 (0x2eaec bytes) 000410 1788637804.263116 fallocate 0x00f8c6 thru 0x01be14 (0xc54f bytes) 000411 1788637804.275212 write 0x0058c8 thru 0x00b95a (0x6093 bytes) 000412 1788637804.282025 fallocate 0x01e125 thru 0x026cd9 (0x8bb5 bytes) 000413 1788637804.294913 mapread 0x00fb42 thru 0x01a3fa (0xa8b9 bytes) 000414 1788637804.494768 trunc from 0x023f8c to 0x026cd9 (0x2d4e bytes) 000415 1788637804.501643 mapread 0x0060f1 thru 0x014ea1 (0xedb1 bytes) 000416 1788637804.524503 MIRROR_RESYNC 9 mirrors 000417 1788637806.034597 trunc from 0x026cda to 0x02122a (0x5aaf bytes) 000418 1788637806.084920 mapwrite 0x00f80e thru 0x01066d (0xe60 bytes) 000419 1788637806.110460 trunc from 0x0109d5 to 0x01d0b2 (0xc6de bytes) 000420 1788637806.115797 write 0x010bb0 thru 0x01531c (0x476d bytes) 000421 1788637806.125529 trunc from 0x01d97b to 0x02122a (0x38b0 bytes) 000422 1788637806.130957 trunc from 0x02122b to 0x01f549 (0x1ce1 bytes) 000424 1788637806.166058 mapwrite 0x033dd5 thru 0x037004 (0x3230 bytes) 000425 1788637806.232934 trunc from 0x037005 to 0x025249 (0x11dbb bytes) 000426 1788637806.255168 mapwrite 0x03909b thru 0x03ffff (0x6f65 bytes) 000428 1788637806.323340 fallocate 0x00e57d thru 0x01bcfb (0xd77f bytes) 000429 1788637806.333889 read 0x03b21e thru 0x03cde2 (0x1bc5 bytes) 000430 1788637806.334835 mapwrite 0x0313d4 thru 0x03cd61 (0xb98e bytes) 000431 1788637806.399747 punch from 0x0312ca to 0x032e4b (0x1b82 bytes) 000432 1788637806.406826 write 0x0325f5 thru 0x03ffff (0xda0b bytes) 000433 1788637806.415680 fallocate 0x000199 thru 0x00c7e5 (0xc64d bytes) 000434 1788637806.424754 mapread 0x017c6c thru 0x024b1e (0xceb3 bytes) 000435 1788637806.471393 read 0x0213fc thru 0x026ec5 (0x5aca bytes) 000436 1788637806.472883 write 0x0351cc thru 0x03ffff (0xae34 bytes) 000437 1788637806.478723 trunc from 0x02b1d9 to 0x034f88 (0x9db0 bytes) 000438 1788637806.508521 punch from 0x02141d to 0x030b10 (0xf6f4 bytes) 000439 1788637806.524786 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe3a3abd1 (stale) Mirror 8: 0xe3a3abd1 (stale) Mirror 10: 0xe3a3abd1 (stale) Mirror 13: 0xe3a3abd1 (stale) Mirror 14: 0xe3a3abd1 (stale) Mirror 15: 0xe3a3abd1 (stale) Mirror 18: 0xe3a3abd1 (stale) Mirror 19: 0xe3a3abd1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b431a12 Mirror 20: 0x4b431a12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000441 1788637807.969087 mapread 0x0002fd thru 0x005101 (0x4e05 bytes) 000442 1788637808.074842 fallocate 0x016809 thru 0x01f08f (0x8887 bytes) 000444 1788637808.158396 trunc from 0x00a0b3 to 0x00f7a4 (0x56f2 bytes) 000446 1788637808.186567 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe3a3abd1 (stale) Mirror 8: 0xe3a3abd1 (stale) Mirror 10: 0xe3a3abd1 (stale) Mirror 13: 0xe3a3abd1 (stale) Mirror 14: 0xe3a3abd1 (stale) Mirror 18: 0xe3a3abd1 (stale) Mirror 19: 0xe3a3abd1 (stale) Mirror 20: 0x4b431a12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe3a3abd1 (stale) Mirror 8: 0xe3a3abd1 (stale) Mirror 10: 0xe3a3abd1 (stale) Mirror 13: 0xe3a3abd1 (stale) Mirror 14: 0xe3a3abd1 (stale) Mirror 18: 0xe3a3abd1 (stale) Mirror 19: 0xe3a3abd1 (stale) Mirror 20: 0x4b431a12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000447 1788637809.067102 write 0x027286 thru 0x032178 (0xaef3 bytes) 000448 1788637809.090194 mapread 0x00b96e thru 0x015efe (0xa591 bytes) 000449 1788637809.139661 mapread 0x00fbd8 thru 0x0137be (0x3be7 bytes) 000450 1788637809.140986 trunc from 0x040000 to 0x009157 (0x36ea8 bytes) 000451 1788637809.183748 MIRROR_RESYNC 9 mirrors 000453 1788637810.634948 mapwrite 0x02987f thru 0x034021 (0xa7a3 bytes) 000455 1788637810.794864 fallocate 0x00bfdc thru 0x01085a (0x487f bytes) 000456 1788637810.806258 write 0x03cc7e thru 0x03ffff (0x3382 bytes) 000457 1788637810.810635 trunc from 0x03d14a to 0x03ffff (0x2eb6 bytes) 000458 1788637810.829606 trunc from 0x011f23 to 0x01bb68 (0x9c46 bytes) 000459 1788637810.839640 mapread 0x0317b1 thru 0x033c53 (0x24a3 bytes) 000460 1788637810.840667 write 0x032f2f thru 0x03be49 (0x8f1b bytes) 000461 1788637810.865383 mapwrite 0x014c7d thru 0x01cd20 (0x80a4 bytes) 000462 1788637810.898291 mapwrite 0x02d4d4 thru 0x02e013 (0xb40 bytes) 000463 1788637810.929455 punch from 0x0347ef to 0x03ffff (0xb811 bytes) 000464 1788637810.942193 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf1e3d511 (stale) Mirror 8: 0xf1e3d511 (stale) Mirror 10: 0xf1e3d511 (stale) Mirror 13: 0xf1e3d511 (stale) Mirror 18: 0xf1e3d511 (stale) Mirror 19: 0xf1e3d511 (stale) Mirror 20: 0xf1e3d511 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf1e3d511 (stale) Mirror 8: 0xf1e3d511 (stale) Mirror 10: 0xf1e3d511 (stale) Mirror 13: 0xf1e3d511 (stale) Mirror 18: 0xf1e3d511 (stale) Mirror 19: 0xf1e3d511 (stale) Mirror 20: 0xf1e3d511 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000465 1788637811.573316 trunc from 0x02e163 to 0x036c47 (0x8ae5 bytes) 000466 1788637811.580570 punch from 0x02b9a2 to 0x03b4a4 (0xfb03 bytes) 000467 1788637811.587684 write 0x025f21 thru 0x026012 (0xf2 bytes) 000468 1788637811.607293 fallocate 0x012b3c thru 0x0144f9 (0x19be bytes) 000469 1788637811.612924 punch from 0x000651 to 0x01026f (0xfc1f bytes) 000470 1788637811.621278 trunc from 0x0189fa to 0x02886e (0xfe75 bytes) 000471 1788637811.635092 mapread 0x017df3 thru 0x0188c5 (0xad3 bytes) 000473 1788637811.659560 MIRROR_RESYNC 8 mirrors 000474 1788637813.733766 write 0x039f82 thru 0x03ffff (0x607e bytes) 000475 1788637813.774561 trunc from 0x040000 to 0x01cc2a (0x233d5 bytes) 000476 1788637813.807319 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cc2b): Mirror 6: 0xb09a167d (stale) Mirror 8: 0xb09a167d (stale) Mirror 10: 0xb09a167d (stale) Mirror 13: 0xb09a167d (stale) Mirror 19: 0xb09a167d (stale) Mirror 20: 0xb09a167d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cc2b): Mirror 6: 0xb09a167d (stale) Mirror 8: 0xb09a167d (stale) Mirror 10: 0xb09a167d (stale) Mirror 13: 0xb09a167d (stale) Mirror 19: 0xb09a167d (stale) Mirror 20: 0xb09a167d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cc2b: skipped 000477 1788637814.262917 write 0x01f00a thru 0x023370 (0x4367 bytes) 000478 1788637814.269529 trunc from 0x01385b to 0x020f20 (0xd6c6 bytes) 000479 1788637814.283896 mapread 0x002b2d thru 0x00f490 (0xc964 bytes) 000480 1788637814.313571 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x23371): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23371): Mirror 6: 0xe527d046 (stale) Mirror 8: 0xe527d046 (stale) Mirror 10: 0xe527d046 (stale) Mirror 13: 0xe527d046 (stale) Mirror 19: 0xe527d046 (stale) Mirror 20: 0xe527d046 (stale) CRC-32 checksum value for chunk [0, 0x23371): Mirror 1: 0x6eef520c Mirror 21: 0x6eef520c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23371: skipped 000481 1788637815.299788 write 0x00d6a8 thru 0x012f63 (0x58bc bytes) 000482 1788637815.372605 trunc from 0x022346 to 0x023370 (0x102b bytes) 000483 1788637815.379601 fallocate 0x037780 thru 0x03ffff (0x8880 bytes) 000484 1788637815.390282 mapread 0x024fa7 thru 0x02d031 (0x808b bytes) 000485 1788637815.425086 read 0x0168f1 thru 0x021b42 (0xb252 bytes) 000486 1788637815.434074 mapwrite 0x027c98 thru 0x03405e (0xc3c7 bytes) 000487 1788637815.482595 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x93012bd3 (stale) Mirror 10: 0x93012bd3 (stale) Mirror 13: 0x93012bd3 (stale) Mirror 19: 0x93012bd3 (stale) Mirror 20: 0x93012bd3 (stale) Mirror 21: 0x4a182181 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x93012bd3 (stale) Mirror 10: 0x93012bd3 (stale) Mirror 13: 0x93012bd3 (stale) Mirror 19: 0x93012bd3 (stale) Mirror 20: 0x93012bd3 (stale) Mirror 21: 0x4a182181 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000488 1788637816.139508 fallocate 0x021354 thru 0x0236b9 (0x2366 bytes) 000489 1788637816.146658 fallocate 0x03c114 thru 0x03ffff (0x3eec bytes) 000490 1788637816.160226 mapwrite 0x01ced9 thru 0x0246fb (0x7823 bytes) 000491 1788637816.215080 fallocate 0x02bf56 thru 0x039f45 (0xdff0 bytes) 000492 1788637816.221407 write 0x0387d5 thru 0x03ffff (0x782b bytes) 000493 1788637816.235665 trunc from 0x040000 to 0x03c316 (0x3ce9 bytes) 000495 1788637816.300995 read 0x015611 thru 0x01c731 (0x7121 bytes) 000496 1788637816.309839 trunc from 0x03c317 to 0x026329 (0x15fed bytes) 000497 1788637816.337152 read 0x01f469 thru 0x026329 (0x6ec1 bytes) 000498 1788637816.350705 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2632a): Mirror 6: 0x34e27f52 (stale) Mirror 10: 0x34e27f52 (stale) Mirror 13: 0x34e27f52 (stale) Mirror 20: 0x34e27f52 (stale) Mirror 21: 0xcd715ea3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2632a): Mirror 6: 0x34e27f52 (stale) Mirror 10: 0x34e27f52 (stale) Mirror 13: 0x34e27f52 (stale) Mirror 20: 0x34e27f52 (stale) Mirror 21: 0xcd715ea3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2632a: skipped 000499 1788637817.081103 write 0x003d20 thru 0x008b95 (0x4e76 bytes) 000500 1788637817.111844 read 0x022928 thru 0x026329 (0x3a02 bytes) 000501 1788637817.125120 punch from 0x023fec to 0x026329 (0x233e bytes) 000502 1788637817.134933 mapwrite 0x00524a thru 0x010693 (0xb44a bytes) 000503 1788637817.210006 write 0x03780a thru 0x03c8a1 (0x5098 bytes) 000504 1788637817.211501 read 0x00bfe3 thru 0x010db6 (0x4dd4 bytes) 000505 1788637817.212094 fallocate 0x030672 thru 0x031545 (0xed4 bytes) 000506 1788637817.221255 mapread 0x023045 thru 0x0316d3 (0xe68f bytes) 000507 1788637817.291762 MIRROR_RESYNC 6 mirrors 000508 1788637819.180141 fallocate 0x01a241 thru 0x027439 (0xd1f9 bytes) 000509 1788637819.246008 mapread 0x01fcdc thru 0x022f06 (0x322b bytes) 000510 1788637819.305938 MIRROR_RESYNC 6 mirrors 000511 1788637821.337186 mapread 0x005dbd thru 0x00d247 (0x748b bytes) 000513 1788637821.381128 fallocate 0x020a02 thru 0x02bfa5 (0xb5a4 bytes) 000514 1788637821.429157 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c8a2): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c8a2): Mirror 6: 0x3b9d8eed (stale) Mirror 10: 0x3b9d8eed (stale) Mirror 13: 0x3b9d8eed (stale) Mirror 20: 0x3b9d8eed (stale) Mirror 21: 0x3b9d8eed (stale) CRC-32 checksum value for chunk [0, 0x3c8a2): Mirror 1: 0x3b9d8eed Mirror 22: 0x3b9d8eed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c8a2: skipped 000515 1788637822.969939 fallocate 0x02db21 thru 0x035e61 (0x8341 bytes) 000516 1788637823.012707 trunc from 0x03c8a2 to 0x023aa8 (0x18df9 bytes) 000517 1788637823.031169 mapread 0x014af2 thru 0x01e48d (0x999c bytes) 000518 1788637823.075959 MIRROR_RESYNC 7 mirrors 000519 1788637824.973801 write 0x02cbb5 thru 0x034f59 (0x83a5 bytes) 000520 1788637825.014051 MIRROR_RESYNC 7 mirrors 000521 1788637827.038598 punch from 0x003adb to 0x00726c (0x3792 bytes) 000522 1788637827.078693 trunc from 0x019771 to 0x02233c (0x8bcc bytes) 000524 1788637827.086647 mapread 0x02d29e thru 0x033f20 (0x6c83 bytes) 000525 1788637827.114965 trunc from 0x030532 to 0x034f59 (0x4a28 bytes) 000526 1788637827.120632 punch from 0x03277c to 0x034f59 (0x27de bytes) 000527 1788637827.127484 trunc from 0x006f39 to 0x00761a (0x6e2 bytes) 000528 1788637827.142577 trunc from 0x034f5a to 0x02f0af (0x5eaa bytes) 000529 1788637827.166772 mapwrite 0x039280 thru 0x03c58f (0x3310 bytes) 000530 1788637827.220652 trunc from 0x03c590 to 0x027abc (0x14ad3 bytes) 000532 1788637827.247524 mapwrite 0x0259b9 thru 0x025b27 (0x16f bytes) 000533 1788637827.267160 punch from 0x025ce6 to 0x027abc (0x1dd7 bytes) 000534 1788637827.277263 trunc from 0x0132b1 to 0x01da3a (0xa78a bytes) 000535 1788637827.287835 read 0x0168e9 thru 0x01ba1f (0x5137 bytes) 000537 1788637827.305989 punch from 0x00e1a6 to 0x0188fc (0xa757 bytes) 000538 1788637827.313734 write 0x02b72b thru 0x034e64 (0x973a bytes) 000539 1788637827.318714 mapread 0x0312a0 thru 0x034e64 (0x3bc5 bytes) 000540 1788637827.321047 read 0x01db1d thru 0x02ba63 (0xdf47 bytes) 000541 1788637827.333025 punch from 0x0317bd to 0x034e64 (0x36a8 bytes) 000542 1788637827.350085 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34e65): Mirror 6: 0x765ad6c9 (stale) Mirror 10: 0x765ad6c9 (stale) Mirror 13: 0x765ad6c9 (stale) Mirror 21: 0x765ad6c9 (stale) Mirror 22: 0x765ad6c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e65): Mirror 6: 0x765ad6c9 (stale) Mirror 10: 0x765ad6c9 (stale) Mirror 13: 0x765ad6c9 (stale) Mirror 21: 0x765ad6c9 (stale) Mirror 22: 0x765ad6c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e65: skipped 000543 1788637827.807863 mapread 0x024c87 thru 0x02cf11 (0x828b bytes) 000544 1788637827.830076 mapread 0x034cc1 thru 0x034e64 (0x1a4 bytes) 000545 1788637827.840278 trunc from 0x034e65 to 0x037de7 (0x2f83 bytes) 000546 1788637827.867320 mapwrite 0x02d728 thru 0x033090 (0x5969 bytes) 000547 1788637827.890608 MIRROR_RESYNC 6 mirrors 000549 1788637830.807723 trunc from 0x017c31 to 0x01b1b6 (0x3586 bytes) 000551 1788637830.845589 punch from 0x017d02 to 0x01c4fc (0x47fb bytes) 000552 1788637830.851841 trunc from 0x02a173 to 0x033976 (0x9804 bytes) 000553 1788637830.866272 MIRROR_RESYNC 6 mirrors 000554 1788637833.899811 fallocate 0x01ea28 thru 0x02e6b8 (0xfc91 bytes) 000555 1788637833.932950 punch from 0x01f5d1 to 0x0296e4 (0xa114 bytes) 000556 1788637833.937904 trunc from 0x037de8 to 0x012a80 (0x25367 bytes) 000557 1788637833.961611 mapwrite 0x0356d2 thru 0x03ffff (0xa92e bytes) 000559 1788637834.021102 trunc from 0x0206c9 to 0x0295a0 (0x8ed8 bytes) 000560 1788637834.026173 fallocate 0x00ee29 thru 0x0181c3 (0x939b bytes) 000561 1788637834.031315 mapread 0x0141bb thru 0x01f1bc (0xb002 bytes) 000562 1788637834.058025 read 0x01c24c thru 0x01d0c2 (0xe77 bytes) 000563 1788637834.058755 trunc from 0x040000 to 0x03348e (0xcb71 bytes) 000565 1788637834.077571 mapread 0x027b1a thru 0x0292c3 (0x17aa bytes) 000566 1788637834.088725 trunc from 0x03348f to 0x025dd5 (0xd6b9 bytes) 000567 1788637834.114267 MIRROR_RESYNC 6 mirrors 000568 1788637835.552959 read 0x023615 thru 0x025dd5 (0x27c1 bytes) 000569 1788637835.562098 write 0x0285f5 thru 0x031b1a (0x9526 bytes) 000570 1788637835.596186 write 0x01a4f9 thru 0x01d01b (0x2b23 bytes) 000572 1788637835.609923 mapwrite 0x03d10e thru 0x03d82f (0x722 bytes) 000573 1788637835.651529 trunc from 0x037372 to 0x03d82f (0x64be bytes) 000574 1788637835.657718 read 0x01e56e thru 0x02bd97 (0xd82a bytes) 000575 1788637835.675172 write 0x038220 thru 0x03fee5 (0x7cc6 bytes) 000576 1788637835.681094 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fee6): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fee6): Mirror 6: 0xe5863075 (stale) Mirror 10: 0xe5863075 (stale) Mirror 13: 0xe5863075 (stale) Mirror 21: 0xe5863075 (stale) Mirror 22: 0xe5863075 (stale) CRC-32 checksum value for chunk [0, 0x3fee6): Mirror 1: 0xe323a41d Mirror 23: 0xe323a41d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fee6: skipped 000577 1788637836.919431 trunc from 0x0033e8 to 0x0078db (0x44f4 bytes) 000578 1788637836.962712 fallocate 0x0381bf thru 0x03ffff (0x7e41 bytes) 000579 1788637836.968725 trunc from 0x012097 to 0x021272 (0xf1dc bytes) 000581 1788637836.973916 fallocate 0x026b0d thru 0x02f59e (0x8a92 bytes) 000582 1788637836.979382 trunc from 0x040000 to 0x002c69 (0x3d396 bytes) 000584 1788637837.002923 mapwrite 0x015731 thru 0x0176cf (0x1f9f bytes) 000585 1788637837.052810 read 0x0128eb thru 0x0176cf (0x4de5 bytes) 000586 1788637837.053611 mapread 0x00abe0 thru 0x0176cf (0xcaf0 bytes) 000587 1788637837.064507 punch from 0x00cdc1 to 0x015f89 (0x91c9 bytes) 000588 1788637837.070050 fallocate 0x0318db thru 0x036f28 (0x564e bytes) 000589 1788637837.076149 mapread 0x01050f thru 0x017391 (0x6e83 bytes) 000590 1788637837.087579 mapwrite 0x019813 thru 0x01b54e (0x1d3c bytes) 000591 1788637837.124859 mapwrite 0x016688 thru 0x01b937 (0x52b0 bytes) 000592 1788637837.173194 write 0x02f2ea thru 0x02f831 (0x548 bytes) 000594 1788637837.175597 mapread 0x0009ae thru 0x006c4d (0x62a0 bytes) 000595 1788637837.182519 fallocate 0x02bb54 thru 0x037c00 (0xc0ad bytes) 000596 1788637837.187221 mapwrite 0x0288b0 thru 0x0369b5 (0xe106 bytes) 000597 1788637837.252157 read 0x022956 thru 0x02ca5d (0xa108 bytes) 000598 1788637837.253443 mapwrite 0x01689c thru 0x018148 (0x18ad bytes) 000599 1788637837.266128 read 0x0256df thru 0x03500a (0xf92c bytes) 000600 1788637837.267219 trunc from 0x0369b6 to 0x02546e (0x11547 bytes) 000601 1788637837.288334 MIRROR_RESYNC 7 mirrors 000602 1788637838.821219 write 0x014fac thru 0x01ec49 (0x9c9e bytes) 000603 1788637838.869410 fallocate 0x0366c1 thru 0x03e44c (0x7d8c bytes) 000604 1788637838.875172 punch from 0x0293d6 to 0x037e2e (0xea59 bytes) 000605 1788637838.880241 mapread 0x00e235 thru 0x01b54e (0xd31a bytes) 000606 1788637838.898876 fallocate 0x032b7c thru 0x039262 (0x66e7 bytes) 000607 1788637838.903941 punch from 0x034603 to 0x036809 (0x2207 bytes) 000608 1788637838.909850 trunc from 0x025386 to 0x033b58 (0xe7d3 bytes) 000609 1788637838.915284 fallocate 0x023438 thru 0x02ae9d (0x7a66 bytes) 000610 1788637838.920296 write 0x020b2d thru 0x027174 (0x6648 bytes) 000611 1788637838.930902 punch from 0x01a58a to 0x01bcbb (0x1732 bytes) 000614 1788637838.954149 trunc from 0x011125 to 0x01b66f (0xa54b bytes) 000615 1788637838.960557 trunc from 0x010a95 to 0x01e62f (0xdb9b bytes) 000616 1788637838.966511 trunc from 0x03e44d to 0x00265a (0x3bdf2 bytes) 000617 1788637838.987055 punch from 0x000074 to 0x00265a (0x25e7 bytes) 000618 1788637838.993119 fallocate 0x003650 thru 0x012d97 (0xf748 bytes) 000619 1788637838.998780 write 0x018c95 thru 0x01ac7f (0x1feb bytes) 000620 1788637838.999860 trunc from 0x00b951 to 0x00ddf7 (0x24a7 bytes) 000622 1788637839.005929 trunc from 0x01ac80 to 0x024653 (0x99d4 bytes) 000623 1788637839.023097 mapread 0x006d4c thru 0x008c4f (0x1f04 bytes) 000624 1788637839.030784 punch from 0x011776 to 0x014731 (0x2fbc bytes) 000625 1788637839.035917 trunc from 0x024654 to 0x03b8f0 (0x1729d bytes) 000626 1788637839.052519 mapread 0x0130cd thru 0x01b1e5 (0x8119 bytes) 000627 1788637839.070724 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8f1): Mirror 6: 0x4ee5717b (stale) Mirror 10: 0x4ee5717b (stale) Mirror 13: 0x4ee5717b (stale) Mirror 21: 0x4ee5717b (stale) Mirror 22: 0x4ee5717b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8f1): Mirror 6: 0x4ee5717b (stale) Mirror 10: 0x4ee5717b (stale) Mirror 13: 0x4ee5717b (stale) Mirror 21: 0x4ee5717b (stale) Mirror 22: 0x4ee5717b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8f1: skipped 000628 1788637839.552802 mapwrite 0x02dea3 thru 0x0323ea (0x4548 bytes) 000629 1788637839.606047 punch from 0x0383f5 to 0x03b8f0 (0x34fc bytes) 000630 1788637839.612011 fallocate 0x00e9a5 thru 0x01d147 (0xe7a3 bytes) 000631 1788637839.619896 trunc from 0x00dec8 to 0x01af6d (0xd0a6 bytes) 000632 1788637839.626980 fallocate 0x027f8c thru 0x02d5bc (0x5631 bytes) 000633 1788637839.633228 mapwrite 0x002542 thru 0x011bfc (0xf6bb bytes) 000634 1788637839.687917 read 0x01dd31 thru 0x020983 (0x2c53 bytes) 000635 1788637839.701573 MIRROR_RESYNC 6 mirrors 000636 1788637842.295318 trunc from 0x03b8f1 to 0x014a94 (0x26e5c bytes) 000637 1788637842.333000 fallocate 0x0012fb thru 0x001374 (0x7a bytes) 000638 1788637842.337448 read 0x0002a6 thru 0x003441 (0x319c bytes) 000640 1788637842.344892 mapread 0x013cf1 thru 0x014a94 (0xda4 bytes) 000641 1788637842.360053 fallocate 0x000fee thru 0x00fc84 (0xec97 bytes) 000642 1788637842.366822 trunc from 0x014a95 to 0x023af9 (0xf065 bytes) 000643 1788637842.399909 punch from 0x00f869 to 0x01a5f6 (0xad8e bytes) 000644 1788637842.407146 read 0x00f05a thru 0x010fb1 (0x1f58 bytes) 000645 1788637842.438107 trunc from 0x023afa to 0x004a80 (0x1f079 bytes) 000646 1788637842.458546 trunc from 0x003e96 to 0x004a80 (0xbeb bytes) 000648 1788637842.465474 trunc from 0x004a81 to 0x0107a1 (0xbd21 bytes) 000649 1788637842.503257 fallocate 0x03b926 thru 0x03ffff (0x46da bytes) 000650 1788637842.516408 MIRROR_RESYNC 6 mirrors 000651 1788637844.371614 read 0x000f0a thru 0x00da3e (0xcb35 bytes) 000653 1788637844.382136 read 0x00547f thru 0x0107a1 (0xb323 bytes) 000654 1788637844.386102 fallocate 0x001777 thru 0x00cfd3 (0xb85d bytes) 000655 1788637844.418598 mapwrite 0x013b3c thru 0x01fdde (0xc2a3 bytes) 000656 1788637844.469090 trunc from 0x01fddf to 0x032458 (0x1267a bytes) 000657 1788637844.486105 punch from 0x017b63 to 0x024ab0 (0xcf4e bytes) 000658 1788637844.491788 trunc from 0x028918 to 0x02ed43 (0x642c bytes) 000659 1788637844.499917 punch from 0x009477 to 0x010e20 (0x79aa bytes) 000660 1788637844.510766 trunc from 0x032459 to 0x036fcd (0x4b75 bytes) 000661 1788637844.542760 read 0x02d5a5 thru 0x02e1ce (0xc2a bytes) 000662 1788637844.558035 fallocate 0x018a3d thru 0x02447c (0xba40 bytes) 000663 1788637844.574007 write 0x0280ee thru 0x03210e (0xa021 bytes) 000664 1788637844.577112 mapread 0x008090 thru 0x017b8b (0xfafc bytes) 000665 1788637844.580609 trunc from 0x006d85 to 0x00d79d (0x6a19 bytes) 000666 1788637844.588966 write 0x02071a thru 0x02b713 (0xaffa bytes) 000667 1788637844.595223 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36fce): Mirror 6: 0x3c04b717 (stale) Mirror 10: 0x3c04b717 (stale) Mirror 21: 0x3c04b717 (stale) Mirror 22: 0x3c04b717 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fce): Mirror 6: 0x3c04b717 (stale) Mirror 10: 0x3c04b717 (stale) Mirror 21: 0x3c04b717 (stale) Mirror 22: 0x3c04b717 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fce: skipped 000668 1788637845.073085 write 0x0132d6 thru 0x01d3e4 (0xa10f bytes) 000669 1788637845.096754 read 0x020a96 thru 0x020b2e (0x99 bytes) 000670 1788637845.110741 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36fce): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fce): Mirror 6: 0x3c04b717 (stale) Mirror 10: 0x3c04b717 (stale) Mirror 21: 0x3c04b717 (stale) Mirror 22: 0x3c04b717 (stale) CRC-32 checksum value for chunk [0, 0x36fce): Mirror 1: 0x16806a9a Mirror 23: 0x16806a9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fce: skipped 000671 1788637846.053184 mapwrite 0x01908a thru 0x0239a2 (0xa919 bytes) 000672 1788637846.185450 mapread 0x0051bf thru 0x00b507 (0x6349 bytes) 000673 1788637846.196658 mapwrite 0x0305c4 thru 0x0330fc (0x2b39 bytes) 000675 1788637846.224443 read 0x018750 thru 0x0261c0 (0xda71 bytes) 000676 1788637846.225250 trunc from 0x016de7 to 0x01a9db (0x3bf5 bytes) 000677 1788637846.230968 mapread 0x02cf8a thru 0x0328af (0x5926 bytes) 000678 1788637846.231894 fallocate 0x0078c0 thru 0x0087b9 (0xefa bytes) 000679 1788637846.237795 trunc from 0x036fce to 0x01bbca (0x1b403 bytes) 000680 1788637846.291145 write 0x01fc2a thru 0x02497e (0x4d55 bytes) 000681 1788637846.292421 write 0x03795f thru 0x03ffff (0x86a1 bytes) 000682 1788637846.294779 mapread 0x0058fc thru 0x0098c1 (0x3fc6 bytes) 000683 1788637846.295828 fallocate 0x018ee3 thru 0x0201c3 (0x72e1 bytes) 000684 1788637846.301978 punch from 0x01c12f to 0x021c1c (0x5aee bytes) 000685 1788637846.317956 fallocate 0x025942 thru 0x0350dd (0xf79c bytes) 000686 1788637846.322914 trunc from 0x040000 to 0x018532 (0x27acd bytes) 000687 1788637846.342183 trunc from 0x00eee0 to 0x0152ae (0x63cf bytes) 000688 1788637846.348811 trunc from 0x018533 to 0x0323f7 (0x19ec5 bytes) 000689 1788637846.378576 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x323f8): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323f8): Mirror 6: 0xe719fef0 (stale) Mirror 10: 0xe719fef0 (stale) Mirror 21: 0xe719fef0 (stale) Mirror 22: 0xe719fef0 (stale) Mirror 23: 0xf45dad81 (stale) CRC-32 checksum value for chunk [0, 0x323f8): Mirror 1: 0xa59407b4 Mirror 24: 0xa59407b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323f8: skipped 000690 1788637847.422908 trunc from 0x0323f8 to 0x02d702 (0x4cf5 bytes) 000691 1788637847.482646 write 0x021dc8 thru 0x0289b8 (0x6bf1 bytes) 000692 1788637847.499540 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d703): Mirror 6: 0x11c6eefc (stale) Mirror 21: 0x11c6eefc (stale) Mirror 22: 0x11c6eefc (stale) Mirror 23: 0x9cd7b739 (stale) Mirror 24: 0xf50012e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d703): Mirror 6: 0x11c6eefc (stale) Mirror 21: 0x11c6eefc (stale) Mirror 22: 0x11c6eefc (stale) Mirror 23: 0x9cd7b739 (stale) Mirror 24: 0xf50012e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d703: skipped 000693 1788637847.982254 read 0x01f255 thru 0x02a511 (0xb2bd bytes) 000694 1788637847.994737 trunc from 0x02d703 to 0x02e98a (0x1288 bytes) 000695 1788637848.025696 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e98b): Mirror 6: 0x95564ec6 (stale) Mirror 21: 0x95564ec6 (stale) Mirror 22: 0x95564ec6 (stale) Mirror 24: 0x34526ae8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e98b): Mirror 6: 0x95564ec6 (stale) Mirror 21: 0x95564ec6 (stale) Mirror 22: 0x95564ec6 (stale) Mirror 24: 0x34526ae8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e98b: skipped 000696 1788637848.409337 mapwrite 0x0337ff thru 0x03f668 (0xbe6a bytes) 000697 1788637848.487945 fallocate 0x00f952 thru 0x0116eb (0x1d9a bytes) 000698 1788637848.494190 write 0x028ccf thru 0x032d33 (0xa065 bytes) 000699 1788637848.501200 mapread 0x028e66 thru 0x033fa2 (0xb13d bytes) 000700 1788637848.502479 punch from 0x02f7a7 to 0x039ed5 (0xa72f bytes) 000701 1788637848.530555 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f669): Mirror 6: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f669): Mirror 6: 0x48a0ef5e (stale) Mirror 21: 0x48a0ef5e (stale) Mirror 22: 0x48a0ef5e (stale) Mirror 24: 0x705b9e5a (stale) CRC-32 checksum value for chunk [0, 0x3f669): Mirror 1: 0x5e1f0826 Mirror 25: 0x5e1f0826 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f669: skipped 000702 1788637849.432814 write 0x024b4d thru 0x02f90a (0xadbe bytes) 000703 1788637849.489839 read 0x00c26a thru 0x00f53b (0x32d2 bytes) 000704 1788637849.496528 trunc from 0x002607 to 0x00848e (0x5e88 bytes) 000705 1788637849.502536 trunc from 0x0307a6 to 0x03c968 (0xc1c3 bytes) 000706 1788637849.507690 read 0x00b84f thru 0x014a58 (0x920a bytes) 000707 1788637849.538598 write 0x02d0fa thru 0x037c02 (0xab09 bytes) 000709 1788637849.540837 mapread 0x014383 thru 0x0178c1 (0x353f bytes) 000711 1788637849.543194 fallocate 0x009132 thru 0x016173 (0xd042 bytes) 000712 1788637849.547777 trunc from 0x025cde to 0x02b388 (0x56ab bytes) 000713 1788637849.566179 read 0x02c9cd thru 0x02e1a1 (0x17d5 bytes) 000714 1788637849.567149 trunc from 0x03f669 to 0x006341 (0x39327 bytes) 000715 1788637849.618250 fallocate 0x008f0b thru 0x009ebd (0xfb3 bytes) 000716 1788637849.624616 mapread 0x004c7e thru 0x009ebd (0x5240 bytes) 000717 1788637849.631127 punch from 0x002f9e to 0x009ebd (0x6f20 bytes) 000718 1788637849.641237 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x9ebe): Mirror 6: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9ebe): Mirror 6: 0x5bafcb17 (stale) Mirror 21: 0x5bafcb17 (stale) Mirror 22: 0x5bafcb17 (stale) Mirror 24: 0x5bafcb17 (stale) Mirror 25: 0x5bafcb17 (stale) CRC-32 checksum value for chunk [0, 0x9ebe): Mirror 1: 0x85fd3d57 Mirror 26: 0x85fd3d57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9ebe: skipped 000719 1788637850.378287 mapwrite 0x009aa8 thru 0x0197d3 (0xfd2c bytes) 000720 1788637850.506497 mapwrite 0x01496d thru 0x01550c (0xba0 bytes) 000721 1788637850.521871 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x197d4): Mirror 6: 0x2cf35c18 (stale) Mirror 21: 0x2cf35c18 (stale) Mirror 22: 0x2cf35c18 (stale) Mirror 25: 0x75f7c42d (stale) Mirror 26: 0x1d81a0fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x197d4): Mirror 6: 0x2cf35c18 (stale) Mirror 21: 0x2cf35c18 (stale) Mirror 22: 0x2cf35c18 (stale) Mirror 25: 0x75f7c42d (stale) Mirror 26: 0x1d81a0fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x197d4: skipped 000722 1788637850.915320 punch from 0x01236d to 0x017174 (0x4e08 bytes) 000723 1788637850.922202 mapwrite 0x0352ca thru 0x03ffff (0xad36 bytes) 000724 1788637850.973286 trunc from 0x040000 to 0x028272 (0x17d8d bytes) 000725 1788637850.993777 write 0x038c32 thru 0x03d4d8 (0x48a7 bytes) 000726 1788637850.995172 mapwrite 0x033a85 thru 0x0367e8 (0x2d64 bytes) 000727 1788637851.019900 mapread 0x01470d thru 0x020151 (0xba45 bytes) 000728 1788637851.038181 fallocate 0x02b96c thru 0x039ce7 (0xe37c bytes) 000730 1788637851.044489 write 0x01ecb1 thru 0x027b20 (0x8e70 bytes) 000732 1788637851.057262 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4d9): Mirror 21: 0x7029f88b (stale) Mirror 22: 0x7029f88b (stale) Mirror 25: 0xfe4b7656 (stale) Mirror 26: 0x5dccf29a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4d9): Mirror 21: 0x7029f88b (stale) Mirror 22: 0x7029f88b (stale) Mirror 25: 0xfe4b7656 (stale) Mirror 26: 0x5dccf29a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4d9: skipped 000733 1788637851.429106 read 0x0010d4 thru 0x00147b (0x3a8 bytes) 000734 1788637851.435774 trunc from 0x00e03d to 0x01d122 (0xf0e6 bytes) 000735 1788637851.441754 fallocate 0x001c2d thru 0x010c24 (0xeff8 bytes) 000736 1788637851.447177 write 0x008d7a thru 0x017955 (0xebdc bytes) 000737 1788637851.465605 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4d9): Mirror 22: 0x7029f88b (stale) Mirror 25: 0xfe4b7656 (stale) Mirror 26: 0x5dccf29a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4d9): Mirror 22: 0x7029f88b (stale) Mirror 25: 0xfe4b7656 (stale) Mirror 26: 0x5dccf29a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4d9: skipped 000738 1788637851.828472 trunc from 0x03d4d9 to 0x021b41 (0x1b997 bytes) 000739 1788637851.850945 trunc from 0x012d19 to 0x01ee45 (0xc12d bytes) 000740 1788637851.861824 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21b42): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b42): Mirror 22: 0xaeb5f702 (stale) Mirror 25: 0x5543be2f (stale) Mirror 26: 0xde334ecb (stale) CRC-32 checksum value for chunk [0, 0x21b42): Mirror 1: 0xb83bb3d9 Mirror 27: 0xb83bb3d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b42: skipped 000741 1788637852.951665 trunc from 0x0078b8 to 0x00d1df (0x5928 bytes) 000742 1788637852.988007 punch from 0x01d38a to 0x021b41 (0x47b8 bytes) 000743 1788637852.993237 mapwrite 0x022c97 thru 0x025281 (0x25eb bytes) 000744 1788637853.072302 mapread 0x00ca24 thru 0x01b427 (0xea04 bytes) 000745 1788637853.105213 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25282): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25282): Mirror 22: 0xd367557 (stale) Mirror 25: 0xee3d5cb5 (stale) Mirror 26: 0x85efba82 (stale) Mirror 27: 0x5b537e8d (stale) CRC-32 checksum value for chunk [0, 0x25282): Mirror 1: 0x44499b45 Mirror 28: 0x44499b45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25282: skipped 000746 1788637854.026288 trunc from 0x004c3f to 0x01397c (0xed3e bytes) 000747 1788637854.071888 mapread 0x000e7f thru 0x00e22a (0xd3ac bytes) 000749 1788637854.095223 mapwrite 0x0318e7 thru 0x03cd6e (0xb488 bytes) 000750 1788637854.225081 trunc from 0x0041d5 to 0x00af03 (0x6d2f bytes) 000752 1788637854.238595 write 0x01b479 thru 0x025d04 (0xa88c bytes) 000753 1788637854.261764 write 0x02235d thru 0x02e950 (0xc5f4 bytes) 000754 1788637854.266805 fallocate 0x00fa13 thru 0x010ea3 (0x1491 bytes) 000755 1788637854.285229 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd6f): Mirror 22: 0x75cfd87a (stale) Mirror 25: 0xf0a1575a (stale) Mirror 27: 0x25dcc5f1 (stale) Mirror 28: 0x50900192 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd6f): Mirror 22: 0x75cfd87a (stale) Mirror 25: 0xf0a1575a (stale) Mirror 27: 0x25dcc5f1 (stale) Mirror 28: 0x50900192 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd6f: skipped 000756 1788637854.762770 punch from 0x00e7ab to 0x01d303 (0xeb59 bytes) 000758 1788637854.772948 trunc from 0x03cd6f to 0x01f15f (0x1dc0f bytes) 000759 1788637854.798813 trunc from 0x01f160 to 0x035ddb (0x16c7c bytes) 000760 1788637854.819880 fallocate 0x02cf62 thru 0x038bb1 (0xbc50 bytes) 000761 1788637854.825747 punch from 0x020907 to 0x025079 (0x4773 bytes) 000762 1788637854.833754 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38bb2): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bb2): Mirror 22: 0x45d67f0b (stale) Mirror 25: 0x1ceb2ea4 (stale) Mirror 27: 0x53ea0fde (stale) Mirror 28: 0x432f00bc (stale) CRC-32 checksum value for chunk [0, 0x38bb2): Mirror 1: 0xd1fed91f Mirror 29: 0xd1fed91f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bb2: skipped 000763 1788637855.740404 trunc from 0x038bb2 to 0x000b79 (0x38038 bytes) 000764 1788637855.830916 trunc from 0x0003a6 to 0x000b79 (0x7d4 bytes) 000765 1788637855.851835 write 0x025071 thru 0x02ccb0 (0x7c40 bytes) 000766 1788637855.857944 fallocate 0x016bf3 thru 0x01d927 (0x6d35 bytes) 000767 1788637855.875153 read 0x020e5c thru 0x024da6 (0x3f4b bytes) 000768 1788637855.900699 punch from 0x005a79 to 0x010894 (0xae1c bytes) 000769 1788637855.915582 read 0x006fbd thru 0x00b763 (0x47a7 bytes) 000770 1788637855.944395 mapwrite 0x01be2b thru 0x0219e4 (0x5bba bytes) 000771 1788637855.999781 punch from 0x01ebd8 to 0x02af0d (0xc336 bytes) 000772 1788637856.005042 mapwrite 0x0264df thru 0x02bfdc (0x5afe bytes) 000773 1788637856.050765 mapread 0x02c6ed thru 0x02ccb0 (0x5c4 bytes) 000774 1788637856.056786 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ccb1): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ccb1): Mirror 22: 0xa191a7ad (stale) Mirror 25: 0x4e4d380c (stale) Mirror 27: 0x60ed44d6 (stale) Mirror 28: 0xaac215b0 (stale) Mirror 29: 0x3213f62b (stale) CRC-32 checksum value for chunk [0, 0x2ccb1): Mirror 1: 0x825af699 Mirror 30: 0x825af699 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ccb1: skipped 000775 1788637856.845941 fallocate 0x032a30 thru 0x03cbb6 (0xa187 bytes) 000776 1788637856.900218 read 0x004dfa thru 0x0060d2 (0x12d9 bytes) 000777 1788637856.907827 trunc from 0x03bfd5 to 0x03cbb6 (0xbe2 bytes) 000778 1788637856.913711 mapread 0x0234b6 thru 0x02a5b4 (0x70ff bytes) 000779 1788637856.937790 write 0x00a11c thru 0x00efea (0x4ecf bytes) 000780 1788637856.951908 fallocate 0x00cd6a thru 0x019ff6 (0xd28d bytes) 000781 1788637856.957537 trunc from 0x01a08d to 0x01dcba (0x3c2e bytes) 000782 1788637856.962912 read 0x02cfea thru 0x03c7bb (0xf7d2 bytes) 000783 1788637856.973735 write 0x00339a thru 0x00fa71 (0xc6d8 bytes) 000784 1788637856.976162 fallocate 0x005962 thru 0x014fc3 (0xf662 bytes) 000785 1788637856.981496 mapread 0x03418e thru 0x03a303 (0x6176 bytes) 000786 1788637856.982867 punch from 0x000c17 to 0x001503 (0x8ed bytes) 000787 1788637856.992387 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cbb7): Mirror 22: 0xe2863088 (stale) Mirror 25: 0xa91aa40a (stale) Mirror 27: 0x8177361d (stale) Mirror 28: 0x89c72765 (stale) Mirror 30: 0xec67162d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbb7): Mirror 22: 0xe2863088 (stale) Mirror 25: 0xa91aa40a (stale) Mirror 27: 0x8177361d (stale) Mirror 28: 0x89c72765 (stale) Mirror 30: 0xec67162d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbb7: skipped 000788 1788637857.408654 trunc from 0x03cbb7 to 0x0388aa (0x430c bytes) 000789 1788637857.452855 write 0x020cff thru 0x030055 (0xf357 bytes) 000790 1788637857.465085 write 0x01e335 thru 0x021df1 (0x3abd bytes) 000792 1788637857.471100 write 0x013944 thru 0x022174 (0xe831 bytes) 000794 1788637857.478734 fallocate 0x00a1a5 thru 0x01593d (0xb799 bytes) 000795 1788637857.484180 mapwrite 0x01a37e thru 0x01b960 (0x15e3 bytes) 000796 1788637857.503067 punch from 0x0327cc to 0x0388aa (0x60df bytes) 000797 1788637857.507759 trunc from 0x027699 to 0x0340c2 (0xca2a bytes) 000798 1788637857.513486 trunc from 0x0388ab to 0x000e87 (0x37a23 bytes) 000799 1788637857.529970 punch from 0x000431 to 0x000e87 (0xa57 bytes) 000800 1788637857.537482 mapread 0x000322 thru 0x000e87 (0xb66 bytes) 000801 1788637857.544756 trunc from 0x000832 to 0x000e87 (0x656 bytes) 000802 1788637857.549903 mapread 0x000ddd thru 0x000e87 (0xab bytes) 000803 1788637857.559199 mapwrite 0x02d184 thru 0x030b94 (0x3a11 bytes) 000804 1788637857.604623 mapread 0x021c3e thru 0x022f36 (0x12f9 bytes) 000805 1788637857.614544 write 0x018f20 thru 0x022a0d (0x9aee bytes) 000806 1788637857.621991 read 0x02354a thru 0x030b94 (0xd64b bytes) 000808 1788637857.624207 trunc from 0x030b95 to 0x021c62 (0xef32 bytes) 000809 1788637857.640850 punch from 0x001ae1 to 0x00b589 (0x9aa9 bytes) 000811 1788637857.655811 MIRROR_RESYNC 6 mirrors 000812 1788637858.848055 trunc from 0x021c63 to 0x000bdb (0x21087 bytes) 000814 1788637858.901837 fallocate 0x01ce30 thru 0x01f278 (0x2449 bytes) 000815 1788637858.922958 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbdc): Mirror 25: 0x1e6cd3c (stale) Mirror 27: 0x1e6cd3c (stale) Mirror 28: 0x1e6cd3c (stale) Mirror 30: 0x1e6cd3c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbdc): Mirror 25: 0x1e6cd3c (stale) Mirror 27: 0x1e6cd3c (stale) Mirror 28: 0x1e6cd3c (stale) Mirror 30: 0x1e6cd3c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbdc: skipped 000816 1788637859.456080 write 0x00aa43 thru 0x015f10 (0xb4ce bytes) 000817 1788637859.466055 write 0x00282d thru 0x0048c9 (0x209d bytes) 000819 1788637859.526774 trunc from 0x00059e to 0x002646 (0x20a9 bytes) 000820 1788637859.575753 write 0x008221 thru 0x00fefb (0x7cdb bytes) 000821 1788637859.600589 write 0x0350ab thru 0x03ffff (0xaf55 bytes) 000822 1788637859.609512 mapread 0x01312a thru 0x022b7d (0xfa54 bytes) 000823 1788637859.707559 punch from 0x01c6dc to 0x02aa91 (0xe3b6 bytes) 000824 1788637859.716863 mapread 0x00820a thru 0x0161fe (0xdff5 bytes) 000825 1788637859.719024 mapread 0x01785a thru 0x01bf46 (0x46ed bytes) 000826 1788637859.719758 mapread 0x02cb4c thru 0x037cde (0xb193 bytes) 000827 1788637859.739191 punch from 0x02f4e1 to 0x03be54 (0xc974 bytes) 000828 1788637859.849505 punch from 0x0159a2 to 0x02544f (0xfaae bytes) 000829 1788637859.870060 trunc from 0x01aae2 to 0x02a843 (0xfd62 bytes) 000830 1788637859.878858 read 0x02e71e thru 0x039d27 (0xb60a bytes) 000831 1788637859.908016 read 0x02a8b4 thru 0x02af30 (0x67d bytes) 000832 1788637859.956727 trunc from 0x02eadd to 0x038a61 (0x9f85 bytes) 000833 1788637859.963970 trunc from 0x040000 to 0x010aac (0x2f553 bytes) 000834 1788637860.010511 fallocate 0x004694 thru 0x009d68 (0x56d5 bytes) 000835 1788637860.024823 trunc from 0x010aad to 0x01057a (0x532 bytes) 000836 1788637860.075799 mapread 0x00ad5f thru 0x01057a (0x581c bytes) 000837 1788637860.093333 MIRROR_RESYNC 5 mirrors 000838 1788637861.677623 mapwrite 0x0384ef thru 0x03ffff (0x7b11 bytes) 000839 1788637861.762107 trunc from 0x018ae5 to 0x01bbfa (0x3116 bytes) 000840 1788637861.768284 fallocate 0x036df1 thru 0x03911a (0x232a bytes) 000842 1788637861.778683 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0x225f883c (stale) Mirror 27: 0x225f883c (stale) Mirror 30: 0x225f883c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0x225f883c (stale) Mirror 27: 0x225f883c (stale) Mirror 30: 0x225f883c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000843 1788637862.203195 mapwrite 0x01a2e9 thru 0x01b652 (0x136a bytes) 000844 1788637862.240442 trunc from 0x040000 to 0x02d5e7 (0x12a18 bytes) 000845 1788637862.270996 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d5e8): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d5e8): Mirror 25: 0x1faf944 (stale) Mirror 27: 0x1faf944 (stale) Mirror 30: 0x1faf944 (stale) CRC-32 checksum value for chunk [0, 0x2d5e8): Mirror 1: 0xa41f0a4c Mirror 31: 0xa41f0a4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d5e8: skipped 000846 1788637863.494071 write 0x03bbd1 thru 0x03ffff (0x442f bytes) 000847 1788637863.596186 mapread 0x03d92e thru 0x03ffff (0x26d2 bytes) 000848 1788637863.602722 mapwrite 0x01c0a2 thru 0x020583 (0x44e2 bytes) 000849 1788637863.716521 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0x225f883c (stale) Mirror 27: 0x225f883c (stale) Mirror 30: 0x225f883c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 25: 0x225f883c (stale) Mirror 27: 0x225f883c (stale) Mirror 30: 0x225f883c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000850 1788637864.152114 read 0x0383c5 thru 0x03c576 (0x41b2 bytes) 000851 1788637864.167605 trunc from 0x040000 to 0x0158f4 (0x2a70b bytes) 000852 1788637864.195231 fallocate 0x024217 thru 0x02909a (0x4e84 bytes) 000853 1788637864.205377 read 0x01213d thru 0x01e5b0 (0xc474 bytes) 000854 1788637864.229020 trunc from 0x02909b to 0x0223c6 (0x6cd4 bytes) 000855 1788637864.245541 write 0x03825e thru 0x03ffff (0x7da2 bytes) 000857 1788637864.250986 fallocate 0x0131c2 thru 0x01472d (0x156c bytes) 000858 1788637864.270137 trunc from 0x040000 to 0x03448d (0xbb72 bytes) 000859 1788637864.294331 write 0x028c05 thru 0x02b8cf (0x2ccb bytes) 000860 1788637864.316645 MIRROR_RESYNC 4 mirrors 000861 1788637865.639611 read 0x019ba0 thru 0x0293f2 (0xf853 bytes) 000862 1788637865.654325 MIRROR_EXTEND to 5 mirrors 000863 1788637866.505445 mapwrite 0x00116f thru 0x00d358 (0xc1ea bytes) 000864 1788637866.763842 trunc from 0x03448e to 0x004a95 (0x2f9f8 bytes) 000865 1788637866.810415 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x4a96): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4a96): Mirror 25: 0x12da9f70 (stale) Mirror 27: 0x12da9f70 (stale) Mirror 30: 0x12da9f70 (stale) Mirror 31: 0x12da9f70 (stale) CRC-32 checksum value for chunk [0, 0x4a96): Mirror 1: 0x24607f9b Mirror 32: 0x24607f9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4a96: skipped 000866 1788637867.936752 trunc from 0x0039b0 to 0x004a95 (0x10e6 bytes) 000868 1788637867.984944 mapwrite 0x01a710 thru 0x028eb5 (0xe7a6 bytes) 000869 1788637868.053555 read 0x025790 thru 0x0271bb (0x1a2c bytes) 000870 1788637868.054165 mapwrite 0x00317a thru 0x00b9d3 (0x885a bytes) 000871 1788637868.081518 mapread 0x021c90 thru 0x028eb5 (0x7226 bytes) 000872 1788637868.082464 fallocate 0x00d31b thru 0x010ba5 (0x388b bytes) 000873 1788637868.095636 write 0x009c44 thru 0x00a9b5 (0xd72 bytes) 000874 1788637868.100852 trunc from 0x028eb6 to 0x012033 (0x16e82 bytes) 000875 1788637868.143751 write 0x008859 thru 0x018666 (0xfe0e bytes) 000876 1788637868.150661 trunc from 0x018667 to 0x010125 (0x8541 bytes) 000878 1788637868.173537 write 0x02c36a thru 0x02d918 (0x15af bytes) 000881 1788637868.177892 punch from 0x021d58 to 0x027943 (0x5bec bytes) 000882 1788637868.183914 trunc from 0x02d919 to 0x02e641 (0xd29 bytes) 000883 1788637868.206602 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e642): Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e642): Mirror 25: 0x2aecdcb5 (stale) Mirror 27: 0x2aecdcb5 (stale) Mirror 30: 0x2aecdcb5 (stale) Mirror 31: 0x2aecdcb5 (stale) Mirror 32: 0xd3a4d88e (stale) CRC-32 checksum value for chunk [0, 0x2e642): Mirror 1: 0x9cc3abbe Mirror 33: 0x9cc3abbe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e642: skipped 000884 1788637869.293024 trunc from 0x003c92 to 0x00bffa (0x8369 bytes) 000885 1788637869.333049 punch from 0x01f757 to 0x02167a (0x1f24 bytes) 000886 1788637869.337173 punch from 0x02b578 to 0x02e641 (0x30ca bytes) 000887 1788637869.341494 mapread 0x0007c8 thru 0x00ddf5 (0xd62e bytes) 000889 1788637869.376018 MIRROR_RESYNC 7 mirrors 000890 1788637870.941463 trunc from 0x02e642 to 0x025c1b (0x8a26 bytes) 000891 1788637870.987649 trunc from 0x0201c9 to 0x021697 (0x14cf bytes) 000892 1788637871.001282 MIRROR_RESYNC 7 mirrors 000893 1788637872.476841 trunc from 0x004088 to 0x008bc8 (0x4b41 bytes) 000894 1788637872.567902 trunc from 0x012865 to 0x013e8e (0x162a bytes) 000895 1788637872.574971 mapread 0x018aa8 thru 0x01c798 (0x3cf1 bytes) 000896 1788637872.606563 mapread 0x01fca0 thru 0x025c1b (0x5f7c bytes) 000897 1788637872.615032 trunc from 0x01f7cb to 0x02599a (0x61d0 bytes) 000898 1788637872.640807 read 0x00d4ac thru 0x0181cf (0xad24 bytes) 000899 1788637872.666401 mapwrite 0x0000e0 thru 0x00d109 (0xd02a bytes) 000900 1788637872.747178 trunc from 0x025c1c to 0x0265d3 (0x9b8 bytes) 000902 1788637872.779879 trunc from 0x0265d4 to 0x00c291 (0x1a342 bytes) 000903 1788637872.800566 mapread 0x00962f thru 0x00c291 (0x2c63 bytes) 000904 1788637872.802142 write 0x033347 thru 0x03ffff (0xccb9 bytes) 000905 1788637872.803914 write 0x03963e thru 0x03ffff (0x69c2 bytes) 000906 1788637872.804786 mapwrite 0x01c1ac thru 0x02381a (0x766f bytes) 000907 1788637872.827309 punch from 0x01238d to 0x019972 (0x75e6 bytes) 000908 1788637872.832546 fallocate 0x0383af thru 0x03f7f3 (0x7445 bytes) 000909 1788637872.837565 mapwrite 0x016d42 thru 0x02365a (0xc919 bytes) 000910 1788637872.871936 trunc from 0x040000 to 0x0150bf (0x2af40 bytes) 000911 1788637872.885908 write 0x0275ce thru 0x02887c (0x12af bytes) 000912 1788637872.889136 punch from 0x012976 to 0x013351 (0x9dc bytes) 000913 1788637872.895000 punch from 0x00e95c to 0x00fc05 (0x12aa bytes) 000914 1788637872.899947 mapread 0x007b26 thru 0x00b333 (0x380e bytes) 000915 1788637872.901018 mapwrite 0x02cc11 thru 0x031dbb (0x51ab bytes) 000916 1788637872.943374 read 0x02c689 thru 0x031dbb (0x5733 bytes) 000917 1788637872.944754 punch from 0x0158e8 to 0x01c23d (0x6956 bytes) 000918 1788637872.950399 read 0x02e7d4 thru 0x031dbb (0x35e8 bytes) 000919 1788637872.956229 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31dbc): Mirror 25: 0x3c0c1162 (stale) Mirror 27: 0x3c0c1162 (stale) Mirror 30: 0x3c0c1162 (stale) Mirror 32: 0x3c0c1162 (stale) Mirror 33: 0x3c0c1162 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31dbc): Mirror 25: 0x3c0c1162 (stale) Mirror 27: 0x3c0c1162 (stale) Mirror 30: 0x3c0c1162 (stale) Mirror 32: 0x3c0c1162 (stale) Mirror 33: 0x3c0c1162 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31dbc: skipped 000920 1788637873.404163 read 0x01e604 thru 0x0258a4 (0x72a1 bytes) 000921 1788637873.420236 fallocate 0x021f05 thru 0x028b83 (0x6c7f bytes) 000922 1788637873.426505 punch from 0x025bf2 to 0x031dbb (0xc1ca bytes) 000923 1788637873.433845 trunc from 0x031dbc to 0x0282ea (0x9ad1 bytes) 000924 1788637873.465305 fallocate 0x01bb3b thru 0x020c77 (0x513d bytes) 000925 1788637873.487894 punch from 0x010752 to 0x01334d (0x2bfc bytes) 000926 1788637873.507979 trunc from 0x0282eb to 0x03dad0 (0x157e6 bytes) 000927 1788637873.537133 punch from 0x02cd2b to 0x02dfcf (0x12a5 bytes) 000928 1788637873.548576 punch from 0x01f98e to 0x02e86c (0xeedf bytes) 000929 1788637873.557211 trunc from 0x03dad1 to 0x00c573 (0x3155d bytes) 000930 1788637873.586626 MIRROR_RESYNC 6 mirrors 000931 1788637874.235853 mapwrite 0x03cd13 thru 0x03ffff (0x32ed bytes) 000932 1788637874.350065 fallocate 0x0308a1 thru 0x032e33 (0x2593 bytes) 000933 1788637874.354511 mapwrite 0x0321b8 thru 0x0330ab (0xef4 bytes) 000934 1788637874.403638 write 0x01fefe thru 0x024af7 (0x4bfa bytes) 000935 1788637874.420943 fallocate 0x029ee9 thru 0x035d65 (0xbe7d bytes) 000936 1788637874.434812 trunc from 0x040000 to 0x0155d8 (0x2aa27 bytes) 000937 1788637874.497527 mapwrite 0x0205a2 thru 0x0233fc (0x2e5b bytes) 000938 1788637874.571921 mapwrite 0x0232c9 thru 0x02d2c0 (0x9ff8 bytes) 000939 1788637874.621685 read 0x0111ff thru 0x0143b4 (0x31b6 bytes) 000940 1788637874.627929 mapwrite 0x008c51 thru 0x0188db (0xfc8b bytes) 000942 1788637874.686120 write 0x0240aa thru 0x027ef4 (0x3e4b bytes) 000943 1788637874.687313 trunc from 0x02d2c1 to 0x00c51c (0x20da4 bytes) 000944 1788637874.725809 write 0x0203e6 thru 0x022d7e (0x2999 bytes) 000945 1788637874.729790 mapwrite 0x00fb91 thru 0x01647a (0x68ea bytes) 000946 1788637874.767020 fallocate 0x02eadf thru 0x03bc84 (0xd1a6 bytes) 000948 1788637874.772628 trunc from 0x01ad46 to 0x02845c (0xd717 bytes) 000949 1788637874.778233 punch from 0x011790 to 0x014ace (0x333f bytes) 000950 1788637874.783935 punch from 0x00af36 to 0x0194a7 (0xe572 bytes) 000952 1788637874.805144 write 0x001676 thru 0x00ef3a (0xd8c5 bytes) 000953 1788637874.821918 trunc from 0x01ff50 to 0x02fd2e (0xfddf bytes) 000954 1788637874.829966 trunc from 0x0019bf to 0x00b822 (0x9e64 bytes) 000955 1788637874.845657 punch from 0x03785f to 0x03bc84 (0x4426 bytes) 000956 1788637874.849148 write 0x02cfb8 thru 0x036195 (0x91de bytes) 000958 1788637874.859186 mapread 0x02aa10 thru 0x032ae5 (0x80d6 bytes) 000959 1788637874.872618 MIRROR_RESYNC 6 mirrors 000960 1788637876.470942 trunc from 0x03bc85 to 0x03d0f3 (0x146f bytes) 000961 1788637876.519770 MIRROR_RESYNC 6 mirrors 000962 1788637878.044124 write 0x023cdc thru 0x031b29 (0xde4e bytes) 000963 1788637878.095194 mapwrite 0x016c1b thru 0x023d17 (0xd0fd bytes) 000964 1788637878.163513 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d0f4): Mirror 25: 0x3bf88bc9 (stale) Mirror 30: 0x3bf88bc9 (stale) Mirror 32: 0x3bf88bc9 (stale) Mirror 33: 0x3bf88bc9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0f4): Mirror 25: 0x3bf88bc9 (stale) Mirror 30: 0x3bf88bc9 (stale) Mirror 32: 0x3bf88bc9 (stale) Mirror 33: 0x3bf88bc9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0f4: skipped 000965 1788637878.549628 fallocate 0x0372f9 thru 0x03ffff (0x8d07 bytes) 000966 1788637878.561150 write 0x0364f1 thru 0x03ffff (0x9b0f bytes) 000967 1788637878.570329 fallocate 0x030a23 thru 0x03b026 (0xa604 bytes) 000968 1788637878.578234 mapwrite 0x001539 thru 0x010e37 (0xf8ff bytes) 000969 1788637878.648091 mapread 0x02a298 thru 0x03739d (0xd106 bytes) 000970 1788637878.668665 trunc from 0x02eb6a to 0x02f988 (0xe1f bytes) 000971 1788637878.681789 write 0x00be8e thru 0x01411e (0x8291 bytes) 000972 1788637878.688827 mapwrite 0x015a15 thru 0x0165af (0xb9b bytes) 000973 1788637878.730327 mapread 0x00d9ea thru 0x013bb6 (0x61cd bytes) 000974 1788637878.731823 trunc from 0x0382ce to 0x039db6 (0x1ae9 bytes) 000975 1788637878.742210 MIRROR_RESYNC 5 mirrors 000976 1788637879.583778 MIRROR_RESYNC 5 mirrors 000977 1788637880.005238 trunc from 0x022542 to 0x02298e (0x44d bytes) 000979 1788637880.042406 read 0x019c39 thru 0x02246e (0x8836 bytes) 000980 1788637880.050190 punch from 0x017ce6 to 0x01f73c (0x7a57 bytes) 000981 1788637880.055422 mapwrite 0x00f2d3 thru 0x01ba71 (0xc79f bytes) 000982 1788637880.103656 mapwrite 0x01f1f7 thru 0x0211fd (0x2007 bytes) 000983 1788637880.115271 read 0x03624a thru 0x037c17 (0x19ce bytes) 000984 1788637880.125339 punch from 0x02ffdd to 0x0322ca (0x22ee bytes) 000985 1788637880.130401 write 0x0103a0 thru 0x01f9e0 (0xf641 bytes) 000986 1788637880.132710 read 0x03a727 thru 0x03d045 (0x291f bytes) 000987 1788637880.133428 read 0x000360 thru 0x00f671 (0xf312 bytes) 000988 1788637880.134534 read 0x01f1b6 thru 0x0263af (0x71fa bytes) 000990 1788637880.136326 mapwrite 0x035f13 thru 0x03b01e (0x510c bytes) 000992 1788637880.167990 punch from 0x008a55 to 0x010879 (0x7e25 bytes) 000993 1788637880.184070 MIRROR_RESYNC 5 mirrors 000994 1788637881.223463 read 0x006fe9 thru 0x0123fd (0xb415 bytes) 000995 1788637881.235438 trunc from 0x040000 to 0x0360d9 (0x9f26 bytes) 000996 1788637881.291265 trunc from 0x0360da to 0x008324 (0x2ddb5 bytes) 000997 1788637881.312260 write 0x01bbfb thru 0x01e3c8 (0x27ce bytes) 000998 1788637881.315244 trunc from 0x0110bc to 0x01cf36 (0xbe7b bytes) 000999 1788637881.329115 mapread 0x00235f thru 0x00bbf5 (0x9897 bytes) 001000 1788637881.351747 trunc from 0x01e3c9 to 0x012414 (0xbfb4 bytes) All operations completed A-OK! PASS 70a (141s) == sanity-flr test 71: check mirror extend parent fid ==== 15:51:27 (1788637887) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x17c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 15:51:37 (1788637897) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168409 s, 12.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112057 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 15:51:46 (1788637906) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6894656 .. Extending file size to 1555968 .. done Extending file size to 6481760 .. Extending file size to 7634336 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8304288 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8432704 .. Extending file size to 8473088 .. Extending file size to 9088192 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9007744 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61324 61325 61327 61328 61329 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) 192.168.201.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:53:05 (1788637985) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 128480 .. Extending file size to 6964704 .. done Extending file size to 5516384 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6486624 .. Extending file size to 7170368 .. Extending file size to 7708544 .. Extending file size to 9103488 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8105952 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8552032 .. Extending file size to 9130240 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8864832 .. Waiting 62754 62755 62756 62759 62760 done 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) 192.168.201.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:54:29 (1788638069) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1268816 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25488 3581532 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17784 3589236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43272 7170768 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2488 1268816 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25488 3581532 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17784 3589236 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43272 7170768 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159232 s, 13.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.344946 s, 6.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcd8d6c92 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:54:52 (1788638092) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112094 s, 18.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 15:55:01 (1788638101) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:55:11 (1788638111) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104645 s, 20.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:55:21 (1788638121) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0975109 s, 21.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:55:31 (1788638131) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11552 s, 18.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:56:06 (1788638166) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511882 s, 20.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:57:01 (1788638221) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0838467 s, 25.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:57:57 (1788638277) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg135-server: Pool lustre.test_204f created Adding targets to pool oleg135-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg135-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123039 s, 17.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg135-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg135-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg135-server: Pool lustre.test_204f destroyed PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:59:09 (1788638349) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:59:17 (1788638357) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:59:26 (1788638366) oleg135-server: Pool lustre.test_206 created Adding targets to pool oleg135-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg135-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg135-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg135-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg135-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:59:56 (1788638396) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132587 s, 7.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052083 s, 20.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x60953421 Mirror 3: 0x60953421 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:00:07 (1788638407) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:00:10 (1788638410) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:00:14 (1788638414) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0345643 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xff0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2440000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2440000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2440000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xbe0000 Read this (4 bytes): BBBB 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:00:54 (1788638454) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0645409 s, 127 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:01:02 (1788638462) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110137 s, 9.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc8 lmm_fid: [0x200000405:0xc8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc8 lmm_fid: [0x200000405:0xc8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x142:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611336 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 16:01:04.000000000 -0400 Modify: 2026-09-05 16:01:04.000000000 -0400 Change: 2026-09-05 16:01:06.000000000 -0400 Birth: 2026-09-05 16:01:05.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:01:12 (1788638472) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754648 s, 13.9 MB/s oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: O/280000401/d4/164: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:01:38 (1788638498) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.295133 s, 139 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.017451 s, 235 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 16:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xcd:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 16:01 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:01:46 (1788638506) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:01:55 (1788638515) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1951 sec =========== 16:02:04 (1788638524) === sanity-flr: start cleanup 16:02:06 (1788638526) === === sanity-flr: finish cleanup 16:02:15 (1788638535) ===