-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 12:07:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg233-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:08:04 (1788624484) === oleg233-client.virtnet: executing check_config_client /mnt/lustre oleg233-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg233-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff938987a12800.idle_timeout=debug osc.lustre-OST0001-osc-ffff938987a12800.idle_timeout=debug disable quota as required oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:08:40 (1788624520) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8232] == sanity-flr test 0a: lfs mirror create with -N option == 12:08:42 (1788624522) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:08:58 (1788624538) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:09:02 (1788624542) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:09:05 (1788624545) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:09:36 (1788624576) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (6s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:09:42 (1788624582) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 12:09:48 (1788624588) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff938987a12800.stats=clear osc.lustre-OST0001-osc-ffff938987a12800.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:09:59 (1788624599) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:10:13 (1788624613) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:10:22 (1788624622) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:10:26 (1788624626) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:10:34 (1788624634) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 12:10:44 (1788624644) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:10:52 (1788624652) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:11:00 (1788624660) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:11:08 (1788624668) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:11:19 (1788624679) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:11:27 (1788624687) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.89867 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121739 s, 8.6 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:11:39 (1788624699) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.136498 s, 30.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:11:48 (1788624708) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:11:59 (1788624719) Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (84s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:13:24 (1788624804) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.640105 s, 13.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:14:09 (1788624849) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:15:22 (1788624922) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.87925 s, 18.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 12:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.876s user 0m0.000s sys 0m0.162s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff938987a12800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.118s user 0m0.002s sys 0m0.196s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.270s user 0m0.000s sys 0m0.177s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff938987a12800.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:16:52 (1788625012) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:16:56 (1788625016) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:16:59 (1788625019) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:17:03 (1788625023) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 12:17:10 (1788625030) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.69518 s, 18.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:17:25 (1788625045) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523547 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402309 s, 26.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:17:33 (1788625053) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:17:42 (1788625062) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103161 s, 993 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122271 s, 837 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123758 s, 827 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00390311 s, 262 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 12:17:52 (1788625072) create test files with size 15745957 .. ae3201ceeb60ad94b31bbcf7d0f80f88 - 999f3840ef644d6aa23b401ab94dbb7f - 840d78373dc07a7d6307bf9b917f8c7b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 12:18:14 (1788625094) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:18:30 (1788625110) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:18:39 (1788625119) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128093 s, 8.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023614 s, 44.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 12:18:49 (1788625129) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151777 s, 13.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.376654 s, 11.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118395 s, 17.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 12:19:05 (1788625145) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 12:19:09 (1788625149) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:19:13 (1788625153) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954615 s, 11.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 12:19:21 (1788625161) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187184 s, 16.8 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:19:32 (1788625172) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:19:43 (1788625183) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.506438 s, 20.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:20:00 (1788625200) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff938985372800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.290459 s, 36.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff938986a19800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:20:19 (1788625219) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:20:59 (1788625259) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:21:38 (1788625298) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:21:42 (1788625302) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.183167 s, 17.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.157191 s, 20.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8cf2794c (nosync,stale) Mirror 2: 0x8cf2794c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2f1bcc4947c94002d3ed287d548512ca - mirror 1 checksum: c183df3fbcb2a4a346a6059ab3b4b328 - mirror 2 checksum: 2f1bcc4947c94002d3ed287d548512ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T12:21:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T12:21:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2f1bcc4947c94002d3ed287d548512ca - mirror 1 checksum: c183df3fbcb2a4a346a6059ab3b4b328 - mirror 2 checksum: c183df3fbcb2a4a346a6059ab3b4b328 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T12:21:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T12:21:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:21:52 (1788625312) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347083 s, 30.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:22:00 (1788625320) Failing mds1 on oleg233-server Stopping /mnt/lustre-mds1 (opts:) on oleg233-server 12:22:12 (1788625332) shut down facet: mds1 facet_host: oleg233-server facet_failover_host: oleg233-server Failover mds1 to oleg233-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 12:22:47 (1788625367) targets are mounted 12:22:47 (1788625367) facet_failover done oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294437 s, 35.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:23:04 (1788625384) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0442921 s, 462 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0249633 s, 820 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:23:15 (1788625395) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0442822 s, 462 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00782104 s, 2.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:23:24 (1788625404) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0106691 s, 1.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:23:33 (1788625413) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0533362 s, 384 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:23:41 (1788625421) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00980559 s, 418 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0321125 s, 128 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:23:49 (1788625429) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:24:09 (1788625449) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:24:40 (1788625480) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:25:05 (1788625505) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:25:29 (1788625529) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 12:25:39 (1788625539) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57577 mirror split pid 57578 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57577 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57578 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 12:26:48 (1788625608) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5611 000001 1788625611.720575 mapwrite 0x032870 thru 0x03c51a (0x9cab bytes) 000002 1788625611.785923 MIRROR_EXTEND to 2 mirrors 000003 1788625612.580031 mapread 0x02ffea thru 0x039455 (0x946c bytes) 000004 1788625612.606395 mapread 0x0227e2 thru 0x0265d3 (0x3df2 bytes) 000006 1788625612.622177 mapread 0x01affc thru 0x023a53 (0x8a58 bytes) zero_range to largest ever: 0x281a3 000007 1788625612.646905 trunc from 0x018d17 to 0x0281a2 (0xf48c bytes) 000008 1788625612.688780 read 0x011d90 thru 0x01533f (0x35b0 bytes) 000009 1788625612.704035 read 0x012766 thru 0x01313f (0x9da bytes) 000010 1788625612.705082 fallocate 0x0199bc thru 0x023a1e (0xa063 bytes) 000011 1788625612.714445 trunc from 0x02e13d to 0x0346aa (0x656e bytes) 000012 1788625612.721190 read 0x019b0b thru 0x019e05 (0x2fb bytes) 000013 1788625612.735961 mapwrite 0x01e119 thru 0x02918e (0xb076 bytes) 000014 1788625612.766821 trunc from 0x016463 to 0x024920 (0xe4be bytes) 000015 1788625612.776966 punch from 0x013a0b to 0x01e75d (0xad53 bytes) 000016 1788625612.783004 mapread 0x010d80 thru 0x01497d (0x3bfe bytes) 000017 1788625612.796882 write 0x00c7a5 thru 0x01c3c8 (0xfc24 bytes) 000018 1788625612.812191 write 0x009164 thru 0x016b3c (0xd9d9 bytes) fallocating to largest ever: 0x40000 000019 1788625612.814946 fallocate 0x03aa43 thru 0x03ffff (0x55bd bytes) 000020 1788625612.821034 read 0x038ac4 thru 0x03c51a (0x3a57 bytes) 000021 1788625612.822256 punch from 0x03863f to 0x03c51a (0x3edc bytes) 000023 1788625612.830441 write 0x02d01e thru 0x03a13c (0xd11f bytes) 000024 1788625612.841035 trunc from 0x01899e to 0x01fd76 (0x73d9 bytes) 000025 1788625612.867384 mapwrite 0x00c232 thru 0x00d26a (0x1039 bytes) 000026 1788625612.881107 write 0x01210e thru 0x013f2a (0x1e1d bytes) 000028 1788625612.883369 trunc from 0x03c51b to 0x00c991 (0x2fb89 bytes) 000029 1788625612.905871 trunc from 0x009dc7 to 0x00c991 (0x2bcb bytes) 000030 1788625612.912906 write 0x008bbc thru 0x0092cd (0x712 bytes) 000031 1788625612.919164 mapread 0x005e63 thru 0x007c65 (0x1e03 bytes) 000032 1788625612.920471 mapread 0x0043f4 thru 0x00c991 (0x859e bytes) 000033 1788625612.939070 punch from 0x00bec5 to 0x00c991 (0xacd bytes) 000034 1788625612.944763 trunc from 0x008170 to 0x00c991 (0x4822 bytes) 000035 1788625612.961863 trunc from 0x00b49f to 0x00c991 (0x14f3 bytes) 000038 1788625612.969610 punch from 0x00a17e to 0x00c991 (0x2814 bytes) 000039 1788625612.975874 mapread 0x001dc4 thru 0x004be6 (0x2e23 bytes) 000041 1788625612.978792 mapread 0x000582 thru 0x00c991 (0xc410 bytes) 000042 1788625613.008024 mapread 0x007605 thru 0x00c991 (0x538d bytes) 000043 1788625613.009138 punch from 0x000eef to 0x00c991 (0xbaa3 bytes) 000044 1788625613.015621 read 0x003c7b thru 0x0063b5 (0x273b bytes) 000045 1788625613.028870 mapread 0x008782 thru 0x00c991 (0x4210 bytes) 000046 1788625613.030190 fallocate 0x03568f thru 0x03c2a1 (0x6c13 bytes) 000047 1788625613.037037 punch from 0x03b5cc to 0x03c2a1 (0xcd6 bytes) 000048 1788625613.043812 mapread 0x00bd3c thru 0x015682 (0x9947 bytes) 000049 1788625613.065041 MIRROR_RESYNC 2 mirrors 000050 1788625613.726026 trunc from 0x03c2a2 to 0x002b06 (0x3979b bytes) 000051 1788625613.774179 trunc from 0x002b07 to 0x01f11c (0x1c616 bytes) 000052 1788625613.792253 trunc from 0x00d124 to 0x01d0b9 (0xff96 bytes) 000053 1788625613.798055 mapread 0x01ee39 thru 0x01efc8 (0x190 bytes) 000054 1788625613.803415 mapread 0x013e61 thru 0x01ef68 (0xb108 bytes) 000055 1788625613.824978 punch from 0x000cf1 to 0x00755c (0x686c bytes) 000056 1788625613.829902 mapread 0x019d05 thru 0x01f11c (0x5418 bytes) 000057 1788625613.831016 fallocate 0x011168 thru 0x015e6e (0x4d07 bytes) 000058 1788625613.838045 read 0x01c0dc thru 0x01f11c (0x3041 bytes) 000059 1788625613.838962 punch from 0x01a370 to 0x01f11c (0x4dad bytes) 000060 1788625613.847854 write 0x02d996 thru 0x033d1c (0x6387 bytes) 000061 1788625613.849151 mapwrite 0x03c43e thru 0x03ffff (0x3bc2 bytes) 000062 1788625613.895850 trunc from 0x040000 to 0x009a22 (0x365dd bytes) 000063 1788625613.933343 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) 000065 1788625614.087782 fallocate 0x026e9a thru 0x02e95d (0x7ac4 bytes) 000067 1788625614.095460 punch from 0x001959 to 0x009a22 (0x80ca bytes) 000069 1788625614.107794 mapread 0x0014ee thru 0x001ee5 (0x9f8 bytes) 000070 1788625614.114751 punch from 0x008a30 to 0x009782 (0xd53 bytes) 000071 1788625614.119995 mapread 0x005e9d thru 0x009a22 (0x3b86 bytes) 000072 1788625614.142890 MIRROR_EXTEND to 2 mirrors 000073 1788625614.776969 write 0x007d69 thru 0x01327e (0xb516 bytes) 000074 1788625614.827410 mapread 0x0078ba thru 0x009da9 (0x24f0 bytes) 000075 1788625614.828320 mapread 0x00b141 thru 0x01327e (0x813e bytes) 000076 1788625614.829167 fallocate 0x03d4aa thru 0x03ffff (0x2b56 bytes) 000077 1788625614.836937 mapwrite 0x01f584 thru 0x023eb6 (0x4933 bytes) 000078 1788625614.929819 read 0x0143d4 thru 0x01ed25 (0xa952 bytes) 000079 1788625614.954963 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) 000080 1788625615.176430 MIRROR_EXTEND to 2 mirrors 000081 1788625616.099624 punch from 0x00da00 to 0x014d01 (0x7302 bytes) 000082 1788625616.143009 fallocate 0x010fc9 thru 0x018296 (0x72ce bytes) 000084 1788625616.149161 punch from 0x015e1d to 0x01a475 (0x4659 bytes) 000085 1788625616.154966 punch from 0x00ecd2 to 0x01a7ee (0xbb1d bytes) 000086 1788625616.160848 read 0x0185ee thru 0x023eb6 (0xb8c9 bytes) 000087 1788625616.170950 mapwrite 0x02dd34 thru 0x0391c6 (0xb493 bytes) 000089 1788625616.245943 punch from 0x013666 to 0x018a0a (0x53a5 bytes) 000090 1788625616.252926 write 0x00d995 thru 0x0145ea (0x6c56 bytes) 000091 1788625616.270818 mapwrite 0x032576 thru 0x03ffff (0xda8a bytes) 000092 1788625616.339040 punch from 0x018c3d to 0x019058 (0x41c bytes) 000093 1788625616.347106 mapwrite 0x03f1e1 thru 0x03f4cb (0x2eb bytes) 000094 1788625616.366152 trunc from 0x029fa7 to 0x02ea4d (0x4aa7 bytes) 000097 1788625616.373985 trunc from 0x004401 to 0x009eeb (0x5aeb bytes) 000098 1788625616.385073 mapread 0x02147a thru 0x02448e (0x3015 bytes) 000099 1788625616.402138 write 0x0326d4 thru 0x03ffff (0xd92c bytes) 000100 1788625616.404853 mapwrite 0x026903 thru 0x035548 (0xec46 bytes) 000101 1788625616.435142 trunc from 0x033cb1 to 0x03bdf0 (0x8140 bytes) 000103 1788625616.442610 mapwrite 0x001eff thru 0x007769 (0x586b bytes) 000104 1788625616.476128 fallocate 0x010d09 thru 0x01c221 (0xb519 bytes) 000105 1788625616.481826 trunc from 0x017cf0 to 0x018b90 (0xea1 bytes) 000106 1788625616.487053 mapread 0x0021c7 thru 0x00d22c (0xb066 bytes) 000107 1788625616.488224 read 0x031a23 thru 0x03fe09 (0xe3e7 bytes) 000108 1788625616.501029 mapread 0x020356 thru 0x0249c4 (0x466f bytes) 000109 1788625616.502777 trunc from 0x040000 to 0x01f1de (0x20e21 bytes) 000110 1788625616.521986 mapwrite 0x003d5f thru 0x011c42 (0xdee4 bytes) 000112 1788625616.549703 MIRROR_RESYNC 2 mirrors 000113 1788625617.200069 fallocate 0x02186a thru 0x0312fa (0xfa91 bytes) 000114 1788625617.244149 write 0x01b1b9 thru 0x02443c (0x9284 bytes) 000115 1788625617.252610 fallocate 0x00ee68 thru 0x01a775 (0xb90e bytes) 000116 1788625617.258035 read 0x002334 thru 0x007394 (0x5061 bytes) 000117 1788625617.265704 fallocate 0x0021c6 thru 0x00c412 (0xa24d bytes) 000118 1788625617.275129 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) 000119 1788625617.431276 read 0x0010b2 thru 0x010d63 (0xfcb2 bytes) 000121 1788625617.450182 write 0x02eabf thru 0x03e64a (0xfb8c bytes) 000122 1788625617.456813 trunc from 0x03e64b to 0x01a4fa (0x24150 bytes) 000124 1788625617.509962 MIRROR_EXTEND to 2 mirrors 000125 1788625618.148866 trunc from 0x00f4d0 to 0x01642e (0x6f5f bytes) 000126 1788625618.215901 punch from 0x017ed0 to 0x01a4fa (0x262b bytes) 000127 1788625618.222695 mapread 0x009ceb thru 0x00a29c (0x5b2 bytes) 000128 1788625618.250892 mapread 0x01a3b2 thru 0x01a4fa (0x149 bytes) 000129 1788625618.276061 punch from 0x00c0df to 0x01328b (0x71ad bytes) 000130 1788625618.298863 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) 000131 1788625618.620955 mapwrite 0x036f5a thru 0x03e60a (0x76b1 bytes) 000132 1788625618.759929 mapread 0x0150e0 thru 0x022356 (0xd277 bytes) 000134 1788625618.787047 trunc from 0x031602 to 0x035e1b (0x481a bytes) 000135 1788625618.794124 write 0x00cf5f thru 0x00d935 (0x9d7 bytes) 000137 1788625618.810649 write 0x0342c2 thru 0x03ffff (0xbd3e bytes) 000138 1788625618.821581 mapread 0x01dc3a thru 0x01df5e (0x325 bytes) 000139 1788625618.822549 write 0x02c4c7 thru 0x037f1c (0xba56 bytes) 000140 1788625618.832570 trunc from 0x01c63c to 0x02b129 (0xeaee bytes) 000141 1788625618.841659 write 0x0353c5 thru 0x03ffff (0xac3b bytes) 000142 1788625618.842996 fallocate 0x02454e thru 0x02f9c2 (0xb475 bytes) 000144 1788625618.852794 write 0x01fc2b thru 0x02aeea (0xb2c0 bytes) 000147 1788625618.886017 trunc from 0x017833 to 0x018fd8 (0x17a6 bytes) 000149 1788625618.904166 write 0x026cc0 thru 0x02bc83 (0x4fc4 bytes) 000150 1788625618.913668 mapwrite 0x03d0d1 thru 0x03ffff (0x2f2f bytes) 000152 1788625618.948948 write 0x038f5b thru 0x038fed (0x93 bytes) 000153 1788625618.951008 punch from 0x009396 to 0x00b2c8 (0x1f33 bytes) 000154 1788625618.961127 fallocate 0x020f0c thru 0x02d63c (0xc731 bytes) 000155 1788625618.972934 mapread 0x02289e thru 0x02b437 (0x8b9a bytes) 000156 1788625618.978193 punch from 0x02827f to 0x02a7a2 (0x2524 bytes) 000157 1788625618.987623 mapread 0x028570 thru 0x02adeb (0x287c bytes) 000159 1788625619.004606 trunc from 0x01d95d to 0x02a45a (0xcafe bytes) 000160 1788625619.027534 punch from 0x01dfa4 to 0x02790d (0x996a bytes) 000161 1788625619.037921 fallocate 0x03fe41 thru 0x03ffff (0x1bf bytes) 000162 1788625619.044652 mapwrite 0x03da43 thru 0x03ffff (0x25bd bytes) 000163 1788625619.089951 mapwrite 0x02226e thru 0x031b44 (0xf8d7 bytes) 000164 1788625619.167367 mapwrite 0x0221c1 thru 0x028cbb (0x6afb bytes) 000165 1788625619.205406 read 0x02b778 thru 0x02fff8 (0x4881 bytes) 000166 1788625619.206707 fallocate 0x013231 thru 0x019263 (0x6033 bytes) 000167 1788625619.225048 mapwrite 0x021c9d thru 0x023aa1 (0x1e05 bytes) 000168 1788625619.268450 write 0x034159 thru 0x0359e3 (0x188b bytes) 000169 1788625619.271674 trunc from 0x040000 to 0x0062fc (0x39d03 bytes) 000170 1788625619.307935 mapread 0x005941 thru 0x0062fc (0x9bc bytes) 000171 1788625619.330639 mapwrite 0x017da8 thru 0x025213 (0xd46c bytes) 000172 1788625619.452805 fallocate 0x004088 thru 0x00baed (0x7a66 bytes) 000175 1788625619.468103 trunc from 0x01569b to 0x01ac1e (0x5584 bytes) 000176 1788625619.483111 write 0x031835 thru 0x03228c (0xa58 bytes) 000177 1788625619.490762 trunc from 0x0280e1 to 0x03228c (0xa1ac bytes) 000178 1788625619.522116 punch from 0x030038 to 0x03228c (0x2255 bytes) 000179 1788625619.536161 fallocate 0x017eac thru 0x017ff4 (0x149 bytes) 000180 1788625619.549527 read 0x02e4da thru 0x03228c (0x3db3 bytes) 000182 1788625619.576951 MIRROR_EXTEND to 2 mirrors 000184 1788625620.733155 trunc from 0x03228d to 0x0213f3 (0x10e99 bytes) 000185 1788625620.802111 mapread 0x00cb1b thru 0x00f687 (0x2b6d bytes) 000187 1788625620.830481 write 0x033259 thru 0x03f5d6 (0xc37e bytes) 000188 1788625620.832901 mapwrite 0x031df5 thru 0x036fdd (0x51e9 bytes) 000189 1788625620.857768 trunc from 0x03f5d7 to 0x00e385 (0x31251 bytes) 000190 1788625620.873718 trunc from 0x006753 to 0x006d0a (0x5b8 bytes) 000191 1788625620.879912 punch from 0x003361 to 0x00cd02 (0x99a2 bytes) 000192 1788625620.885912 mapwrite 0x02064f thru 0x025f8e (0x5940 bytes) 000193 1788625620.942641 trunc from 0x025f8f to 0x038caa (0x12d1c bytes) 000194 1788625620.964066 write 0x03411f thru 0x03c5a4 (0x8486 bytes) 000196 1788625620.971863 read 0x010a6a thru 0x01d1bc (0xc753 bytes) 000197 1788625620.981665 mapwrite 0x009c62 thru 0x012086 (0x8425 bytes) 000198 1788625621.016861 MIRROR_RESYNC 2 mirrors 000199 1788625622.114832 mapwrite 0x022e17 thru 0x029e29 (0x7013 bytes) 000200 1788625622.279933 fallocate 0x004350 thru 0x012dba (0xea6b bytes) 000201 1788625622.288028 trunc from 0x03c5a5 to 0x010d09 (0x2b89b bytes) 000202 1788625622.336737 read 0x007999 thru 0x010d09 (0x9371 bytes) 000204 1788625622.358416 read 0x00f95d thru 0x010d09 (0x13ad bytes) 000205 1788625622.370090 read 0x003897 thru 0x00f088 (0xb7f2 bytes) 000206 1788625622.404921 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10d0a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d0a): Mirror 2: 0x8b9a1121 (stale) CRC-32 checksum value for chunk [0, 0x10d0a): Mirror 1: 0x8b9a1121 Mirror 3: 0x8b9a1121 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d0a: skipped 000207 1788625623.695985 punch from 0x002b62 to 0x006479 (0x3918 bytes) 000208 1788625623.793136 mapwrite 0x00992d thru 0x00a11e (0x7f2 bytes) 000209 1788625623.828798 trunc from 0x004f80 to 0x005084 (0x105 bytes) 000210 1788625623.849073 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10d0a): Mirror 3: 0x8b9a1121 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d0a): Mirror 3: 0x8b9a1121 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d0a: skipped 000211 1788625624.181096 mapread 0x003937 thru 0x00c81c (0x8ee6 bytes) 000212 1788625624.223026 trunc from 0x00d71a to 0x010d09 (0x35f0 bytes) 000213 1788625624.240853 fallocate 0x0175ac thru 0x020f8a (0x99df bytes) 000214 1788625624.260681 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) 000215 1788625624.496876 fallocate 0x03a229 thru 0x03ef77 (0x4d4f bytes) 000216 1788625624.506605 read 0x00f773 thru 0x010c85 (0x1513 bytes) 000217 1788625624.513214 mapread 0x009c19 thru 0x010d09 (0x70f1 bytes) 000218 1788625624.559659 trunc from 0x010d0a to 0x005bf4 (0xb115 bytes) 000219 1788625624.616720 trunc from 0x002cc1 to 0x005bf4 (0x2f34 bytes) 000221 1788625624.632254 mapwrite 0x01b047 thru 0x02638b (0xb345 bytes) 000222 1788625624.749225 trunc from 0x02638c to 0x019972 (0xca19 bytes) 000224 1788625624.780679 write 0x0150f5 thru 0x01b224 (0x6130 bytes) 000226 1788625624.790677 trunc from 0x01b225 to 0x02b007 (0xfde3 bytes) 000227 1788625624.825729 write 0x023015 thru 0x02e07b (0xb067 bytes) 000228 1788625624.851882 trunc from 0x01959a to 0x024bd8 (0xb63f bytes) 000229 1788625624.888413 mapwrite 0x023492 thru 0x0322e9 (0xee58 bytes) 000230 1788625624.976103 MIRROR_EXTEND to 2 mirrors 000231 1788625625.789050 mapread 0x008c6d thru 0x0153e3 (0xc777 bytes) 000232 1788625625.828956 trunc from 0x020890 to 0x02416d (0x38de bytes) 000233 1788625625.932682 mapread 0x01e375 thru 0x024ab4 (0x6740 bytes) 000234 1788625625.984647 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x322ea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x322ea): Mirror 2: 0x41901033 (stale) CRC-32 checksum value for chunk [0, 0x322ea): Mirror 1: 0x1513199d Mirror 3: 0x1513199d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x322ea: skipped 000235 1788625627.587225 fallocate 0x03eaa0 thru 0x03ffff (0x1560 bytes) 000236 1788625627.648812 trunc from 0x040000 to 0x02acff (0x15300 bytes) 000237 1788625627.681830 read 0x019271 thru 0x025280 (0xc010 bytes) 000239 1788625627.707985 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ad00): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad00): Mirror 2: 0x69ad97d6 (stale) Mirror 3: 0xa92d1a8a (stale) CRC-32 checksum value for chunk [0, 0x2ad00): Mirror 1: 0xa92d1a8a Mirror 4: 0xa92d1a8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad00: skipped 000240 1788625629.008710 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ad00): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad00): Mirror 2: 0x69ad97d6 (stale) Mirror 3: 0xa92d1a8a (stale) CRC-32 checksum value for chunk [0, 0x2ad00): Mirror 1: 0xa92d1a8a Mirror 4: 0xa92d1a8a Mirror 5: 0xa92d1a8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad00: skipped 000241 1788625630.484532 fallocate 0x02bf50 thru 0x02c8a6 (0x957 bytes) 000242 1788625630.572486 punch from 0x02a149 to 0x02acff (0xbb7 bytes) 000243 1788625630.591888 read 0x0016af thru 0x006f0f (0x5861 bytes) 000244 1788625630.644909 MIRROR_RESYNC 5 mirrors 000245 1788625632.464618 punch from 0x0233e9 to 0x025355 (0x1f6d bytes) 000246 1788625632.523601 trunc from 0x02ad00 to 0x00f726 (0x1b5d9 bytes) 000247 1788625632.575034 trunc from 0x00ba3e to 0x00f726 (0x3ce9 bytes) 000248 1788625632.588382 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf727): Mirror 2: 0x351375c7 (stale) Mirror 4: 0x351375c7 (stale) Mirror 5: 0x351375c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf727): Mirror 2: 0x351375c7 (stale) Mirror 4: 0x351375c7 (stale) Mirror 5: 0x351375c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf727: skipped 000249 1788625632.950606 trunc from 0x00f727 to 0x001ec6 (0xd860 bytes) 000250 1788625632.988210 trunc from 0x001ba0 to 0x001ec6 (0x327 bytes) 000251 1788625632.999873 read 0x001c62 thru 0x001ec6 (0x265 bytes) 000252 1788625633.009904 trunc from 0x001ec7 to 0x03f9a9 (0x3dae3 bytes) 000253 1788625633.026696 trunc from 0x03f9aa to 0x007034 (0x38975 bytes) 000254 1788625633.049977 mapread 0x0045e4 thru 0x007034 (0x2a51 bytes) 000255 1788625633.070221 punch from 0x003d79 to 0x007034 (0x32bc bytes) 000256 1788625633.076734 mapread 0x00352c thru 0x007034 (0x3b09 bytes) 000257 1788625633.084968 trunc from 0x003bfe to 0x007034 (0x3437 bytes) 000258 1788625633.090811 fallocate 0x037d24 thru 0x03ffff (0x82dc bytes) 000259 1788625633.099457 read 0x02e2a6 thru 0x039c22 (0xb97d bytes) 000260 1788625633.110153 trunc from 0x040000 to 0x00b656 (0x349a9 bytes) 000262 1788625633.131221 mapwrite 0x0341f6 thru 0x03c089 (0x7e94 bytes) 000263 1788625633.232479 fallocate 0x015afd thru 0x01aa53 (0x4f57 bytes) 000264 1788625633.238904 fallocate 0x012c69 thru 0x01b5d2 (0x896a bytes) 000265 1788625633.245989 read 0x030f93 thru 0x03c089 (0xb0f7 bytes) 000266 1788625633.259490 read 0x008939 thru 0x011a71 (0x9139 bytes) 000267 1788625633.260062 write 0x0256de thru 0x034adc (0xf3ff bytes) 000268 1788625633.267717 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c08a): Mirror 2: 0x92aa9d9d (stale) Mirror 4: 0x92aa9d9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c08a): Mirror 2: 0x92aa9d9d (stale) Mirror 4: 0x92aa9d9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c08a: skipped 000269 1788625633.777394 trunc from 0x024e2a to 0x02d86a (0x8a41 bytes) 000270 1788625633.794172 mapwrite 0x02f666 thru 0x03ca0b (0xd3a6 bytes) 000271 1788625633.931218 trunc from 0x03ca0c to 0x00e664 (0x2e3a7 bytes) 000272 1788625633.957967 write 0x03170a thru 0x03bf81 (0xa878 bytes) 000273 1788625633.969159 trunc from 0x03bf82 to 0x0385ba (0x39c7 bytes) 000275 1788625633.993807 fallocate 0x0140d5 thru 0x014110 (0x3c bytes) 000276 1788625634.000864 punch from 0x03340d to 0x0385ba (0x51ae bytes) 000277 1788625634.019582 fallocate 0x010dc1 thru 0x01c755 (0xb995 bytes) 000278 1788625634.027558 mapwrite 0x03ef1d thru 0x03ffff (0x10e3 bytes) 000279 1788625634.057507 trunc from 0x01e09d to 0x02cb17 (0xea7b bytes) 000280 1788625634.064658 write 0x004e94 thru 0x0142a8 (0xf415 bytes) 000281 1788625634.076195 write 0x019520 thru 0x01f3a5 (0x5e86 bytes) 000282 1788625634.086152 mapread 0x01cbc1 thru 0x01d9e5 (0xe25 bytes) 000283 1788625634.087613 punch from 0x000b4e to 0x004951 (0x3e04 bytes) 000284 1788625634.109594 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbaf94acb (stale) Mirror 4: 0xbaf94acb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5103e019 Mirror 5: 0x5103e019 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000285 1788625635.097769 punch from 0x02cf92 to 0x0397df (0xc84e bytes) 000286 1788625635.150223 read 0x00c52e thru 0x01c2df (0xfdb2 bytes) 000287 1788625635.162756 fallocate 0x03603b thru 0x03ffff (0x9fc5 bytes) 000288 1788625635.168106 mapread 0x02f508 thru 0x02f6d3 (0x1cc bytes) 000289 1788625635.173560 trunc from 0x036520 to 0x03ffff (0x9ae0 bytes) 000292 1788625635.187680 punch from 0x01d85f to 0x020bbb (0x335d bytes) 000293 1788625635.195922 trunc from 0x040000 to 0x03315a (0xcea5 bytes) 000294 1788625635.283688 read 0x00c319 thru 0x0160b0 (0x9d98 bytes) 000295 1788625635.284569 trunc from 0x03315b to 0x017c1c (0x1b53e bytes) 000297 1788625635.308527 trunc from 0x01737e to 0x017c1c (0x89f bytes) 000298 1788625635.314352 trunc from 0x001625 to 0x002949 (0x1325 bytes) 000299 1788625635.320806 read 0x014a29 thru 0x017c1c (0x31f4 bytes) 000300 1788625635.339092 write 0x009757 thru 0x00b2e4 (0x1b8e bytes) 000301 1788625635.340086 punch from 0x00109d to 0x004383 (0x32e7 bytes) 000302 1788625635.344978 trunc from 0x001fd1 to 0x00a07d (0x80ad bytes) 000304 1788625635.365182 mapwrite 0x00dedd thru 0x01685f (0x8983 bytes) 000305 1788625635.400637 punch from 0x0165ce to 0x017c1c (0x164f bytes) 000306 1788625635.426439 mapwrite 0x00b87c thru 0x014ba0 (0x9325 bytes) 000308 1788625635.497672 trunc from 0x001c70 to 0x004c3b (0x2fcc bytes) 000309 1788625635.505397 fallocate 0x00a8c2 thru 0x011f52 (0x7691 bytes) 000310 1788625635.516563 mapread 0x012535 thru 0x017c1c (0x56e8 bytes) 000311 1788625635.542086 trunc from 0x017c1d to 0x030c98 (0x1907c bytes) 000312 1788625635.571650 write 0x0110cc thru 0x014d5e (0x3c93 bytes) 000313 1788625635.572893 mapwrite 0x01d2af thru 0x01e1ff (0xf51 bytes) 000314 1788625635.623844 MIRROR_RESYNC 4 mirrors 000315 1788625637.410181 mapread 0x0136cb thru 0x0170d0 (0x3a06 bytes) 000316 1788625637.426734 write 0x036f87 thru 0x03ba54 (0x4ace bytes) 000317 1788625637.462461 read 0x00c020 thru 0x01a7ae (0xe78f bytes) 000318 1788625637.476589 mapwrite 0x032b06 thru 0x0372c3 (0x47be bytes) 000319 1788625637.507518 fallocate 0x011d92 thru 0x01cf0d (0xb17c bytes) 000320 1788625637.514721 mapwrite 0x0254b0 thru 0x02a358 (0x4ea9 bytes) 000321 1788625637.546164 read 0x035000 thru 0x03ba54 (0x6a55 bytes) 000322 1788625637.547120 fallocate 0x02004b thru 0x025338 (0x52ee bytes) 000323 1788625637.553937 mapread 0x0104a8 thru 0x0108dc (0x435 bytes) 000324 1788625637.554753 trunc from 0x02b47f to 0x038b8d (0xd70f bytes) 000326 1788625637.563118 fallocate 0x00bd1f thru 0x01317f (0x7461 bytes) 000327 1788625637.569561 punch from 0x01abf6 to 0x025307 (0xa712 bytes) 000328 1788625637.576251 read 0x01ed4a thru 0x021bf2 (0x2ea9 bytes) 000329 1788625637.594130 write 0x000734 thru 0x00fb8e (0xf45b bytes) 000330 1788625637.596718 punch from 0x00466e to 0x00dc9a (0x962d bytes) 000331 1788625637.613943 mapwrite 0x01c53c thru 0x029a11 (0xd4d6 bytes) 000332 1788625637.647581 read 0x010428 thru 0x010adb (0x6b4 bytes) 000333 1788625637.648549 write 0x02d164 thru 0x03a4fd (0xd39a bytes) 000334 1788625637.676168 MIRROR_RESYNC 4 mirrors 000335 1788625639.323020 write 0x02d85a thru 0x030094 (0x283b bytes) 000336 1788625639.363522 trunc from 0x03ba55 to 0x002dc6 (0x38c8e bytes) 000337 1788625639.386228 mapread 0x000947 thru 0x002dc6 (0x2480 bytes) 000338 1788625639.400750 fallocate 0x016391 thru 0x020709 (0xa379 bytes) 000339 1788625639.406331 trunc from 0x0036fb to 0x00ff3a (0xc840 bytes) 000341 1788625639.411494 punch from 0x008648 to 0x009650 (0x1009 bytes) 000342 1788625639.415590 fallocate 0x03249e thru 0x037f59 (0x5abc bytes) 000343 1788625639.420658 trunc from 0x037f5a to 0x00aec2 (0x2d097 bytes) 000344 1788625639.435119 mapwrite 0x03f4a0 thru 0x03ffff (0xb60 bytes) 000345 1788625639.475385 mapwrite 0x02a10d thru 0x036cda (0xcbce bytes) 000347 1788625639.522074 read 0x01ff8b thru 0x02329c (0x3312 bytes) 000348 1788625639.535772 MIRROR_RESYNC 4 mirrors 000349 1788625640.528248 write 0x02e023 thru 0x03a5ba (0xc598 bytes) 000350 1788625640.578973 read 0x036132 thru 0x03d454 (0x7323 bytes) 000351 1788625640.583605 fallocate 0x003ed0 thru 0x00cbec (0x8d1d bytes) 000352 1788625640.591270 punch from 0x034e4d to 0x03f8ea (0xaa9e bytes) 000353 1788625640.611701 trunc from 0x02e527 to 0x033541 (0x501b bytes) 000354 1788625640.620398 trunc from 0x0069d2 to 0x00a84a (0x3e79 bytes) 000355 1788625640.637245 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa40d43e6 (stale) Mirror 4: 0xa40d43e6 (stale) Mirror 5: 0xa40d43e6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc7075452 Mirror 6: 0xc7075452 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000356 1788625642.051703 read 0x032e8f thru 0x03706a (0x41dc bytes) 000357 1788625642.070928 punch from 0x003d3c to 0x004706 (0x9cb bytes) 000358 1788625642.129856 trunc from 0x040000 to 0x01a748 (0x258b7 bytes) 000360 1788625642.157538 write 0x02def0 thru 0x03ab59 (0xcc6a bytes) 000361 1788625642.159855 mapread 0x002f7d thru 0x0120db (0xf15f bytes) 000362 1788625642.192439 mapread 0x018245 thru 0x021f18 (0x9cd4 bytes) 000363 1788625642.200065 read 0x00ac24 thru 0x015dd8 (0xb1b5 bytes) 000364 1788625642.203799 MIRROR_RESYNC 5 mirrors 000365 1788625643.538807 trunc from 0x03ab5a to 0x01bf3c (0x1ec1d bytes) 000366 1788625643.631655 read 0x013839 thru 0x01bf3c (0x8704 bytes) 000367 1788625643.646842 punch from 0x00970e to 0x00b9b2 (0x22a5 bytes) 000368 1788625643.655812 mapwrite 0x00fd49 thru 0x014db2 (0x506a bytes) 000369 1788625643.711081 write 0x00a3db thru 0x013986 (0x95ac bytes) 000370 1788625643.723988 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bf3d): Mirror 2: 0x52df1315 (stale) Mirror 4: 0x52df1315 (stale) Mirror 5: 0x52df1315 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bf3d): Mirror 2: 0x52df1315 (stale) Mirror 4: 0x52df1315 (stale) Mirror 5: 0x52df1315 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bf3d: skipped 000371 1788625644.355515 MIRROR_RESYNC 4 mirrors 000372 1788625645.746354 trunc from 0x008987 to 0x00f5cc (0x6c46 bytes) 000373 1788625645.829385 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bf3d): Mirror 4: 0xc5e213b4 (stale) Mirror 5: 0xc5e213b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bf3d): Mirror 4: 0xc5e213b4 (stale) Mirror 5: 0xc5e213b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bf3d: skipped 000374 1788625646.339400 punch from 0x00fbc1 to 0x01bf3c (0xc37c bytes) 000375 1788625646.352246 MIRROR_RESYNC 3 mirrors 000376 1788625647.417225 trunc from 0x0153c7 to 0x015eac (0xae6 bytes) 000377 1788625647.459513 write 0x038d8a thru 0x03ffff (0x7276 bytes) 000378 1788625647.462437 mapread 0x03e631 thru 0x03ffff (0x19cf bytes) 000379 1788625647.463959 mapread 0x018c8c thru 0x01b696 (0x2a0b bytes) 000380 1788625647.504467 punch from 0x008a90 to 0x00f08b (0x65fc bytes) 000381 1788625647.509749 write 0x0290f3 thru 0x02a966 (0x1874 bytes) 000382 1788625647.524928 fallocate 0x02fb8e thru 0x034be2 (0x5055 bytes) 000383 1788625647.531591 write 0x032259 thru 0x03d80f (0xb5b7 bytes) 000384 1788625647.539644 fallocate 0x02b540 thru 0x02d52f (0x1ff0 bytes) 000385 1788625647.544946 trunc from 0x040000 to 0x00b2a5 (0x34d5a bytes) 000386 1788625647.575268 fallocate 0x0324bf thru 0x0391a9 (0x6ceb bytes) 000387 1788625647.581343 write 0x024e20 thru 0x02f8cb (0xaaac bytes) 000388 1788625647.586657 write 0x034afa thru 0x03ffff (0xb506 bytes) 000389 1788625647.591880 punch from 0x0015d7 to 0x0112ac (0xfcd6 bytes) 000390 1788625647.598484 trunc from 0x040000 to 0x039cf9 (0x6306 bytes) 000391 1788625647.618691 mapwrite 0x0277f0 thru 0x02bdc8 (0x45d9 bytes) 000392 1788625647.655759 mapread 0x022a80 thru 0x02b9b5 (0x8f36 bytes) 000393 1788625647.685224 fallocate 0x019bf7 thru 0x024899 (0xaca3 bytes) 000394 1788625647.693732 trunc from 0x039cfa to 0x024366 (0x15993 bytes) 000395 1788625647.729179 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24367): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24367): Mirror 4: 0xdb3b6590 (stale) Mirror 5: 0xdb3b6590 (stale) CRC-32 checksum value for chunk [0, 0x24367): Mirror 1: 0xd5e558c Mirror 6: 0xd5e558c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24367: skipped 000396 1788625649.002057 fallocate 0x02d9bf thru 0x0346d8 (0x6d1a bytes) 000397 1788625649.058410 trunc from 0x0346d9 to 0x01038a (0x2434e bytes) 000398 1788625649.095660 mapwrite 0x02255c thru 0x028ada (0x657f bytes) 000399 1788625649.161400 punch from 0x021b90 to 0x028ada (0x6f4b bytes) 000400 1788625649.171550 write 0x02c27a thru 0x035765 (0x94ec bytes) 000401 1788625649.178468 fallocate 0x006aa2 thru 0x015dfe (0xf35d bytes) 000402 1788625649.198907 write 0x02c290 thru 0x034a32 (0x87a3 bytes) 000404 1788625649.200539 trunc from 0x035766 to 0x00ce50 (0x28915 bytes) 000405 1788625649.219668 mapread 0x002772 thru 0x00ce50 (0xa6df bytes) 000407 1788625649.247443 mapwrite 0x00265b thru 0x005060 (0x2a06 bytes) 000408 1788625649.289286 MIRROR_RESYNC 4 mirrors 000409 1788625650.639561 read 0x009436 thru 0x00bb1c (0x26e7 bytes) 000410 1788625650.643654 write 0x021b3b thru 0x0246a3 (0x2b69 bytes) 000411 1788625650.679688 write 0x00e27e thru 0x00e4b8 (0x23b bytes) 000412 1788625650.690951 punch from 0x01cb01 to 0x01d18f (0x68f bytes) 000414 1788625650.697813 mapread 0x020ef3 thru 0x0246a3 (0x37b1 bytes) 000415 1788625650.708967 write 0x01d7fa thru 0x0277eb (0x9ff2 bytes) 000416 1788625650.718723 mapread 0x01447e thru 0x01f115 (0xac98 bytes) 000417 1788625650.733151 write 0x031b12 thru 0x03291e (0xe0d bytes) 000418 1788625650.735010 write 0x03d215 thru 0x03ffff (0x2deb bytes) 000420 1788625650.737912 read 0x005870 thru 0x015637 (0xfdc8 bytes) 000421 1788625650.751454 mapread 0x03f0fd thru 0x03ffff (0xf03 bytes) 000422 1788625650.752327 write 0x00d260 thru 0x00fd10 (0x2ab1 bytes) 000423 1788625650.753273 punch from 0x00eddd to 0x00f514 (0x738 bytes) 000425 1788625650.772443 punch from 0x0279cd to 0x02f6c2 (0x7cf6 bytes) 000426 1788625650.778687 trunc from 0x009cb7 to 0x00c382 (0x26cc bytes) 000427 1788625650.792962 punch from 0x026f5d to 0x029836 (0x28da bytes) 000428 1788625650.799221 read 0x037ec2 thru 0x039956 (0x1a95 bytes) 000429 1788625650.800161 write 0x021f95 thru 0x024b77 (0x2be3 bytes) 000430 1788625650.802951 fallocate 0x004018 thru 0x009a9c (0x5a85 bytes) 000431 1788625650.808117 trunc from 0x029d2a to 0x02ea3a (0x4d11 bytes) 000432 1788625650.814195 read 0x014387 thru 0x01caa0 (0x871a bytes) 000433 1788625650.815278 trunc from 0x01c565 to 0x0299fb (0xd497 bytes) 000434 1788625650.827439 fallocate 0x001004 thru 0x008edc (0x7ed9 bytes) 000437 1788625650.843785 mapwrite 0x0118b2 thru 0x0196d4 (0x7e23 bytes) 000438 1788625650.869021 trunc from 0x0359ac to 0x03ffff (0xa654 bytes) 000439 1788625650.875523 write 0x013635 thru 0x017273 (0x3c3f bytes) 000440 1788625650.903650 MIRROR_RESYNC 4 mirrors 000441 1788625652.479870 mapwrite 0x023d50 thru 0x028644 (0x48f5 bytes) 000442 1788625652.594131 mapwrite 0x0284e8 thru 0x02e516 (0x602f bytes) 000443 1788625652.631169 fallocate 0x010e6d thru 0x018763 (0x78f7 bytes) 000444 1788625652.639733 mapwrite 0x0286a2 thru 0x0349d0 (0xc32f bytes) 000445 1788625652.663808 trunc from 0x040000 to 0x028273 (0x17d8c bytes) 000446 1788625652.688830 read 0x009aba thru 0x00bd42 (0x2289 bytes) 000447 1788625652.694589 read 0x003950 thru 0x004f36 (0x15e7 bytes) 000448 1788625652.705197 trunc from 0x00c2e2 to 0x01b94d (0xf66c bytes) 000449 1788625652.712557 read 0x024e3d thru 0x028273 (0x3437 bytes) 000450 1788625652.713713 trunc from 0x028274 to 0x03c753 (0x144e0 bytes) 000451 1788625652.732430 mapwrite 0x01640c thru 0x022caa (0xc89f bytes) 000452 1788625652.784157 mapwrite 0x019a99 thru 0x0209b3 (0x6f1b bytes) 000453 1788625652.806452 write 0x03be13 thru 0x03ffff (0x41ed bytes) 000454 1788625652.811628 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe33f6a91 (stale) Mirror 5: 0xe33f6a91 (stale) Mirror 6: 0xe33f6a91 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x23ed2124 Mirror 7: 0x23ed2124 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1788625653.875329 read 0x016248 thru 0x020ff4 (0xadad bytes) 000457 1788625653.902476 write 0x016b27 thru 0x01a614 (0x3aee bytes) 000458 1788625653.961909 fallocate 0x02c6f0 thru 0x02d22f (0xb40 bytes) 000459 1788625653.965938 fallocate 0x004a4a thru 0x012c85 (0xe23c bytes) 000460 1788625653.971118 fallocate 0x00714b thru 0x0102f7 (0x91ad bytes) 000461 1788625653.979240 write 0x035be2 thru 0x03b98c (0x5dab bytes) 000462 1788625653.987510 write 0x0057f0 thru 0x009842 (0x4053 bytes) 000463 1788625653.994975 trunc from 0x01fd63 to 0x026a0d (0x6cab bytes) 000464 1788625654.002389 fallocate 0x03cdb1 thru 0x03ffff (0x324f bytes) 000465 1788625654.007210 punch from 0x02f1e9 to 0x030b72 (0x198a bytes) 000466 1788625654.012805 mapread 0x026fad thru 0x03698f (0xf9e3 bytes) 000467 1788625654.053931 punch from 0x00e882 to 0x017574 (0x8cf3 bytes) 000468 1788625654.093057 punch from 0x00392e to 0x012a74 (0xf147 bytes) 000469 1788625654.100092 mapwrite 0x00f9b1 thru 0x01b4f9 (0xbb49 bytes) 000470 1788625654.215584 trunc from 0x040000 to 0x03ba15 (0x45ea bytes) 000471 1788625654.253981 punch from 0x009aad to 0x01134c (0x78a0 bytes) 000472 1788625654.264361 trunc from 0x004e14 to 0x0147aa (0xf997 bytes) 000473 1788625654.275343 fallocate 0x0169f2 thru 0x0174dc (0xaeb bytes) 000474 1788625654.282899 fallocate 0x03965b thru 0x03ab2f (0x14d5 bytes) 000475 1788625654.287524 trunc from 0x026d30 to 0x02b865 (0x4b36 bytes) 000476 1788625654.292120 write 0x030bcb thru 0x0326b5 (0x1aeb bytes) 000477 1788625654.293187 read 0x02ae42 thru 0x0361db (0xb39a bytes) 000478 1788625654.300981 mapwrite 0x029c4e thru 0x0330a4 (0x9457 bytes) 000480 1788625654.333113 MIRROR_RESYNC 5 mirrors 000481 1788625656.062840 write 0x008a7e thru 0x00f340 (0x68c3 bytes) 000482 1788625656.107516 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ba16): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba16): Mirror 4: 0x3402d1a0 (stale) Mirror 5: 0x3402d1a0 (stale) Mirror 6: 0x3402d1a0 (stale) Mirror 7: 0x3402d1a0 (stale) CRC-32 checksum value for chunk [0, 0x3ba16): Mirror 1: 0x1f7cc972 Mirror 8: 0x1f7cc972 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba16: skipped 000483 1788625657.349132 punch from 0x001b0e to 0x0083da (0x68cd bytes) 000484 1788625657.412511 fallocate 0x03aa0c thru 0x03ffff (0x55f4 bytes) 000485 1788625657.432641 mapread 0x011fa6 thru 0x0219e0 (0xfa3b bytes) 000486 1788625657.472149 write 0x0338c3 thru 0x03904c (0x578a bytes) 000487 1788625657.486403 fallocate 0x024050 thru 0x0336fb (0xf6ac bytes) 000488 1788625657.496855 mapread 0x01b91c thru 0x01d6e1 (0x1dc6 bytes) 000489 1788625657.498381 read 0x018264 thru 0x01e6db (0x6478 bytes) 000491 1788625657.500942 fallocate 0x036d72 thru 0x03ffff (0x928e bytes) 000492 1788625657.506235 read 0x00beb8 thru 0x00f0c0 (0x3209 bytes) 000493 1788625657.520828 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x219028e4 (stale) Mirror 6: 0x219028e4 (stale) Mirror 7: 0x219028e4 (stale) Mirror 8: 0xa9b389e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x219028e4 (stale) Mirror 6: 0x219028e4 (stale) Mirror 7: 0x219028e4 (stale) Mirror 8: 0xa9b389e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000494 1788625658.068278 write 0x01c47e thru 0x0235e6 (0x7169 bytes) 000495 1788625658.097126 read 0x01e281 thru 0x02d965 (0xf6e5 bytes) 000496 1788625658.128907 mapread 0x01c0a8 thru 0x02be48 (0xfda1 bytes) 000497 1788625658.132124 write 0x0300e5 thru 0x039f71 (0x9e8d bytes) 000498 1788625658.165998 write 0x021b24 thru 0x02413d (0x261a bytes) 000499 1788625658.171090 trunc from 0x040000 to 0x01853e (0x27ac1 bytes) 000500 1788625658.212868 trunc from 0x01853f to 0x00337b (0x151c3 bytes) 000501 1788625658.241943 write 0x036bab thru 0x03ffff (0x9455 bytes) 000502 1788625658.247231 fallocate 0x027eca thru 0x033c2c (0xbd63 bytes) 000503 1788625658.256547 trunc from 0x03c5c1 to 0x03ffff (0x3a3f bytes) 000504 1788625658.272008 read 0x038a3d thru 0x03ffff (0x75c3 bytes) 000505 1788625658.284379 mapwrite 0x01ff5c thru 0x02e9d1 (0xea76 bytes) 000506 1788625658.312240 mapwrite 0x0374b6 thru 0x03e9de (0x7529 bytes) 000507 1788625658.332645 fallocate 0x03c97d thru 0x03ffff (0x3683 bytes) 000508 1788625658.339083 mapwrite 0x038127 thru 0x03c999 (0x4873 bytes) 000509 1788625658.356261 fallocate 0x022032 thru 0x028061 (0x6030 bytes) 000510 1788625658.362572 mapwrite 0x00bead thru 0x01bd55 (0xfea9 bytes) 000511 1788625658.385326 punch from 0x01efff to 0x02e3b6 (0xf3b8 bytes) 000512 1788625658.399212 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x219028e4 (stale) Mirror 6: 0x219028e4 (stale) Mirror 8: 0xa9b389e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x219028e4 (stale) Mirror 6: 0x219028e4 (stale) Mirror 8: 0xa9b389e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000513 1788625658.740689 write 0x03f0f2 thru 0x03ffff (0xf0e bytes) 000514 1788625658.752327 write 0x030c0a thru 0x03c4eb (0xb8e2 bytes) 000515 1788625658.766915 punch from 0x0216fb to 0x022a39 (0x133f bytes) 000516 1788625658.772415 trunc from 0x040000 to 0x039e27 (0x61d8 bytes) 000517 1788625658.792486 trunc from 0x03050b to 0x03324d (0x2d43 bytes) 000518 1788625658.819508 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e28): Mirror 6: 0xc150a343 (stale) Mirror 8: 0xfe407f44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e28): Mirror 6: 0xc150a343 (stale) Mirror 8: 0xfe407f44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e28: skipped 000520 1788625659.289080 fallocate 0x031e08 thru 0x03ac04 (0x8dfd bytes) 000521 1788625659.311615 MIRROR_RESYNC 3 mirrors 000522 1788625660.735399 trunc from 0x02784b to 0x02a996 (0x314c bytes) 000523 1788625660.838441 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39e28): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e28): Mirror 6: 0x928c60ca (stale) Mirror 8: 0x928c60ca (stale) CRC-32 checksum value for chunk [0, 0x39e28): Mirror 1: 0x928c60ca Mirror 9: 0x928c60ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e28: skipped 000524 1788625662.606753 mapread 0x032189 thru 0x039e27 (0x7c9f bytes) 000525 1788625662.683409 write 0x02dcb8 thru 0x03c2f6 (0xe63f bytes) 000526 1788625662.778835 MIRROR_RESYNC 4 mirrors 000527 1788625664.267498 mapread 0x02b6ec thru 0x02c64c (0xf61 bytes) 000528 1788625664.286772 fallocate 0x02ee84 thru 0x037506 (0x8683 bytes) 000529 1788625664.323228 mapread 0x02efa0 thru 0x036e88 (0x7ee9 bytes) 000530 1788625664.334609 mapwrite 0x019fbb thru 0x02319a (0x91e0 bytes) 000531 1788625664.383070 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c2f7): Mirror 6: 0x6237cfe4 (stale) Mirror 9: 0x6237cfe4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2f7): Mirror 6: 0x6237cfe4 (stale) Mirror 9: 0x6237cfe4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2f7: skipped 000532 1788625664.799067 read 0x015eb2 thru 0x01f925 (0x9a74 bytes) 000534 1788625664.827034 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c2f7): Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2f7): Mirror 6: 0x6237cfe4 (stale) Mirror 9: 0x6237cfe4 (stale) CRC-32 checksum value for chunk [0, 0x3c2f7): Mirror 1: 0x4c3e2d53 Mirror 10: 0x4c3e2d53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2f7: skipped 000535 1788625665.884507 write 0x0250b8 thru 0x02cf8d (0x7ed6 bytes) 000536 1788625665.945540 write 0x00b4ea thru 0x01a50b (0xf022 bytes) 000537 1788625665.954051 mapwrite 0x03af90 thru 0x03ffff (0x5070 bytes) 000539 1788625666.051804 write 0x016d84 thru 0x01f7c2 (0x8a3f bytes) 000540 1788625666.065417 trunc from 0x040000 to 0x0147ef (0x2b810 bytes) 000541 1788625666.112840 read 0x000225 thru 0x00c3e4 (0xc1c0 bytes) 000542 1788625666.150720 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x147f0): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x147f0): Mirror 6: 0x8977ade6 (stale) Mirror 9: 0x8977ade6 (stale) Mirror 10: 0x8977ade6 (stale) CRC-32 checksum value for chunk [0, 0x147f0): Mirror 1: 0x73573232 Mirror 11: 0x73573232 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x147f0: skipped 000543 1788625667.387055 trunc from 0x000a08 to 0x00a674 (0x9c6d bytes) 000544 1788625667.437426 trunc from 0x0147f0 to 0x00be44 (0x89ab bytes) 000545 1788625667.497640 mapwrite 0x000783 thru 0x00c14b (0xb9c9 bytes) 000546 1788625667.564615 write 0x01570b thru 0x016cf5 (0x15eb bytes) 000547 1788625667.565839 punch from 0x001cb5 to 0x00efa2 (0xd2ee bytes) 000548 1788625667.573282 fallocate 0x00b8c3 thru 0x00f352 (0x3a90 bytes) 000549 1788625667.579587 write 0x000dca thru 0x0070ae (0x62e5 bytes) 000550 1788625667.586329 punch from 0x00b3de to 0x016cf5 (0xb918 bytes) 000551 1788625667.604285 mapwrite 0x03b8e3 thru 0x03ffff (0x471d bytes) 000552 1788625667.662814 punch from 0x027582 to 0x03591d (0xe39c bytes) 000553 1788625667.675944 MIRROR_RESYNC 5 mirrors 000554 1788625669.293945 MIRROR_EXTEND to 6 mirrors 000555 1788625670.522352 write 0x02a3aa thru 0x03181d (0x7474 bytes) 000556 1788625670.593193 write 0x02bf97 thru 0x02fa4d (0x3ab7 bytes) 000557 1788625670.594989 mapread 0x039efc thru 0x03ffff (0x6104 bytes) 000558 1788625670.616999 write 0x00aabd thru 0x00c4f2 (0x1a36 bytes) 000560 1788625670.644947 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb70e401 (stale) Mirror 9: 0xb70e401 (stale) Mirror 10: 0xb70e401 (stale) Mirror 11: 0xb70e401 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb70e401 (stale) Mirror 9: 0xb70e401 (stale) Mirror 10: 0xb70e401 (stale) Mirror 11: 0xb70e401 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000561 1788625671.280444 write 0x032dfb thru 0x03e5dd (0xb7e3 bytes) 000562 1788625671.305818 mapwrite 0x00557a thru 0x008c05 (0x368c bytes) 000563 1788625671.388207 mapread 0x01b169 thru 0x02003c (0x4ed4 bytes) 000564 1788625671.415252 mapread 0x0179df thru 0x01c8e7 (0x4f09 bytes) 000565 1788625671.421231 mapwrite 0x00a3df thru 0x014d0b (0xa92d bytes) 000566 1788625671.473653 MIRROR_RESYNC 5 mirrors 000568 1788625673.175444 trunc from 0x040000 to 0x013686 (0x2c979 bytes) 000569 1788625673.228654 read 0x00f440 thru 0x013686 (0x4247 bytes) 000570 1788625673.257531 mapwrite 0x00cdfc thru 0x011476 (0x467b bytes) 000571 1788625673.316586 trunc from 0x00e714 to 0x013686 (0x4f73 bytes) 000572 1788625673.329780 read 0x00d523 thru 0x013686 (0x6164 bytes) 000573 1788625673.357738 write 0x012189 thru 0x018d08 (0x6b80 bytes) 000574 1788625673.359626 trunc from 0x00e714 to 0x018a69 (0xa356 bytes) 000575 1788625673.388439 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x18d09): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18d09): Mirror 6: 0x2486f640 (stale) Mirror 9: 0x2486f640 (stale) Mirror 10: 0x2486f640 (stale) Mirror 11: 0x2486f640 (stale) CRC-32 checksum value for chunk [0, 0x18d09): Mirror 1: 0xe12a261f Mirror 12: 0xe12a261f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18d09: skipped 000576 1788625674.372752 mapread 0x009e8b thru 0x00b4f8 (0x166e bytes) 000577 1788625674.426636 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x18d09): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18d09): Mirror 6: 0x2486f640 (stale) Mirror 9: 0x2486f640 (stale) Mirror 10: 0x2486f640 (stale) Mirror 11: 0x2486f640 (stale) CRC-32 checksum value for chunk [0, 0x18d09): Mirror 1: 0xe12a261f Mirror 12: 0xe12a261f Mirror 13: 0xe12a261f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18d09: skipped 000578 1788625675.926987 trunc from 0x018d09 to 0x00eb73 (0xa195 bytes) 000579 1788625676.044030 mapwrite 0x03a319 thru 0x03c1d2 (0x1eba bytes) 000580 1788625676.140174 fallocate 0x038b23 thru 0x03ffff (0x74dd bytes) 000581 1788625676.150451 fallocate 0x01e6ca thru 0x025291 (0x6bc8 bytes) 000582 1788625676.161088 trunc from 0x03c1d3 to 0x02d55e (0xec74 bytes) 000583 1788625676.188789 read 0x019ae2 thru 0x020e6f (0x738e bytes) 000584 1788625676.202392 read 0x027e9f thru 0x02d55e (0x56c0 bytes) 000585 1788625676.214866 punch from 0x026901 to 0x02d55e (0x6c5e bytes) 000586 1788625676.220742 read 0x026581 thru 0x02d55e (0x6fde bytes) 000587 1788625676.241782 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2d55f): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d55f): Mirror 6: 0xa5670b83 (stale) Mirror 9: 0xa5670b83 (stale) Mirror 10: 0xa5670b83 (stale) Mirror 11: 0xa5670b83 (stale) Mirror 12: 0x8ee2a7b0 (stale) Mirror 13: 0x8ee2a7b0 (stale) CRC-32 checksum value for chunk [0, 0x2d55f): Mirror 1: 0xc6a8b3b0 Mirror 14: 0xc6a8b3b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d55f: skipped 000588 1788625677.375837 punch from 0x011532 to 0x014ea2 (0x3971 bytes) 000589 1788625677.440459 trunc from 0x02d55f to 0x00fde5 (0x1d779 bytes) 000590 1788625677.503368 read 0x009a9c thru 0x00fde5 (0x634a bytes) 000591 1788625677.515760 trunc from 0x00323f to 0x007b1f (0x48e1 bytes) 000592 1788625677.521888 mapread 0x007b3c thru 0x00fde5 (0x82aa bytes) 000593 1788625677.534171 trunc from 0x002be7 to 0x00b1fc (0x8616 bytes) 000595 1788625677.544717 MIRROR_RESYNC 8 mirrors 000596 1788625679.230204 write 0x031fe0 thru 0x03c523 (0xa544 bytes) 000597 1788625679.266539 mapread 0x0169d7 thru 0x0188fc (0x1f26 bytes) 000598 1788625679.285055 mapread 0x02f1bb thru 0x03ab78 (0xb9be bytes) 000599 1788625679.293953 mapwrite 0x03e9b6 thru 0x03f630 (0xc7b bytes) 000601 1788625679.336617 fallocate 0x01918e thru 0x0254f6 (0xc369 bytes) 000602 1788625679.342720 trunc from 0x029c64 to 0x02cc6a (0x3007 bytes) 000603 1788625679.348870 read 0x027645 thru 0x02e905 (0x72c1 bytes) 000604 1788625679.358695 fallocate 0x0113eb thru 0x0212e9 (0xfeff bytes) 000605 1788625679.364395 punch from 0x00a53b to 0x013ad0 (0x9596 bytes) 000606 1788625679.372685 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f631): Mirror 9: 0xd5113444 (stale) Mirror 10: 0xd5113444 (stale) Mirror 11: 0xd5113444 (stale) Mirror 12: 0xd5113444 (stale) Mirror 13: 0xd5113444 (stale) Mirror 14: 0xd5113444 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f631): Mirror 9: 0xd5113444 (stale) Mirror 10: 0xd5113444 (stale) Mirror 11: 0xd5113444 (stale) Mirror 12: 0xd5113444 (stale) Mirror 13: 0xd5113444 (stale) Mirror 14: 0xd5113444 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f631: skipped 000608 1788625679.812361 mapread 0x0351c4 thru 0x03d8b8 (0x86f5 bytes) 000609 1788625679.833054 fallocate 0x013c3c thru 0x01c9dc (0x8da1 bytes) 000610 1788625679.839141 fallocate 0x01d2bb thru 0x02b929 (0xe66f bytes) 000611 1788625679.844918 trunc from 0x03f631 to 0x01c069 (0x235c7 bytes) 000612 1788625679.876584 mapwrite 0x016022 thru 0x02349b (0xd47a bytes) 000613 1788625679.935446 trunc from 0x0101ff to 0x012f07 (0x2d09 bytes) 000614 1788625679.941743 mapwrite 0x00b61c thru 0x017472 (0xbe57 bytes) 000615 1788625679.976560 read 0x010ad3 thru 0x02046a (0xf998 bytes) 000616 1788625679.977372 punch from 0x005c5c to 0x006d6c (0x1111 bytes) 000618 1788625679.986948 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2349c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2349c): Mirror 9: 0x2719f369 (stale) Mirror 10: 0x2719f369 (stale) Mirror 11: 0x2719f369 (stale) Mirror 12: 0x2719f369 (stale) Mirror 13: 0x2719f369 (stale) Mirror 14: 0x2719f369 (stale) CRC-32 checksum value for chunk [0, 0x2349c): Mirror 1: 0x236c67d9 Mirror 15: 0x236c67d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2349c: skipped 000620 1788625681.015858 punch from 0x00df3f to 0x0147cd (0x688f bytes) 000621 1788625681.064843 fallocate 0x00a19f thru 0x011043 (0x6ea5 bytes) 000622 1788625681.071061 read 0x000475 thru 0x009e82 (0x9a0e bytes) 000623 1788625681.079409 mapread 0x01e92c thru 0x02349b (0x4b70 bytes) 000624 1788625681.099079 mapread 0x00e527 thru 0x01cb4c (0xe626 bytes) 000625 1788625681.114902 fallocate 0x01098b thru 0x01d13f (0xc7b5 bytes) 000626 1788625681.121101 mapread 0x019860 thru 0x02349b (0x9c3c bytes) 000627 1788625681.122894 trunc from 0x02349c to 0x025c49 (0x27ae bytes) 000628 1788625681.153036 mapread 0x020795 thru 0x025c49 (0x54b5 bytes) 000629 1788625681.160463 write 0x016a31 thru 0x018fc4 (0x2594 bytes) 000630 1788625681.163064 mapwrite 0x01ce02 thru 0x026ca9 (0x9ea8 bytes) 000631 1788625681.216172 mapwrite 0x02a54a thru 0x035f2e (0xb9e5 bytes) 000632 1788625681.273108 mapread 0x01a4c7 thru 0x027b35 (0xd66f bytes) 000633 1788625681.276342 mapwrite 0x02e1d1 thru 0x035771 (0x75a1 bytes) 000634 1788625681.299892 fallocate 0x00fea3 thru 0x01d149 (0xd2a7 bytes) 000635 1788625681.305759 punch from 0x02916f to 0x0346e6 (0xb578 bytes) 000636 1788625681.313091 trunc from 0x035f2f to 0x033523 (0x2a0b bytes) 000637 1788625681.329411 trunc from 0x004922 to 0x011715 (0xcdf4 bytes) 000638 1788625681.336863 trunc from 0x01ebd5 to 0x027a18 (0x8e44 bytes) 000639 1788625681.344604 fallocate 0x036db8 thru 0x0380c7 (0x1310 bytes) 000640 1788625681.356505 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33524): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33524): Mirror 9: 0x915b03c8 (stale) Mirror 10: 0x915b03c8 (stale) Mirror 11: 0x915b03c8 (stale) Mirror 12: 0x915b03c8 (stale) Mirror 13: 0x915b03c8 (stale) Mirror 14: 0x915b03c8 (stale) Mirror 15: 0xd117e3da (stale) CRC-32 checksum value for chunk [0, 0x33524): Mirror 1: 0x80bb6465 Mirror 16: 0x80bb6465 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33524: skipped 000641 1788625682.853717 write 0x0084fb thru 0x00851a (0x20 bytes) 000642 1788625682.930595 trunc from 0x0066ef to 0x00841f (0x1d31 bytes) 000643 1788625682.957440 punch from 0x00fb47 to 0x01c7c3 (0xcc7d bytes) 000644 1788625682.965624 punch from 0x02be3a to 0x033523 (0x76ea bytes) 000645 1788625682.977887 fallocate 0x019049 thru 0x02230f (0x92c7 bytes) 000646 1788625682.991899 fallocate 0x028a9f thru 0x035c19 (0xd17b bytes) 000647 1788625682.999041 punch from 0x025001 to 0x031729 (0xc729 bytes) 000648 1788625683.008216 write 0x013a5d thru 0x023885 (0xfe29 bytes) 000649 1788625683.051595 MIRROR_RESYNC 9 mirrors 000650 1788625685.618460 trunc from 0x027006 to 0x02ffcb (0x8fc6 bytes) 000651 1788625685.652221 mapread 0x01edeb thru 0x029a76 (0xac8c bytes) 000652 1788625685.682331 fallocate 0x039d8f thru 0x03f084 (0x52f6 bytes) 000653 1788625685.687274 mapread 0x018076 thru 0x01f6b4 (0x763f bytes) 000654 1788625685.696934 punch from 0x031604 to 0x033523 (0x1f20 bytes) 000656 1788625685.710913 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33524): Mirror 9: 0x4851bf81 (stale) Mirror 11: 0x4851bf81 (stale) Mirror 12: 0x4851bf81 (stale) Mirror 13: 0x4851bf81 (stale) Mirror 14: 0x4851bf81 (stale) Mirror 15: 0x4851bf81 (stale) Mirror 16: 0x4851bf81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33524): Mirror 9: 0x4851bf81 (stale) Mirror 11: 0x4851bf81 (stale) Mirror 12: 0x4851bf81 (stale) Mirror 13: 0x4851bf81 (stale) Mirror 14: 0x4851bf81 (stale) Mirror 15: 0x4851bf81 (stale) Mirror 16: 0x4851bf81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33524: skipped 000657 1788625686.206181 trunc from 0x023d8f to 0x03166e (0xd8e0 bytes) 000658 1788625686.211973 fallocate 0x02f30b thru 0x03bd03 (0xc9f9 bytes) 000659 1788625686.217348 read 0x02d115 thru 0x035460 (0x834c bytes) 000660 1788625686.226920 trunc from 0x03bd04 to 0x0255b7 (0x1674c bytes) 000661 1788625686.246117 write 0x03e8a2 thru 0x03ffff (0x175e bytes) 000663 1788625686.247902 trunc from 0x040000 to 0x02173b (0x1e8c4 bytes) 000664 1788625686.263905 trunc from 0x01a4e7 to 0x02173b (0x7255 bytes) 000665 1788625686.268111 trunc from 0x02173c to 0x02ed62 (0xd627 bytes) 000666 1788625686.278745 write 0x020232 thru 0x023e11 (0x3be0 bytes) 000667 1788625686.288842 trunc from 0x02ed63 to 0x02eb9f (0x1c3 bytes) 000671 1788625686.303619 fallocate 0x0148f5 thru 0x020594 (0xbca0 bytes) 000672 1788625686.308015 trunc from 0x02eba0 to 0x00400f (0x2ab90 bytes) 000673 1788625686.322420 punch from 0x00318d to 0x00400f (0xe83 bytes) 000674 1788625686.327408 mapread 0x001b22 thru 0x00400f (0x24ee bytes) 000675 1788625686.348673 mapread 0x003960 thru 0x00400f (0x6b0 bytes) 000676 1788625686.349969 mapwrite 0x00eb3e thru 0x019574 (0xaa37 bytes) 000677 1788625686.420080 fallocate 0x037b21 thru 0x039961 (0x1e41 bytes) 000678 1788625686.434158 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x19575): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19575): Mirror 9: 0x8b151140 (stale) Mirror 11: 0x8b151140 (stale) Mirror 12: 0x8b151140 (stale) Mirror 13: 0x8b151140 (stale) Mirror 14: 0x8b151140 (stale) Mirror 15: 0x8b151140 (stale) Mirror 16: 0x8b151140 (stale) CRC-32 checksum value for chunk [0, 0x19575): Mirror 1: 0x7d00f042 Mirror 17: 0x7d00f042 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19575: skipped 000680 1788625687.440243 mapread 0x00e0c5 thru 0x019574 (0xb4b0 bytes) 000681 1788625687.492772 mapread 0x00f0df thru 0x00fa2a (0x94c bytes) 000682 1788625687.509097 mapread 0x012562 thru 0x015f94 (0x3a33 bytes) 000683 1788625687.526154 read 0x00ce56 thru 0x01769e (0xa849 bytes) 000684 1788625687.535194 mapread 0x0182d8 thru 0x019574 (0x129d bytes) 000685 1788625687.554458 fallocate 0x023a1a thru 0x02473d (0xd24 bytes) 000686 1788625687.601967 trunc from 0x019575 to 0x017434 (0x2140 bytes) 000687 1788625687.655949 write 0x031706 thru 0x03bfbe (0xa8b9 bytes) 000688 1788625687.659162 write 0x00caab thru 0x00cf9a (0x4f0 bytes) 000689 1788625687.667026 read 0x02ff90 thru 0x03bfbe (0xc02f bytes) 000690 1788625687.680437 read 0x019bed thru 0x022024 (0x8438 bytes) 000691 1788625687.686437 MIRROR_RESYNC 9 mirrors 000692 1788625690.155834 fallocate 0x0353f1 thru 0x03b9c5 (0x65d5 bytes) 000693 1788625690.211994 punch from 0x01c22a to 0x0227e3 (0x65ba bytes) 000694 1788625690.226149 mapread 0x027876 thru 0x0281e8 (0x973 bytes) 000695 1788625690.269557 read 0x010f1e thru 0x014d09 (0x3dec bytes) 000696 1788625690.299798 read 0x018b04 thru 0x01df44 (0x5441 bytes) 000697 1788625690.323900 read 0x02a2cf thru 0x03923e (0xef70 bytes) 000698 1788625690.325868 fallocate 0x02d406 thru 0x031850 (0x444b bytes) 000699 1788625690.340828 mapread 0x032c40 thru 0x03bfbe (0x937f bytes) 000700 1788625690.346860 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3bfbf): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfbf): Mirror 9: 0x4c97f88d (stale) Mirror 11: 0x4c97f88d (stale) Mirror 12: 0x4c97f88d (stale) Mirror 13: 0x4c97f88d (stale) Mirror 14: 0x4c97f88d (stale) Mirror 15: 0x4c97f88d (stale) Mirror 16: 0x4c97f88d (stale) Mirror 17: 0x4c97f88d (stale) CRC-32 checksum value for chunk [0, 0x3bfbf): Mirror 1: 0x4c97f88d Mirror 18: 0x4c97f88d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfbf: skipped 000701 1788625691.592887 fallocate 0x01379d thru 0x01ee7f (0xb6e3 bytes) 000702 1788625691.668364 read 0x031261 thru 0x036966 (0x5706 bytes) 000703 1788625691.689554 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfbf): Mirror 9: 0x4c97f88d (stale) Mirror 11: 0x4c97f88d (stale) Mirror 12: 0x4c97f88d (stale) Mirror 14: 0x4c97f88d (stale) Mirror 15: 0x4c97f88d (stale) Mirror 16: 0x4c97f88d (stale) Mirror 17: 0x4c97f88d (stale) Mirror 18: 0x4c97f88d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfbf): Mirror 9: 0x4c97f88d (stale) Mirror 11: 0x4c97f88d (stale) Mirror 12: 0x4c97f88d (stale) Mirror 14: 0x4c97f88d (stale) Mirror 15: 0x4c97f88d (stale) Mirror 16: 0x4c97f88d (stale) Mirror 17: 0x4c97f88d (stale) Mirror 18: 0x4c97f88d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfbf: skipped 000704 1788625692.423955 trunc from 0x03bfbf to 0x00a48c (0x31b32 bytes) 000705 1788625692.463192 trunc from 0x00a48d to 0x005af1 (0x499b bytes) 000706 1788625692.498164 fallocate 0x028a08 thru 0x032bc5 (0xa1be bytes) 000707 1788625692.507856 fallocate 0x030c16 thru 0x037fcd (0x73b8 bytes) 000708 1788625692.517224 write 0x00390a thru 0x007049 (0x3740 bytes) 000709 1788625692.536130 write 0x02e3d2 thru 0x03c835 (0xe464 bytes) 000710 1788625692.558231 write 0x0357a1 thru 0x03f16d (0x99cd bytes) 000711 1788625692.561236 read 0x00530a thru 0x008e7f (0x3b76 bytes) 000712 1788625692.571047 trunc from 0x03f16e to 0x01223c (0x2cf31 bytes) 000713 1788625692.591376 punch from 0x006e71 to 0x01223c (0xb3cc bytes) 000714 1788625692.613983 write 0x03f82e thru 0x03ffff (0x7d2 bytes) 000715 1788625692.628228 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x31673b98 (stale) Mirror 11: 0x31673b98 (stale) Mirror 12: 0x31673b98 (stale) Mirror 14: 0x31673b98 (stale) Mirror 15: 0x31673b98 (stale) Mirror 16: 0x31673b98 (stale) Mirror 17: 0x31673b98 (stale) Mirror 18: 0x31673b98 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9c79be3a Mirror 19: 0x9c79be3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000716 1788625694.200181 punch from 0x03d26e to 0x03ffff (0x2d92 bytes) 000717 1788625694.253032 trunc from 0x00b838 to 0x00d017 (0x17e0 bytes) 000718 1788625694.259671 mapread 0x017e7e thru 0x01f926 (0x7aa9 bytes) 000719 1788625694.295774 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x31673b98 (stale) Mirror 11: 0x31673b98 (stale) Mirror 12: 0x31673b98 (stale) Mirror 14: 0x31673b98 (stale) Mirror 15: 0x31673b98 (stale) Mirror 16: 0x31673b98 (stale) Mirror 17: 0x31673b98 (stale) Mirror 18: 0x31673b98 (stale) Mirror 19: 0x9c79be3a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8d2da092 Mirror 20: 0x8d2da092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000721 1788625695.488794 mapread 0x016a1e thru 0x022aa2 (0xc085 bytes) 000722 1788625695.564562 mapread 0x02f80a thru 0x0307f7 (0xfee bytes) 000723 1788625695.593103 write 0x01124a thru 0x01bd83 (0xab3a bytes) 000724 1788625695.653161 trunc from 0x0279c0 to 0x02f29b (0x78dc bytes) 000725 1788625695.658945 fallocate 0x01d7e3 thru 0x027ede (0xa6fc bytes) 000726 1788625695.668555 MIRROR_RESYNC 11 mirrors 000727 1788625697.744972 read 0x011e74 thru 0x01acd6 (0x8e63 bytes) 000728 1788625697.754717 write 0x0209fb thru 0x024425 (0x3a2b bytes) 000729 1788625697.803762 write 0x0373b9 thru 0x03f9b7 (0x85ff bytes) 000730 1788625697.821000 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9681ae9d (stale) Mirror 11: 0x9681ae9d (stale) Mirror 12: 0x9681ae9d (stale) Mirror 14: 0x9681ae9d (stale) Mirror 15: 0x9681ae9d (stale) Mirror 16: 0x9681ae9d (stale) Mirror 17: 0x9681ae9d (stale) Mirror 18: 0x9681ae9d (stale) Mirror 19: 0x9681ae9d (stale) Mirror 20: 0x9681ae9d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59688e2 Mirror 21: 0x59688e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000732 1788625699.244877 mapread 0x00249b thru 0x004fd7 (0x2b3d bytes) 000733 1788625699.266475 mapread 0x006ae6 thru 0x00727c (0x797 bytes) 000734 1788625699.291473 read 0x026500 thru 0x02c761 (0x6262 bytes) 000735 1788625699.324395 fallocate 0x00df2b thru 0x01a611 (0xc6e7 bytes) 000736 1788625699.405890 trunc from 0x039d69 to 0x03ffff (0x6297 bytes) 000737 1788625699.413555 punch from 0x014d1f to 0x024396 (0xf678 bytes) 000738 1788625699.422044 trunc from 0x040000 to 0x02f820 (0x107df bytes) 000739 1788625699.459184 punch from 0x02cdf3 to 0x02f820 (0x2a2e bytes) 000740 1788625699.464112 mapwrite 0x00c49b thru 0x00c7dd (0x343 bytes) 000741 1788625699.486221 trunc from 0x02f821 to 0x021267 (0xe5b9 bytes) 000742 1788625699.502099 read 0x0011f7 thru 0x010577 (0xf381 bytes) 000743 1788625699.510708 trunc from 0x0207d0 to 0x021267 (0xa98 bytes) 000744 1788625699.514403 mapwrite 0x03a2f8 thru 0x03ffff (0x5d08 bytes) 000745 1788625699.575772 punch from 0x02c78b to 0x03c288 (0xfafe bytes) 000747 1788625699.584891 trunc from 0x040000 to 0x029679 (0x16986 bytes) 000748 1788625699.612917 trunc from 0x01172e to 0x01cee4 (0xb7b7 bytes) 000749 1788625699.622905 write 0x022c2c thru 0x02589c (0x2c71 bytes) 000750 1788625699.633511 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2967a): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2967a): Mirror 9: 0xc1bc756d (stale) Mirror 11: 0xc1bc756d (stale) Mirror 12: 0xc1bc756d (stale) Mirror 14: 0xc1bc756d (stale) Mirror 15: 0xc1bc756d (stale) Mirror 16: 0xc1bc756d (stale) Mirror 17: 0xc1bc756d (stale) Mirror 18: 0xc1bc756d (stale) Mirror 19: 0xc1bc756d (stale) Mirror 20: 0xc1bc756d (stale) Mirror 21: 0x71d93963 (stale) CRC-32 checksum value for chunk [0, 0x2967a): Mirror 1: 0xea07a15f Mirror 22: 0xea07a15f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2967a: skipped 000751 1788625701.186982 write 0x00a784 thru 0x00ac80 (0x4fd bytes) 000752 1788625701.266123 punch from 0x01575d to 0x01da34 (0x82d8 bytes) 000753 1788625701.276884 trunc from 0x02967a to 0x008751 (0x20f28 bytes) 000754 1788625701.315019 trunc from 0x006b07 to 0x008751 (0x1c4b bytes) 000756 1788625701.337320 mapread 0x00695c thru 0x008751 (0x1df6 bytes) 000757 1788625701.362724 trunc from 0x0066d7 to 0x008751 (0x207b bytes) 000758 1788625701.368886 read 0x006007 thru 0x008751 (0x274b bytes) 000759 1788625701.376189 fallocate 0x0247cb thru 0x02e658 (0x9e8e bytes) 000760 1788625701.383283 read 0x0200ba thru 0x0265a3 (0x64ea bytes) 000761 1788625701.408776 punch from 0x02c149 to 0x02e658 (0x2510 bytes) 000762 1788625701.423014 mapread 0x002f8f thru 0x00cc28 (0x9c9a bytes) 000763 1788625701.424574 punch from 0x00c2ab to 0x00cbb1 (0x907 bytes) 000764 1788625701.433263 write 0x012f86 thru 0x014fa6 (0x2021 bytes) 000765 1788625701.434820 write 0x00efec thru 0x010bce (0x1be3 bytes) 000766 1788625701.437654 mapwrite 0x032326 thru 0x03aff8 (0x8cd3 bytes) 000767 1788625701.488927 mapread 0x01ebb1 thru 0x02e233 (0xf683 bytes) 000768 1788625701.505224 punch from 0x0100e2 to 0x01d051 (0xcf70 bytes) 000769 1788625701.511200 read 0x01582b thru 0x01e16c (0x8942 bytes) 000771 1788625701.529792 fallocate 0x018977 thru 0x02407d (0xb707 bytes) 000772 1788625701.535267 trunc from 0x03aff9 to 0x0135b7 (0x27a41 bytes) 000773 1788625701.576985 mapread 0x013598 thru 0x0135b7 (0x20 bytes) 000774 1788625701.578222 mapwrite 0x0282d1 thru 0x02ff0f (0x7c3f bytes) 000775 1788625701.666873 mapwrite 0x02f6cb thru 0x03f69e (0xffd4 bytes) 000776 1788625701.818820 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f69f): Mirror 9: 0xbe63360e (stale) Mirror 11: 0xbe63360e (stale) Mirror 12: 0xbe63360e (stale) Mirror 14: 0xbe63360e (stale) Mirror 15: 0xbe63360e (stale) Mirror 16: 0xbe63360e (stale) Mirror 17: 0xbe63360e (stale) Mirror 18: 0xbe63360e (stale) Mirror 19: 0xbe63360e (stale) Mirror 20: 0xbe63360e (stale) Mirror 21: 0x8f7b1c77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f69f): Mirror 9: 0xbe63360e (stale) Mirror 11: 0xbe63360e (stale) Mirror 12: 0xbe63360e (stale) Mirror 14: 0xbe63360e (stale) Mirror 15: 0xbe63360e (stale) Mirror 16: 0xbe63360e (stale) Mirror 17: 0xbe63360e (stale) Mirror 18: 0xbe63360e (stale) Mirror 19: 0xbe63360e (stale) Mirror 20: 0xbe63360e (stale) Mirror 21: 0x8f7b1c77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f69f: skipped 000777 1788625702.721498 mapread 0x0332a0 thru 0x03545d (0x21be bytes) 000779 1788625702.755662 mapwrite 0x0330a6 thru 0x03ffff (0xcf5a bytes) 000780 1788625702.836981 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9681ae9d (stale) Mirror 11: 0x9681ae9d (stale) Mirror 12: 0x9681ae9d (stale) Mirror 14: 0x9681ae9d (stale) Mirror 15: 0x9681ae9d (stale) Mirror 16: 0x9681ae9d (stale) Mirror 17: 0x9681ae9d (stale) Mirror 18: 0x9681ae9d (stale) Mirror 19: 0x9681ae9d (stale) Mirror 20: 0x9681ae9d (stale) Mirror 21: 0x59688e2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x115b05 Mirror 22: 0x115b05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1788625704.282793 punch from 0x027348 to 0x034e49 (0xdb02 bytes) 000782 1788625704.341723 trunc from 0x03afff to 0x03ffff (0x5001 bytes) 000783 1788625704.352057 MIRROR_RESYNC 13 mirrors 000784 1788625707.651276 mapwrite 0x038ccf thru 0x03ffff (0x7331 bytes) 000786 1788625707.777325 trunc from 0x004040 to 0x00554f (0x1510 bytes) 000787 1788625707.784788 trunc from 0x040000 to 0x002016 (0x3dfe9 bytes) 000788 1788625707.806702 mapread 0x00001d thru 0x002016 (0x1ffa bytes) 000789 1788625707.813640 fallocate 0x019e5f thru 0x022c5a (0x8dfc bytes) 000790 1788625707.821916 punch from 0x018d1d to 0x02153b (0x881f bytes) 000791 1788625707.829114 trunc from 0x022c5b to 0x00eab3 (0x141a7 bytes) 000792 1788625707.858989 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0xeab4): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeab4): Mirror 9: 0xa6d69b43 (stale) Mirror 11: 0xa6d69b43 (stale) Mirror 12: 0xa6d69b43 (stale) Mirror 14: 0xa6d69b43 (stale) Mirror 15: 0xa6d69b43 (stale) Mirror 16: 0xa6d69b43 (stale) Mirror 17: 0xa6d69b43 (stale) Mirror 18: 0xa6d69b43 (stale) Mirror 19: 0xa6d69b43 (stale) Mirror 20: 0xa6d69b43 (stale) Mirror 21: 0xa6d69b43 (stale) Mirror 22: 0xa6d69b43 (stale) CRC-32 checksum value for chunk [0, 0xeab4): Mirror 1: 0x2af3b047 Mirror 23: 0x2af3b047 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeab4: skipped 000793 1788625709.596750 MIRROR_RESYNC 14 mirrors 000794 1788625712.192365 trunc from 0x00ab19 to 0x00eab3 (0x3f9b bytes) 000795 1788625712.270904 trunc from 0x00d8e2 to 0x00eab3 (0x11d2 bytes) 000796 1788625712.276576 mapwrite 0x030e21 thru 0x03e7a6 (0xd986 bytes) 000797 1788625712.350110 write 0x03a39e thru 0x03ffff (0x5c62 bytes) 000799 1788625712.357853 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x249e321e (stale) Mirror 11: 0x249e321e (stale) Mirror 12: 0x249e321e (stale) Mirror 14: 0x249e321e (stale) Mirror 15: 0x249e321e (stale) Mirror 16: 0x249e321e (stale) Mirror 17: 0x249e321e (stale) Mirror 19: 0x249e321e (stale) Mirror 20: 0x249e321e (stale) Mirror 21: 0x249e321e (stale) Mirror 22: 0x249e321e (stale) Mirror 23: 0x249e321e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x249e321e (stale) Mirror 11: 0x249e321e (stale) Mirror 12: 0x249e321e (stale) Mirror 14: 0x249e321e (stale) Mirror 15: 0x249e321e (stale) Mirror 16: 0x249e321e (stale) Mirror 17: 0x249e321e (stale) Mirror 19: 0x249e321e (stale) Mirror 20: 0x249e321e (stale) Mirror 21: 0x249e321e (stale) Mirror 22: 0x249e321e (stale) Mirror 23: 0x249e321e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000800 1788625713.045057 trunc from 0x040000 to 0x0369f2 (0x960d bytes) 000802 1788625713.085736 mapread 0x02112c thru 0x025705 (0x45da bytes) 000803 1788625713.114076 punch from 0x022ed9 to 0x0237d9 (0x901 bytes) 000804 1788625713.119857 read 0x02e6d2 thru 0x0369f2 (0x8321 bytes) 000805 1788625713.131686 read 0x00c19f thru 0x00d4af (0x1311 bytes) 000807 1788625713.147256 punch from 0x014d45 to 0x02325c (0xe518 bytes) 000808 1788625713.153878 read 0x02b56e thru 0x0308b6 (0x5349 bytes) 000809 1788625713.155024 trunc from 0x0369f3 to 0x038c89 (0x2297 bytes) 000810 1788625713.176400 read 0x01a4e0 thru 0x01e4b3 (0x3fd4 bytes) 000811 1788625713.197794 MIRROR_RESYNC 13 mirrors 000812 1788625715.885080 trunc from 0x0309f2 to 0x034eb4 (0x44c3 bytes) 000814 1788625715.937443 trunc from 0x038c8a to 0x031b0c (0x717d bytes) 000815 1788625715.958189 read 0x008646 thru 0x00f113 (0x6ace bytes) 000816 1788625715.973966 MIRROR_RESYNC 13 mirrors 000817 1788625718.556661 mapwrite 0x03db5a thru 0x03ffff (0x24a6 bytes) 000818 1788625718.644936 trunc from 0x0211f2 to 0x024d31 (0x3b40 bytes) 000819 1788625718.652751 fallocate 0x02e4fa thru 0x02f624 (0x112b bytes) 000821 1788625718.659691 mapread 0x009c97 thru 0x0149ad (0xad17 bytes) 000822 1788625718.682661 punch from 0x026e16 to 0x027b95 (0xd80 bytes) 000823 1788625718.687024 fallocate 0x032de8 thru 0x036891 (0x3aaa bytes) 000824 1788625718.692019 trunc from 0x040000 to 0x000ea4 (0x3f15b bytes) 000825 1788625718.715916 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0xea5): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea5): Mirror 9: 0xe8432afc (stale) Mirror 11: 0xe8432afc (stale) Mirror 12: 0xe8432afc (stale) Mirror 14: 0xe8432afc (stale) Mirror 15: 0xe8432afc (stale) Mirror 16: 0xe8432afc (stale) Mirror 17: 0xe8432afc (stale) Mirror 19: 0xe8432afc (stale) Mirror 20: 0xe8432afc (stale) Mirror 21: 0xe8432afc (stale) Mirror 22: 0xe8432afc (stale) Mirror 23: 0xe8432afc (stale) CRC-32 checksum value for chunk [0, 0xea5): Mirror 1: 0xe8432afc Mirror 24: 0xe8432afc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea5: skipped 000826 1788625719.555914 fallocate 0x02519c thru 0x02b081 (0x5ee6 bytes) 000827 1788625719.622118 trunc from 0x000ea5 to 0x02955a (0x286b6 bytes) 000828 1788625719.647249 punch from 0x01b679 to 0x026807 (0xb18f bytes) 000829 1788625719.651368 mapread 0x00a449 thru 0x00c25f (0x1e17 bytes) 000830 1788625719.670286 mapread 0x00e7fe thru 0x01a315 (0xbb18 bytes) 000831 1788625719.681161 trunc from 0x02955b to 0x001769 (0x27df1 bytes) 000832 1788625719.698107 mapwrite 0x03a263 thru 0x03ffff (0x5d9d bytes) 000833 1788625719.755722 trunc from 0x014e95 to 0x01f746 (0xa8b2 bytes) 000834 1788625719.767699 trunc from 0x040000 to 0x01568e (0x2a971 bytes) 000835 1788625719.807230 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1568f): Mirror 11: 0x22e942bf (stale) Mirror 12: 0x22e942bf (stale) Mirror 14: 0x22e942bf (stale) Mirror 15: 0x22e942bf (stale) Mirror 16: 0x22e942bf (stale) Mirror 17: 0x22e942bf (stale) Mirror 19: 0x22e942bf (stale) Mirror 20: 0x22e942bf (stale) Mirror 21: 0x22e942bf (stale) Mirror 22: 0x22e942bf (stale) Mirror 23: 0x22e942bf (stale) Mirror 24: 0x6a1d08da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1568f): Mirror 11: 0x22e942bf (stale) Mirror 12: 0x22e942bf (stale) Mirror 14: 0x22e942bf (stale) Mirror 15: 0x22e942bf (stale) Mirror 16: 0x22e942bf (stale) Mirror 17: 0x22e942bf (stale) Mirror 19: 0x22e942bf (stale) Mirror 20: 0x22e942bf (stale) Mirror 21: 0x22e942bf (stale) Mirror 22: 0x22e942bf (stale) Mirror 23: 0x22e942bf (stale) Mirror 24: 0x6a1d08da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1568f: skipped 000836 1788625720.651584 mapwrite 0x024b07 thru 0x031e2b (0xd325 bytes) 000837 1788625720.735322 fallocate 0x00afa5 thru 0x015a01 (0xaa5d bytes) 000838 1788625720.739879 punch from 0x01a5f6 to 0x01b213 (0xc1e bytes) 000839 1788625720.743901 mapwrite 0x03fc63 thru 0x03ffff (0x39d bytes) 000840 1788625720.783064 write 0x02e156 thru 0x03ddbb (0xfc66 bytes) 000842 1788625720.804358 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x249e321e (stale) Mirror 12: 0x249e321e (stale) Mirror 14: 0x249e321e (stale) Mirror 15: 0x249e321e (stale) Mirror 16: 0x249e321e (stale) Mirror 17: 0x249e321e (stale) Mirror 19: 0x249e321e (stale) Mirror 20: 0x249e321e (stale) Mirror 21: 0x249e321e (stale) Mirror 22: 0x249e321e (stale) Mirror 23: 0x249e321e (stale) Mirror 24: 0x7823f187 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd6bebe32 Mirror 25: 0xd6bebe32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1788625722.313919 punch from 0x0305ec to 0x035e6a (0x587f bytes) 000846 1788625722.368529 mapwrite 0x00be24 thru 0x01aa5c (0xec39 bytes) 000847 1788625722.420221 write 0x02f142 thru 0x0380f7 (0x8fb6 bytes) 000848 1788625722.437924 mapwrite 0x02e825 thru 0x03cdaa (0xe586 bytes) 000849 1788625722.496786 mapread 0x0276e5 thru 0x028e1f (0x173b bytes) 000850 1788625722.497722 mapwrite 0x01f38e thru 0x02aa08 (0xb67b bytes) 000851 1788625722.529874 trunc from 0x040000 to 0x019087 (0x26f78 bytes) 000852 1788625722.569563 MIRROR_RESYNC 14 mirrors 000853 1788625724.606543 trunc from 0x00b7ca to 0x019087 (0xd8be bytes) 000854 1788625724.669651 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x19088): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19088): Mirror 11: 0x1cce466 (stale) Mirror 12: 0x1cce466 (stale) Mirror 14: 0x1cce466 (stale) Mirror 15: 0x1cce466 (stale) Mirror 16: 0x1cce466 (stale) Mirror 17: 0x1cce466 (stale) Mirror 19: 0x1cce466 (stale) Mirror 20: 0x1cce466 (stale) Mirror 21: 0x1cce466 (stale) Mirror 22: 0x1cce466 (stale) Mirror 23: 0x1cce466 (stale) Mirror 24: 0x1cce466 (stale) Mirror 25: 0x1cce466 (stale) CRC-32 checksum value for chunk [0, 0x19088): Mirror 1: 0x3ab95eb0 Mirror 26: 0x3ab95eb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19088: skipped 000855 1788625726.084866 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19088): Mirror 11: 0x1cce466 (stale) Mirror 12: 0x1cce466 (stale) Mirror 14: 0x1cce466 (stale) Mirror 15: 0x1cce466 (stale) Mirror 16: 0x1cce466 (stale) Mirror 17: 0x1cce466 (stale) Mirror 19: 0x1cce466 (stale) Mirror 20: 0x1cce466 (stale) Mirror 21: 0x1cce466 (stale) Mirror 23: 0x1cce466 (stale) Mirror 24: 0x1cce466 (stale) Mirror 25: 0x1cce466 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19088): Mirror 11: 0x1cce466 (stale) Mirror 12: 0x1cce466 (stale) Mirror 14: 0x1cce466 (stale) Mirror 15: 0x1cce466 (stale) Mirror 16: 0x1cce466 (stale) Mirror 17: 0x1cce466 (stale) Mirror 19: 0x1cce466 (stale) Mirror 20: 0x1cce466 (stale) Mirror 21: 0x1cce466 (stale) Mirror 23: 0x1cce466 (stale) Mirror 24: 0x1cce466 (stale) Mirror 25: 0x1cce466 (stale) CRC-32 checksum value for chunk [0, 0x19088): Mirror 1: 0x3ab95eb0 Mirror 26: 0x3ab95eb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19088: skipped 000856 1788625726.868392 punch from 0x00ff21 to 0x019087 (0x9167 bytes) 000857 1788625726.933035 mapwrite 0x0190c0 thru 0x01f673 (0x65b4 bytes) 000858 1788625727.010972 mapwrite 0x02f165 thru 0x032261 (0x30fd bytes) 000859 1788625727.051364 mapwrite 0x007466 thru 0x0087c5 (0x1360 bytes) 000860 1788625727.077924 mapread 0x02fe02 thru 0x032261 (0x2460 bytes) 000861 1788625727.079032 trunc from 0x032262 to 0x0375f9 (0x5398 bytes) 000862 1788625727.096868 read 0x00b985 thru 0x0188d7 (0xcf53 bytes) 000863 1788625727.102129 trunc from 0x0375fa to 0x038f7e (0x1985 bytes) 000864 1788625727.122891 mapwrite 0x021404 thru 0x02ac4d (0x984a bytes) 000865 1788625727.147506 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f7f): Mirror 11: 0xc3108000 (stale) Mirror 12: 0xc3108000 (stale) Mirror 14: 0xc3108000 (stale) Mirror 15: 0xc3108000 (stale) Mirror 16: 0xc3108000 (stale) Mirror 19: 0xc3108000 (stale) Mirror 20: 0xc3108000 (stale) Mirror 21: 0xc3108000 (stale) Mirror 23: 0xc3108000 (stale) Mirror 24: 0xc3108000 (stale) Mirror 25: 0xc3108000 (stale) Mirror 26: 0x671340d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f7f): Mirror 11: 0xc3108000 (stale) Mirror 12: 0xc3108000 (stale) Mirror 14: 0xc3108000 (stale) Mirror 15: 0xc3108000 (stale) Mirror 16: 0xc3108000 (stale) Mirror 19: 0xc3108000 (stale) Mirror 20: 0xc3108000 (stale) Mirror 21: 0xc3108000 (stale) Mirror 23: 0xc3108000 (stale) Mirror 24: 0xc3108000 (stale) Mirror 25: 0xc3108000 (stale) Mirror 26: 0x671340d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f7f: skipped 000866 1788625727.938024 fallocate 0x03dfa6 thru 0x03ffff (0x205a bytes) 000867 1788625727.945676 write 0x032540 thru 0x03ffff (0xdac0 bytes) 000868 1788625727.958779 mapwrite 0x016ddb thru 0x019156 (0x237c bytes) 000869 1788625728.011457 mapwrite 0x0194c3 thru 0x028ca3 (0xf7e1 bytes) 000870 1788625728.057740 mapwrite 0x00d1cc thru 0x0143bb (0x71f0 bytes) 000871 1788625728.117275 mapwrite 0x015046 thru 0x01743a (0x23f5 bytes) 000872 1788625728.166035 trunc from 0x040000 to 0x00ee7d (0x31182 bytes) 000873 1788625728.194677 fallocate 0x0103c2 thru 0x013893 (0x34d2 bytes) 000874 1788625728.204615 trunc from 0x013894 to 0x01bfac (0x8719 bytes) 000876 1788625728.239081 trunc from 0x01bfad to 0x012dfe (0x91ae bytes) 000877 1788625728.274349 trunc from 0x012dff to 0x00bac9 (0x7335 bytes) 000878 1788625728.317134 mapread 0x0008c2 thru 0x006244 (0x5983 bytes) 000879 1788625728.330287 mapwrite 0x00eb27 thru 0x01ea98 (0xff72 bytes) 000880 1788625728.438622 trunc from 0x01ea99 to 0x01bf88 (0x2b10 bytes) 000881 1788625728.471837 trunc from 0x01bf89 to 0x0110c0 (0xaec8 bytes) 000882 1788625728.498744 trunc from 0x0001da to 0x0054b7 (0x52de bytes) 000883 1788625728.510532 mapwrite 0x0025a6 thru 0x00e424 (0xbe7f bytes) 000884 1788625728.578223 trunc from 0x0110c1 to 0x024d35 (0x13c75 bytes) 000885 1788625728.637970 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24d36): Mirror 11: 0xbafe3856 (stale) Mirror 12: 0xbafe3856 (stale) Mirror 14: 0xbafe3856 (stale) Mirror 15: 0xbafe3856 (stale) Mirror 16: 0xbafe3856 (stale) Mirror 19: 0xbafe3856 (stale) Mirror 20: 0xbafe3856 (stale) Mirror 21: 0xbafe3856 (stale) Mirror 23: 0xbafe3856 (stale) Mirror 25: 0xbafe3856 (stale) Mirror 26: 0x1bfa2527 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d36): Mirror 11: 0xbafe3856 (stale) Mirror 12: 0xbafe3856 (stale) Mirror 14: 0xbafe3856 (stale) Mirror 15: 0xbafe3856 (stale) Mirror 16: 0xbafe3856 (stale) Mirror 19: 0xbafe3856 (stale) Mirror 20: 0xbafe3856 (stale) Mirror 21: 0xbafe3856 (stale) Mirror 23: 0xbafe3856 (stale) Mirror 25: 0xbafe3856 (stale) Mirror 26: 0x1bfa2527 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d36: skipped 000886 1788625729.513149 trunc from 0x002628 to 0x002719 (0xf2 bytes) 000888 1788625729.523654 read 0x00379a thru 0x00db01 (0xa368 bytes) 000889 1788625729.534024 read 0x01e9e9 thru 0x024d35 (0x634d bytes) 000890 1788625729.560155 fallocate 0x034b13 thru 0x03848b (0x3979 bytes) 000892 1788625729.566551 trunc from 0x035b14 to 0x03848b (0x2978 bytes) 000893 1788625729.573528 read 0x018908 thru 0x026d6d (0xe466 bytes) 000894 1788625729.585646 trunc from 0x03848c to 0x03d6d0 (0x5245 bytes) 000895 1788625729.617730 write 0x02c105 thru 0x03a599 (0xe495 bytes) 000896 1788625729.631500 mapread 0x009f7b thru 0x019ce3 (0xfd69 bytes) 000897 1788625729.632996 read 0x00d668 thru 0x013be1 (0x657a bytes) 000898 1788625729.636590 write 0x0366a8 thru 0x03ffff (0x9958 bytes) 000899 1788625729.639003 mapwrite 0x03c8ac thru 0x03ffff (0x3754 bytes) 000900 1788625729.675042 read 0x0123ba thru 0x02117b (0xedc2 bytes) 000901 1788625729.687666 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5ea8b977 (stale) Mirror 12: 0x5ea8b977 (stale) Mirror 14: 0x5ea8b977 (stale) Mirror 15: 0x5ea8b977 (stale) Mirror 16: 0x5ea8b977 (stale) Mirror 19: 0x5ea8b977 (stale) Mirror 20: 0x5ea8b977 (stale) Mirror 23: 0x5ea8b977 (stale) Mirror 25: 0x5ea8b977 (stale) Mirror 26: 0x7823f187 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5ea8b977 (stale) Mirror 12: 0x5ea8b977 (stale) Mirror 14: 0x5ea8b977 (stale) Mirror 15: 0x5ea8b977 (stale) Mirror 16: 0x5ea8b977 (stale) Mirror 19: 0x5ea8b977 (stale) Mirror 20: 0x5ea8b977 (stale) Mirror 23: 0x5ea8b977 (stale) Mirror 25: 0x5ea8b977 (stale) Mirror 26: 0x7823f187 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000903 1788625730.500604 trunc from 0x03b865 to 0x03ffff (0x479b bytes) 000904 1788625730.508839 mapread 0x01248c thru 0x01dc47 (0xb7bc bytes) 000905 1788625730.557949 punch from 0x00a152 to 0x00cf2c (0x2ddb bytes) 000906 1788625730.567482 read 0x024496 thru 0x02d73e (0x92a9 bytes) 000907 1788625730.584569 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5ea8b977 (stale) Mirror 12: 0x5ea8b977 (stale) Mirror 14: 0x5ea8b977 (stale) Mirror 15: 0x5ea8b977 (stale) Mirror 16: 0x5ea8b977 (stale) Mirror 19: 0x5ea8b977 (stale) Mirror 20: 0x5ea8b977 (stale) Mirror 23: 0x5ea8b977 (stale) Mirror 25: 0x5ea8b977 (stale) Mirror 26: 0x7823f187 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfa069692 Mirror 27: 0xfa069692 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000908 1788625732.330077 trunc from 0x040000 to 0x01be9d (0x24162 bytes) 000909 1788625732.450641 trunc from 0x013dab to 0x015089 (0x12df bytes) 000910 1788625732.475953 punch from 0x01ac6b to 0x01be9d (0x1233 bytes) 000911 1788625732.492162 trunc from 0x00b50f to 0x0195ee (0xe0e0 bytes) 000912 1788625732.522287 mapread 0x00d887 thru 0x01a939 (0xd0b3 bytes) 000915 1788625732.585225 punch from 0x011f91 to 0x01be9d (0x9f0d bytes) 000916 1788625732.591819 mapread 0x0110d0 thru 0x016ab3 (0x59e4 bytes) 000917 1788625732.602895 trunc from 0x01be9e to 0x0256da (0x983d bytes) 000918 1788625732.631447 trunc from 0x0256db to 0x018264 (0xd476 bytes) 000919 1788625732.654715 read 0x015a94 thru 0x018264 (0x27d1 bytes) 000920 1788625732.655646 mapwrite 0x0125a5 thru 0x01a52e (0x7f8a bytes) 000921 1788625732.692468 fallocate 0x0218c9 thru 0x02448d (0x2bc5 bytes) 000922 1788625732.697597 mapread 0x004b3d thru 0x007503 (0x29c7 bytes) 000923 1788625732.709443 mapwrite 0x03368d thru 0x0356b6 (0x202a bytes) 000924 1788625732.753618 mapwrite 0x018e63 thru 0x024bf9 (0xbd97 bytes) 000925 1788625732.786507 read 0x01626e thru 0x01ef0e (0x8ca1 bytes) 000926 1788625732.786992 write 0x001c62 thru 0x00ca01 (0xada0 bytes) 000927 1788625732.788823 trunc from 0x0356b7 to 0x013e2c (0x2188a bytes) 000929 1788625732.812920 mapwrite 0x0339d6 thru 0x033adf (0x10a bytes) 000930 1788625732.866614 trunc from 0x002dba to 0x00fe80 (0xd0c7 bytes) 000931 1788625732.874019 read 0x00bce3 thru 0x010fe5 (0x5303 bytes) 000933 1788625732.888456 fallocate 0x022a71 thru 0x030b7e (0xe10e bytes) 000934 1788625732.895712 fallocate 0x03b0f5 thru 0x03ffff (0x4f0b bytes) 000935 1788625732.908951 write 0x004e3c thru 0x009fcc (0x5191 bytes) 000936 1788625732.910231 punch from 0x01eaad to 0x027f78 (0x94cc bytes) 000937 1788625732.921589 trunc from 0x00117a to 0x0067ad (0x5634 bytes) 000938 1788625732.958164 trunc from 0x033ae0 to 0x00f00b (0x24ad4 bytes) 000939 1788625733.010542 read 0x006e5c thru 0x00cc85 (0x5e2a bytes) 000940 1788625733.021903 read 0x009b91 thru 0x00d864 (0x3cd4 bytes) 000941 1788625733.022767 trunc from 0x00b23d to 0x00ccab (0x1a6f bytes) 000942 1788625733.032490 fallocate 0x03cb08 thru 0x03ffff (0x34f8 bytes) 000943 1788625733.038528 mapread 0x01bbf7 thru 0x0215e2 (0x59ec bytes) 000944 1788625733.061456 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5ea8b977 (stale) Mirror 12: 0x5ea8b977 (stale) Mirror 14: 0x5ea8b977 (stale) Mirror 15: 0x5ea8b977 (stale) Mirror 16: 0x5ea8b977 (stale) Mirror 19: 0x5ea8b977 (stale) Mirror 20: 0x5ea8b977 (stale) Mirror 23: 0x5ea8b977 (stale) Mirror 26: 0x7823f187 (stale) Mirror 27: 0xfa069692 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5ea8b977 (stale) Mirror 12: 0x5ea8b977 (stale) Mirror 14: 0x5ea8b977 (stale) Mirror 15: 0x5ea8b977 (stale) Mirror 16: 0x5ea8b977 (stale) Mirror 19: 0x5ea8b977 (stale) Mirror 20: 0x5ea8b977 (stale) Mirror 23: 0x5ea8b977 (stale) Mirror 26: 0x7823f187 (stale) Mirror 27: 0xfa069692 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000945 1788625733.999012 fallocate 0x028ab8 thru 0x02edc1 (0x630a bytes) 000947 1788625734.006383 trunc from 0x00462e to 0x004da9 (0x77c bytes) 000948 1788625734.011215 trunc from 0x040000 to 0x002d99 (0x3d266 bytes) 000949 1788625734.035877 mapread 0x000fc2 thru 0x002d99 (0x1dd8 bytes) 000950 1788625734.054651 mapwrite 0x03e41e thru 0x03ffff (0x1be2 bytes) 000951 1788625734.103867 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5ea8b977 (stale) Mirror 12: 0x5ea8b977 (stale) Mirror 14: 0x5ea8b977 (stale) Mirror 15: 0x5ea8b977 (stale) Mirror 16: 0x5ea8b977 (stale) Mirror 19: 0x5ea8b977 (stale) Mirror 20: 0x5ea8b977 (stale) Mirror 26: 0x7823f187 (stale) Mirror 27: 0xfa069692 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5ea8b977 (stale) Mirror 12: 0x5ea8b977 (stale) Mirror 14: 0x5ea8b977 (stale) Mirror 15: 0x5ea8b977 (stale) Mirror 16: 0x5ea8b977 (stale) Mirror 19: 0x5ea8b977 (stale) Mirror 20: 0x5ea8b977 (stale) Mirror 26: 0x7823f187 (stale) Mirror 27: 0xfa069692 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000952 1788625734.853857 trunc from 0x002fec to 0x00648f (0x34a4 bytes) 000953 1788625734.860885 trunc from 0x024e28 to 0x02f451 (0xa62a bytes) 000954 1788625734.866007 mapwrite 0x001db8 thru 0x010e27 (0xf070 bytes) 000955 1788625734.908477 trunc from 0x013b2b to 0x014c39 (0x110f bytes) 000956 1788625734.915767 fallocate 0x010d79 thru 0x017a0a (0x6c92 bytes) 000957 1788625734.928784 trunc from 0x040000 to 0x007f1c (0x380e3 bytes) 000958 1788625734.976389 trunc from 0x00378c to 0x007f1c (0x4791 bytes) 000959 1788625734.990129 MIRROR_RESYNC 10 mirrors 000960 1788625737.306765 trunc from 0x007f1d to 0x033513 (0x2b5f7 bytes) 000961 1788625737.361906 fallocate 0x014b74 thru 0x015849 (0xcd6 bytes) 000962 1788625737.374816 mapread 0x00ff66 thru 0x015dd3 (0x5e6e bytes) 000963 1788625737.408884 fallocate 0x021d6a thru 0x02592d (0x3bc4 bytes) 000965 1788625737.415582 trunc from 0x00ebde to 0x010343 (0x1766 bytes) 000966 1788625737.424631 mapwrite 0x000e62 thru 0x010a31 (0xfbd0 bytes) 000967 1788625737.537051 write 0x00679a thru 0x00871a (0x1f81 bytes) 000968 1788625737.538365 punch from 0x003342 to 0x00bbdf (0x889e bytes) 000969 1788625737.569631 mapwrite 0x02c7f0 thru 0x03410d (0x791e bytes) 000970 1788625737.651743 trunc from 0x013f1c to 0x023dbe (0xfea3 bytes) 000971 1788625737.671793 MIRROR_RESYNC 10 mirrors 000972 1788625740.566654 mapwrite 0x00926e thru 0x010b5a (0x78ed bytes) 000974 1788625740.678770 write 0x01afcc thru 0x01e3c8 (0x33fd bytes) 000975 1788625740.697112 MIRROR_RESYNC 10 mirrors 000976 1788625743.088563 MIRROR_EXTEND to 11 mirrors 000977 1788625744.451803 fallocate 0x038eb1 thru 0x03ffff (0x714f bytes) 000978 1788625744.507891 fallocate 0x039abe thru 0x03ffff (0x6542 bytes) 000979 1788625744.513992 fallocate 0x03aa33 thru 0x03ffff (0x55cd bytes) 000980 1788625744.519582 mapread 0x034b26 thru 0x03ffff (0xb4da bytes) 000981 1788625744.546153 fallocate 0x0251fe thru 0x02ff31 (0xad34 bytes) 000982 1788625744.552379 mapwrite 0x029e9e thru 0x02d68f (0x37f2 bytes) 000986 1788625744.581015 fallocate 0x0210fc thru 0x02d7dd (0xc6e2 bytes) 000987 1788625744.592196 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1f2c5256 (stale) Mirror 14: 0x1f2c5256 (stale) Mirror 15: 0x1f2c5256 (stale) Mirror 16: 0x1f2c5256 (stale) Mirror 19: 0x1f2c5256 (stale) Mirror 20: 0x1f2c5256 (stale) Mirror 26: 0x1f2c5256 (stale) Mirror 27: 0x1f2c5256 (stale) Mirror 28: 0x1f2c5256 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1f2c5256 (stale) Mirror 14: 0x1f2c5256 (stale) Mirror 15: 0x1f2c5256 (stale) Mirror 16: 0x1f2c5256 (stale) Mirror 19: 0x1f2c5256 (stale) Mirror 20: 0x1f2c5256 (stale) Mirror 26: 0x1f2c5256 (stale) Mirror 27: 0x1f2c5256 (stale) Mirror 28: 0x1f2c5256 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000988 1788625745.233529 trunc from 0x040000 to 0x03eab3 (0x154c bytes) 000989 1788625745.258182 punch from 0x0265c1 to 0x0290cb (0x2b0b bytes) 000990 1788625745.264970 fallocate 0x00935d thru 0x00cd81 (0x3a25 bytes) 000991 1788625745.270148 mapread 0x01aaab thru 0x01c8dc (0x1e32 bytes) 000992 1788625745.292422 mapread 0x01a6df thru 0x0231c2 (0x8ae4 bytes) 000993 1788625745.293596 read 0x0380d0 thru 0x03eab3 (0x69e4 bytes) 000995 1788625745.307275 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x175:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eab4): Mirror 14: 0x3db7888e (stale) Mirror 15: 0x3db7888e (stale) Mirror 16: 0x3db7888e (stale) Mirror 19: 0x3db7888e (stale) Mirror 20: 0x3db7888e (stale) Mirror 26: 0x3db7888e (stale) Mirror 27: 0x3db7888e (stale) Mirror 28: 0x3db7888e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eab4): Mirror 14: 0x3db7888e (stale) Mirror 15: 0x3db7888e (stale) Mirror 16: 0x3db7888e (stale) Mirror 19: 0x3db7888e (stale) Mirror 20: 0x3db7888e (stale) Mirror 26: 0x3db7888e (stale) Mirror 27: 0x3db7888e (stale) Mirror 28: 0x3db7888e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eab4: skipped 000996 1788625745.808754 read 0x025f8a thru 0x02cf38 (0x6faf bytes) 000997 1788625745.820129 fallocate 0x017f4d thru 0x01f375 (0x7429 bytes) 000998 1788625745.826434 trunc from 0x03eab4 to 0x038ba5 (0x5f0e bytes) All operations completed A-OK! PASS 70a (143s) == sanity-flr test 71: check mirror extend parent fid ==== 12:29:11 (1788625751) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1bb:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 12:29:23 (1788625763) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274794 s, 7.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112229 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 12:29:34 (1788625774) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7845952 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2438176 .. done Extending file size to 5967520 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8448896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9087744 .. Extending file size to 8471136 .. Extending file size to 9141024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61273 61274 61276 61278 61282 failed 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:30:57 (1788625857) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7579680 .. Extending file size to 479104 .. Extending file size to 3326176 .. Extending file size to 4371264 .. Extending file size to 6051808 .. Extending file size to 7275552 .. Extending file size to 8977344 .. Extending file size to 7542880 .. Extending file size to 7895616 .. Extending file size to 8084928 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8805664 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62818 62819 62820 62821 62825 failed 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:32:18 (1788625938) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2268 1269036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18528 3588492 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23948 3583072 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42476 7171564 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2268 1269036 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18528 3588492 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23948 3583072 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42476 7171564 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0933566 s, 22.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215993 s, 9.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc60106c1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:32:39 (1788625959) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136947 s, 15.3 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 12:32:48 (1788625968) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:32:56 (1788625976) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0953509 s, 22.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x209:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:33:06 (1788625986) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0959492 s, 21.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x207:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x207:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:33:16 (1788625996) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109693 s, 19.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:33:50 (1788626030) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606265 s, 17.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x208:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x208:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:34:42 (1788626082) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124827 s, 16.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:35:39 (1788626139) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg233-server: Pool lustre.test_204f created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145075 s, 14.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg233-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:36:54 (1788626214) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:37:02 (1788626222) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:37:10 (1788626230) oleg233-server: Pool lustre.test_206 created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg233-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:37:37 (1788626257) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701309 s, 15.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276279 s, 38.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc19d85c5 Mirror 3: 0xc19d85c5 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:37:48 (1788626268) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:37:52 (1788626272) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:37:55 (1788626275) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0691507 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2220000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1030000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x8d0000 Read this (4 bytes): BBBB 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:38:35 (1788626315) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0241238 s, 340 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:38:42 (1788626322) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537885 s, 19.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 12:38:44.000000000 -0400 Modify: 2026-09-05 12:38:44.000000000 -0400 Change: 2026-09-05 12:38:46.000000000 -0400 Birth: 2026-09-05 12:38:44.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:38:51 (1788626331) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295478 s, 35.5 MB/s oleg233-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg233-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg233-server: O/280000401/d27/13b: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:39:12 (1788626352) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.187875 s, 218 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0437867 s, 93.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 12:39 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 12:39 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:39:21 (1788626361) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:39:29 (1788626369) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1914 sec =========== 12:39:42 (1788626382) === sanity-flr: start cleanup 12:39:44 (1788626384) === === sanity-flr: finish cleanup 12:39:52 (1788626392) ===