-----============= acceptance-small: sanity-flr ============----- Sun Sep 6 21:23:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg129-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:24:01 (1788744241) === oleg129-client.virtnet: executing check_config_client /mnt/lustre oleg129-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg129-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9512c525d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9512c525d800.idle_timeout=debug disable quota as required oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:24:45 (1788744285) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8209] == sanity-flr test 0a: lfs mirror create with -N option == 21:24:48 (1788744288) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:25:08 (1788744308) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:25:13 (1788744313) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:25:16 (1788744316) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:25:45 (1788744345) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:25:49 (1788744349) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 21:25:53 (1788744353) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9512c525d800.stats=clear osc.lustre-OST0001-osc-ffff9512c525d800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:26:03 (1788744363) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:26:15 (1788744375) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:26:24 (1788744384) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:26:28 (1788744388) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:26:37 (1788744397) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 21:26:48 (1788744408) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:26:57 (1788744417) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:27:06 (1788744426) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:27:15 (1788744435) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:27:26 (1788744446) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:27:33 (1788744453) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.842538 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140019 s, 7.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:27:42 (1788744462) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.120887 s, 26.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:27:51 (1788744471) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:28:02 (1788744482) Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg129-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (143s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:30:24 (1788744624) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.649146 s, 16.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:31:13 (1788744673) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:32:39 (1788744759) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.79036 s, 13.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 6 21:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.935s user 0m0.001s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9512c525d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.870s user 0m0.000s sys 0m0.151s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg129-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg129-server oleg129-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.578s user 0m0.001s sys 0m0.196s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg129-server: oleg129-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg129-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg129-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9512c525d800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:34:16 (1788744856) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:34:19 (1788744859) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:34:23 (1788744863) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 21:34:27 (1788744867) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 21:34:35 (1788744875) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.558311 s, 22.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:34:47 (1788744887) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544208 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294602 s, 35.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:34:55 (1788744895) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:35:03 (1788744903) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010318 s, 992 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121878 s, 840 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00298497 s, 343 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00117491 s, 872 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 21:35:12 (1788744912) create test files with size 15741177 .. 4a8f1a42ffcb4726e1d50a6bdc0b9389 - 77e840a5f97f33f32930b3cc6e70225e - 574534816dce157fcbb6356709dd3071 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 21:35:38 (1788744938) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:35:53 (1788744953) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:36:00 (1788744960) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570314 s, 18.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342096 s, 30.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 21:36:09 (1788744969) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0620273 s, 33.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.242486 s, 17.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134839 s, 15.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 21:36:23 (1788744983) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 21:36:27 (1788744987) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:36:31 (1788744991) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292959 s, 35.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 21:36:40 (1788745000) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.289276 s, 10.9 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:36:51 (1788745011) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xda:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:37:02 (1788745022) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.434076 s, 24.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:37:14 (1788745034) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9512d550a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.381566 s, 27.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg129-client.virtnet /mnt/lustre (opts:) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9512c8791800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:37:32 (1788745052) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:38:13 (1788745093) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:38:54 (1788745134) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 21:38:59 (1788745139) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.242099 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.079557 s, 39.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x992f9dd7 (nosync,stale) Mirror 2: 0x992f9dd7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 870181d75b88f13a4bdea7714dc9b521 - mirror 1 checksum: 9e98096952cf8824e9d0db2b8652c035 - mirror 2 checksum: 870181d75b88f13a4bdea7714dc9b521 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T21:39:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T21:39:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 870181d75b88f13a4bdea7714dc9b521 - mirror 1 checksum: 9e98096952cf8824e9d0db2b8652c035 - mirror 2 checksum: 9e98096952cf8824e9d0db2b8652c035 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T21:39:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T21:39:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:39:12 (1788745152) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426198 s, 24.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:39:21 (1788745161) Failing mds1 on oleg129-server Stopping /mnt/lustre-mds1 (opts:) on oleg129-server 21:39:35 (1788745175) shut down facet: mds1 facet_host: oleg129-server facet_failover_host: oleg129-server Failover mds1 to oleg129-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 21:40:08 (1788745208) targets are mounted 21:40:08 (1788745208) facet_failover done oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284856 s, 36.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:40:26 (1788745226) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0470044 s, 436 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0289858 s, 707 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:40:37 (1788745237) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0805254 s, 254 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00988511 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:40:49 (1788745249) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00783402 s, 2.1 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:40:59 (1788745259) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0620898 s, 330 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (11s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:41:10 (1788745270) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00632319 s, 648 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0775917 s, 52.8 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:41:20 (1788745280) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:41:47 (1788745307) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:42:16 (1788745336) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:42:41 (1788745361) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:43:06 (1788745386) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 21:43:14 (1788745394) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57576 mirror split pid 57577 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5139: 57576 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5139: 57577 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 21:44:22 (1788745462) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 5464 skipping zero size read skipping zero length punch hole 000005 1788745464.583168 mapwrite 0x00bc05 thru 0x0150e8 (0x94e4 bytes) 000006 1788745464.647577 read 0x010321 thru 0x0150e8 (0x4dc8 bytes) 000007 1788745464.649030 read 0x00fff1 thru 0x0101d9 (0x1e9 bytes) 000008 1788745464.651078 mapread 0x00e991 thru 0x0150e8 (0x6758 bytes) truncating to largest ever: 0x36622 000009 1788745464.653286 trunc from 0x0150e9 to 0x036621 (0x21539 bytes) 000010 1788745464.675312 mapwrite 0x00873a thru 0x009bad (0x1474 bytes) 000011 1788745464.688765 trunc from 0x01aeef to 0x02474e (0x9860 bytes) 000012 1788745464.694633 mapread 0x02afa8 thru 0x02e21f (0x3278 bytes) 000013 1788745464.708584 mapwrite 0x0147fd thru 0x01bc1a (0x741e bytes) 000014 1788745464.727807 mapread 0x002e44 thru 0x00ab7b (0x7d38 bytes) 000015 1788745464.729160 write 0x015d9f thru 0x0195cc (0x382e bytes) 000017 1788745464.732139 mapread 0x01b0da thru 0x0219ab (0x68d2 bytes) 000018 1788745464.734343 trunc from 0x036622 to 0x003ab3 (0x32b6e bytes) 000020 1788745464.753430 punch from 0x000d50 to 0x003ab3 (0x2d64 bytes) 000021 1788745464.759044 mapread 0x001387 thru 0x003ab3 (0x272d bytes) 000022 1788745464.777571 trunc from 0x003ab4 to 0x0268e0 (0x22e2d bytes) 000023 1788745464.798317 write 0x0174af thru 0x026870 (0xf3c2 bytes) 000024 1788745464.809931 mapwrite 0x02732f thru 0x031c34 (0xa906 bytes) 000025 1788745464.912006 mapwrite 0x005e31 thru 0x006622 (0x7f2 bytes) 000026 1788745464.937727 read 0x01306e thru 0x0186f7 (0x568a bytes) 000027 1788745464.953096 read 0x01b861 thru 0x0207e5 (0x4f85 bytes) 000028 1788745464.953905 read 0x02c7bd thru 0x0317a1 (0x4fe5 bytes) 000029 1788745464.955091 trunc from 0x031c35 to 0x004047 (0x2dbed bytes) 000030 1788745464.972168 mapwrite 0x01effc thru 0x02a3b3 (0xb3b8 bytes) 000031 1788745465.027366 MIRROR_EXTEND to 2 mirrors 000032 1788745465.651618 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) 000033 1788745465.812929 trunc from 0x02a3b4 to 0x0357c9 (0xb416 bytes) 000035 1788745465.851965 mapwrite 0x019cfb thru 0x02160f (0x7915 bytes) 000036 1788745465.888815 read 0x01e7bc thru 0x02d2a1 (0xeae6 bytes) fallocating to largest ever: 0x3d1d8 000037 1788745465.896792 fallocate 0x02dccd thru 0x03d1d7 (0xf50b bytes) 000038 1788745465.902180 write 0x0207eb thru 0x02be2f (0xb645 bytes) 000039 1788745465.903607 trunc from 0x020cab to 0x030c4f (0xffa5 bytes) 000040 1788745465.919282 fallocate 0x027bed thru 0x02a8dc (0x2cf0 bytes) 000041 1788745465.924523 trunc from 0x01f96e to 0x02f0c0 (0xf753 bytes) 000042 1788745465.930740 mapwrite 0x027b1d thru 0x036dde (0xf2c2 bytes) 000043 1788745465.973244 mapread 0x019eaa thru 0x023b86 (0x9cdd bytes) 000045 1788745465.980432 mapread 0x02ce2c thru 0x033776 (0x694b bytes) 000046 1788745465.981642 read 0x03452f thru 0x037dc3 (0x3895 bytes) 000047 1788745465.982990 mapread 0x017d86 thru 0x01bd73 (0x3fee bytes) 000049 1788745465.986231 punch from 0x0386b0 to 0x03d1d7 (0x4b28 bytes) 000050 1788745465.991357 trunc from 0x011283 to 0x0144e1 (0x325f bytes) 000051 1788745465.996629 trunc from 0x03d1d8 to 0x01d888 (0x1f94f bytes) 000054 1788745466.015876 write 0x00aa8f thru 0x00ca08 (0x1f7a bytes) 000055 1788745466.036030 read 0x01afd4 thru 0x01d888 (0x28b5 bytes) 000056 1788745466.036957 trunc from 0x00a74d to 0x00bc4e (0x1502 bytes) 000057 1788745466.049093 mapwrite 0x0255da thru 0x02ac2e (0x5655 bytes) 000058 1788745466.102037 fallocate 0x02100c thru 0x02380e (0x2803 bytes) 000059 1788745466.107638 fallocate 0x004ca8 thru 0x0105df (0xb938 bytes) 000060 1788745466.113190 mapread 0x000233 thru 0x008940 (0x870e bytes) 000062 1788745466.114809 read 0x01e6bd thru 0x023bfe (0x5542 bytes) 000064 1788745466.118082 fallocate 0x01b940 thru 0x029a65 (0xe126 bytes) 000065 1788745466.123954 trunc from 0x00ebd9 to 0x0115a6 (0x29ce bytes) 000066 1788745466.128040 mapwrite 0x020c7d thru 0x021641 (0x9c5 bytes) 000067 1788745466.141295 fallocate 0x013db7 thru 0x01827d (0x44c7 bytes) 000068 1788745466.146241 trunc from 0x02ac2f to 0x012e5c (0x17dd2 bytes) 000069 1788745466.161546 mapwrite 0x00ddcf thru 0x014962 (0x6b94 bytes) 000070 1788745466.210317 write 0x017487 thru 0x02606d (0xebe7 bytes) 000071 1788745466.212700 fallocate 0x00b7b3 thru 0x00d6c7 (0x1f15 bytes) fallocating to largest ever: 0x40000 000072 1788745466.222701 fallocate 0x038357 thru 0x03ffff (0x7ca9 bytes) 000073 1788745466.228434 mapwrite 0x02136b thru 0x02b0c0 (0x9d56 bytes) 000074 1788745466.283399 trunc from 0x03b98e to 0x03c689 (0xcfc bytes) 000075 1788745466.289751 trunc from 0x025775 to 0x02c6c8 (0x6f54 bytes) 000076 1788745466.298554 write 0x01f8de thru 0x02d990 (0xe0b3 bytes) 000077 1788745466.302406 mapread 0x0299fe thru 0x02ee35 (0x5438 bytes) 000078 1788745466.304162 trunc from 0x01aaa8 to 0x027ec2 (0xd41b bytes) 000079 1788745466.323931 fallocate 0x00f37e thru 0x0185db (0x925e bytes) 000080 1788745466.329441 write 0x018416 thru 0x024238 (0xbe23 bytes) 000081 1788745466.342340 write 0x0250aa thru 0x02bc90 (0x6be7 bytes) 000082 1788745466.357359 MIRROR_EXTEND to 2 mirrors 000083 1788745467.306624 read 0x004879 thru 0x00c51a (0x7ca2 bytes) 000084 1788745467.352313 MIRROR_EXTEND to 3 mirrors 000085 1788745468.275344 punch from 0x03fb8b to 0x03ffff (0x475 bytes) 000086 1788745468.340255 read 0x0080be thru 0x013b39 (0xba7c bytes) 000087 1788745468.358962 trunc from 0x040000 to 0x034490 (0xbb6f bytes) 000088 1788745468.478097 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34491): Mirror 3: 0x8cfa55a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34491): Mirror 3: 0x8cfa55a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34491: skipped 000089 1788745468.936130 trunc from 0x014977 to 0x0176ae (0x2d38 bytes) 000090 1788745468.952575 MIRROR_RESYNC 2 mirrors 000092 1788745470.318325 trunc from 0x026287 to 0x02d5a3 (0x731d bytes) 000093 1788745470.388374 mapwrite 0x0222e0 thru 0x025f38 (0x3c59 bytes) 000094 1788745470.501126 trunc from 0x034491 to 0x0370bd (0x2c2d bytes) 000095 1788745470.586482 read 0x014da1 thru 0x021e51 (0xd0b1 bytes) 000097 1788745470.606288 trunc from 0x0370be to 0x012ab4 (0x24609 bytes) 000098 1788745470.665109 trunc from 0x0000c1 to 0x00d733 (0xd673 bytes) 000099 1788745470.676704 mapread 0x00fa37 thru 0x012ab4 (0x307e bytes) 000100 1788745470.741425 MIRROR_RESYNC 2 mirrors 000101 1788745471.618369 mapwrite 0x018f30 thru 0x01afcb (0x209c bytes) 000102 1788745471.712260 trunc from 0x01afcc to 0x037a5c (0x1ca91 bytes) 000103 1788745471.735877 fallocate 0x0251e6 thru 0x02d033 (0x7e4e bytes) 000104 1788745471.746133 fallocate 0x03e097 thru 0x03ffff (0x1f69 bytes) 000105 1788745471.751633 write 0x008422 thru 0x00f088 (0x6c67 bytes) 000106 1788745471.762635 trunc from 0x040000 to 0x017381 (0x28c7e bytes) 000107 1788745471.787244 trunc from 0x017382 to 0x02d2d4 (0x15f53 bytes) 000108 1788745471.806892 trunc from 0x02d2d5 to 0x02d0bb (0x219 bytes) 000109 1788745471.833369 mapwrite 0x00d446 thru 0x00d888 (0x443 bytes) 000110 1788745471.854311 write 0x0201a4 thru 0x02808c (0x7ee9 bytes) 000111 1788745471.869612 MIRROR_RESYNC 2 mirrors 000112 1788745472.528916 fallocate 0x00ef61 thru 0x01b853 (0xc8f3 bytes) 000113 1788745472.561006 punch from 0x005c80 to 0x012675 (0xc9f6 bytes) 000114 1788745472.567908 read 0x0230ec thru 0x02533f (0x2254 bytes) 000115 1788745472.573614 mapread 0x028815 thru 0x02d0bb (0x48a7 bytes) 000116 1788745472.593547 trunc from 0x02d0bc to 0x034b42 (0x7a87 bytes) 000117 1788745472.622062 trunc from 0x025feb to 0x033944 (0xd95a bytes) 000118 1788745472.628766 trunc from 0x034b43 to 0x02177c (0x133c6 bytes) 000119 1788745472.645401 write 0x00218a thru 0x00f72d (0xd5a4 bytes) 000120 1788745472.658181 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2177d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2177d): Mirror 3: 0xeb023286 (stale) CRC-32 checksum value for chunk [0, 0x2177d): Mirror 1: 0x195d610b Mirror 4: 0x195d610b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2177d: skipped 000121 1788745473.393275 MIRROR_RESYNC 3 mirrors 000124 1788745474.102575 write 0x03808f thru 0x03ffff (0x7f71 bytes) 000126 1788745474.163400 trunc from 0x01528b to 0x022015 (0xcd8b bytes) 000127 1788745474.170015 read 0x0366c1 thru 0x03ffff (0x993f bytes) 000129 1788745474.177550 trunc from 0x040000 to 0x02cd7e (0x13281 bytes) 000130 1788745474.226032 trunc from 0x014a86 to 0x01cdd7 (0x8352 bytes) 000132 1788745474.233243 write 0x02ee0b thru 0x0383ca (0x95c0 bytes) 000133 1788745474.234775 mapwrite 0x03ca5b thru 0x03f3fa (0x29a0 bytes) 000134 1788745474.299510 mapwrite 0x005de9 thru 0x012ee4 (0xd0fc bytes) 000135 1788745474.347972 write 0x023824 thru 0x02fe5b (0xc638 bytes) 000136 1788745474.355512 mapread 0x0015d2 thru 0x007a23 (0x6452 bytes) 000137 1788745474.356919 mapread 0x019a59 thru 0x025e21 (0xc3c9 bytes) 000139 1788745474.366924 write 0x019686 thru 0x01bef9 (0x2874 bytes) 000140 1788745474.373514 MIRROR_RESYNC 3 mirrors 000141 1788745476.248368 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) 000142 1788745476.797097 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) 000144 1788745477.030489 read 0x0230aa thru 0x03303b (0xff92 bytes) 000145 1788745477.053020 write 0x00b39a thru 0x017fa1 (0xcc08 bytes) 000146 1788745477.095770 trunc from 0x01471e to 0x01a353 (0x5c36 bytes) 000147 1788745477.124390 mapwrite 0x000c09 thru 0x0047b2 (0x3baa bytes) 000148 1788745477.216081 trunc from 0x03f3fb to 0x01d55d (0x21e9d bytes) 000149 1788745477.239604 read 0x00472a thru 0x006b7d (0x2454 bytes) 000150 1788745477.240219 trunc from 0x01d55e to 0x01dedd (0x980 bytes) 000151 1788745477.270186 read 0x00c4cb thru 0x00e7be (0x22f4 bytes) 000152 1788745477.271389 fallocate 0x02f150 thru 0x03a620 (0xb4d1 bytes) 000153 1788745477.299562 MIRROR_EXTEND to 2 mirrors 000154 1788745478.662721 trunc from 0x01dede to 0x038356 (0x1a479 bytes) 000155 1788745478.820769 fallocate 0x00b3c4 thru 0x00fef3 (0x4b30 bytes) 000156 1788745478.835233 fallocate 0x02f339 thru 0x032209 (0x2ed1 bytes) 000157 1788745478.850164 read 0x0266bc thru 0x02aa8f (0x43d4 bytes) 000158 1788745478.877545 trunc from 0x038357 to 0x0108c7 (0x27a8f bytes) 000159 1788745478.934517 trunc from 0x001102 to 0x004aa9 (0x39a8 bytes) 000160 1788745478.951755 fallocate 0x020d94 thru 0x02499d (0x3c0a bytes) 000161 1788745478.966519 fallocate 0x01899d thru 0x01d537 (0x4b9b bytes) 000162 1788745478.975748 punch from 0x0048cc to 0x00e777 (0x9eac bytes) 000163 1788745478.981079 write 0x01fdd0 thru 0x02a54b (0xa77c bytes) 000164 1788745478.995989 punch from 0x00eaf1 to 0x017ea1 (0x93b1 bytes) 000167 1788745479.003702 trunc from 0x00a3f0 to 0x00abf6 (0x807 bytes) 000168 1788745479.010205 trunc from 0x02a54c to 0x00284f (0x27cfc bytes) 000169 1788745479.036133 punch from 0x0008cd to 0x00284f (0x1f83 bytes) 000170 1788745479.058984 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2850): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2850): Mirror 2: 0x251717f6 (stale) CRC-32 checksum value for chunk [0, 0x2850): Mirror 1: 0x2c2481b8 Mirror 3: 0x2c2481b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2850: skipped 000171 1788745479.909343 punch from 0x0025e7 to 0x00284f (0x269 bytes) 000172 1788745479.948095 fallocate 0x020a21 thru 0x029327 (0x8907 bytes) 000173 1788745479.953613 write 0x03ca4f thru 0x03f2e4 (0x2896 bytes) 000174 1788745479.959193 mapwrite 0x01d7dc thru 0x0284fc (0xad21 bytes) 000175 1788745479.999116 read 0x03d00d thru 0x03f2e4 (0x22d8 bytes) 000176 1788745480.000899 write 0x0330d8 thru 0x03ec13 (0xbb3c bytes) 000177 1788745480.007003 fallocate 0x0002ff thru 0x00eb96 (0xe898 bytes) 000178 1788745480.012112 trunc from 0x0281b6 to 0x0377b9 (0xf604 bytes) 000179 1788745480.026949 fallocate 0x0025cd thru 0x00ef5d (0xc991 bytes) 000180 1788745480.043927 write 0x00df24 thru 0x014009 (0x60e6 bytes) 000181 1788745480.061568 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2e5): Mirror 3: 0xde307af2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2e5): Mirror 3: 0xde307af2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2e5: skipped 000183 1788745480.342465 MIRROR_RESYNC 2 mirrors 000184 1788745481.106261 read 0x027431 thru 0x029180 (0x1d50 bytes) 000185 1788745481.119830 punch from 0x0310da to 0x03a03a (0x8f61 bytes) 000186 1788745481.178821 read 0x0187d7 thru 0x0194c4 (0xcee bytes) 000187 1788745481.211435 read 0x0057c2 thru 0x010b70 (0xb3af bytes) 000188 1788745481.212311 trunc from 0x03f2e5 to 0x022401 (0x1cee3 bytes) 000189 1788745481.245708 mapwrite 0x03178a thru 0x035be6 (0x445d bytes) 000190 1788745481.325009 trunc from 0x035be7 to 0x01fc5d (0x15f89 bytes) 000191 1788745481.349516 fallocate 0x0372a4 thru 0x0392d4 (0x2031 bytes) 000192 1788745481.365594 punch from 0x001f13 to 0x009280 (0x736e bytes) 000193 1788745481.385244 read 0x033b5e thru 0x03689f (0x2d42 bytes) 000194 1788745481.410987 fallocate 0x024d49 thru 0x0312f0 (0xc5a8 bytes) 000195 1788745481.426200 mapwrite 0x02743b thru 0x034688 (0xd24e bytes) 000196 1788745481.457809 read 0x001c19 thru 0x0041fb (0x25e3 bytes) 000198 1788745481.459026 punch from 0x02841e to 0x02f9fc (0x75df bytes) 000199 1788745481.466379 trunc from 0x01d081 to 0x0254b9 (0x8439 bytes) 000200 1788745481.481814 MIRROR_RESYNC 2 mirrors 000201 1788745482.537435 trunc from 0x00bb43 to 0x0172aa (0xb768 bytes) 000202 1788745482.571960 mapwrite 0x037873 thru 0x03ffff (0x878d bytes) 000203 1788745482.643629 read 0x027af8 thru 0x02eeec (0x73f5 bytes) 000204 1788745482.657922 mapread 0x02b2ca thru 0x038a66 (0xd79d bytes) 000205 1788745482.664948 write 0x001d2f thru 0x0094e7 (0x77b9 bytes) 000207 1788745482.677946 mapwrite 0x023d4c thru 0x029895 (0x5b4a bytes) 000208 1788745482.717964 trunc from 0x040000 to 0x0149e6 (0x2b619 bytes) 000209 1788745482.744811 read 0x0133e1 thru 0x0149e6 (0x1606 bytes) 000210 1788745482.761789 trunc from 0x0009c9 to 0x00fba9 (0xf1e1 bytes) 000212 1788745482.768911 punch from 0x002b64 to 0x007d10 (0x51ad bytes) 000213 1788745482.774440 mapread 0x00e899 thru 0x0149e6 (0x614e bytes) 000214 1788745482.793784 punch from 0x0095c5 to 0x013663 (0xa09f bytes) 000215 1788745482.816334 mapwrite 0x036be8 thru 0x03ffff (0x9418 bytes) 000216 1788745482.875854 mapwrite 0x01258f thru 0x0211fc (0xec6e bytes) 000217 1788745482.917765 fallocate 0x0228ba thru 0x02daad (0xb1f4 bytes) 000218 1788745482.938132 punch from 0x013847 to 0x01a4ca (0x6c84 bytes) 000219 1788745482.951798 trunc from 0x02a4ad to 0x0326c2 (0x8216 bytes) 000220 1788745482.959074 mapwrite 0x0225fe thru 0x025ed3 (0x38d6 bytes) 000221 1788745482.987932 MIRROR_RESYNC 2 mirrors 000222 1788745483.840124 mapread 0x03e259 thru 0x03ffff (0x1da7 bytes) 000223 1788745483.860880 read 0x01a558 thru 0x01e981 (0x442a bytes) 000224 1788745483.879429 mapread 0x005d97 thru 0x009a5c (0x3cc6 bytes) 000225 1788745483.921821 mapread 0x0256c8 thru 0x02dd7b (0x86b4 bytes) 000226 1788745483.933895 write 0x020500 thru 0x02c9a4 (0xc4a5 bytes) 000227 1788745484.054706 mapwrite 0x03b1f1 thru 0x03ffff (0x4e0f bytes) 000228 1788745484.178282 punch from 0x01c699 to 0x02bfd1 (0xf939 bytes) 000229 1788745484.186527 trunc from 0x03acc7 to 0x03c7f0 (0x1b2a bytes) 000230 1788745484.193820 mapwrite 0x022603 thru 0x02c20a (0x9c08 bytes) 000231 1788745484.238214 write 0x03b7e9 thru 0x03ffff (0x4817 bytes) 000232 1788745484.244744 mapread 0x005d3a thru 0x0127c7 (0xca8e bytes) 000233 1788745484.268293 punch from 0x02b568 to 0x039d36 (0xe7cf bytes) 000234 1788745484.274580 punch from 0x0259a1 to 0x029306 (0x3966 bytes) 000235 1788745484.281296 punch from 0x0184a4 to 0x020f81 (0x8ade bytes) 000236 1788745484.286602 write 0x00f326 thru 0x01bb9f (0xc87a bytes) 000237 1788745484.293790 write 0x02f63b thru 0x03034c (0xd12 bytes) 000239 1788745484.305762 write 0x023280 thru 0x03220b (0xef8c bytes) 000240 1788745484.313421 trunc from 0x040000 to 0x03565a (0xa9a5 bytes) 000241 1788745484.335934 write 0x03f700 thru 0x03ffff (0x900 bytes) 000242 1788745484.337478 fallocate 0x01a7c1 thru 0x01e6cd (0x3f0d bytes) 000243 1788745484.343480 read 0x015e90 thru 0x01eb5d (0x8cce bytes) 000244 1788745484.352047 fallocate 0x008747 thru 0x01526f (0xcb29 bytes) 000246 1788745484.357492 mapwrite 0x03c2e9 thru 0x03ffff (0x3d17 bytes) 000247 1788745484.377547 fallocate 0x01f046 thru 0x02ec3c (0xfbf7 bytes) 000248 1788745484.383045 punch from 0x033bad to 0x03ffff (0xc453 bytes) 000249 1788745484.388804 punch from 0x031b54 to 0x034f92 (0x343f bytes) 000250 1788745484.395358 mapread 0x003b2b thru 0x0109cd (0xcea3 bytes) 000251 1788745484.396414 trunc from 0x040000 to 0x014885 (0x2b77a bytes) 000252 1788745484.415861 mapread 0x00242c thru 0x0070a0 (0x4c75 bytes) 000253 1788745484.416845 mapread 0x01193a thru 0x0139d2 (0x2099 bytes) 000255 1788745484.418103 mapread 0x004755 thru 0x00b188 (0x6a34 bytes) 000256 1788745484.420320 trunc from 0x00c97a to 0x014885 (0x7f0c bytes) 000257 1788745484.426290 write 0x03233d thru 0x0326a9 (0x36d bytes) 000258 1788745484.427377 punch from 0x00669f to 0x008378 (0x1cda bytes) 000259 1788745484.432856 trunc from 0x006c4c to 0x016b54 (0xff09 bytes) 000261 1788745484.443711 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x326aa): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326aa): Mirror 3: 0xa6b65831 (stale) CRC-32 checksum value for chunk [0, 0x326aa): Mirror 1: 0xabbec44 Mirror 4: 0xabbec44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326aa: skipped 000262 1788745485.138635 write 0x00650c thru 0x0107fd (0xa2f2 bytes) 000263 1788745485.224157 trunc from 0x0326aa to 0x03c3a9 (0x9d00 bytes) 000264 1788745485.289641 punch from 0x0301ce to 0x033b76 (0x39a9 bytes) 000265 1788745485.298339 write 0x00e51d thru 0x00f34c (0xe30 bytes) 000266 1788745485.299745 mapwrite 0x03545d thru 0x03b7f9 (0x639d bytes) 000267 1788745485.341318 fallocate 0x039dbe thru 0x03ffff (0x6242 bytes) 000268 1788745485.347765 mapread 0x03e9fe thru 0x03fa16 (0x1019 bytes) 000269 1788745485.356852 fallocate 0x02ed36 thru 0x038338 (0x9603 bytes) 000270 1788745485.367264 punch from 0x01c8e4 to 0x025c87 (0x93a4 bytes) 000271 1788745485.373141 punch from 0x030a0f to 0x03674b (0x5d3d bytes) 000274 1788745485.382327 punch from 0x02fb51 to 0x03922f (0x96df bytes) 000276 1788745485.395875 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4cc642bd (stale) Mirror 4: 0x32685dab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3d33eab2 Mirror 5: 0x3d33eab2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1788745486.555573 mapread 0x00c283 thru 0x01a9ec (0xe76a bytes) 000278 1788745486.614089 mapwrite 0x02b3eb thru 0x035c5b (0xa871 bytes) 000279 1788745486.727851 mapwrite 0x0321f1 thru 0x033184 (0xf94 bytes) 000281 1788745486.762046 read 0x0333fc thru 0x03ffff (0xcc04 bytes) 000282 1788745486.772428 mapwrite 0x00bb42 thru 0x01b44c (0xf90b bytes) 000283 1788745486.838543 punch from 0x019c29 to 0x01b8df (0x1cb7 bytes) 000284 1788745486.846178 punch from 0x039528 to 0x03c5e5 (0x30be bytes) 000286 1788745486.853438 punch from 0x014225 to 0x01ed51 (0xab2d bytes) 000287 1788745486.861067 write 0x009621 thru 0x00eec5 (0x58a5 bytes) 000288 1788745486.869050 read 0x020017 thru 0x02d896 (0xd880 bytes) 000289 1788745486.881965 fallocate 0x02d724 thru 0x0364d5 (0x8db2 bytes) 000290 1788745486.888446 mapwrite 0x029d93 thru 0x039c2e (0xfe9c bytes) 000291 1788745486.925278 mapwrite 0x00afb4 thru 0x013ac7 (0x8b14 bytes) 000292 1788745486.952619 trunc from 0x0059a7 to 0x013f13 (0xe56d bytes) 000293 1788745486.964247 fallocate 0x02e8a8 thru 0x03d972 (0xf0cb bytes) 000294 1788745486.969801 mapwrite 0x014b9c thru 0x01cd92 (0x81f7 bytes) 000296 1788745487.014206 punch from 0x006c45 to 0x00f2fd (0x86b9 bytes) 000297 1788745487.028799 trunc from 0x024aa1 to 0x02a090 (0x55f0 bytes) 000298 1788745487.048820 mapwrite 0x0261ff thru 0x02626c (0x6e bytes) 000299 1788745487.091918 fallocate 0x03bed3 thru 0x03c448 (0x576 bytes) 000300 1788745487.100824 trunc from 0x013cb9 to 0x0142b9 (0x601 bytes) 000302 1788745487.107310 mapread 0x007034 thru 0x010372 (0x933f bytes) 000304 1788745487.160761 trunc from 0x040000 to 0x021998 (0x1e667 bytes) 000305 1788745487.207510 read 0x0194ca thru 0x01d97b (0x44b2 bytes) 000306 1788745487.209047 mapread 0x015389 thru 0x016fab (0x1c23 bytes) 000307 1788745487.210943 mapread 0x01c7dc thru 0x021998 (0x51bd bytes) 000308 1788745487.212076 mapwrite 0x01397c thru 0x020b12 (0xd197 bytes) 000309 1788745487.258249 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21999): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21999): Mirror 3: 0x4d2e206d (stale) Mirror 4: 0xd906acf (stale) Mirror 5: 0x9322b9db (stale) CRC-32 checksum value for chunk [0, 0x21999): Mirror 1: 0x4b9e1fdc Mirror 6: 0x4b9e1fdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21999: skipped 000310 1788745488.216879 trunc from 0x000d39 to 0x00fbe0 (0xeea8 bytes) 000311 1788745488.264319 punch from 0x017226 to 0x020984 (0x975f bytes) 000312 1788745488.274815 write 0x02fc90 thru 0x0339ab (0x3d1c bytes) 000313 1788745488.279631 trunc from 0x00596d to 0x007275 (0x1909 bytes) 000314 1788745488.285055 read 0x00198e thru 0x00def3 (0xc566 bytes) 000315 1788745488.299232 MIRROR_RESYNC 5 mirrors 000316 1788745489.669249 read 0x02e3f8 thru 0x0339ab (0x55b4 bytes) 000317 1788745489.679827 read 0x01495c thru 0x014f65 (0x60a bytes) 000318 1788745489.689241 write 0x0063a3 thru 0x00869c (0x22fa bytes) 000319 1788745489.723996 read 0x00771e thru 0x0083f8 (0xcdb bytes) 000320 1788745489.724534 punch from 0x007acd to 0x0139af (0xbee3 bytes) 000321 1788745489.738982 punch from 0x019e76 to 0x01fdfd (0x5f88 bytes) 000322 1788745489.744311 punch from 0x00eead to 0x0108b2 (0x1a06 bytes) 000323 1788745489.749484 mapread 0x018a5a thru 0x01d473 (0x4a1a bytes) 000324 1788745489.761396 write 0x02bd3c thru 0x039ba1 (0xde66 bytes) 000325 1788745489.768134 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39ba2): Mirror 4: 0x1aa10476 (stale) Mirror 5: 0x1aa10476 (stale) Mirror 6: 0x1aa10476 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ba2): Mirror 4: 0x1aa10476 (stale) Mirror 5: 0x1aa10476 (stale) Mirror 6: 0x1aa10476 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ba2: skipped 000326 1788745490.137371 MIRROR_RESYNC 4 mirrors 000327 1788745491.339845 read 0x032a9c thru 0x039ba1 (0x7106 bytes) 000329 1788745491.350705 fallocate 0x010f57 thru 0x01e781 (0xd82b bytes) 000330 1788745491.385429 mapwrite 0x014795 thru 0x022cfb (0xe567 bytes) 000331 1788745491.437404 write 0x00879b thru 0x0149c5 (0xc22b bytes) 000332 1788745491.443322 read 0x01371b thru 0x018807 (0x50ed bytes) 000333 1788745491.444691 trunc from 0x039ba2 to 0x01f2d0 (0x1a8d1 bytes) 000334 1788745491.464836 fallocate 0x035a27 thru 0x03ffff (0xa5d9 bytes) 000335 1788745491.471006 trunc from 0x040000 to 0x037069 (0x8f96 bytes) 000336 1788745491.488475 trunc from 0x00f4ef to 0x0171a2 (0x7cb4 bytes) 000337 1788745491.509263 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3706a): Mirror 5: 0x2bf5c5c2 (stale) Mirror 6: 0x2bf5c5c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3706a): Mirror 5: 0x2bf5c5c2 (stale) Mirror 6: 0x2bf5c5c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3706a: skipped 000338 1788745491.886115 write 0x0080fd thru 0x00ead6 (0x69da bytes) 000339 1788745491.908648 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3706a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3706a): Mirror 5: 0x2bf5c5c2 (stale) Mirror 6: 0x2bf5c5c2 (stale) CRC-32 checksum value for chunk [0, 0x3706a): Mirror 1: 0x8bc719d6 Mirror 7: 0x8bc719d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3706a: skipped 000340 1788745493.195419 MIRROR_RESYNC 4 mirrors 000341 1788745494.287221 mapread 0x0194f2 thru 0x024a6d (0xb57c bytes) 000343 1788745494.322721 mapwrite 0x02a3da thru 0x03681f (0xc446 bytes) 000344 1788745494.445061 fallocate 0x02dcbc thru 0x032ea7 (0x51ec bytes) 000345 1788745494.452752 trunc from 0x03706a to 0x00e2fa (0x28d6f bytes) 000346 1788745494.488539 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe2fb): Mirror 6: 0x55668f7e (stale) Mirror 7: 0x55668f7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe2fb): Mirror 6: 0x55668f7e (stale) Mirror 7: 0x55668f7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe2fb: skipped 000347 1788745494.823362 write 0x007e1e thru 0x008e7d (0x1060 bytes) 000348 1788745494.837815 mapwrite 0x0085c1 thru 0x011a2f (0x946f bytes) 000349 1788745494.893283 trunc from 0x0043aa to 0x0101bc (0xbe13 bytes) 000350 1788745494.900641 fallocate 0x00597c thru 0x009b27 (0x41ac bytes) 000351 1788745494.907756 trunc from 0x011a30 to 0x037bd6 (0x261a7 bytes) 000352 1788745494.926013 mapwrite 0x0160b7 thru 0x01fd1c (0x9c66 bytes) 000353 1788745494.968558 fallocate 0x03ee54 thru 0x03ffff (0x11ac bytes) 000354 1788745494.973855 trunc from 0x040000 to 0x002f0e (0x3d0f1 bytes) 000355 1788745494.994307 trunc from 0x0010fb to 0x001eca (0xdd0 bytes) 000356 1788745494.999590 trunc from 0x002f0f to 0x01d9a5 (0x1aa97 bytes) 000357 1788745495.016293 trunc from 0x013cdf to 0x0188e6 (0x4c08 bytes) 000358 1788745495.020954 trunc from 0x01417a to 0x01949e (0x5325 bytes) 000359 1788745495.025621 read 0x00f063 thru 0x015873 (0x6811 bytes) 000360 1788745495.033908 mapread 0x008e98 thru 0x0134be (0xa627 bytes) 000361 1788745495.043883 mapread 0x00a2a5 thru 0x016c70 (0xc9cc bytes) 000362 1788745495.054251 punch from 0x00a234 to 0x0141fb (0x9fc8 bytes) 000363 1788745495.059799 fallocate 0x010e90 thru 0x016486 (0x55f7 bytes) 000364 1788745495.065421 mapread 0x01c2a6 thru 0x01d9a5 (0x1700 bytes) 000365 1788745495.067787 punch from 0x0082c0 to 0x00d01d (0x4d5e bytes) 000366 1788745495.073832 mapwrite 0x02a99f thru 0x0392f0 (0xe952 bytes) 000367 1788745495.145523 MIRROR_RESYNC 3 mirrors 000368 1788745495.796555 MIRROR_EXTEND to 4 mirrors 000369 1788745496.499408 trunc from 0x0392f1 to 0x02b16e (0xe182 bytes) 000370 1788745496.579644 fallocate 0x00e349 thru 0x01c327 (0xdfdf bytes) 000371 1788745496.585320 trunc from 0x02b16f to 0x021dca (0x93a4 bytes) 000372 1788745496.608555 write 0x034c62 thru 0x03af4a (0x62e9 bytes) 000373 1788745496.610830 punch from 0x035b91 to 0x03af4a (0x53ba bytes) 000375 1788745496.626846 trunc from 0x0095cd to 0x01644d (0xce81 bytes) 000376 1788745496.632410 punch from 0x036dd6 to 0x03af4a (0x4175 bytes) 000377 1788745496.638817 write 0x030fb3 thru 0x03e486 (0xd4d4 bytes) 000378 1788745496.647712 punch from 0x008c79 to 0x0122ab (0x9633 bytes) 000379 1788745496.652811 trunc from 0x02f508 to 0x036aca (0x75c3 bytes) 000380 1788745496.669797 read 0x03dd41 thru 0x03e486 (0x746 bytes) 000381 1788745496.670608 read 0x006f3b thru 0x009f3c (0x3002 bytes) 000382 1788745496.688762 fallocate 0x025693 thru 0x028193 (0x2b01 bytes) 000383 1788745496.700780 mapread 0x0114f2 thru 0x015c8c (0x479b bytes) 000384 1788745496.705873 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e487): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e487): Mirror 6: 0x3c482d6f (stale) Mirror 7: 0x3c482d6f (stale) Mirror 8: 0x3c482d6f (stale) CRC-32 checksum value for chunk [0, 0x3e487): Mirror 1: 0xcb2e2a3c Mirror 9: 0xcb2e2a3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e487: skipped 000385 1788745497.905618 MIRROR_RESYNC 5 mirrors 000386 1788745498.679543 mapwrite 0x033c16 thru 0x03ffff (0xc3ea bytes) 000387 1788745498.787502 write 0x022ddf thru 0x0319e6 (0xec08 bytes) 000388 1788745498.797944 mapread 0x01fc70 thru 0x02a63d (0xa9ce bytes) 000389 1788745498.823242 mapread 0x02fa7a thru 0x03745b (0x79e2 bytes) 000390 1788745498.824340 trunc from 0x040000 to 0x012fee (0x2d011 bytes) 000391 1788745498.849508 read 0x011f51 thru 0x012fee (0x109e bytes) 000392 1788745498.858337 write 0x027e81 thru 0x02fb2e (0x7cae bytes) 000393 1788745498.860770 punch from 0x0083e4 to 0x01606a (0xdc87 bytes) 000394 1788745498.867012 fallocate 0x009537 thru 0x014106 (0xabd0 bytes) 000395 1788745498.876580 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fb2f): Mirror 7: 0x9457d978 (stale) Mirror 8: 0x9457d978 (stale) Mirror 9: 0x9457d978 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb2f): Mirror 7: 0x9457d978 (stale) Mirror 8: 0x9457d978 (stale) Mirror 9: 0x9457d978 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb2f: skipped 000397 1788745499.190241 read 0x026466 thru 0x02fb2e (0x96c9 bytes) 000398 1788745499.203968 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fb2f): Mirror 7: 0x9457d978 (stale) Mirror 9: 0x9457d978 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb2f): Mirror 7: 0x9457d978 (stale) Mirror 9: 0x9457d978 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb2f: skipped 000399 1788745499.591380 read 0x014c8d thru 0x016976 (0x1cea bytes) 000400 1788745499.607154 MIRROR_RESYNC 3 mirrors 000401 1788745500.411456 punch from 0x0107a9 to 0x01e0b4 (0xd90c bytes) 000402 1788745500.435058 trunc from 0x029ae1 to 0x02a628 (0xb48 bytes) 000403 1788745500.439033 fallocate 0x014808 thru 0x0235c4 (0xedbd bytes) 000404 1788745500.442627 trunc from 0x02fb2f to 0x0214e7 (0xe647 bytes) 000405 1788745500.459136 mapread 0x00d57c thru 0x015c70 (0x86f5 bytes) 000406 1788745500.479026 trunc from 0x000934 to 0x0029f4 (0x20c1 bytes) 000407 1788745500.486019 mapread 0x00259a thru 0x00faac (0xd513 bytes) 000409 1788745500.508145 MIRROR_RESYNC 3 mirrors 000410 1788745501.390943 trunc from 0x0004a2 to 0x00b8c5 (0xb424 bytes) 000411 1788745501.425414 read 0x01e379 thru 0x0214e7 (0x316f bytes) 000413 1788745501.436278 fallocate 0x01844c thru 0x01cb90 (0x4745 bytes) 000414 1788745501.442831 trunc from 0x00e2a8 to 0x014a7b (0x67d4 bytes) 000415 1788745501.449434 mapread 0x013f74 thru 0x0214e7 (0xd574 bytes) 000416 1788745501.471564 write 0x02bf94 thru 0x03b165 (0xf1d2 bytes) 000417 1788745501.475514 mapwrite 0x01a1af thru 0x02640a (0xc25c bytes) 000418 1788745501.514034 fallocate 0x007fe5 thru 0x00f911 (0x792d bytes) 000419 1788745501.523811 MIRROR_RESYNC 3 mirrors 000420 1788745502.394691 trunc from 0x03b166 to 0x01c403 (0x1ed62 bytes) 000421 1788745502.439964 fallocate 0x0175e3 thru 0x0237bd (0xc1db bytes) 000422 1788745502.445629 trunc from 0x01c404 to 0x03a123 (0x1dd20 bytes) 000425 1788745502.475944 trunc from 0x03a124 to 0x02fd44 (0xa3df bytes) 000426 1788745502.499321 mapwrite 0x0060a0 thru 0x0131e0 (0xd141 bytes) 000427 1788745502.606616 read 0x02dd90 thru 0x02fd44 (0x1fb5 bytes) 000428 1788745502.624548 trunc from 0x000605 to 0x002263 (0x1c5f bytes) 000429 1788745502.654552 MIRROR_RESYNC 3 mirrors 000430 1788745503.588231 punch from 0x00f6b0 to 0x01a0fd (0xaa4e bytes) 000431 1788745503.627570 trunc from 0x02fd45 to 0x01fe0b (0xff39 bytes) 000432 1788745503.648121 mapread 0x01948b thru 0x01b48b (0x2001 bytes) 000433 1788745503.673278 trunc from 0x019569 to 0x01fe0b (0x68a3 bytes) 000434 1788745503.682253 MIRROR_RESYNC 3 mirrors 000435 1788745504.548378 trunc from 0x01fe0c to 0x0030e4 (0x1cd27 bytes) 000436 1788745504.618151 trunc from 0x0030e5 to 0x01c50c (0x19428 bytes) 000437 1788745504.648124 read 0x010b9f thru 0x01c50c (0xb96e bytes) 000438 1788745504.662426 write 0x019b0a thru 0x0223c8 (0x88bf bytes) 000439 1788745504.664566 trunc from 0x0223c9 to 0x03de7b (0x1bab3 bytes) 000440 1788745504.684010 mapwrite 0x00864b thru 0x01546e (0xce24 bytes) 000441 1788745504.730347 trunc from 0x03de7c to 0x02fee5 (0xdf96 bytes) 000442 1788745504.757403 MIRROR_RESYNC 3 mirrors 000443 1788745505.788371 mapread 0x017f37 thru 0x027331 (0xf3fb bytes) 000444 1788745505.857368 trunc from 0x00a72c to 0x012441 (0x7d16 bytes) 000445 1788745505.894798 mapread 0x02887f thru 0x028ddb (0x55d bytes) 000446 1788745505.913585 read 0x025093 thru 0x02fee5 (0xae53 bytes) 000447 1788745505.914449 trunc from 0x02fee6 to 0x032754 (0x286f bytes) 000449 1788745505.942869 punch from 0x030101 to 0x032754 (0x2654 bytes) 000450 1788745505.948931 trunc from 0x029bd1 to 0x032754 (0x8b84 bytes) 000451 1788745505.959493 mapread 0x00736c thru 0x00ec2f (0x78c4 bytes) 000452 1788745505.970530 trunc from 0x032755 to 0x00dbf9 (0x24b5b bytes) 000454 1788745505.997907 punch from 0x0024a0 to 0x00a81f (0x8380 bytes) 000455 1788745506.004032 read 0x00ccbb thru 0x00dbf9 (0xf3f bytes) 000456 1788745506.005042 mapwrite 0x03449d thru 0x03ffff (0xbb63 bytes) 000457 1788745506.062105 MIRROR_RESYNC 3 mirrors 000458 1788745506.687760 fallocate 0x01d4b9 thru 0x0274cb (0xa013 bytes) 000459 1788745506.729822 read 0x00b444 thru 0x00b621 (0x1de bytes) 000460 1788745506.740089 trunc from 0x009be0 to 0x010975 (0x6d96 bytes) 000462 1788745506.763905 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6d184928 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6d184928 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000464 1788745507.086064 write 0x007115 thru 0x00d460 (0x634c bytes) 000465 1788745507.101444 trunc from 0x040000 to 0x03d306 (0x2cf9 bytes) 000466 1788745507.128065 read 0x01c8d5 thru 0x025b92 (0x92be bytes) 000467 1788745507.135969 trunc from 0x03d307 to 0x004e6e (0x38498 bytes) 000468 1788745507.148461 trunc from 0x004e6f to 0x021017 (0x1c1a9 bytes) 000470 1788745507.166253 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x21018): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21018): Mirror 9: 0xcf93824a (stale) CRC-32 checksum value for chunk [0, 0x21018): Mirror 1: 0xcf93824a Mirror 10: 0xcf93824a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21018: skipped 000471 1788745507.760795 fallocate 0x03ed4d thru 0x03ffff (0x12b3 bytes) 000472 1788745507.803062 write 0x021438 thru 0x02ac52 (0x981b bytes) 000473 1788745507.818640 trunc from 0x008b0c to 0x01424d (0xb742 bytes) 000474 1788745507.823462 mapread 0x000836 thru 0x00f731 (0xeefc bytes) 000475 1788745507.853573 write 0x017d61 thru 0x01c7a7 (0x4a47 bytes) 000476 1788745507.866122 punch from 0x00e142 to 0x01944f (0xb30e bytes) 000477 1788745507.884948 fallocate 0x00269d thru 0x0105d5 (0xdf39 bytes) 000478 1788745507.891228 punch from 0x029dc0 to 0x02ac52 (0xe93 bytes) 000479 1788745507.897569 punch from 0x00fabb to 0x01c5f1 (0xcb37 bytes) 000480 1788745507.903739 punch from 0x00684a to 0x00ca6d (0x6224 bytes) 000481 1788745507.912353 mapwrite 0x02ae13 thru 0x035104 (0xa2f2 bytes) 000482 1788745508.041408 mapread 0x0152d9 thru 0x02265a (0xd382 bytes) 000483 1788745508.073938 read 0x02e8aa thru 0x035104 (0x685b bytes) 000484 1788745508.078163 read 0x030ac4 thru 0x032401 (0x193e bytes) 000486 1788745508.083462 read 0x008f86 thru 0x00f79b (0x6816 bytes) 000487 1788745508.100197 write 0x025c18 thru 0x027aaa (0x1e93 bytes) 000488 1788745508.106241 mapread 0x01e30b thru 0x0272df (0x8fd5 bytes) 000489 1788745508.114402 trunc from 0x01b7b3 to 0x0290b5 (0xd903 bytes) 000490 1788745508.140928 fallocate 0x0205f6 thru 0x0288b5 (0x82c0 bytes) 000492 1788745508.146941 read 0x029468 thru 0x02f8de (0x6477 bytes) 000494 1788745508.161635 fallocate 0x00bca2 thru 0x018d3e (0xd09d bytes) 000495 1788745508.168314 fallocate 0x01e922 thru 0x02d8d5 (0xefb4 bytes) 000497 1788745508.182417 punch from 0x012bd8 to 0x017336 (0x475f bytes) 000498 1788745508.189574 trunc from 0x028629 to 0x02cb99 (0x4571 bytes) 000499 1788745508.204296 write 0x035899 thru 0x03ffff (0xa767 bytes) 000500 1788745508.209987 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000501 1788745508.664148 trunc from 0x040000 to 0x00cff0 (0x3300f bytes) 000502 1788745508.692149 mapread 0x001249 thru 0x009da8 (0x8b60 bytes) 000504 1788745508.720492 trunc from 0x00cff1 to 0x01224c (0x525c bytes) 000505 1788745508.739608 fallocate 0x02814d thru 0x02d442 (0x52f6 bytes) 000506 1788745508.745995 mapwrite 0x03d497 thru 0x03ffff (0x2b69 bytes) 000507 1788745508.790388 punch from 0x03a7ea to 0x03ffff (0x5816 bytes) 000508 1788745508.798272 trunc from 0x031315 to 0x03ffff (0xeceb bytes) 000509 1788745508.803861 trunc from 0x003963 to 0x00dc71 (0xa30f bytes) 000510 1788745508.820736 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 11: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1788745509.970415 trunc from 0x040000 to 0x026a87 (0x19578 bytes) 000512 1788745510.013320 fallocate 0x01d82e thru 0x02225a (0x4a2d bytes) 000513 1788745510.018174 trunc from 0x02443e to 0x025469 (0x102c bytes) 000515 1788745510.025973 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26a88): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a88): Mirror 10: 0xff4761d1 (stale) Mirror 11: 0xff4761d1 (stale) CRC-32 checksum value for chunk [0, 0x26a88): Mirror 1: 0xff4761d1 Mirror 12: 0xff4761d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a88: skipped 000516 1788745511.090095 punch from 0x003a3a to 0x01262c (0xebf3 bytes) 000517 1788745511.147114 trunc from 0x02649e to 0x026a87 (0x5ea bytes) 000518 1788745511.153474 trunc from 0x021d44 to 0x026a87 (0x4d44 bytes) 000519 1788745511.162842 punch from 0x00f5a1 to 0x01bcc1 (0xc721 bytes) 000520 1788745511.180455 punch from 0x017b3c to 0x026a87 (0xef4c bytes) 000521 1788745511.187896 mapread 0x0221f0 thru 0x022549 (0x35a bytes) 000522 1788745511.211939 read 0x016663 thru 0x01daa5 (0x7443 bytes) 000523 1788745511.225770 mapwrite 0x00082f thru 0x008a55 (0x8227 bytes) 000524 1788745511.302612 trunc from 0x026a88 to 0x0011b5 (0x258d2 bytes) 000525 1788745511.369645 mapread 0x000db2 thru 0x0011b5 (0x404 bytes) 000526 1788745511.372715 fallocate 0x02beea thru 0x02e7b3 (0x28ca bytes) 000527 1788745511.402104 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e7b4): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7b4): Mirror 10: 0x35bbc796 (stale) Mirror 11: 0x35bbc796 (stale) Mirror 12: 0x35bbc796 (stale) CRC-32 checksum value for chunk [0, 0x2e7b4): Mirror 1: 0xbbff4641 Mirror 13: 0xbbff4641 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7b4: skipped 000528 1788745512.252635 punch from 0x023c0f to 0x02e7b3 (0xaba5 bytes) 000529 1788745512.282084 write 0x006277 thru 0x00aa2f (0x47b9 bytes) 000530 1788745512.293059 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e7b4): Mirror 11: 0x35bbc796 (stale) Mirror 12: 0x35bbc796 (stale) Mirror 13: 0xbbff4641 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7b4): Mirror 11: 0x35bbc796 (stale) Mirror 12: 0x35bbc796 (stale) Mirror 13: 0xbbff4641 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7b4: skipped 000531 1788745512.685801 mapwrite 0x021a75 thru 0x028223 (0x67af bytes) 000532 1788745512.735313 trunc from 0x02e7b4 to 0x0091bb (0x255f8 bytes) 000533 1788745512.771042 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x91bc): Mirror 11: 0xb524a58b (stale) Mirror 13: 0xae6f4920 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x91bc): Mirror 11: 0xb524a58b (stale) Mirror 13: 0xae6f4920 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x91bc: skipped 000534 1788745513.286100 punch from 0x00510e to 0x0091bb (0x40ae bytes) 000535 1788745513.304548 write 0x01de43 thru 0x02291a (0x4ad8 bytes) 000536 1788745513.316972 fallocate 0x02dbb3 thru 0x036880 (0x8cce bytes) 000537 1788745513.327760 read 0x01ff08 thru 0x02221f (0x2318 bytes) 000538 1788745513.328921 mapwrite 0x024e3d thru 0x02ac8b (0x5e4f bytes) 000539 1788745513.406600 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36881): Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36881): Mirror 11: 0xc2be5f69 (stale) Mirror 13: 0x3ac97283 (stale) CRC-32 checksum value for chunk [0, 0x36881): Mirror 1: 0xfac8d9c5 Mirror 14: 0xfac8d9c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36881: skipped 000540 1788745514.609185 trunc from 0x0219d8 to 0x02306d (0x1696 bytes) 000541 1788745514.666237 MIRROR_RESYNC 4 mirrors 000542 1788745515.781716 punch from 0x033a2d to 0x036880 (0x2e54 bytes) 000543 1788745515.831049 MIRROR_RESYNC 4 mirrors 000544 1788745517.199806 fallocate 0x02e654 thru 0x037936 (0x92e3 bytes) 000545 1788745517.232885 fallocate 0x01fb95 thru 0x021a13 (0x1e7f bytes) 000546 1788745517.238161 trunc from 0x0266d4 to 0x0289b1 (0x22de bytes) 000547 1788745517.243513 fallocate 0x015950 thru 0x016c1a (0x12cb bytes) 000548 1788745517.248424 write 0x0106ac thru 0x017555 (0x6eaa bytes) 000549 1788745517.258776 punch from 0x016462 to 0x01b892 (0x5431 bytes) 000550 1788745517.271635 punch from 0x016c33 to 0x021099 (0xa467 bytes) 000551 1788745517.282751 MIRROR_RESYNC 4 mirrors 000552 1788745518.661287 MIRROR_EXTEND to 5 mirrors 000553 1788745519.942151 read 0x03725f thru 0x037936 (0x6d8 bytes) 000554 1788745519.951120 fallocate 0x01723d thru 0x019bf8 (0x29bc bytes) 000555 1788745519.988806 mapread 0x021a1c thru 0x022532 (0xb17 bytes) 000556 1788745520.000231 fallocate 0x00b4ea thru 0x0176a0 (0xc1b7 bytes) 000557 1788745520.006196 punch from 0x02fc2f to 0x0305de (0x9b0 bytes) 000558 1788745520.016462 write 0x01ff11 thru 0x0245df (0x46cf bytes) 000559 1788745520.039485 trunc from 0x036d70 to 0x037936 (0xbc7 bytes) 000560 1788745520.056865 punch from 0x016b4e to 0x01e125 (0x75d8 bytes) 000561 1788745520.062797 read 0x001c42 thru 0x00ea40 (0xcdff bytes) 000562 1788745520.075359 punch from 0x0003ea to 0x0025d7 (0x21ee bytes) 000564 1788745520.080977 mapwrite 0x023efa thru 0x02cd9c (0x8ea3 bytes) 000565 1788745520.103322 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37937): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37937): Mirror 11: 0xbc678216 (stale) Mirror 13: 0xbc678216 (stale) Mirror 14: 0xbc678216 (stale) Mirror 15: 0xbc678216 (stale) CRC-32 checksum value for chunk [0, 0x37937): Mirror 1: 0xcf4c7502 Mirror 16: 0xcf4c7502 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37937: skipped 000566 1788745521.319357 write 0x021542 thru 0x0272bf (0x5d7e bytes) 000567 1788745521.394436 trunc from 0x037937 to 0x0117bd (0x26179 bytes) 000568 1788745521.451178 trunc from 0x0117be to 0x0095d7 (0x81e6 bytes) 000569 1788745521.472443 write 0x035ebd thru 0x03b15c (0x52a0 bytes) 000570 1788745521.475084 write 0x0226fc thru 0x02accb (0x85d0 bytes) 000571 1788745521.493607 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b15d): Mirror 13: 0x783150a4 (stale) Mirror 14: 0x783150a4 (stale) Mirror 15: 0x783150a4 (stale) Mirror 16: 0x6bb52a12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b15d): Mirror 13: 0x783150a4 (stale) Mirror 14: 0x783150a4 (stale) Mirror 15: 0x783150a4 (stale) Mirror 16: 0x6bb52a12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b15d: skipped 000572 1788745522.201061 trunc from 0x0158fb to 0x0245e9 (0xecef bytes) 000573 1788745522.219029 write 0x014294 thru 0x01a622 (0x638f bytes) 000574 1788745522.243206 mapwrite 0x011c54 thru 0x01d09d (0xb44a bytes) 000575 1788745522.302042 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b15d): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b15d): Mirror 13: 0x783150a4 (stale) Mirror 14: 0x783150a4 (stale) Mirror 15: 0x783150a4 (stale) Mirror 16: 0x6bb52a12 (stale) CRC-32 checksum value for chunk [0, 0x3b15d): Mirror 1: 0x7c5a45b0 Mirror 17: 0x7c5a45b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b15d: skipped 000576 1788745523.404610 punch from 0x0222e8 to 0x025daf (0x3ac8 bytes) 000577 1788745523.457535 fallocate 0x03261a thru 0x038ce0 (0x66c7 bytes) 000579 1788745523.466330 mapread 0x003395 thru 0x0107d7 (0xd443 bytes) 000581 1788745523.507317 trunc from 0x01b817 to 0x01ef66 (0x3750 bytes) 000582 1788745523.515565 read 0x00bea7 thru 0x013949 (0x7aa3 bytes) 000583 1788745523.534493 mapread 0x0221a7 thru 0x0229ee (0x848 bytes) 000584 1788745523.551167 read 0x0274b5 thru 0x036e29 (0xf975 bytes) 000585 1788745523.571625 mapread 0x01c37a thru 0x023b54 (0x77db bytes) 000586 1788745523.574160 punch from 0x025150 to 0x0325dd (0xd48e bytes) 000587 1788745523.585187 write 0x03c3cc thru 0x03ffff (0x3c34 bytes) 000588 1788745523.603724 write 0x032b02 thru 0x03a884 (0x7d83 bytes) 000589 1788745523.620923 write 0x0136e1 thru 0x023034 (0xf954 bytes) 000591 1788745523.644152 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xebf5ffb8 (stale) Mirror 14: 0xebf5ffb8 (stale) Mirror 15: 0xebf5ffb8 (stale) Mirror 16: 0x6dbc0fd2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xebf5ffb8 (stale) Mirror 14: 0xebf5ffb8 (stale) Mirror 15: 0xebf5ffb8 (stale) Mirror 16: 0x6dbc0fd2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000592 1788745524.360800 trunc from 0x040000 to 0x015700 (0x2a8ff bytes) 000593 1788745524.455189 write 0x027a27 thru 0x029c48 (0x2222 bytes) 000594 1788745524.463128 fallocate 0x006d22 thru 0x00e865 (0x7b44 bytes) 000595 1788745524.470930 read 0x0112c0 thru 0x01177e (0x4bf bytes) 000596 1788745524.480377 punch from 0x00ab6b to 0x01ab0b (0xffa1 bytes) 000597 1788745524.496401 punch from 0x0282f4 to 0x029c48 (0x1955 bytes) 000598 1788745524.516145 read 0x01dfcb thru 0x029c48 (0xbc7e bytes) 000599 1788745524.565426 punch from 0x01a65f to 0x027f78 (0xd91a bytes) 000600 1788745524.580389 mapread 0x008862 thru 0x00ccb0 (0x444f bytes) 000602 1788745524.584219 write 0x018849 thru 0x01fb97 (0x734f bytes) 000603 1788745524.595475 write 0x011fd9 thru 0x018007 (0x602f bytes) 000604 1788745524.602951 trunc from 0x008af6 to 0x0168a3 (0xddae bytes) 000605 1788745524.626438 fallocate 0x0046d7 thru 0x0112ae (0xcbd8 bytes) 000606 1788745524.634625 read 0x003d95 thru 0x008ee6 (0x5152 bytes) 000608 1788745524.657229 punch from 0x013bee to 0x014ec1 (0x12d4 bytes) 000609 1788745524.663380 trunc from 0x029c49 to 0x008430 (0x21818 bytes) 000610 1788745524.689391 mapread 0x002f79 thru 0x008430 (0x54b8 bytes) 000611 1788745524.693598 fallocate 0x0147b1 thru 0x01e69f (0x9eef bytes) 000612 1788745524.706987 write 0x026ccc thru 0x02fa23 (0x8d58 bytes) 000613 1788745524.709268 mapread 0x00e136 thru 0x01617a (0x8045 bytes) 000614 1788745524.843118 trunc from 0x02fa24 to 0x0254e2 (0xa541 bytes) 000615 1788745524.882754 read 0x01cfbf thru 0x025472 (0x84b4 bytes) 000616 1788745524.900327 trunc from 0x0003ad to 0x00f977 (0xf5cb bytes) 000617 1788745524.912250 mapwrite 0x0142a9 thru 0x01846e (0x41c6 bytes) 000618 1788745524.936537 mapwrite 0x00c3ed thru 0x01820e (0xbe22 bytes) 000619 1788745525.031571 punch from 0x017b71 to 0x0254e2 (0xd972 bytes) 000620 1788745525.043623 punch from 0x00b566 to 0x01a818 (0xf2b3 bytes) 000621 1788745525.056549 write 0x018804 thru 0x019129 (0x926 bytes) 000622 1788745525.073096 fallocate 0x0146be thru 0x02011a (0xba5d bytes) 000623 1788745525.101051 MIRROR_RESYNC 5 mirrors 000624 1788745526.693485 trunc from 0x01ae7b to 0x01f5ae (0x4734 bytes) 000625 1788745526.747119 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x254e3): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x254e3): Mirror 13: 0xe514408a (stale) Mirror 14: 0xe514408a (stale) Mirror 15: 0xe514408a (stale) Mirror 16: 0xe514408a (stale) CRC-32 checksum value for chunk [0, 0x254e3): Mirror 1: 0xe514408a Mirror 17: 0xe514408a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x254e3: skipped 000626 1788745527.850370 write 0x03e10f thru 0x03ffff (0x1ef1 bytes) 000627 1788745527.893174 mapread 0x030abe thru 0x03d321 (0xc864 bytes) 000628 1788745527.915941 mapread 0x02589d thru 0x02d6cc (0x7e30 bytes) 000629 1788745527.928075 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3f68b6ed (stale) Mirror 15: 0x3f68b6ed (stale) Mirror 16: 0x3f68b6ed (stale) Mirror 17: 0x3f68b6ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3f68b6ed (stale) Mirror 15: 0x3f68b6ed (stale) Mirror 16: 0x3f68b6ed (stale) Mirror 17: 0x3f68b6ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1788745528.254574 trunc from 0x002844 to 0x00b79e (0x8f5b bytes) 000631 1788745528.263985 MIRROR_RESYNC 5 mirrors 000632 1788745529.236822 mapread 0x02df4a thru 0x03bb96 (0xdc4d bytes) 000633 1788745529.261000 write 0x011a2a thru 0x01cc86 (0xb25d bytes) 000634 1788745529.293287 mapread 0x020050 thru 0x022517 (0x24c8 bytes) 000635 1788745529.308196 mapwrite 0x012474 thru 0x01742a (0x4fb7 bytes) 000636 1788745529.324108 mapwrite 0x025d5a thru 0x035627 (0xf8ce bytes) 000637 1788745529.360599 write 0x023321 thru 0x02a915 (0x75f5 bytes) 000638 1788745529.361936 trunc from 0x00f9b6 to 0x010326 (0x971 bytes) 000639 1788745529.368040 mapwrite 0x010ca5 thru 0x012eb6 (0x2212 bytes) 000640 1788745529.395224 trunc from 0x00cef1 to 0x017311 (0xa421 bytes) 000642 1788745529.408744 mapread 0x00a113 thru 0x011dbd (0x7cab bytes) 000643 1788745529.423454 mapread 0x006450 thru 0x013f58 (0xdb09 bytes) 000644 1788745529.424409 write 0x010202 thru 0x014d64 (0x4b63 bytes) 000645 1788745529.425438 mapwrite 0x00f6e3 thru 0x01ccc1 (0xd5df bytes) 000646 1788745529.461418 mapread 0x013e83 thru 0x0170bf (0x323d bytes) 000647 1788745529.468323 punch from 0x00ab1f to 0x010c60 (0x6142 bytes) 000649 1788745529.482039 fallocate 0x03cdd0 thru 0x03ffff (0x3230 bytes) 000650 1788745529.490976 read 0x012075 thru 0x019508 (0x7494 bytes) 000651 1788745529.491392 read 0x00db7c thru 0x010772 (0x2bf7 bytes) 000653 1788745529.513975 mapwrite 0x027c33 thru 0x02868c (0xa5a bytes) 000654 1788745529.528537 trunc from 0x040000 to 0x02cbb3 (0x1344c bytes) 000656 1788745529.559089 trunc from 0x01be13 to 0x022f9b (0x7189 bytes) 000657 1788745529.571629 write 0x03c6e3 thru 0x03ffff (0x391d bytes) 000658 1788745529.574916 write 0x01639d thru 0x0242fa (0xdf5e bytes) 000659 1788745529.583040 mapwrite 0x015631 thru 0x020024 (0xa9f4 bytes) 000660 1788745529.617524 fallocate 0x007c6d thru 0x00a008 (0x239c bytes) 000661 1788745529.621183 trunc from 0x035076 to 0x036311 (0x129c bytes) 000662 1788745529.624575 write 0x004537 thru 0x007f05 (0x39cf bytes) 000663 1788745529.630454 MIRROR_RESYNC 5 mirrors 000664 1788745530.565100 trunc from 0x040000 to 0x028a55 (0x175aa bytes) 000665 1788745530.600563 fallocate 0x0046ef thru 0x008ab9 (0x43cb bytes) 000666 1788745530.606030 read 0x01b6ae thru 0x022c93 (0x75e6 bytes) 000667 1788745530.616761 read 0x00df5f thru 0x01dc45 (0xfce7 bytes) 000668 1788745530.631117 trunc from 0x028a56 to 0x03cf90 (0x1453b bytes) 000669 1788745530.655617 mapread 0x0311f3 thru 0x03cf90 (0xbd9e bytes) 000670 1788745530.673639 mapread 0x00ece2 thru 0x0188f4 (0x9c13 bytes) 000671 1788745530.676181 trunc from 0x03cf91 to 0x02ca3a (0x10556 bytes) 000672 1788745530.695292 mapwrite 0x021697 thru 0x02edb7 (0xd721 bytes) 000673 1788745530.761544 mapwrite 0x01ab68 thru 0x023e8e (0x9327 bytes) 000674 1788745530.778041 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: 0xa452e8ac (stale) Mirror 16: 0xa452e8ac (stale) Mirror 17: 0xa452e8ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: 0xa452e8ac (stale) Mirror 16: 0xa452e8ac (stale) Mirror 17: 0xa452e8ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edb8: skipped 000675 1788745531.007979 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: 0xa452e8ac (stale) Mirror 16: 0xa452e8ac (stale) Mirror 17: 0xa452e8ac (stale) CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 1: 0xd547cd69 Mirror 18: 0xd547cd69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edb8: skipped 000676 1788745531.854979 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: 0xa452e8ac (stale) Mirror 16: 0xa452e8ac (stale) Mirror 17: 0xa452e8ac (stale) CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 1: 0xd547cd69 Mirror 18: 0xd547cd69 Mirror 19: 0xd547cd69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edb8: skipped 000677 1788745533.579128 fallocate 0x005abf thru 0x011c52 (0xc194 bytes) 000678 1788745533.621088 trunc from 0x01cb3a to 0x01e9cd (0x1e94 bytes) 000679 1788745533.627257 fallocate 0x00151e thru 0x003ce2 (0x27c5 bytes) 000680 1788745533.634193 read 0x02e29a thru 0x02edb7 (0xb1e bytes) 000681 1788745533.653057 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 14: 0xa452e8ac (stale) Mirror 16: 0xa452e8ac (stale) Mirror 17: 0xa452e8ac (stale) Mirror 18: 0xd547cd69 (stale) Mirror 19: 0xd547cd69 (stale) CRC-32 checksum value for chunk [0, 0x2edb8): Mirror 1: 0xd33ba6db Mirror 20: 0xd33ba6db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edb8: skipped 000682 1788745534.764060 read 0x0276ef thru 0x02b7d6 (0x40e8 bytes) 000683 1788745534.784742 mapwrite 0x009428 thru 0x00c7d6 (0x33af bytes) 000684 1788745534.936926 punch from 0x024612 to 0x029fa6 (0x5995 bytes) 000685 1788745534.954462 fallocate 0x03cb38 thru 0x03ffff (0x34c8 bytes) 000686 1788745534.974385 fallocate 0x014fdd thru 0x0205e1 (0xb605 bytes) 000687 1788745534.985784 trunc from 0x02edb8 to 0x03252e (0x3777 bytes) 000688 1788745535.036144 mapread 0x02a569 thru 0x03252e (0x7fc6 bytes) 000689 1788745535.063499 fallocate 0x0206d2 thru 0x02d22b (0xcb5a bytes) 000690 1788745535.073715 trunc from 0x03252f to 0x029f38 (0x85f6 bytes) 000693 1788745535.140093 read 0x022850 thru 0x029f38 (0x76e9 bytes) 000694 1788745535.155148 trunc from 0x029f39 to 0x03cdfd (0x12ec5 bytes) 000695 1788745535.189309 fallocate 0x00ffe9 thru 0x01c440 (0xc458 bytes) 000696 1788745535.204942 read 0x00b944 thru 0x019726 (0xdde3 bytes) 000697 1788745535.205597 read 0x01b807 thru 0x01d619 (0x1e13 bytes) 000698 1788745535.215463 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cdfe): Mirror 16: 0xea456110 (stale) Mirror 17: 0xea456110 (stale) Mirror 18: 0xc22fb5af (stale) Mirror 19: 0xc22fb5af (stale) Mirror 20: 0x6a11ec9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdfe): Mirror 16: 0xea456110 (stale) Mirror 17: 0xea456110 (stale) Mirror 18: 0xc22fb5af (stale) Mirror 19: 0xc22fb5af (stale) Mirror 20: 0x6a11ec9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdfe: skipped 000699 1788745536.101033 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cdfe): Mirror 16: 0xea456110 (stale) Mirror 17: 0xea456110 (stale) Mirror 18: 0xc22fb5af (stale) Mirror 20: 0x6a11ec9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdfe): Mirror 16: 0xea456110 (stale) Mirror 17: 0xea456110 (stale) Mirror 18: 0xc22fb5af (stale) Mirror 20: 0x6a11ec9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdfe: skipped 000700 1788745536.957164 punch from 0x011810 to 0x01656c (0x4d5d bytes) 000701 1788745536.965784 trunc from 0x03cdfe to 0x03b9d7 (0x1426 bytes) 000702 1788745536.994056 mapwrite 0x003c17 thru 0x00ffdc (0xc3c6 bytes) 000703 1788745537.063407 fallocate 0x025416 thru 0x02a94d (0x5538 bytes) 000704 1788745537.091196 fallocate 0x02dfd1 thru 0x03d710 (0xf740 bytes) 000705 1788745537.099974 trunc from 0x026350 to 0x02a2f4 (0x3fa5 bytes) 000706 1788745537.115108 write 0x012353 thru 0x021397 (0xf045 bytes) 000707 1788745537.136200 mapread 0x024eda thru 0x0270b1 (0x21d8 bytes) 000708 1788745537.150794 trunc from 0x03123d to 0x03bb6e (0xa932 bytes) 000709 1788745537.164185 mapread 0x012075 thru 0x014d9d (0x2d29 bytes) 000710 1788745537.175339 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d711): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d711): Mirror 16: 0x84f9dba1 (stale) Mirror 17: 0x84f9dba1 (stale) Mirror 18: 0x56204c1c (stale) Mirror 20: 0x100689dd (stale) CRC-32 checksum value for chunk [0, 0x3d711): Mirror 1: 0x4c640c85 Mirror 21: 0x4c640c85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d711: skipped 000711 1788745538.440435 trunc from 0x03d711 to 0x005511 (0x381ff bytes) 000712 1788745538.600222 trunc from 0x0012c6 to 0x005511 (0x424c bytes) 000715 1788745538.612629 trunc from 0x002be8 to 0x005511 (0x292a bytes) 000716 1788745538.633212 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x5512): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5512): Mirror 16: 0x61a95d55 (stale) Mirror 17: 0x61a95d55 (stale) Mirror 18: 0x61a95d55 (stale) Mirror 20: 0x61a95d55 (stale) Mirror 21: 0xe992ec14 (stale) CRC-32 checksum value for chunk [0, 0x5512): Mirror 1: 0xd5f7fd35 Mirror 22: 0xd5f7fd35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5512: skipped 000717 1788745540.278481 trunc from 0x003273 to 0x005511 (0x229f bytes) 000718 1788745540.321963 fallocate 0x0225cf thru 0x02ee5d (0xc88f bytes) 000719 1788745540.337074 write 0x004c2a thru 0x0088ff (0x3cd6 bytes) 000720 1788745540.359464 fallocate 0x035ea8 thru 0x03ace5 (0x4e3e bytes) 000721 1788745540.364815 mapwrite 0x009b41 thru 0x00fa13 (0x5ed3 bytes) 000722 1788745540.457926 mapread 0x01fabe thru 0x02c873 (0xcdb6 bytes) 000723 1788745540.514547 trunc from 0x034fd2 to 0x03635a (0x1389 bytes) 000725 1788745540.525651 write 0x00e9e8 thru 0x01b9eb (0xd004 bytes) 000726 1788745540.527423 fallocate 0x017388 thru 0x01fcb4 (0x892d bytes) 000727 1788745540.539411 trunc from 0x02cbbb to 0x03ab1d (0xdf63 bytes) 000728 1788745540.556035 write 0x01b13f thru 0x0258a9 (0xa76b bytes) 000729 1788745540.565035 MIRROR_RESYNC 7 mirrors 000730 1788745542.475904 trunc from 0x03ace6 to 0x029b98 (0x1114d bytes) 000731 1788745542.606367 trunc from 0x00e574 to 0x00ea8f (0x51c bytes) 000732 1788745542.633963 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29b99): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b99): Mirror 16: 0x1daf12ca (stale) Mirror 17: 0x1daf12ca (stale) Mirror 18: 0x1daf12ca (stale) Mirror 20: 0x1daf12ca (stale) Mirror 21: 0x1daf12ca (stale) Mirror 22: 0x1daf12ca (stale) CRC-32 checksum value for chunk [0, 0x29b99): Mirror 1: 0xc8a15c97 Mirror 23: 0xc8a15c97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b99: skipped 000733 1788745544.416386 write 0x01d724 thru 0x0272e6 (0x9bc3 bytes) 000734 1788745544.481779 write 0x00e498 thru 0x010c56 (0x27bf bytes) 000735 1788745544.495781 mapread 0x023afe thru 0x029b98 (0x609b bytes) 000736 1788745544.514617 mapread 0x016a3c thru 0x02041e (0x99e3 bytes) 000737 1788745544.528516 punch from 0x013026 to 0x0167df (0x37ba bytes) 000738 1788745544.535705 trunc from 0x029b99 to 0x026afe (0x309a bytes) 000739 1788745544.567842 mapwrite 0x03aa9b thru 0x03e77e (0x3ce4 bytes) 000740 1788745544.622195 mapread 0x026e98 thru 0x02fcb8 (0x8e21 bytes) 000741 1788745544.638184 read 0x03ca1e thru 0x03e77e (0x1d61 bytes) 000743 1788745544.639272 punch from 0x034fc5 to 0x03b24f (0x628b bytes) 000744 1788745544.645979 write 0x035f41 thru 0x03c904 (0x69c4 bytes) 000745 1788745544.653650 punch from 0x01f293 to 0x027eb8 (0x8c26 bytes) 000746 1788745544.660636 mapwrite 0x026a9f thru 0x032b05 (0xc067 bytes) 000747 1788745544.716784 mapwrite 0x03cc8c thru 0x03ffff (0x3374 bytes) 000748 1788745544.767035 punch from 0x0070ab to 0x0135fc (0xc552 bytes) 000749 1788745544.773081 read 0x02b3e9 thru 0x0306ff (0x5317 bytes) 000750 1788745544.774271 trunc from 0x040000 to 0x0374a5 (0x8b5a bytes) 000751 1788745544.797581 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x374a6): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374a6): Mirror 16: 0x81613abe (stale) Mirror 17: 0x81613abe (stale) Mirror 18: 0x81613abe (stale) Mirror 20: 0x81613abe (stale) Mirror 21: 0x81613abe (stale) Mirror 22: 0x81613abe (stale) Mirror 23: 0x2471214c (stale) CRC-32 checksum value for chunk [0, 0x374a6): Mirror 1: 0x66bbd9d8 Mirror 24: 0x66bbd9d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374a6: skipped 000752 1788745546.585069 trunc from 0x0374a6 to 0x0058a6 (0x31bff bytes) 000753 1788745546.750782 MIRROR_RESYNC 9 mirrors 000754 1788745548.360225 mapwrite 0x03845f thru 0x03ffff (0x7ba1 bytes) 000755 1788745548.468198 punch from 0x01dbea to 0x02bc56 (0xe06d bytes) 000759 1788745548.476209 write 0x015c4b thru 0x02492c (0xece2 bytes) 000760 1788745548.489011 read 0x0096a3 thru 0x00df43 (0x48a1 bytes) 000761 1788745548.497236 mapread 0x00a613 thru 0x00becf (0x18bd bytes) 000762 1788745548.498502 read 0x034109 thru 0x03fadf (0xb9d7 bytes) 000763 1788745548.500824 read 0x02d8a7 thru 0x02dbf5 (0x34f bytes) 000764 1788745548.518631 fallocate 0x019571 thru 0x027f3a (0xe9ca bytes) 000765 1788745548.524320 trunc from 0x040000 to 0x00c1a3 (0x33e5c bytes) 000766 1788745548.549601 trunc from 0x003fb4 to 0x004cd2 (0xd1f bytes) 000767 1788745548.555848 fallocate 0x033c96 thru 0x037f55 (0x42c0 bytes) 000768 1788745548.565119 mapwrite 0x022b5c thru 0x02f6bc (0xcb61 bytes) 000770 1788745548.674869 trunc from 0x00f15f to 0x01df78 (0xee1a bytes) 000771 1788745548.685260 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f56): Mirror 16: 0xdc5274d1 (stale) Mirror 17: 0xdc5274d1 (stale) Mirror 18: 0xdc5274d1 (stale) Mirror 20: 0xdc5274d1 (stale) Mirror 21: 0xdc5274d1 (stale) Mirror 22: 0xdc5274d1 (stale) Mirror 23: 0xdc5274d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f56): Mirror 16: 0xdc5274d1 (stale) Mirror 17: 0xdc5274d1 (stale) Mirror 18: 0xdc5274d1 (stale) Mirror 20: 0xdc5274d1 (stale) Mirror 21: 0xdc5274d1 (stale) Mirror 22: 0xdc5274d1 (stale) Mirror 23: 0xdc5274d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f56: skipped 000772 1788745549.268923 write 0x0129ba thru 0x014e19 (0x2460 bytes) 000773 1788745549.278211 fallocate 0x001736 thru 0x009c76 (0x8541 bytes) 000774 1788745549.281459 write 0x03cf8f thru 0x03ffff (0x3071 bytes) 000775 1788745549.284099 mapwrite 0x006409 thru 0x009cf1 (0x38e9 bytes) 000776 1788745549.362829 punch from 0x03575e to 0x03ffff (0xa8a2 bytes) 000777 1788745549.366850 mapwrite 0x0383c5 thru 0x03e69c (0x62d8 bytes) 000778 1788745549.394163 mapread 0x00ad97 thru 0x00c8c3 (0x1b2d bytes) 000779 1788745549.394837 fallocate 0x034df1 thru 0x03d12d (0x833d bytes) 000780 1788745549.403475 read 0x02c75d thru 0x0356d2 (0x8f76 bytes) 000781 1788745549.414267 write 0x020ed0 thru 0x025038 (0x4169 bytes) 000782 1788745549.430135 fallocate 0x019fe6 thru 0x01d0eb (0x3106 bytes) 000783 1788745549.456266 write 0x033a71 thru 0x0344b7 (0xa47 bytes) 000785 1788745549.466596 MIRROR_RESYNC 8 mirrors 000786 1788745552.111353 trunc from 0x040000 to 0x03db6d (0x2492 bytes) 000787 1788745552.198572 trunc from 0x034815 to 0x03db6d (0x9359 bytes) 000788 1788745552.208868 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db6e): Mirror 16: 0xe4e2e7a5 (stale) Mirror 17: 0xe4e2e7a5 (stale) Mirror 18: 0xe4e2e7a5 (stale) Mirror 20: 0xe4e2e7a5 (stale) Mirror 21: 0xe4e2e7a5 (stale) Mirror 22: 0xe4e2e7a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db6e): Mirror 16: 0xe4e2e7a5 (stale) Mirror 17: 0xe4e2e7a5 (stale) Mirror 18: 0xe4e2e7a5 (stale) Mirror 20: 0xe4e2e7a5 (stale) Mirror 21: 0xe4e2e7a5 (stale) Mirror 22: 0xe4e2e7a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db6e: skipped 000790 1788745552.910283 punch from 0x003261 to 0x010f3c (0xdcdc bytes) 000791 1788745552.924477 trunc from 0x0337c6 to 0x03db6d (0xa3a8 bytes) 000792 1788745552.946064 punch from 0x00e453 to 0x011005 (0x2bb3 bytes) 000793 1788745552.954095 trunc from 0x028578 to 0x030287 (0x7d10 bytes) 000794 1788745552.961256 read 0x035ac8 thru 0x03db6d (0x80a6 bytes) 000795 1788745552.991117 trunc from 0x03db6e to 0x039a0f (0x415e bytes) 000796 1788745553.034606 trunc from 0x00dbd1 to 0x00fe7d (0x22ad bytes) 000797 1788745553.061770 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39a10): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a10): Mirror 16: 0x38a4d48d (stale) Mirror 17: 0x38a4d48d (stale) Mirror 18: 0x38a4d48d (stale) Mirror 20: 0x38a4d48d (stale) Mirror 21: 0x38a4d48d (stale) Mirror 22: 0x38a4d48d (stale) CRC-32 checksum value for chunk [0, 0x39a10): Mirror 1: 0x61051fe1 Mirror 23: 0x61051fe1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a10: skipped 000798 1788745555.336726 trunc from 0x017e49 to 0x02408a (0xc242 bytes) 000799 1788745555.475837 mapwrite 0x033508 thru 0x038066 (0x4b5f bytes) 000800 1788745555.567456 fallocate 0x024c16 thru 0x02cca7 (0x8092 bytes) 000801 1788745555.572823 write 0x03b986 thru 0x03ffff (0x467a bytes) 000802 1788745555.587356 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xe4ba7012 (stale) Mirror 17: 0xe4ba7012 (stale) Mirror 18: 0xe4ba7012 (stale) Mirror 20: 0xe4ba7012 (stale) Mirror 21: 0xe4ba7012 (stale) Mirror 22: 0xe4ba7012 (stale) Mirror 23: 0x62d10f2c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb5272dc Mirror 24: 0xbb5272dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000804 1788745556.997592 mapread 0x03a202 thru 0x03ffff (0x5dfe bytes) 000805 1788745557.053957 fallocate 0x023d8f thru 0x02eb6f (0xade1 bytes) 000807 1788745557.157295 mapwrite 0x028808 thru 0x02d110 (0x4909 bytes) 000808 1788745557.229180 mapwrite 0x035069 thru 0x03b611 (0x65a9 bytes) 000809 1788745557.318227 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xe4ba7012 (stale) Mirror 17: 0xe4ba7012 (stale) Mirror 18: 0xe4ba7012 (stale) Mirror 20: 0xe4ba7012 (stale) Mirror 21: 0xe4ba7012 (stale) Mirror 22: 0xe4ba7012 (stale) Mirror 23: 0x62d10f2c (stale) Mirror 24: 0xbb5272dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4cbdf86b Mirror 25: 0x4cbdf86b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1788745559.702999 punch from 0x01e07f to 0x02d7cb (0xf74d bytes) 000811 1788745559.776168 mapread 0x01179c thru 0x0185fa (0x6e5f bytes) 000812 1788745559.800634 trunc from 0x040000 to 0x0343ba (0xbc45 bytes) 000813 1788745559.839917 trunc from 0x0343bb to 0x027037 (0xd383 bytes) 000814 1788745559.859303 trunc from 0x010da1 to 0x0179de (0x6c3e bytes) 000816 1788745559.871891 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27038): Mirror 16: 0x74990bad (stale) Mirror 17: 0x74990bad (stale) Mirror 20: 0x74990bad (stale) Mirror 21: 0x74990bad (stale) Mirror 22: 0x74990bad (stale) Mirror 23: 0x518093a0 (stale) Mirror 24: 0xa1ef1ab4 (stale) Mirror 25: 0xa1ef1ab4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27038): Mirror 16: 0x74990bad (stale) Mirror 17: 0x74990bad (stale) Mirror 20: 0x74990bad (stale) Mirror 21: 0x74990bad (stale) Mirror 22: 0x74990bad (stale) Mirror 23: 0x518093a0 (stale) Mirror 24: 0xa1ef1ab4 (stale) Mirror 25: 0xa1ef1ab4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27038: skipped 000818 1788745560.405951 trunc from 0x01b849 to 0x027037 (0xb7ef bytes) 000819 1788745560.411160 write 0x007149 thru 0x007b32 (0x9ea bytes) 000820 1788745560.424318 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27038): Mirror 16: 0x74990bad (stale) Mirror 20: 0x74990bad (stale) Mirror 21: 0x74990bad (stale) Mirror 22: 0x74990bad (stale) Mirror 23: 0x518093a0 (stale) Mirror 24: 0xa1ef1ab4 (stale) Mirror 25: 0xa1ef1ab4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27038): Mirror 16: 0x74990bad (stale) Mirror 20: 0x74990bad (stale) Mirror 21: 0x74990bad (stale) Mirror 22: 0x74990bad (stale) Mirror 23: 0x518093a0 (stale) Mirror 24: 0xa1ef1ab4 (stale) Mirror 25: 0xa1ef1ab4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27038: skipped 000821 1788745560.936974 read 0x014620 thru 0x01c6e7 (0x80c8 bytes) 000822 1788745560.947876 mapwrite 0x033f36 thru 0x037e99 (0x3f64 bytes) 000823 1788745561.041947 trunc from 0x016f10 to 0x020d97 (0x9e88 bytes) 000825 1788745561.048996 trunc from 0x025328 to 0x035086 (0xfd5f bytes) 000826 1788745561.055341 mapwrite 0x00ae75 thru 0x01668c (0xb818 bytes) 000827 1788745561.110103 trunc from 0x037e9a to 0x03fc54 (0x7dbb bytes) 000828 1788745561.144055 write 0x03019d thru 0x039d03 (0x9b67 bytes) 000829 1788745561.156946 read 0x00d3e7 thru 0x01722d (0x9e47 bytes) 000830 1788745561.157944 trunc from 0x03fc55 to 0x02553c (0x1a718 bytes) 000831 1788745561.184035 trunc from 0x02553d to 0x02973c (0x4200 bytes) 000832 1788745561.203091 mapread 0x015bf3 thru 0x020e70 (0xb27e bytes) 000833 1788745561.218576 trunc from 0x022c87 to 0x0244ad (0x1827 bytes) 000834 1788745561.225021 fallocate 0x033d98 thru 0x03b714 (0x797d bytes) 000836 1788745561.231882 read 0x0050ca thru 0x00fe49 (0xad80 bytes) 000837 1788745561.232930 trunc from 0x02973d to 0x028133 (0x1609 bytes) 000838 1788745561.259320 punch from 0x00800c to 0x008989 (0x97e bytes) 000839 1788745561.265069 write 0x02f939 thru 0x035eae (0x6576 bytes) 000840 1788745561.267905 mapwrite 0x03ff72 thru 0x03ffff (0x8e bytes) 000841 1788745561.315980 read 0x03feb3 thru 0x03ffff (0x14d bytes) 000842 1788745561.316986 trunc from 0x040000 to 0x005afc (0x3a503 bytes) 000843 1788745561.337418 read 0x00555b thru 0x005afc (0x5a2 bytes) 000844 1788745561.338249 fallocate 0x03e0a8 thru 0x03e29c (0x1f5 bytes) 000845 1788745561.349686 MIRROR_RESYNC 8 mirrors 000846 1788745562.619935 trunc from 0x003ffb to 0x005afc (0x1b02 bytes) 000847 1788745562.664549 mapwrite 0x01e57c thru 0x02606c (0x7af1 bytes) 000848 1788745562.737183 fallocate 0x00c373 thru 0x0140c7 (0x7d55 bytes) 000850 1788745562.749091 trunc from 0x022bb5 to 0x024f7d (0x23c9 bytes) 000851 1788745562.758569 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2606d): Mirror 20: 0xc3186bcd (stale) Mirror 21: 0xc3186bcd (stale) Mirror 22: 0xc3186bcd (stale) Mirror 23: 0xc3186bcd (stale) Mirror 24: 0xc3186bcd (stale) Mirror 25: 0xc3186bcd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2606d): Mirror 20: 0xc3186bcd (stale) Mirror 21: 0xc3186bcd (stale) Mirror 22: 0xc3186bcd (stale) Mirror 23: 0xc3186bcd (stale) Mirror 24: 0xc3186bcd (stale) Mirror 25: 0xc3186bcd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2606d: skipped 000852 1788745563.175333 read 0x01ca4e thru 0x02082a (0x3ddd bytes) 000853 1788745563.186315 mapwrite 0x0042ad thru 0x0101c0 (0xbf14 bytes) 000854 1788745563.242217 read 0x01083a thru 0x0118e9 (0x10b0 bytes) 000856 1788745563.254691 read 0x025231 thru 0x02606c (0xe3c bytes) 000857 1788745563.255730 fallocate 0x023932 thru 0x02fda1 (0xc470 bytes) 000858 1788745563.260924 trunc from 0x00c587 to 0x01a6ab (0xe125 bytes) 000859 1788745563.267509 mapread 0x008401 thru 0x016467 (0xe067 bytes) 000860 1788745563.288614 punch from 0x01233b to 0x01c083 (0x9d49 bytes) 000861 1788745563.295742 read 0x01bc15 thru 0x01dedb (0x22c7 bytes) 000862 1788745563.311162 read 0x02c5d1 thru 0x02fda1 (0x37d1 bytes) 000863 1788745563.311840 read 0x024acf thru 0x02cf8e (0x84c0 bytes) 000865 1788745563.319746 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fda2): Mirror 20: 0x958408c9 (stale) Mirror 22: 0x958408c9 (stale) Mirror 23: 0x958408c9 (stale) Mirror 24: 0x958408c9 (stale) Mirror 25: 0x958408c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fda2): Mirror 20: 0x958408c9 (stale) Mirror 22: 0x958408c9 (stale) Mirror 23: 0x958408c9 (stale) Mirror 24: 0x958408c9 (stale) Mirror 25: 0x958408c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fda2: skipped 000866 1788745563.632101 trunc from 0x02fda2 to 0x01aa48 (0x15359 bytes) 000867 1788745563.660463 write 0x033ec5 thru 0x03ffff (0xc13b bytes) 000868 1788745563.667527 mapread 0x016f9f thru 0x0171ec (0x24e bytes) 000869 1788745563.684031 fallocate 0x02f100 thru 0x0351e8 (0x60e9 bytes) 000871 1788745563.691845 write 0x00e85c thru 0x00f817 (0xfbc bytes) 000873 1788745563.709617 trunc from 0x040000 to 0x036b86 (0x9479 bytes) 000874 1788745563.731407 write 0x002fb3 thru 0x01294a (0xf998 bytes) 000875 1788745563.747344 mapwrite 0x0084e2 thru 0x00a10e (0x1c2d bytes) 000877 1788745563.771624 mapwrite 0x038adc thru 0x03efac (0x64d1 bytes) 000878 1788745563.836571 fallocate 0x00409b thru 0x006647 (0x25ad bytes) 000879 1788745563.846312 fallocate 0x026320 thru 0x030773 (0xa454 bytes) 000880 1788745563.857387 write 0x02b53c thru 0x03909d (0xdb62 bytes) 000881 1788745563.875645 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3efad): Mirror 20: 0x2e029295 (stale) Mirror 22: 0x2e029295 (stale) Mirror 24: 0x2e029295 (stale) Mirror 25: 0x2e029295 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efad): Mirror 20: 0x2e029295 (stale) Mirror 22: 0x2e029295 (stale) Mirror 24: 0x2e029295 (stale) Mirror 25: 0x2e029295 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efad: skipped 000882 1788745564.458188 mapwrite 0x0399ef thru 0x03be9f (0x24b1 bytes) 000883 1788745564.510012 read 0x01704d thru 0x01d1f2 (0x61a6 bytes) 000884 1788745564.520760 mapwrite 0x0104c6 thru 0x01b64c (0xb187 bytes) 000885 1788745564.549926 trunc from 0x03efad to 0x0243ab (0x1ac01 bytes) 000886 1788745564.572102 mapwrite 0x01bcc2 thru 0x01be20 (0x15f bytes) 000887 1788745564.592378 mapread 0x00ff43 thru 0x019729 (0x97e7 bytes) 000888 1788745564.594394 trunc from 0x021d74 to 0x0243ab (0x2638 bytes) 000889 1788745564.608255 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x243ac): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x243ac): Mirror 20: 0x17276f86 (stale) Mirror 22: 0x17276f86 (stale) Mirror 24: 0x17276f86 (stale) Mirror 25: 0x17276f86 (stale) CRC-32 checksum value for chunk [0, 0x243ac): Mirror 1: 0xe38862a4 Mirror 26: 0xe38862a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x243ac: skipped 000890 1788745565.675777 write 0x00249d thru 0x008564 (0x60c8 bytes) 000891 1788745565.765215 trunc from 0x0243ac to 0x031783 (0xd3d8 bytes) 000892 1788745565.797588 mapwrite 0x032001 thru 0x033a05 (0x1a05 bytes) 000893 1788745565.873280 trunc from 0x004c86 to 0x005acb (0xe46 bytes) 000895 1788745565.880766 trunc from 0x02d41d to 0x033268 (0x5e4c bytes) 000896 1788745565.902716 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a06): Mirror 22: 0xa072b84f (stale) Mirror 24: 0xa072b84f (stale) Mirror 25: 0xa072b84f (stale) Mirror 26: 0xa9f2e32d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a06): Mirror 22: 0xa072b84f (stale) Mirror 24: 0xa072b84f (stale) Mirror 25: 0xa072b84f (stale) Mirror 26: 0xa9f2e32d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a06: skipped 000898 1788745566.481962 MIRROR_RESYNC 5 mirrors 000900 1788745568.425496 trunc from 0x033a06 to 0x005fb0 (0x2da55 bytes) 000901 1788745568.513918 write 0x034597 thru 0x0369ae (0x2418 bytes) 000902 1788745568.516629 trunc from 0x0369af to 0x02c6d8 (0xa2d6 bytes) 000904 1788745568.546833 fallocate 0x01f4ac thru 0x024427 (0x4f7c bytes) 000907 1788745568.554496 trunc from 0x006829 to 0x007ae5 (0x12bd bytes) 000909 1788745568.561901 punch from 0x01218a to 0x0187a5 (0x661c bytes) 000910 1788745568.566961 read 0x01b3dc thru 0x023006 (0x7c2b bytes) 000911 1788745568.575517 read 0x006ff7 thru 0x00ed2b (0x7d35 bytes) 000912 1788745568.586580 mapread 0x01df61 thru 0x02c6d8 (0xe778 bytes) 000914 1788745568.588483 trunc from 0x02c6d9 to 0x03690b (0xa233 bytes) 000915 1788745568.604419 fallocate 0x03146e thru 0x03c25e (0xadf1 bytes) 000917 1788745568.612105 read 0x02c8cc thru 0x0348d3 (0x8008 bytes) 000918 1788745568.625552 trunc from 0x03c25f to 0x038536 (0x3d28 bytes) 000919 1788745568.647874 mapwrite 0x015074 thru 0x01e9f3 (0x9980 bytes) 000920 1788745568.666462 trunc from 0x017bed to 0x01f508 (0x791c bytes) 000921 1788745568.671385 punch from 0x0189ea to 0x0252bc (0xc8d3 bytes) 000922 1788745568.680926 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38537): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38537): Mirror 22: 0xe2f4869f (stale) Mirror 24: 0xe2f4869f (stale) Mirror 25: 0xe2f4869f (stale) Mirror 26: 0xe2f4869f (stale) CRC-32 checksum value for chunk [0, 0x38537): Mirror 1: 0xcedb4992 Mirror 27: 0xcedb4992 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38537: skipped 000923 1788745569.540643 fallocate 0x01b4ed thru 0x01f78f (0x42a3 bytes) 000924 1788745569.571490 mapread 0x01ab1f thru 0x0247ed (0x9ccf bytes) 000925 1788745569.593316 read 0x034aa4 thru 0x035a55 (0xfb2 bytes) 000926 1788745569.603434 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38537): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38537): Mirror 22: 0xe2f4869f (stale) Mirror 24: 0xe2f4869f (stale) Mirror 25: 0xe2f4869f (stale) Mirror 26: 0xe2f4869f (stale) Mirror 27: 0xcedb4992 (stale) CRC-32 checksum value for chunk [0, 0x38537): Mirror 1: 0xcedb4992 Mirror 28: 0xcedb4992 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38537: skipped 000927 1788745571.202418 mapread 0x0047f0 thru 0x004d5d (0x56e bytes) 000928 1788745571.256840 read 0x0379e4 thru 0x038536 (0xb53 bytes) 000929 1788745571.296600 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38537): Mirror 22: 0xe2f4869f (stale) Mirror 24: 0xe2f4869f (stale) Mirror 25: 0xe2f4869f (stale) Mirror 27: 0xcedb4992 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38537): Mirror 22: 0xe2f4869f (stale) Mirror 24: 0xe2f4869f (stale) Mirror 25: 0xe2f4869f (stale) Mirror 27: 0xcedb4992 (stale) CRC-32 checksum value for chunk [0, 0x38537): Mirror 1: 0xcedb4992 Mirror 28: 0xcedb4992 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38537: skipped 000930 1788745572.207989 mapread 0x0013d4 thru 0x003ed3 (0x2b00 bytes) 000931 1788745572.363584 MIRROR_RESYNC 6 mirrors 000932 1788745573.828417 fallocate 0x0379cf thru 0x03ffff (0x8631 bytes) 000933 1788745573.869292 fallocate 0x028347 thru 0x028752 (0x40c bytes) 000934 1788745573.876077 fallocate 0x017686 thru 0x025072 (0xd9ed bytes) 000935 1788745573.881235 fallocate 0x03ea6c thru 0x03ffff (0x1594 bytes) 000936 1788745573.889866 MIRROR_RESYNC 6 mirrors 000937 1788745575.576120 mapwrite 0x0148e7 thru 0x020ea0 (0xc5ba bytes) 000938 1788745575.685573 punch from 0x01f0ac to 0x02edf6 (0xfd4b bytes) 000939 1788745575.698310 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xeaaf7489 (stale) Mirror 24: 0xeaaf7489 (stale) Mirror 25: 0xeaaf7489 (stale) Mirror 27: 0xeaaf7489 (stale) Mirror 28: 0xeaaf7489 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x258897d4 Mirror 29: 0x258897d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000940 1788745576.763099 fallocate 0x00e1fc thru 0x016b97 (0x899c bytes) 000941 1788745576.811779 mapread 0x036a72 thru 0x03ffff (0x958e bytes) 000942 1788745576.831245 mapwrite 0x00aeb1 thru 0x00f89e (0x49ee bytes) 000943 1788745576.861494 read 0x030a01 thru 0x039bcb (0x91cb bytes) 000944 1788745576.874231 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xeaaf7489 (stale) Mirror 24: 0xeaaf7489 (stale) Mirror 25: 0xeaaf7489 (stale) Mirror 27: 0xeaaf7489 (stale) Mirror 28: 0xeaaf7489 (stale) Mirror 29: 0x258897d4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd4237cca Mirror 30: 0xd4237cca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000945 1788745578.145941 trunc from 0x040000 to 0x007150 (0x38eaf bytes) 000946 1788745578.226430 mapread 0x006c43 thru 0x007150 (0x50e bytes) 000947 1788745578.261148 mapwrite 0x001cfd thru 0x00ab9f (0x8ea3 bytes) 000948 1788745578.342980 fallocate 0x012e42 thru 0x01ceb2 (0xa071 bytes) 000949 1788745578.347747 mapwrite 0x0315ff thru 0x03e207 (0xcc09 bytes) 000950 1788745578.428368 fallocate 0x0256a6 thru 0x0311bb (0xbb16 bytes) 000951 1788745578.446618 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e208): Mirror 22: 0xf5975dc0 (stale) Mirror 25: 0xf5975dc0 (stale) Mirror 27: 0xf5975dc0 (stale) Mirror 28: 0xf5975dc0 (stale) Mirror 29: 0xfd13a8c4 (stale) Mirror 30: 0x12201d20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e208): Mirror 22: 0xf5975dc0 (stale) Mirror 25: 0xf5975dc0 (stale) Mirror 27: 0xf5975dc0 (stale) Mirror 28: 0xf5975dc0 (stale) Mirror 29: 0xfd13a8c4 (stale) Mirror 30: 0x12201d20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e208: skipped 000952 1788745579.089238 trunc from 0x03e208 to 0x0035b3 (0x3ac54 bytes) 000953 1788745579.113231 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35b4): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b4): Mirror 22: 0xd56718e3 (stale) Mirror 25: 0xd56718e3 (stale) Mirror 27: 0xd56718e3 (stale) Mirror 28: 0xd56718e3 (stale) Mirror 29: 0xd56718e3 (stale) Mirror 30: 0xd56718e3 (stale) CRC-32 checksum value for chunk [0, 0x35b4): Mirror 1: 0x6778e1e7 Mirror 31: 0x6778e1e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b4: skipped 000954 1788745580.054160 trunc from 0x001c8d to 0x0035b3 (0x1927 bytes) 000955 1788745580.133149 trunc from 0x0035b4 to 0x032708 (0x2f155 bytes) 000956 1788745580.178798 fallocate 0x02ea95 thru 0x02fb98 (0x1104 bytes) 000957 1788745580.184011 write 0x02bffa thru 0x03bdc9 (0xfdd0 bytes) 000958 1788745580.189453 write 0x00efa7 thru 0x01131f (0x2379 bytes) 000959 1788745580.201366 read 0x03283c thru 0x03bdc9 (0x958e bytes) 000960 1788745580.202056 punch from 0x03bc44 to 0x03bdc9 (0x186 bytes) 000961 1788745580.216125 write 0x03e048 thru 0x03ffff (0x1fb8 bytes) 000962 1788745580.217144 trunc from 0x040000 to 0x0358b3 (0xa74c bytes) 000963 1788745580.234247 mapread 0x025a25 thru 0x02e652 (0x8c2e bytes) 000964 1788745580.253257 fallocate 0x0232b1 thru 0x02d34f (0xa09f bytes) 000965 1788745580.258929 trunc from 0x0358b4 to 0x035968 (0xb5 bytes) 000966 1788745580.277050 write 0x030c65 thru 0x03ba1b (0xadb7 bytes) 000967 1788745580.283156 read 0x03a4b8 thru 0x03ba1b (0x1564 bytes) 000968 1788745580.291964 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba1c): Mirror 22: 0x284193b7 (stale) Mirror 25: 0x284193b7 (stale) Mirror 28: 0x284193b7 (stale) Mirror 29: 0xb593d93b (stale) Mirror 30: 0xa1c86ff7 (stale) Mirror 31: 0x29545e5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba1c): Mirror 22: 0x284193b7 (stale) Mirror 25: 0x284193b7 (stale) Mirror 28: 0x284193b7 (stale) Mirror 29: 0xb593d93b (stale) Mirror 30: 0xa1c86ff7 (stale) Mirror 31: 0x29545e5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba1c: skipped 000969 1788745580.816053 MIRROR_RESYNC 7 mirrors 000970 1788745583.037634 fallocate 0x005011 thru 0x00d79c (0x878c bytes) 000972 1788745583.096267 mapread 0x03af00 thru 0x03ba1b (0xb1c bytes) 000973 1788745583.111588 mapwrite 0x022611 thru 0x02f86c (0xd25c bytes) 000974 1788745583.169914 read 0x02971b thru 0x02e505 (0x4deb bytes) 000975 1788745583.170604 fallocate 0x00aba4 thru 0x010c0a (0x6067 bytes) 000976 1788745583.179351 trunc from 0x0078e2 to 0x014775 (0xce94 bytes) 000977 1788745583.185061 fallocate 0x00948b thru 0x00fc65 (0x67db bytes) 000978 1788745583.197035 trunc from 0x023b89 to 0x032b5f (0xefd7 bytes) 000979 1788745583.202982 trunc from 0x03ba1c to 0x00e7e2 (0x2d239 bytes) 000980 1788745583.233880 trunc from 0x00e7e3 to 0x00cc85 (0x1b5d bytes) 000981 1788745583.254149 write 0x0026fc thru 0x00c1c3 (0x9ac8 bytes) 000982 1788745583.263114 write 0x01f21b thru 0x0203a3 (0x1189 bytes) 000983 1788745583.264189 punch from 0x001a02 to 0x01180f (0xfe0e bytes) 000984 1788745583.277611 mapread 0x003cd6 thru 0x00b065 (0x7390 bytes) 000985 1788745583.295507 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x203a4): Mirror 22: 0x819b6481 (stale) Mirror 25: 0x819b6481 (stale) Mirror 28: 0x819b6481 (stale) Mirror 29: 0x819b6481 (stale) Mirror 31: 0x819b6481 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x203a4): Mirror 22: 0x819b6481 (stale) Mirror 25: 0x819b6481 (stale) Mirror 28: 0x819b6481 (stale) Mirror 29: 0x819b6481 (stale) Mirror 31: 0x819b6481 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x203a4: skipped 000988 1788745583.730283 MIRROR_RESYNC 6 mirrors 000989 1788745584.814092 MIRROR_RESYNC 6 mirrors 000990 1788745585.474156 mapwrite 0x00dc3b thru 0x01705b (0x9421 bytes) 000991 1788745585.726057 read 0x00d624 thru 0x015ea0 (0x887d bytes) 000992 1788745585.726974 mapwrite 0x0384d0 thru 0x03884a (0x37b bytes) 000993 1788745585.877592 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3884b): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3884b): Mirror 22: 0x83af27c4 (stale) Mirror 25: 0x83af27c4 (stale) Mirror 28: 0x83af27c4 (stale) Mirror 29: 0x83af27c4 (stale) Mirror 31: 0x83af27c4 (stale) CRC-32 checksum value for chunk [0, 0x3884b): Mirror 1: 0x2fefe310 Mirror 32: 0x2fefe310 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3884b: skipped 000994 1788745587.245107 read 0x0121a0 thru 0x017e11 (0x5c72 bytes) 000995 1788745587.267381 fallocate 0x016bf8 thru 0x0190ad (0x24b6 bytes) 000996 1788745587.313119 write 0x02f887 thru 0x03d6d1 (0xde4b bytes) 000997 1788745587.325109 trunc from 0x03d6d2 to 0x0085d1 (0x35100 bytes) 000998 1788745587.345101 mapwrite 0x01ae4d thru 0x023d4a (0x8efe bytes) 000999 1788745587.396253 trunc from 0x023d4b to 0x03b457 (0x1770d bytes) 001000 1788745587.414236 trunc from 0x002164 to 0x00c693 (0xa530 bytes) All operations completed A-OK! PASS 70a (131s) == sanity-flr test 71: check mirror extend parent fid ==== 21:46:33 (1788745593) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1ed:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 21:46:45 (1788745605) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158948 s, 13.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986503 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 21:46:54 (1788745614) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2545248 .. Extending file size to 1278560 .. failed Extending file size to 8080544 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8091744 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9343872 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7847712 .. Extending file size to 8361248 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8732832 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61451 61453 61455 61456 61458 failed 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:48:14 (1788745694) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 300896 .. Extending file size to 3952832 .. done Extending file size to 7020640 .. Extending file size to 5511200 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7336416 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8281504 .. Extending file size to 9099360 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8607360 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62995 62997 62999 63000 63001 failed 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) 192.168.201.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:49:32 (1788745772) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24580 3582440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18860 3588160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43440 7170600 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24580 3582440 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18860 3588160 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43440 7170600 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120192 s, 17.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25394 s, 8.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa301f78f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:49:53 (1788745793) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149667 s, 14.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 21:50:01 (1788745801) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:50:11 (1788745811) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0801725 s, 26.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:50:19 (1788745819) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166789 s, 12.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24d:0x0] } PASS 204b (13s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:50:32 (1788745832) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125926 s, 16.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:51:08 (1788745868) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941536 s, 11.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24e:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:51:59 (1788745919) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131149 s, 16.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:52:47 (1788745967) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg129-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg129-server: Pool lustre.test_204f created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172391 s, 12.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg129-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:53:59 (1788746039) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:54:08 (1788746048) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:54:15 (1788746055) oleg129-server: Pool lustre.test_206 created Adding targets to pool oleg129-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg129-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg129-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg129-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:54:41 (1788746081) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710688 s, 14.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252157 s, 41.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x834d5787 Mirror 3: 0x834d5787 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:54:51 (1788746091) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:54:55 (1788746095) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:54:59 (1788746099) Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.112665 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1c90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1c90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1c90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2190000 Read this (4 bytes): BBBB 192.168.201.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg129-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:55:37 (1788746137) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0405514 s, 202 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:55:46 (1788746146) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985291 s, 10.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 21:55:48.000000000 -0400 Modify: 2026-09-06 21:55:48.000000000 -0400 Change: 2026-09-06 21:55:50.000000000 -0400 Birth: 2026-09-06 21:55:48.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:55:55 (1788746155) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640055 s, 16.4 MB/s oleg129-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg129-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg129-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:56:19 (1788746179) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.271648 s, 151 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0253233 s, 162 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 6 21:56 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 6 21:56 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:56:28 (1788746188) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:56:38 (1788746198) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1989 sec =========== 21:56:48 (1788746208) === sanity-flr: start cleanup 21:56:50 (1788746210) === === sanity-flr: finish cleanup 21:56:59 (1788746219) ===